Excel "not responding" when try to sort by cell color

Hey all... I'm new to this forum but it seems to have some very bright people here... (hope I don't bring the IQ level down too much.)

Anyway, I have a very large excel file (about 590,000 rows, whittled down from over 865,000). Each row is basically the details of each property in a specific county. I've deleted a lot of rows I don't need. Then I highlighted all the duplicates in one column ie. owners name who owns multiple properties. I did this by highlighting the entire column and then going to Conditional Formatting, Highlight Cell Rules, Duplicate Values. I just kept with the default Light Red Fill with Dark Red Text. When I try to sort by cell color or font color it just freezes. The top of the sort box simply says "Sort (Not Responding)". I've read some of the other posts and I've gone in and deleted a couple of the old files with the xlsb extension. (I've redownloaded the data several times and sorted it to the point I can but always get hung up when I try to sort by cell color.)

What can be causing this? Or better yet, how can I fix it (as I really don't care what's causing it... I just want it to stop...)

Any help will be GREATLY appreciated.