It can get down to personal preference. I had a Kenwood R-5000 years ago
that I KNOW was superior to anything else I have ever owned,. Sensitive,
solid selectivity, many tuning and filter features, and like you, I
tried and built speakers, headphones and whatever else I could do. But,
I HATED listening to it after an hour or 2. Just something
psych-acoustic, whatever. I later got Yeasu FRG-8800 and after a
lightning hit, the FRG-100. Later, a Lowe 225. Admittedly, they might be
considered a notch down on qualiity, but the important thing is: I am
happy with them.
You might try tuning AM stations off-freq a couple Khz as your filters
may allow, to get the audio into the higher range, but this is not
really possible on SSB unless the Drake has a carrier offset tuning (as
does my FRG-100).
And, I really doubt if the AOR is any more sensitive than the Drake in
the real world, this sounds more like an antenna issue. Tropical bands
like long antennas.