Paul:

Thanks for the info....that's an interesting balun approach. I was
thinking about the two-connector approach and using a good microwave
load for the termination, but decided it wasn't likely needed for 50
MHz and below. Though with the results you are getting it's tempting to
build another one for VHF/UHF. But I need a good signal generator
first !

I have since obtained much better results than I was initially getting,
with two main approaches to improvement:
(1) I changed the balun to 12 bifilar turns on an FT37-43 core, which
now has enough inductance for reasonable measurement accuracy at the
lower frequencies (though the degradation is still pretty noticeable
with good loads that read 35 dB return loss at 10-30 MHz showing as 25
dB at 1.8 MHz). A few more turns would probably be better.
(2) I spent lots of time chasing down leakage paths and sensitivity to
leakage...grounding cables, reducing attenuator on signal generator,
scraping paint from inside front panel of signal generator, removing
microphone from transceiver in use as a detector, etc.

I don't really understand the leakage paths, but I was able to get
their impact down to a much more usable level. Clearly a high standard
of shielding on all components of the test setup is a must. I'm now
getting around 0.5 dB change from open to short at 30-50 MHz
(insignificant at lower frequencies) and my good loads read 31-34 dB
return loss at 50 MHz, which is probably about what should be expected
due to the adapters to PL-259 plug. I am not yet quite fully confident
when measuring antennas though, due to somewhat greater leakage effects
into the receiver.

73,
Steve VE3SMA