You can get the loss down to 4 dB by using RG-8X "mini" coax.
Standard RG-8 cable will lose 2 dB, which ought not to be a problem.

Is it worth it to shoot the extra cash on LMR 400 or Heliax?

I doubt it, in the case of LMR 400.
No way, in the case of Heliax.

On the topic of antennae, I see mentions of a "no-ground-plane" antenna.
I assume I need one of these since it will be up in a tree?

You can either use a half-wave end-fed radiator such as a J-pole
(which needs no ground plane).

Or, you can use a quarter-wave or stacked-collinear antenna, which
does need a ground plane. Many of these will come with a set of
radials which are attached at the bottom of the radiator, and which
serve as an elevated ground plane.

Hoping to do this one without a climb. I was thinking of shooting
a guide wire over a branch and using it to haul the antenna up, with a
few wires to stake it down with.

In that case, you're definitely going to want a relatively light
feedline, since the antenna and its connector are going to have to
support the weight of the feedline all the way to the ground. I
wouldn't use anything heavier than RG-8X in this situation.

Right, I think FRS radios come without the removable antenna for this
very reason.

Correct.

--
Dave Platt AE6EO
Hosting the Jade Warrior home page: http://www.radagast.org/jade-warrior
I do _not_ wish to receive unsolicited commercial email, and I will
boycott any company which has the gall to send me such ads!