Well, yes, length is in general important in a vector network analyzer.
Let's say you have port 1 of an S-parameter test set, or a
reflection-transmission test set, accurately calibrated at its
connector. What is S11 if you connect nothing there? What is S11 if
you connect an inch of coax there? If you tell me it's different in
the two cases (which you better!), then you should pay attention to the
length of the line to the open or short.

Another way to look at it: on a Smith chart, where will your 50 ohm
termination be? Where will a short be? Where will an open be? If the
"short" and "open" are at different distances down a line from their
connector, even if you calibrate so the "open" is where it should be on
the reflection coefficient display (Smith chart display), where then
_should_ the short be, if you don't have the line lengths exactly the
same?

And while I'm asking questions, what happens if you put your 50 ohm
load, made with really good 0.1% 0805 resistors, at the end of a
section of 50 ohm line which is not actually 50 ohms, but is instead 55
ohms or 45 ohms? The impedance tolerance on typical coax is pretty
poor. Beware. It's actually unlikely to be as good as a carefully
made load as I've described before here. (It's a real pain in the rear
trying to set up a system to test things like VNAs accurately and
guarantee your calibrations because of stuff like this.)

Cheers,
Tom