A double balanced mixer will ideally have infinite isolation from port to port. What you are measuring as isolation is therefore the defect in balancing all of the mixer components.
Some elements of the mixer will leak regardless of whether it has an LO drive or not, for instance the geometry of the package and wires. The switching elements will change when driven. The mixer will only be used when driven, so this is the more important case of the two to measure.
Typically it's the unbalance in the stray capacitance of the elements that would leak in the undriven case. When driven, it's the unbalance in their conductivity. This difference makes a measure of undriven leakage more or less irrelevant.
As a VNA is typically a superheterodyne receiver, expecting to receive just the one specific frequency, it's likely it can be upset by some odd unexpected signals leaking into it. You may find that at some LO drive frequencies, the VNA produces obviously wrong results. It should be possible to alter the LO drive slightly to avoid these problems.
