You are comparing a device with front end filtering to devices without front end filtering.
I’m going to guess that the MixNV internally uses a RFFC5072 (85 to 4200 MHz Wideband Synthesizer / VCO with Integrated 30 MHz to 6 GHz RF Mixer) since it’s oscillator can tune from 85MHz to 4.2GHz, which is the same chip used in a HackRF and a Othernet moRFeus.
If you read the “Specifications” in the documentation for the MixNV they suggest using a filter “If lower harmonic levels are needed, Windfreak Technologies suggest the use of low cost SMA filters from Crystek. A $25 1GHz example would be the CLPFL-1000. There are usually many different frequency cut-offs in stock at Digikey”.
So my guess would be, assuming that you have not added any filtering between the output of the MixNV and the input of the LimeSDR/XTRX, that you are seeing intermodulation distortions from the fundamental frequency and the second or third order harmonics generated by the MixNV. And with filtering, or enough attenuation (which is in effect what a filter does), the second or third order harmonics generated by the MixNV will be suppressed enough to no longer generate intermodulation distortions.
EDIT: And the reason I mentioned the moRFeus is because the hardware would at a guess be extremely similar to a MixNV. In that there is probably a ATmega32U4 in the MixNV to provide the serial interface to control the RFFC5072. And I would guess a Skyworks Si5351C-GM1 to provide the 10MHz to 100MHz reference frequency input similar to the chip in a HackRF (Si5351C). Oh and people have used the third harmonic from the moRFeus to transmit CW at 10.8 GHz (The fundamental frequency was set to 3602000 kHz with output power at maximum).