Chip to chip synchronization

Hello,
Our system comprises multiple LMS chips. The DIQ interfaces are validated by sending known data and then feeding a signal from a signal generator.
There is a definite phase difference across multiple LMS ics. I was hoping that since the same clock is used for both CGEN and SXR (and we are using a low skew clock distribution chip to distribute the reference clock) the phase difference across the chips would not be much. However there is a significant phase difference. However this can be calibrated out to an extent.
Our measurement technique is to simply take the FFT (1024 pt FFT, 2MHz from LO input signal, sampling at 100MHz) of the signal and measure the phase and then subtract them from the reference LMS.

First concern: Across multiple measurements, within the LMS (between channels A and B) the phase difference variation is less than a degree. However across LMS chips the variation is around 5 degrees over measurements (time varying).

Second concern: This phase difference is not same across power cycles. This is a huge problem. We can calibrate our array in the lab, but how will we fix this issue in the field?

Looking forward to your inputs as we are very close to finishing the project (we would like to think so!). Thank you for your time,
Regards,

Hello @Zack,
Looking forward to your inputs on this.
Regards,

Hello @Zack,
Sorry to bother again, but we are close to our project completion and could use your valuable input.
Regards,