Expected phase error when synchronizing multiple xtrxs [phase coherent processing]

[spet]

My setup is as picturesd below - i’m synchronizing two xtrx’s for 4 channel phase coherent receive. I split a CW tone close to my LO frequency and feed it into all 4 channels across 2 xtrx’s

Does anyone know the expected phase error across boards? I’d expect it to be, maybe, at worst case, close to 2x the integrated phase noise at the LO i’m operating at, but this is only about 0.6 degrees and i’m experience about 10-15 degrees

i’m getting about 0.20-0.3 radians which is quite substantial and i imagine thats from the fractional-N PLLs in each lms7002 which would be non-deterministic across boards. Is it advised to use non-fractional modes for better coherence?

it also seem that there is no deterministic phase coherence from start up to start up or LO change. That is the lms7002 is capable of phase lock (to some degree of error inter-chip), but if you wanted phase coherence (deterministic phase lock every startup/between LO changes) that is simply not possible as the PLL randomly relocks its fractional-N pll between chips thus making phase difference between channels across chips random and requiring a recalibration step after every LO change?

[andrewback]

Tagging @VytautasB

[spet]

heres some actual data if it helps. i have also confirmed an LO tuning means non-deterministic phase difference between all channels, i suspect there is no way to remedy this and any LO change would need a calibration step.Theres also a lot of phase drift between two chips after calibration (see gif below).

I’ve also found that interchip phase noise increases as sample rate increases, but i haven’t pinpointed why

over 10 seconds, in varying temp

[spet]

Anyone from Lime have any recommendations on this issue? i can reproduce this over multiple custom boards and with multiple XTRXs @andrewback @VytautasB @Zack

[VytautasB]

Hi @spet,

This looks more like an application / RF system integration question than a GW-specific one.

Can you give more details on your setup:

1. Which version of LimeSDR XTRX are you using? Have you configured second XTRX to accept external clock?
2. Is clock network is similar to LimeSDR XTRX on your custom boards?

Regards,

Vytautas

[spet]

Thanks @VytautasB. Yes definitely more of a a RF systems question, but specifically wondering if theres anyway to get determinism out of the PLLs across chips, and if this temperature behavior is expected it suggests the LO is moving a ton with very little temperature change.

1. XTRX is GW 3.5 (litex version), second xtrx register 0018 has been configured to 0004 iusing limeSPI.

2. clock network is similar on custom boards.

   has anyone at lime looked into phase lock between multiple chips?

[andrewback]

There was a document for LimeSDR USB. Not sure if this helps:

LimeSDR-USB_channel_alignment_v01r00.pdf

[spet]

@andrewback

There was a document for LimeSDR USB. Not sure if this helps:

LMS7002M-docs/LimeSDR-USB_channel_alignment_v01r00.pdf at master · myriadrf/LMS7002M-docs · GitHub

i believe this document is describing MIMO on a single chip? i.e. 2 channel rx/tx MIMO?

a feature that would be super nice is direct LO injection and sidestepping PLL. any roadmap for such a feature?

[andrewback]

@spet

i believe this document is describing MIMO on a single chip? i.e. 2 channel rx/tx MIMO?

Apologies, you are right and I posted in haste.

@spet

a feature that would be super nice is direct LO injection and sidestepping PLL. any roadmap for such a feature?

I don’t think that is possible with LMS7002M and I’m not able to advise re silicon roadmap.

[spet]

thanks, understood. can any engineers from lime focused on the rf side + the lo/pll comment on multi-chip synchronization, phase error, and phase drift of the LO / PLLs with temperature.

[andrewback]

Tagging @Karolis.

[Karolis]

Hello,

in short - you won’t be able to phase sync two xtrx boards. For that, you would need, as you mentioned, LO sharing capabilities, which LMS7002 does not have.

Could you fully describe how do you setup two xtrx boards for reference syncing? What steps you took to source from one, and sync the other. I believe you may still be running on two separate internal xco’s, and want to rule that possibility out.

[spet]

@Karolis Thanks for the response! While phase synchronization would be great, constant phase offset between all channels would suffice, but the lms7002 doesnt seem capable of that across multiple chips? is that correct?

I’m almost certain I’m using 1 clock, but my setup is as follows:

1. on XTRX 0, connected X7 to XTRX 1. I have ensured R156 is installed as below
2. on XTRX 1, I’ve ensured the x7 path that feeds IC15 is all good. I’ve then set FPGA register 0018 from 0002 to 0004 which sets it up for ext clk usage. i.e. limeSPI -d e5 write -c FPGA -s 00180004
3. i’ve verified that switching this register from 0002->0004 and back to 0002 means the second xtrx 1 drops out and doesnt produce data

Questions:

1. Is there a way to synchronize the two PLLs across chips or ensure phase lock between the two PLLs? (I believe phase lock would mean we do not see phase drift over temperature/time)
2. what is the expected LO drift over small temperature changes? as shown in original post, i see that the inter-chip phase drift is quite extreme when i blow a fan on the XTRXs indicating both chips PLL/LO are changing quite dramatically with respect to one another (but channels on the chip are staying phase synchronized, i.e. the LO is changing on the same chip but the exact same between two channels which makes sense).
3. when lime talks about “mimo” with multiple chips does it strictly mean non-coherent mimo?

if y’all at lime have two xtrx’s lying around i encourage you try try this setup:

1. use 1 clock for both xtrxs as described above
2. drop down two limesuiteng source blocks in gnuradio and plot:
   1. angle((xtrx0 chan A) * conj(xtrx0 chan B))
   2. angle((xtrx0 chan A) * conj(xtrx1 chan A))
   3. angle((xtrx0 chan A) * conj(xtrx1 chan B))
   4. angle((xtrx1 chan A) * conj(xtrx1 chan B))
3. set them to any sample rate and center frequency + feed any sinusoid into all channels (offset from the LO)
4. blow a fan on the xtrxs and see if phase drifts for plots (2) and (3) described in point (2)

here is a python flow graph to compare:

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
#
# SPDX-License-Identifier: GPL-3.0
#
# 4-channel GNU Radio flowgraph — direct limesuiteng 
#
# Usage:
#   python3 phase_check_4_ch.py [--serial0 aaa] [--serial1 e5]

from PyQt5 import Qt, QtCore
from gnuradio import qtgui
from gnuradio import gr
from gnuradio.fft import window
from gnuradio import blocks
from gnuradio import limesuiteng
import numpy as np
import sys
import signal
import sip
import threading
from collections import deque
from argparse import ArgumentParser

import pyqtgraph as pg
pg.setConfigOption('background', 'k')
pg.setConfigOption('foreground', 'w')


class phase_cal_cc(gr.sync_block):
    """Accumulate N complex samples, compute mean phase, then rotate stream by exp(-j*mean).
    Passes samples through unchanged until calibration completes.
    After calibration, tracks min/max of output phase in degrees.
    Call trigger() to start a new calibration run (resets min/max too)."""

    def __init__(self, n_cal=500):
        gr.sync_block.__init__(self, name="phase_cal_cc",
                               in_sig=[np.complex64], out_sig=[np.complex64])
        self._n_cal = n_cal
        self._buf = []
        self._collecting = False
        self._calibrated = False
        self._rot = np.complex64(1 + 0j)
        self._done_cb = None
        self.phase_min = None
        self.phase_max = None

    def trigger(self, done_cb=None):
        self._done_cb = done_cb
        self._buf = []
        self._collecting = True
        self._calibrated = False
        self.phase_min = None
        self.phase_max = None

    def offset_rad(self):
        return -np.angle(self._rot)

    def work(self, input_items, output_items):
        inp = input_items[0]
        out = output_items[0]

        if self._collecting:
            need = self._n_cal - len(self._buf)
            self._buf.extend(inp[:need].tolist())
            if len(self._buf) >= self._n_cal:
                self._collecting = False
                self._calibrated = True
                offset = float(np.angle(np.mean(self._buf)))
                self._rot = np.complex64(np.exp(-1j * offset))
                if self._done_cb:
                    self._done_cb(offset)

        out[:] = inp * self._rot

        if self._calibrated:
            phases_deg = np.degrees(np.angle(out))
            lo = float(np.min(phases_deg))
            hi = float(np.max(phases_deg))
            self.phase_min = lo if self.phase_min is None else min(self.phase_min, lo)
            self.phase_max = hi if self.phase_max is None else max(self.phase_max, hi)

        return len(inp)


class RollingPhaseSink(gr.sync_block, Qt.QWidget):
    """Rolling 10-second phase plot. work() feeds a deque ring buffer;
    a QTimer repaints at 20 Hz so the display scrolls continuously."""

    def __init__(self, n_channels, rate, window_sec=10,
                 title="", labels=None, colors=None):
        gr.sync_block.__init__(self, "rolling_phase_sink",
                               in_sig=[np.float32] * n_channels, out_sig=[])
        Qt.QWidget.__init__(self)

        self._maxlen = int(window_sec * rate)
        self._window_sec = window_sec
        self._bufs = [deque(maxlen=self._maxlen) for _ in range(n_channels)]
        self._lock = threading.Lock()

        _colors = colors or ['r', 'g', 'm', 'c']
        _labels = labels or [f'ch{i}' for i in range(n_channels)]

        self._pw = pg.PlotWidget(title=title)
        self._pw.setYRange(-185, 185)
        self._pw.showGrid(x=True, y=True)
        self._pw.setLabel('left',   'Phase (deg)')
        self._pw.setLabel('bottom', 'Time (s)')
        self._pw.setMinimumHeight(200)

        self._curves = [
            self._pw.plot(pen=pg.mkPen(_colors[i], width=2))
            for i in range(n_channels)
        ]

        # Legend as a row of colored labels below the plot — no overlap
        legend_row = Qt.QHBoxLayout()
        legend_row.addStretch()
        for i in range(n_channels):
            swatch = Qt.QLabel(f'— {_labels[i]}')
            swatch.setStyleSheet(f'color: {_colors[i]}; font-weight: bold;')
            legend_row.addWidget(swatch)
        legend_row.addStretch()

        layout = Qt.QVBoxLayout()
        layout.setContentsMargins(0, 0, 0, 0)
        self.setLayout(layout)
        layout.addWidget(self._pw)
        layout.addLayout(legend_row)
        self.setMinimumHeight(240)

        self._timer = Qt.QTimer()
        self._timer.timeout.connect(self._redraw)
        self._timer.start(50)   # 20 Hz

    def work(self, input_items, output_items):
        n = len(input_items[0])
        with self._lock:
            for i, buf in enumerate(self._bufs):
                buf.extend(input_items[i][:n])
        return n

    def _redraw(self):
        with self._lock:
            snaps = [np.array(b, dtype=np.float32) for b in self._bufs]
        for curve, data in zip(self._curves, snaps):
            if len(data):
                t = np.linspace(-self._window_sec * len(data) / self._maxlen,
                                0, len(data), dtype=np.float32)
                curve.setData(t, data)


class phase_check_4_ch(gr.top_block, Qt.QWidget):

    def __init__(self, serial0='aaa', serial1='e5'):
        gr.top_block.__init__(self, "4x XTRX Receiver (tachyon)", catch_exceptions=True)
        Qt.QWidget.__init__(self)
        self.setWindowTitle("4x XTRX Receiver (tachyon)")
        qtgui.util.check_set_qss()
        try:
            self.setWindowIcon(Qt.QIcon.fromTheme('gnuradio-grc'))
        except BaseException as exc:
            print(f"Qt GUI: Could not set Icon: {str(exc)}", file=sys.stderr)

        self.top_scroll_layout = Qt.QVBoxLayout()
        self.setLayout(self.top_scroll_layout)
        self.top_scroll = Qt.QScrollArea()
        self.top_scroll.setFrameStyle(Qt.QFrame.NoFrame)
        self.top_scroll_layout.addWidget(self.top_scroll)
        self.top_scroll.setWidgetResizable(True)
        self.top_widget = Qt.QWidget()
        self.top_scroll.setWidget(self.top_widget)
        self.top_layout = Qt.QVBoxLayout(self.top_widget)
        self.top_grid_layout = Qt.QGridLayout()
        self.top_layout.addLayout(self.top_grid_layout)

        self.settings = Qt.QSettings("gnuradio/flowgraphs", "phase_check_4_ch")
        try:
            geometry = self.settings.value("geometry")
            if geometry:
                self.restoreGeometry(geometry)
        except BaseException as exc:
            print(f"Qt GUI: Could not restore geometry: {str(exc)}", file=sys.stderr)

        self.flowgraph_started = threading.Event()

        ##################################################
        # Variables
        ##################################################
        self.samp_rate = samp_rate = .10e6
        self.center_freq = center_freq = 2.43e9
        self.gain = gain = 50
        # LMS7002M RX LPF minimum is 1.5 MHz
        _rx_bw = 0 #max(samp_rate * 0.75, 1.5e6)
        _rad2deg = 180.0 / 3.141592653589793

        ##################################################
        # GUI Controls
        ##################################################
        self._gain_range = qtgui.Range(0, 100, 1, gain, 200)
        self._gain_win = qtgui.RangeWidget(
            self._gain_range, self.set_gain, "RX Gain (all channels)",
            "counter_slider", float, QtCore.Qt.Horizontal)
        self.top_layout.addWidget(self._gain_win)

        # Center frequency field + relock button
        _freq_row = Qt.QHBoxLayout()
        _freq_label = Qt.QLabel("Center Freq (Hz):")
        self._freq_edit = Qt.QLineEdit(str(int(center_freq)))
        self._freq_edit.setFixedWidth(160)
        self._freq_edit.returnPressed.connect(self._on_freq_enter)
        _relock_btn = Qt.QPushButton("Relock LO")
        _relock_btn.clicked.connect(self._relock_lo)
        _freq_row.addWidget(_freq_label)
        _freq_row.addWidget(self._freq_edit)
        _freq_row.addWidget(_relock_btn)
        _freq_row.addStretch()
        self.top_layout.addLayout(_freq_row)

        ##################################################
        # Sources — direct limesuiteng, 2 channels each
        ##################################################
        # XTRX 0 — serial0 (default: aaa)
        self.lime_src_0 = limesuiteng.sdrdevice_source(
            '', serial0, 0, [0, 1], "complex32f_t", "complex12_t", samp_rate, 0)
        self.lime_src_0.set_lo_frequency(center_freq)
        self.lime_src_0.set_gfir_bandwidth(0)
        self.lime_src_0.set_antenna('auto')
        self.lime_src_0.set_gain_generic(gain)
        self.lime_src_0.set_nco_frequency(0)
        self.lime_src_0.set_lpf_bandwidth(_rx_bw)
        self.lime_src_0.set_calibration_enable(1)

        # XTRX 1 — serial1 (default: e5)
        self.lime_src_1 = limesuiteng.sdrdevice_source(
            '', serial1, 0, [0, 1], "complex32f_t", "complex12_t", samp_rate, 0)
        self.lime_src_1.set_lo_frequency(center_freq)
        self.lime_src_1.set_gfir_bandwidth(0)
        self.lime_src_1.set_antenna('auto')
        self.lime_src_1.set_gain_generic(gain)
        self.lime_src_1.set_nco_frequency(0)
        self.lime_src_1.set_lpf_bandwidth(_rx_bw)
        self.lime_src_1.set_calibration_enable(1)

        ##################################################
        # Frequency sink — all 4 channels
        ##################################################
        self.qtgui_freq_sink = qtgui.freq_sink_c(
            1024,
            window.WIN_BLACKMAN_hARRIS,
            center_freq,
            samp_rate,
            "4x XTRX",
            4,
            None,
        )
        self.qtgui_freq_sink.set_update_time(0.10)
        self.qtgui_freq_sink.set_y_axis(-140, 10)
        self.qtgui_freq_sink.set_y_label('Relative Gain', 'dB')
        self.qtgui_freq_sink.set_trigger_mode(qtgui.TRIG_MODE_FREE, 0.0, 0, "")
        self.qtgui_freq_sink.enable_autoscale(False)
        self.qtgui_freq_sink.enable_grid(True)
        self.qtgui_freq_sink.set_fft_average(0.10)
        self.qtgui_freq_sink.enable_axis_labels(True)
        ch_labels = ['XTRX0-ch0', 'XTRX0-ch1', 'XTRX1-ch0', 'XTRX1-ch1']
        ch_colors = ['blue', 'red', 'green', 'magenta']
        for i in range(4):
            self.qtgui_freq_sink.set_line_label(i, ch_labels[i])
            self.qtgui_freq_sink.set_line_color(i, ch_colors[i])
            self.qtgui_freq_sink.set_line_width(i, 1)
            self.qtgui_freq_sink.set_line_alpha(i, 1.0)
        self.top_layout.addWidget(
            sip.wrapinstance(self.qtgui_freq_sink.qwidget(), Qt.QWidget))

        ##################################################
        # I/Q time sinks
        ##################################################
        n_samps = max(1024, int(samp_rate * 0.010))

        self.c2r_0 = blocks.complex_to_real(1)
        self.c2r_1 = blocks.complex_to_real(1)
        self.c2r_2 = blocks.complex_to_real(1)
        self.c2r_3 = blocks.complex_to_real(1)
        self.c2i_0 = blocks.complex_to_imag(1)
        self.c2i_1 = blocks.complex_to_imag(1)
        self.c2i_2 = blocks.complex_to_imag(1)
        self.c2i_3 = blocks.complex_to_imag(1)

        self.qtgui_time_sink_real = qtgui.time_sink_f(
            n_samps, samp_rate, "Real (I) — all channels", 4, None)
        self.qtgui_time_sink_real.set_update_time(0.10)
        self.qtgui_time_sink_real.set_y_axis(-1, 1)
        self.qtgui_time_sink_real.set_y_label('Amplitude', '')
        self.qtgui_time_sink_real.enable_tags(False)
        self.qtgui_time_sink_real.set_trigger_mode(
            qtgui.TRIG_MODE_FREE, qtgui.TRIG_SLOPE_POS, 0.0, 0, 0, '')
        self.qtgui_time_sink_real.enable_autoscale(True)
        self.qtgui_time_sink_real.enable_grid(True)
        self.qtgui_time_sink_real.enable_axis_labels(True)
        self.qtgui_time_sink_real.enable_control_panel(False)
        self.qtgui_time_sink_real.enable_stem_plot(False)
        for i, (lbl, col) in enumerate(zip(ch_labels, ch_colors)):
            self.qtgui_time_sink_real.set_line_label(i, lbl)
            self.qtgui_time_sink_real.set_line_color(i, col)
            self.qtgui_time_sink_real.set_line_width(i, 1)
            self.qtgui_time_sink_real.set_line_alpha(i, 1.0)
        self.top_layout.addWidget(
            sip.wrapinstance(self.qtgui_time_sink_real.qwidget(), Qt.QWidget))

        self.qtgui_time_sink_imag = qtgui.time_sink_f(
            n_samps, samp_rate, "Imag (Q) — all channels", 4, None)
        self.qtgui_time_sink_imag.set_update_time(0.10)
        self.qtgui_time_sink_imag.set_y_axis(-1, 1)
        self.qtgui_time_sink_imag.set_y_label('Amplitude', '')
        self.qtgui_time_sink_imag.enable_tags(False)
        self.qtgui_time_sink_imag.set_trigger_mode(
            qtgui.TRIG_MODE_FREE, qtgui.TRIG_SLOPE_POS, 0.0, 0, 0, '')
        self.qtgui_time_sink_imag.enable_autoscale(True)
        self.qtgui_time_sink_imag.enable_grid(True)
        self.qtgui_time_sink_imag.enable_axis_labels(True)
        self.qtgui_time_sink_imag.enable_control_panel(False)
        self.qtgui_time_sink_imag.enable_stem_plot(False)
        for i, (lbl, col) in enumerate(zip(ch_labels, ch_colors)):
            self.qtgui_time_sink_imag.set_line_label(i, lbl)
            self.qtgui_time_sink_imag.set_line_color(i, col)
            self.qtgui_time_sink_imag.set_line_width(i, 1)
            self.qtgui_time_sink_imag.set_line_alpha(i, 1.0)
        self.top_layout.addWidget(
            sip.wrapinstance(self.qtgui_time_sink_imag.qwidget(), Qt.QWidget))

        ##################################################
        # Phase difference blocks
        ##################################################
        self.mult_conj_1  = blocks.multiply_conjugate_cc(1)  # ch1 rel ch0 (XTRX0 intra)
        self.mult_conj_2  = blocks.multiply_conjugate_cc(1)  # ch2 rel ch0 (inter-chip)
        self.mult_conj_3  = blocks.multiply_conjugate_cc(1)  # ch3 rel ch0 (inter-chip)
        self.mult_conj_32 = blocks.multiply_conjugate_cc(1)  # ch3 rel ch2 (XTRX1 intra)
        self.c2arg_1  = blocks.complex_to_arg(1)
        self.c2arg_2  = blocks.complex_to_arg(1)
        self.c2arg_3  = blocks.complex_to_arg(1)
        self.c2arg_32 = blocks.complex_to_arg(1)
        self.deg_1  = blocks.multiply_const_ff(_rad2deg)
        self.deg_2  = blocks.multiply_const_ff(_rad2deg)
        self.deg_3  = blocks.multiply_const_ff(_rad2deg)
        self.deg_32 = blocks.multiply_const_ff(_rad2deg)

        # Decimate phase to a fixed display rate so the 10 s window stays
        # manageable regardless of samp_rate.  Only samp_rate needs changing.
        PHASE_RATE  = 1000                          # display samples/sec
        phase_decim = max(1, int(samp_rate / PHASE_RATE))
        phase_rate  = samp_rate / phase_decim       # actual rate after decim

        self.kin_phase_1  = blocks.keep_one_in_n(gr.sizeof_float, phase_decim)
        self.kin_phase_2  = blocks.keep_one_in_n(gr.sizeof_float, phase_decim)
        self.kin_phase_3  = blocks.keep_one_in_n(gr.sizeof_float, phase_decim)
        self.kin_phase_32 = blocks.keep_one_in_n(gr.sizeof_float, phase_decim)

        phase_labels = ['ch1 rel ch0 (XTRX0)', 'ch2 rel ch0 (inter-chip)',
                        'ch3 rel ch0 (inter-chip)', 'ch3 rel ch2 (XTRX1)']
        phase_colors = ['red', 'green', 'magenta', 'cyan']

        self.qtgui_time_sink_phase = RollingPhaseSink(
            n_channels=4, rate=phase_rate, window_sec=10,
            title="Phase diff (deg)",
            labels=phase_labels, colors=phase_colors)
        self.top_layout.addWidget(self.qtgui_time_sink_phase)

        ##################################################
        # Calibrated phase plot
        ##################################################
        self.cal_1  = phase_cal_cc(500)
        self.cal_2  = phase_cal_cc(500)
        self.cal_3  = phase_cal_cc(500)
        self.cal_32 = phase_cal_cc(500)
        self.cal_c2arg_1  = blocks.complex_to_arg(1)
        self.cal_c2arg_2  = blocks.complex_to_arg(1)
        self.cal_c2arg_3  = blocks.complex_to_arg(1)
        self.cal_c2arg_32 = blocks.complex_to_arg(1)
        self.cal_deg_1  = blocks.multiply_const_ff(_rad2deg)
        self.cal_deg_2  = blocks.multiply_const_ff(_rad2deg)
        self.cal_deg_3  = blocks.multiply_const_ff(_rad2deg)
        self.cal_deg_32 = blocks.multiply_const_ff(_rad2deg)

        self.kin_cal_1  = blocks.keep_one_in_n(gr.sizeof_float, phase_decim)
        self.kin_cal_2  = blocks.keep_one_in_n(gr.sizeof_float, phase_decim)
        self.kin_cal_3  = blocks.keep_one_in_n(gr.sizeof_float, phase_decim)
        self.kin_cal_32 = blocks.keep_one_in_n(gr.sizeof_float, phase_decim)

        self.qtgui_time_sink_cal = RollingPhaseSink(
            n_channels=4, rate=phase_rate, window_sec=10,
            title="Phase diff — calibrated (deg)",
            labels=phase_labels, colors=phase_colors)

        # Calibration button + status label
        _cal_row = Qt.QHBoxLayout()
        _cal_btn = Qt.QPushButton("Calibrate Phase")
        _cal_btn.clicked.connect(self._calibrate)
        self._cal_label = Qt.QLabel("Not calibrated")
        _cal_row.addWidget(_cal_btn)
        _cal_row.addWidget(self._cal_label)
        _cal_row.addStretch()
        self.top_layout.addLayout(_cal_row)
        self.top_layout.addWidget(self.qtgui_time_sink_cal)

        # Min/max bars — one row per channel, color-matched, updated by QTimer
        _minmax_grid = Qt.QGridLayout()
        _minmax_grid.setColumnStretch(1, 1)
        _minmax_grid.addWidget(Qt.QLabel(""), 0, 0)
        _minmax_grid.addWidget(Qt.QLabel("min"), 0, 1, QtCore.Qt.AlignCenter)
        _minmax_grid.addWidget(Qt.QLabel("max"), 0, 2, QtCore.Qt.AlignCenter)
        self._minmax_labels = []
        _bar_names = ['ch1/ch0 (XTRX0)', 'ch2/ch0 (inter)', 'ch3/ch0 (inter)', 'ch3/ch2 (XTRX1)']
        for i, (name, col) in enumerate(zip(_bar_names, phase_colors)):
            _name_lbl = Qt.QLabel(name)
            _name_lbl.setStyleSheet(f"color: {col}; font-weight: bold;")
            _min_lbl = Qt.QLabel("—")
            _min_lbl.setAlignment(QtCore.Qt.AlignCenter)
            _min_lbl.setStyleSheet(f"color: {col};")
            _max_lbl = Qt.QLabel("—")
            _max_lbl.setAlignment(QtCore.Qt.AlignCenter)
            _max_lbl.setStyleSheet(f"color: {col};")
            _minmax_grid.addWidget(_name_lbl, i + 1, 0)
            _minmax_grid.addWidget(_min_lbl,  i + 1, 1)
            _minmax_grid.addWidget(_max_lbl,  i + 1, 2)
            self._minmax_labels.append((_min_lbl, _max_lbl))
        self.top_layout.addLayout(_minmax_grid)

        self._minmax_timer = Qt.QTimer()
        self._minmax_timer.timeout.connect(self._update_minmax)
        self._minmax_timer.start(200)

        ##################################################
        # Connections
        ##################################################
        self.connect((self.lime_src_0, 0), (self.qtgui_freq_sink, 0))
        self.connect((self.lime_src_0, 1), (self.qtgui_freq_sink, 1))
        self.connect((self.lime_src_1, 0), (self.qtgui_freq_sink, 2))
        self.connect((self.lime_src_1, 1), (self.qtgui_freq_sink, 3))

        # Real (I)
        self.connect((self.lime_src_0, 0), (self.c2r_0, 0))
        self.connect((self.lime_src_0, 1), (self.c2r_1, 0))
        self.connect((self.lime_src_1, 0), (self.c2r_2, 0))
        self.connect((self.lime_src_1, 1), (self.c2r_3, 0))
        self.connect((self.c2r_0, 0), (self.qtgui_time_sink_real, 0))
        self.connect((self.c2r_1, 0), (self.qtgui_time_sink_real, 1))
        self.connect((self.c2r_2, 0), (self.qtgui_time_sink_real, 2))
        self.connect((self.c2r_3, 0), (self.qtgui_time_sink_real, 3))

        # Imag (Q)
        self.connect((self.lime_src_0, 0), (self.c2i_0, 0))
        self.connect((self.lime_src_0, 1), (self.c2i_1, 0))
        self.connect((self.lime_src_1, 0), (self.c2i_2, 0))
        self.connect((self.lime_src_1, 1), (self.c2i_3, 0))
        self.connect((self.c2i_0, 0), (self.qtgui_time_sink_imag, 0))
        self.connect((self.c2i_1, 0), (self.qtgui_time_sink_imag, 1))
        self.connect((self.c2i_2, 0), (self.qtgui_time_sink_imag, 2))
        self.connect((self.c2i_3, 0), (self.qtgui_time_sink_imag, 3))

        # Phase diffs → decimate → rolling sink
        self.connect((self.lime_src_0, 1), (self.mult_conj_1, 0))
        self.connect((self.lime_src_0, 0), (self.mult_conj_1, 1))
        self.connect((self.mult_conj_1, 0), (self.c2arg_1, 0))
        self.connect((self.c2arg_1, 0), (self.deg_1, 0))
        self.connect((self.deg_1, 0), (self.kin_phase_1, 0))
        self.connect((self.kin_phase_1, 0), (self.qtgui_time_sink_phase, 0))

        self.connect((self.lime_src_1, 0), (self.mult_conj_2, 0))
        self.connect((self.lime_src_0, 0), (self.mult_conj_2, 1))
        self.connect((self.mult_conj_2, 0), (self.c2arg_2, 0))
        self.connect((self.c2arg_2, 0), (self.deg_2, 0))
        self.connect((self.deg_2, 0), (self.kin_phase_2, 0))
        self.connect((self.kin_phase_2, 0), (self.qtgui_time_sink_phase, 1))

        self.connect((self.lime_src_1, 1), (self.mult_conj_3, 0))
        self.connect((self.lime_src_0, 0), (self.mult_conj_3, 1))
        self.connect((self.mult_conj_3, 0), (self.c2arg_3, 0))
        self.connect((self.c2arg_3, 0), (self.deg_3, 0))
        self.connect((self.deg_3, 0), (self.kin_phase_3, 0))
        self.connect((self.kin_phase_3, 0), (self.qtgui_time_sink_phase, 2))

        self.connect((self.lime_src_1, 1), (self.mult_conj_32, 0))
        self.connect((self.lime_src_1, 0), (self.mult_conj_32, 1))
        self.connect((self.mult_conj_32, 0), (self.c2arg_32, 0))
        self.connect((self.c2arg_32, 0), (self.deg_32, 0))
        self.connect((self.deg_32, 0), (self.kin_phase_32, 0))
        self.connect((self.kin_phase_32, 0), (self.qtgui_time_sink_phase, 3))

        # Calibrated path → decimate → rolling sink
        self.connect((self.mult_conj_1,  0), (self.cal_1,  0))
        self.connect((self.mult_conj_2,  0), (self.cal_2,  0))
        self.connect((self.mult_conj_3,  0), (self.cal_3,  0))
        self.connect((self.mult_conj_32, 0), (self.cal_32, 0))
        self.connect((self.cal_1,  0), (self.cal_c2arg_1,  0))
        self.connect((self.cal_2,  0), (self.cal_c2arg_2,  0))
        self.connect((self.cal_3,  0), (self.cal_c2arg_3,  0))
        self.connect((self.cal_32, 0), (self.cal_c2arg_32, 0))
        self.connect((self.cal_c2arg_1,  0), (self.cal_deg_1,  0))
        self.connect((self.cal_c2arg_2,  0), (self.cal_deg_2,  0))
        self.connect((self.cal_c2arg_3,  0), (self.cal_deg_3,  0))
        self.connect((self.cal_c2arg_32, 0), (self.cal_deg_32, 0))
        self.connect((self.cal_deg_1,  0), (self.kin_cal_1,  0))
        self.connect((self.cal_deg_2,  0), (self.kin_cal_2,  0))
        self.connect((self.cal_deg_3,  0), (self.kin_cal_3,  0))
        self.connect((self.cal_deg_32, 0), (self.kin_cal_32, 0))
        self.connect((self.kin_cal_1,  0), (self.qtgui_time_sink_cal, 0))
        self.connect((self.kin_cal_2,  0), (self.qtgui_time_sink_cal, 1))
        self.connect((self.kin_cal_3,  0), (self.qtgui_time_sink_cal, 2))
        self.connect((self.kin_cal_32, 0), (self.qtgui_time_sink_cal, 3))

    def _calibrate(self):
        self._cal_label.setText("Calibrating...")
        self._cal_done = [False, False, False, False]

        def _make_cb(idx):
            def _cb(_offset_rad):
                self._cal_done[idx] = True
                if all(self._cal_done):
                    offsets = [b.offset_rad() for b in (self.cal_1, self.cal_2, self.cal_3, self.cal_32)]
                    Qt.QTimer.singleShot(0, lambda: self._cal_label.setText(
                        f"Cal: ch1={np.degrees(offsets[0]):.1f}° "
                        f"ch2={np.degrees(offsets[1]):.1f}° "
                        f"ch3={np.degrees(offsets[2]):.1f}° "
                        f"ch3/2={np.degrees(offsets[3]):.1f}°"))
            return _cb

        self.cal_1.trigger(_make_cb(0))
        self.cal_2.trigger(_make_cb(1))
        self.cal_3.trigger(_make_cb(2))
        self.cal_32.trigger(_make_cb(3))

    def _update_minmax(self):
        for (min_lbl, max_lbl), cal in zip(self._minmax_labels,
                                            (self.cal_1, self.cal_2, self.cal_3, self.cal_32)):
            lo = cal.phase_min
            hi = cal.phase_max
            min_lbl.setText(f"{lo:.1f}°" if lo is not None else "—")
            max_lbl.setText(f"{hi:.1f}°" if hi is not None else "—")

    def _retune(self, freq):
        self.lime_src_0.set_lo_frequency(freq)
        self.lime_src_1.set_lo_frequency(freq)
        self.center_freq = freq
        self._freq_edit.setText(str(int(freq)))
        self.qtgui_freq_sink.set_frequency_range(freq, self.samp_rate)

    def _on_freq_enter(self):
        try:
            freq = float(self._freq_edit.text())
        except ValueError:
            self._freq_edit.setText(str(int(self.center_freq)))
            return
        self._retune(freq)

    def _relock_lo(self):
        freq = self.center_freq
        step = 10e6
        self._retune(freq + step)
        Qt.QTimer.singleShot(300, lambda: self._retune(freq))

    def closeEvent(self, event):
        self.settings = Qt.QSettings("gnuradio/flowgraphs", "phase_check_4_ch")
        self.settings.setValue("geometry", self.saveGeometry())
        self.stop()
        self.wait()
        event.accept()

    def get_samp_rate(self):
        return self.samp_rate

    def set_samp_rate(self, samp_rate):
        self.samp_rate = samp_rate
        self.qtgui_freq_sink.set_frequency_range(self.center_freq, samp_rate)

    def get_gain(self):
        return self.gain

    def set_gain(self, gain):
        self.gain = gain
        self.lime_src_0.set_gain_generic(gain)
        self.lime_src_1.set_gain_generic(gain)


def main(top_block_cls=phase_check_4_ch, options=None):
    if gr.enable_realtime_scheduling() != gr.RT_OK:
        gr.logger("realtime").warn("Error: failed to enable real-time scheduling.")

    parser = ArgumentParser()
    parser.add_argument('--serial0', default='aaa', help='Serial for XTRX 0 (default: aaa)')
    parser.add_argument('--serial1', default='e5',  help='Serial for XTRX 1 (default: e5)')
    args = parser.parse_args()

    qapp = Qt.QApplication(sys.argv)

    tb = top_block_cls(serial0=args.serial0, serial1=args.serial1)
    tb.start()
    tb.flowgraph_started.set()
    tb.show()

    def sig_handler(sig=None, frame=None):
        tb.stop()
        tb.wait()
        Qt.QApplication.quit()

    signal.signal(signal.SIGINT, sig_handler)
    signal.signal(signal.SIGTERM, sig_handler)

    timer = Qt.QTimer()
    timer.start(500)
    timer.timeout.connect(lambda: None)

    qapp.exec_()


if __name__ == '__main__':
    main()

[Karolis]

@spet

constant phase offset between all channels would suffice

Hello,

I will try to replicate your setup on Thursday. Will come back to you with a full response then.

[Karolis]

Hello,

I replicated your tests. The phase coherence remains stable, as long as the both XTRX board temperature remains stable. Both XTRX RX paths are fed the same CW with an offset of ~20kHz, through a 0deg splitter. XTRX0 (name arbitrary) is feeding XTRX1 the reference clock.

The first video (below): one XTRX is sandwiched between a NIC and GPU, the other is bellow the NIC. One is running hotter then the other, but at start of video and after calibration both show somewhat stable phase between both XTRX. When fan is turned on, at the middle of the video, phase drifts drastically between both XTRXs.

Microsoft OneDrive

Second video (below): same setup after 1min, just screen grabbed after letting both XTRXs reach stable temperature.

Microsoft OneDrive

So, the best bet for a stable phase relation - keep both XTRXs at stable temperature.

General advise - maybe try looking at the upcoming LimeSDR Micro, which has a phase detector circuitry on the board to phase align multiple board from a common reference source although this could also require thermal management to some degree.

[spet]

@Karolis

vise - maybe try looking at the upcoming LimeSDR Micro, which has a phase detector circuitry on the board to phase align multiple board from a common reference source although this could also require thermal management to s

Really appreciate you looking into this @Karolis! So there are no lms7002 configuration we can take advantage of to keep this stable? Additionally, do we know why there is phase noise of about +/-10-20 degrees even when it is temperature stable? Trying to track down where exactly this comes from if they are both using the same clk, it seems the PLL/LO is jumping around relative to another device quite a bit

[spet]

just following up to see if theres any advice regarding configuration which can help mitigate this issue and if we know why the phase noise is so large between devices?

[Karolis]

Hello,

I just did a quick test, where I added a third XTRX as a common reference source for the other two XTRX boards - equal reference path lengths and so on. The result was as expected - similar behavior as in the initial test.

So the general answer remains the same - under stringent thermal management, XTRX have an phase coherence board to board around ±5deg based on your script.

Phase noise is same for both boards - and it is <1 deg integrated in 0.012-20MHz range (values 0.1deg for lower frequencies and nearing 1deg near 3GHz and above). This is measured separately, on a single channel - not comparing XTRX0 to XTRX1.

I would say there will be no relevant configuration which could help in this case. There are a lot of variables that will effect phase relation between both boards - reference biasing, PLL settings, LNA settings and so on. But the effects will be mainly board to board and environment dependent, hence not consistently repeatable.

I am sorry for the not so favorable response.

Regards.