Question regarding how 2 SDRs running simultaneously are timed

I have 2 LimeSDR Minis (v 2.4) that are connected to the same clock. I am running them simultaneously and am not surprised that their output is not synchronized in time. What does puzzle me is the following: there is a time offset between the two that is constant over multiple loops of LMS_SendStream - this is what is sending out the TX stream and even though the offset between the transmit streams (from the two different SDRs) varies from one initialization to the next (brought on by restarting the program).

In other words, I start the program and all of the stream set commands occur, then I start looping through the LMS_SendStream commands and observe a constant time offset between the SDRs that only changes if I stop the program and restart it. For that reason I can synchronize them by offsetting the TX timestamp of one relative to the other, but the actual time offset still varies from one initialization to the next, making the synchronization impossible. I want to make it clear I am not actually thinking I can synchronize them this way, but am trying to understand how they work.

I guess what I don’t understand is why the time offset doesn’t vary for each loop? I have tried restructuring the code I am running by giving the LMS_SendStream commands so they are given in a separate order, etc, however the relative offset of one particular SDR relative to the other remains (i.e. one SDR is always timed before the other, and the offset time is on the order of 4 msec).

Maybe I have described this poorly, but if anyone can explain to me why I see this (or give me any ideas), please let me know.

Thanks in advance for all of your help.

This was written with LimeSDR-USB in mind and describes where offsets arise across the two channels and how these may be aligned.

However, Mini uses the same RFIC and hence it may provide some useful insights.

Thank you - I will take a look at this. For clarity, I am talking about 2 completely separate devices (separate usb connections) and not 2 channels on the same device.

Indeed, but I nevertheless thought it might have some useful insights.

You have both SDRs running off the same clock, so the samples counting will be synchronized on both boards, and assuming you’re sending the Tx packets with a designated timestamp when they should be transmitted, then there will be a constant offset in time between the boards, as they are running at the same rate, from the same clock source.

The time offset between the boards is the result of host system and USB data transmission latency. Each board hardware activates it’s own stream and starts counting samples once their FPGA register 0x000A[0] is set to 1.
So the time offset your seeing is the time difference between that register change in both boards.

Such huge time difference is a flaw of the legacy LimeSuite API, the LMS_StartStream() function is creating threads for data processing, which is a relatively expensive operation, thus it takes milliseconds. Therefore once the first board is started and already running, the second board start takes couple of milliseconds to create it’s threads and only then it activates. After that point due to shared clock the boards sampling rates are synchronized in time, but have different start points in time.
The new GitHub - myriadrf/LimeSuiteNG API creates threads preemptively, so the Start() operation is minimal, essentially just a write opration to that register. That would reduce the time offset significantly (to micro seconds), but the boards would still not be trully started at the same time.
The boards are independent from each other and the USB is a serial bus shared among all other connected devices, so there is no guaranteed way to send a time point synchronized command to both boards from the PC without having some latency.
The only way to trully synchronize both boards would have to involve something at the hardware level, like one board signaling the stream start to the other through GPIO or something like that.

Thank you very much for this explanation - it makes a lot of sense. I do realize that a hardware signal is needed ideally to synchronize the two boards, but I had not looked into LimeSuiteNG previously - I will do that now.

Hi,

How would one go about trying to implement the GPIO trigger? I am trying to set up a similar system using GNU radio. New to SDRs but tryingg to my best to learn. Should I try to use limesuitNG instead of gnuradio if I am using limesdrs?

FPGA gateware would need to be modified to generate/receive such trigger.

LimeSuiteNG provides GNU Radio plugin, so It doesn’t matter which one you use, as the boards synchronization has to be done at hardware level.

1 Like

Okay, so after some initial tests, I am realizing that I have some issues with my setup that would go away if I could implement this HW trigger. Unfortunately I have done 1 course in Verilog 10 years ago… Would you dare make an estimate on how complex it would be to try and implement something like this? It sounds kind of easy but that’s just me thinking about what would be needed, in order to do it in embedded c.

@VytautasB is probably the best person to advise.

Hello @malj,

It should be fairly easy to implement. I will check what is needed and get back to you.

regards,
Vytautas

2 Likes

I am also interested in that, so if you could let me know I would also appreciate it. Thanks!

1 Like

Hello @VytautasB, dont want to pester you, but have you had time too have a look at this?

Hello @malj,

yes, I did some work on this. Probably I can give you updated gateware to test sometime next week.

Regards,
Vytautas

1 Like

oh, amazing!

Hi @malj, @tkp,

you can find updated GW here.

Use latest LimeSuiteNG from “develop” branch to update GW for LimeSDR-Mini V2

FPGA_EGPIO description:
FPGA_EGPIO[0] - Stream Enable trigger output
FPGA_EGPIO[1] - Stream Enable trigger input

Prepare hardware:
Connect FPGA_EGPIO[0] output from “master board” to FPGA_EGPIO[1] input on “slave board”

Usage:

  1. Set FPGA SPI register bit 0x000A[3] to ‘1’ for slave board before starting data stream.
  2. Start data stream on slave board. (Now Slave board is ready and waits trigger from FPGA_EGPIO[1] input)
  3. Start data stream on master board

After this procedure data stream and Sample counters should start at the same time on both boards.
If you do the testing let me know if it works.

Regards,
Vytautas

Thank you so much for this - it will take me a bit to test it as I have just had shoulder surgery, but I will be testing it in the next month or so.

Amazing! Flu is running in the family, trying to get some time to test this week I hope…

One update here before I forget: I did write up some LimeSuiteNG code and indeed the start streaming commands take much less time: on the order of 600 usec rather than close to 4.5 msec (MacBook M4 Pro). The variation in the time taken is also much less: on the order of 40 usec rather than 100’s of usec previously. So this is indeed a huge improvement - it gets the boards closer without hardware synchronization (and is even close enough for some applications), but doesn’t get them all the way there as expected.

Hello again! I am finally about to test this, just want to double check how to revert it if I have to. Can I just flash the *_golden.bit file? I assume I just use the limeFLASH util?