Synchronize two LimeSDR

[cmichal]

The easiest thing, I think, to do is to install libLimeSuite.so in /usr/local/lib
(on linux it seems to need to have two symlinks, so there is libLimeSuite.so, libLimeSuite.so.18.01.0 and libLimeSuite.18.01-1, or whatever version numbers you’ve got, all pointing to the same file).

Then there are two include files that need to be found at compile time:
LMS7002M_parameters.h and LimeSuite.h.
If you drop the first into /usr/local/include and the second into
/usr/local/include/lime
Then:
g++ dualRXTX.cpp -lLimeSuite -o dualRXTX

will do the job.

[mc955]

Thanks, that worked a treat.

In terms of cables I have designed a tiny PCB that solders onto the LimeSDR-Mini’s 0.1" GPIO header and then exposes an SMA connector allowing easy connection to the PPS output of a GPS unit via a standard SMA cable.

If anyone would like a board then let me know (UK only).

[mc955]

@cmichal - When sampling at 30.72MS/s using the I16 data format, the function LMS_GetStreamStatus reports around 1077952577 dropped packets each second. I assume this is just the driver being confused by the modified time stamps as the link rate seems to stay the same at 123.404 MB/s when I disable the PPS sync functionality (e.g. hold gpio 0 low). I can’t quite work out however where that number comes from given the increase in the time stamp of 8x16^15 and the sample rate and number of samples per packet.

[cmichal]

I’m afraid I can’t help you on that one. I don’t think I’ve looked at the dropped packets field - I tend to just watch the packet timestamps. It isn’t obvious to me where that number is coming from. I expect you’re right that it is just the driver subtracting something that’s negative - and maybe it gets turned into a 32 bit number somewhere? I expect a dive into the LimeSuite source looking for where that field gets set would illuminate it somewhat.
Good luck!

[mc955]

@cmichal Yeah looks like a mess, I won’t worry about it.

On a separate note, have you or anybody else used this method to trigger a TX? What would be very useful for my application is to also be able to have TX triggered by the PPS pulse, allowing time of flight measurement.

With the work we have done so far we know the index of the rx sample corresponding to the 0->1 transition of the trigger. Now in the case of the TX stream the timestamp in the metadata is defined as ’ time when the first sample in the submitted buffer should be sent’. My question now I guess is are these the same counters? From looking at the vhdl I think this is the case, but it isn’t immediately clear. From what I can gather in the Limesuite source code, when the first stream is opened the FPGA is setup for streaming, which I guess starts the counter running, and from then on isn’t reconfigured or disabled provided there is one active stream.

That would suggest you should be able to:

1. Enable RX channel
2. Enable TX channel
3. Setup RX stream
4. Record index of PPS event
5. Setup TX stream, then optionally stop & destroy RX stream, but leave channel enabled to tick counter
6. Send buffer of samples to SDR with a time stamp of PPS event + sample rate, so the packet is sent at exactly the next PPS event.

I’d be grateful if anyone with more understanding of how the synchronization works would be able to advise on this in some more depth.

[cmichal]

Hi,

If I understand you, then the answer is yes, this can be done, and I’ve done it. Yes, the TX timestamp is based on the RX counter.

I have a digital pulse sequencer that produces a pulse and supplies it to the LimeSDR GPIO, causing capture of the timestamp. The start of that pulse is my t=0. I then produce pulsed output on the limeSDRs that are reliably synchronized with later digital pulse from the digital pulse sequencer.
I allow a fair bit of latency between the capture and the real start of TX (~200ms?), 1 second will be plenty.

My ordering isn’t exactly what you have, it is:

1. Enable RX and TX channels
2. Set up RX and TX streams
3. Start receiving data on RX stream, waiting for GPIO/PPS event
4. After timestamp capture, send TX packets with timestamps at desired point in the future
5. continue (always) receiving on RX

This actually works very well. There are a few gotchas to watch out for, like it seems that you need to make sure that the TX buffer is empty before you send a TX packets with a discontinuous timestamp.

This works well down to single timestamp precision. If you want to do better than that, it can be done too, though its a fair bit more work.

I’m not sure if I tried disabling the RX stream or not in the SISO case. At some point I had both channels running and found that if you then disabled one of the channels, nothing good happened. I think you’re best off just continuing to stream RX data, even if you’re just throwing it away.

I’ve thought a tiny bit about getting rid of the latency - about having a mode where the timestamp counter doesn’t increment until the GPIO signal is received so that you could transmit pulses one counter tick after the GPIO signal, but haven’t looked seriously into implementing it.

[mc955]

@cmichal Great to hear, that’s exactly what I’m looking for.

For my application, I have a transmitter and multiple receivers, all Lime-Mini’s. The source will be on a plane and the receivers scattered around an airfield. The latency between PPS occurring and transmission isn’t a problem as I can just offset the two events by a known number of samples and adjust my t=0 accordingly.

Single sample precision is fine, the jitter of the PPS is about one sample period (30ns) so there would be no benefit to trying to go further.

In that case I’ll leave the RX enabled - do I still need to read it from the stream with LMS_RecvStream or will the PC side (the one assigned in lms_stream_t) FIFO just overwrite itself and not complain?

Also finally would you just be able to clarify what you meant by ‘you need to make sure that the TX buffer is empty before you send a TX packets with a discontinuous timestamp’. Are you referring to the TX buffer in the FPGA or the PC side FIFO whose size is defined in lms_stream_t streamId? Either way how can you flush it to ensure it is empty before TX?

Thanks,
Matt

[cmichal]

I think you probably do need to read the data with LMS_RecvStream - otherwise it will just pile up in a buffer somewhere - on the host computer I expect. It may not be totally necessary, but I think you’re less likely to find weird problems if you just read and throw away the data.

On the tx side, if you send a packet with a timestamp of 2 seconds, then immediately send another packet with a timestamp of 3 seconds (before the previous one has actually been transmitted), I think what happens is the second packet is transmitted as soon as the first one is finished - it won’t wait.

[mc955]

Thanks for your help, it seems to be working.

Interestingly as I am only using the TX stream to send a 8 buffers of 1360 samples once every second, nothing happens unless I set tx_metadata.flushPartialPacket = true;

In the comments of Lime.h it says ‘send samples to HW even if packet is not completely filled’. I assume this is referring to the 4096 byte packets sent over USB but I am confused because surely I should be able to completely fill 8?

[mc955]

@cmichal Actually I take that earlier comment back, it seems to be sending late for some reason. Think it might be due to flushPartialPacket being set true. Any advice on how to use the TX stream in a bursty way without enabling this?

[cmichal]

I thought when I looked into it, flushPartialPacket was only useful if your tx buffer size was not commensurate with the FPGA packet size. I have always kept my buffer size to be an integral multiple of the packet size, and so I think flushPartialPacket doesn’t do anything.

When you say late - is it late by a fixed amount? Or does the delay vary? I wouldn’t be at all surprised if there is some offset between the tx and rx times, but it should be fixed.

If you don’t have the tx buffer size set to a multiple of the packet size, I strongly recommend it.

[mc955]

@cmichal So the delay doesn’t seem that consistent, however I have noticed something else that seems to be a bigger problem.

Firstly here is a link to the code I am running - https://github.com/mattcoates/iib-project/blob/master/software/pps_sync_tx/main.cpp

TX and RX are connected together externally via a 30dB attenuator.

An example output for one second is:
PPS sync occured at sample 136397134
Samples since last PPS = 30720016
Offset = 14
TX scheduled for sample 143197134
Capture scheduled for sample 143195760
popping from TX, samples popped 0/1360
RX Data rate: 92.5368 MB/s
TX Dropped: 0
Capturing at 143195760

The resulting plot of the output file looks like this:

The output of the python plotting script is:
File contains 21760 samples.
Duration: 708.3333 us

2018-09-24 20:55:08
File begins with sample 143195760
PPS sync occured at sample 136397134
TX occured at sample 143197134
TX offset = 1360 + 14 = 1374

This adds up with the above output from the C++. The offset is stated as being 1374 samples. The below image however shows that the TX event does not occur until sample 4196. In addition whilst the TX buffer is 1360*8=10880 samples long, and all 10880 are sent to the stream, the burst in image 1 above only lasts from 4196 to 12360, which is 8164 samples. In addition you can see discontinuities in the output waveform below.

Writing the TX buffer out to file and plotting looks like this.

Now setting the flushPartialPacket to false and running the same code above gives the following output showing that TX has not occured.

In this case in the terminal each second the message popping from TX, samples popped 0/1360 appears again.

I can’t seem to get my head around what is going on here, any help would be greatly appreciated.

[cmichal]

I don’t think I’ve seen messages about TX samples being popped - so that’s a bit weird. There are two things jumping out at me, neither of which feels likely to address your immediate problem:

1. make sure you don’t schedule TX to start on an odd-numbered timestamp.
2. if TX and RX are the same frequency, you should disable one of the PLL’s. Maybe you’re doing that somewhere I don’t see? There’s a post on the forum here [LimeSDR-USB] Synchronization and Toggle TDD/FDD Mode that gives a couple of lines of code to do it.

The only other thing that’s coming to mind is that at some point in June I fetched the then-latest LimeSuite source, and immediately started having some weird synchronization problems. I fell back to the version I had been using from Jan 2. (commit e5cde28f53ab838a140ea32a67e3f6d569ff4bce).
I didn’t pursue the problems, and haven’t looked at the current code, but you might try that older version of the library.

[KarlL]

@mc955

popping from TX, samples popped 0/1360

This should mean that TX tried to get samples to send, but couldn’t get anything, as if you didn’t give enough samples or you didn’t give them fast enough.

[mc955]

@KarlL I think this message may just be a result of the streamer thread that manages the fifo looking for some samples but not finding any, which makes sense because we send them very infrequently. This message only appears frequently during program execution if the flushPartialPacket flag is set to false.

@cmichal Thanks for the tips, I have added in the snippet to use only one PLL, however as expected nothing changed. May I ask why the timestamp must always be even?

With regards to the LimeSuite version, based off the git commit messages it looks like there have been lots of bug fixes for the Lime-Mini since January this year which makes me reluctant to revert back as it may introduce some errors elsewhere.

I have however made some progress. I flashed the original gateware back and made a short program to do the following:

1. Setup streams and receive a few buffers to allow everything to stabilize
2. Read a buffer of samples and record the timestamp
3. Send a buffer of samples with a timestamp 256 * 1360 samples in the future
4. Record the next 300 buffers and plot the output

The code can be found here:
https://github.com/mattcoates/iib-project/blob/master/software/tx_testing/main.cpp

An example output is:

Devices found: 1 0: LimeSDR Mini, media=USB 3.0, module=FT601, addr=24607:1027, serial=1D3AC940C7E517

Reference clock 40.00 MHz

RX Center frequency: 868 MHz
TX Center frequency: 868 MHz
Selected RX path 3: LNAW
Selected TX path 2: BAND2
Host interface sample rate: 30.72 MHz
RF ADC sample rate: 122.88MHz
RX LPF configured
RX LPF bandwitdh: 10 MHz
TX LPF configured
TX LPF bandwitdh: 10 MHz
Normalized RX Gain: 0.739726
RX Gain: 54 dB
Normalized TX Gain: 0.394737
TX Gain: 30 dB
Rx calibration finished
Tx calibration finished
RX Event at 199920
TX Scheduled for 548080
Delta = 348160
Final RX stamp: 606560
RX dropped: 0
TX dropped: 0
Device closed

The output graph is:

The tx_buffer is 1360 samples long. Here the burst lasts from sample 348260 to 349620 which gives a length of 1360 as required. I ran this program a few times, and in each case the TX event looked to start at 348260, which is 100 samples (or 3.26us) after the scheduled start of 348160. I wonder where this latency comes from, and if it can be predicted.

As before this only works if flushPartialPacket is set to true, even if I make the stream FIFO and tx_buffer the same size. I think I may have to look into the source to see what happens here.

Im also confused about the strange DC offset that has appeared in the output.

Anyway, I’ll keep trying at it and see what happens. I have a suspicion that it may not be wise to shedule a TX event at some point when the most current HW timestamp has the MSB set, so I may move that a few packets into the future.

Thanks.

[cmichal]

Does the DC offset correlate with turning off the second PLL?

I don’t know your FPGA modifications, but for mine, the MSB is only set in a header for received data, the timestamps that get looked at for TX are read out upstream from there, and so should never see the MSB.

The odd-number issue is because in single channel mode the timestamp counter increments after two pairs of IQ samples are received, then to actually count the correct number of samples, the count gets shifted up one bit. So in that mode, the timestamp is always an even number.

Again, this is from the LimeSDR (not mini) FPGA, things may be different on the mini.

I think I have seen a latency of 3ish microseconds, similar to you, though with the LMS7002 configuration unchanged that’s been constant.

I have always had flushPartialPacket set to False. Weird.

[mc955]

I don’t think it has anything to do with the PLL as I saw it before I added in the code to turn it off, and also I RX 1000 buffers from the stream and throw them away prior to reading a timestamp and sending the TX samples.

So my FPGA modifications are the same, but in either case the LMS driver’s most current HW timestamp has the MSB flipped, and if the driver does some sort of sanity check before sending the packet to the FPGA it may discard or modify it.

The timestamp thing makes sense. Lots of the VHDL for the mini is the same as the full size version, it even has the original comments and references to the cyclone family…

Yeah its puzzling why flushPartialPacket must be set true, or even why it matters at all if our packet size is a multiple of 1360. Its really annoying that there isn’t a loop back example for the LMS API that I can have a look at.

[mc955]

So I think I have it working now. The reason why we were seeing a pulse that was shorter than 10880 was because the first packet of 1360 samples was being sent immediately after the SendStream call and the remaining 1360*9 were being transmitted at the correct time, so the first sample of the TX we see in our output is at tx_metadata.timestamp + 1360 + 100, which makes sense.

I had a feeling this was due to the streamer thread not liking being active for so long with nothing to send, so what I did to fix the issue was the following:

1. Start TX stream
2. Send TX samples with a timestamp 75 packets in the future (e.g. 1360*75)
3. Wait until the rx_metadata.timestamp was equal to the time when the TX pulse has been sent
4. Stop TX stream

This seems to work well now, and each second a buffer of 10880 samples is transmitted at exactly a predetermined number of samples after the PPS event. The code can be found here in case anyone is interested.

Finally, @cmichal you mentioned that in single channel mode, the sample counter is incremented only every other sample. I think this is occurring on the Lime-Mini too as when I try to time the number of samples between PPS events, I always get an even number even though there is some small variance due to the VCTCXO. Do you think there would be a way to modify the gate ware easily to increment the counter on every received sample?

[cmichal]

@mc955 Nice going! I’m pretty surprised you have to do that - it seems very odd that there’s something ok with having 75 packets delay with no data, but not longer… But maybe there’s something more subtle going on. With my setup, I send some packets, leave long delays with nothing, the send a few more packets, and things seem to work. Though maybe what you’ve found is related to the problems I started seeing with a later LimeSuite library. I’ll have to get back to that someday, though it will be a while.

For the question of odd numbered packets, I did find a solution to have odd numbered samples recorded. Then if I want transmit to start on an odd numbered packet, I just include a zero or three at the beginning of the TX packet, and set the packet to go 1 or 3 samples early.

The modification to the gateware I made is a little more invasive that what I posted earlier. Basically I’ve got a new register in the rx_path_top that keeps track of how many clock ticks have passed since the sample packet number was incremented. Then when the GPIO trigger is observed, you can tell if you’re on an even or odd numbered sample. I tried to make this so it would work generally, but there appear to be several different modes (SISO, MIMO, DDR etc) for which there are several flags. And honestly from the documentation in the code, I couldn’t quite resolve all of what all of those flags do.

You can see the changes to rx_path_top.vhd here:
http://www.phas.ubc.ca/~michal/Lime/rx_path_top.diff
and the complete file here
http://www.phas.ubc.ca/~michal/Lime/rx_path_top.vhd

There are some bits in there that you don’t need. Anything involving trigger_seen, last_bit3, trigger_time, trig_pulse_duration, trig_out_pulse, trig_sync, trig_sync_sync, and div_clk shouldn’t matter to you and can be removed. I’ll apologize for the faulty indentation in this code - Quartus seems to mess it up whether I use tabs or spaces…

[KarlL]

I took the time to retry compiling your 1st version and it does work. I guess I must have forgotten to update some files the previous time I tried.

I just found some small mistakes in the readme.txt:

It should be
cp /path/to/rx_path_top.vhd src/rx_path_top/rx_path/synth/
instead of
cp /path/to/rx_path_path_top.vhd src/rx_path_top/synth/

Also instead of opening LimeSDR-USB_lms7_trx.qsf I opened the project file lms7_trx.qpf and built it.

I’ve also compiled your 2nd version and the synchronization seems to be working too. I’m using my own C++ code, not LimeSuite.

Thanks again for sharing your code.