# Synchronize two LimeSDR

**URL:** <https://discourse.myriadrf.org/t/synchronize-two-limesdr/1714>\
**Category:** LimeSDR\
**Created:** [18 September 2017 09:52 UTC](https://discourse.myriadrf.org/t/synchronize-two-limesdr/1714 "2017-09-18T09:52:56Z")\
**Posts on this page:** 20\
**Page:** 2

<div class="post-metadata">

**Author:** ![Zero](https://avatars.discourse-cdn.com/v4/letter/z/a88e57/32.png) [@Zero](https://discourse.myriadrf.org/u/Zero)\
**Post date:** [19 September 2017 13:39 UTC](https://discourse.myriadrf.org/t/synchronize-two-limesdr/1714/22 "2017-09-19T13:39:43Z")

</div>

Improved kludge: use the PPS to drive an RF switch. Run a carrier out the second TX, through the switch, and into the second RX. When the PPS occurs you see the tone in the receiver; you can find the leading edge and correlate the PPS with buffer sample number.

---

<div class="post-metadata">

**Author:** ![mitch](https://avatars.discourse-cdn.com/v4/letter/m/dec6dc/32.png) [@mitch](https://discourse.myriadrf.org/u/mitch)\
**Post date:** [23 October 2017 22:57 UTC](https://discourse.myriadrf.org/t/synchronize-two-limesdr/1714/23 "2017-10-23T22:57:19Z")

</div>

> [@Zero](#):
>
> drive an RF switch. Run a carrier out the second TX, through the switch, and into the second RX. When the PPS occurs you see the tone in the receiver; you can find the leading edge and

Hi @Zero did you guys implement the external 10MHz reference in the LimeSDR ? We could really use that. We are using OAI and need external clock synchronization.

If not, what would it take to make the Lime accept the external clock from OAI? I can make the change on the OAI side but have less knowledge of the LimeSDR code.

Thanks,  
Mitch

---

<div class="post-metadata">

**Author:** ![Zero](https://avatars.discourse-cdn.com/v4/letter/z/a88e57/32.png) [@Zero](https://discourse.myriadrf.org/u/Zero)\
**Post date:** [23 October 2017 23:46 UTC](https://discourse.myriadrf.org/t/synchronize-two-limesdr/1714/24 "2017-10-23T23:46:07Z")

</div>

The external reference input already exists, if all you need is frequency coherence. What’s missing is an external time sync input.

I didn’t implement it, but don’t think it would be such a big deal. My thinking is that time syncronized sample reception/transmission could be achieved using the real time tracking counters in the RX/TX burst control hardware. Set the software up to send at some time (say, 1e20 ns) into the future - much to long from the present to happen due to elapsed time. Then, the external signal occurs (brought in via one of the GPIO lines) and sets the real time counters to 1e20 - and there goes/comes your burst.

Alternatively you could just zero the counters with the external input, and transmit at some time after zero - but there’s a race condition. The counter is zeroed when the stream is opened, and starts running - it’s possible you would get to your transmit time before you received your sync input.

I’d be happy to hack something together, but I’m not in a position to test it - my burst-driven code still doesn’t work, and I still don’t know why. (You don’t know of any documentation that explains the ‘C’ api doing burst scheduling, by any chance?)

---

<div class="post-metadata">

**Author:** ![mpb](https://yyz2.discourse-cdn.com/flex034/user_avatar/discourse.myriadrf.org/mpb/32/1381_2.png) [@mpb](https://discourse.myriadrf.org/u/mpb)\
**Post date:** [5 November 2017 12:25 UTC](https://discourse.myriadrf.org/t/synchronize-two-limesdr/1714/25 "2017-11-05T12:25:16Z")

</div>

Hi all,

I’m currently refactoring the output logic of [ODR-DabMod](https://github.com/Opendigitalradio/ODR-DabMod/), a Digital Audio Broadcasting modulator, which has separate UHD and SoapySDR blocks. The intention is to make all features available to both USRPs and the LimeSDR. These include a digital predistortion prototype (which needs feedback after power amplifier, implemented with continuous TX and occasional RX bursts, with relative timestamps), and the ability to sync the transmission to a specific GPS-derived time (for single-frequency networks, where several units are transmitting from different remote sites). The latter also needs an external disciplined REFCLK, which I understood is supported.

Long story short: I’m also interested in the ability to set the HW time on the PPS rising edge.

My code assumes system time is synced using NTP, and the second change (in system time) is close to the PPS edge. To summarise: software waits until the second changes (+ margin, say 100ms), and tells the device “on the next PPS edge set your HW time to current\_second+1”.

If I understand the discussion correctly, the LimeSDR doesn’t support anything like that for now. Is anybody working on this already? Could I help in any way (FPGA/SW dev)?

Cheers  
mpb

---

<div class="post-metadata">

**Author:** ![andrewback](https://avatars.discourse-cdn.com/v4/letter/a/22d042/32.png) [@andrewback](https://discourse.myriadrf.org/u/andrewback)\
**Post date:** [6 November 2017 11:35 UTC](https://discourse.myriadrf.org/t/synchronize-two-limesdr/1714/26 "2017-11-06T11:35:27Z")

</div>

So you need a PPS input on the LimeSDR that can be used to set O/S time on the host? There is a GPIO header and API to access this, but I’m not certain that this is what you want.

> <https://github.com/myriadrf/LimeSuite/blob/master/src/examples/gpio_example.cpp>

Adding in @Zack as he is best positioned to comment.

---

<div class="post-metadata">

**Author:** ![mpb](https://yyz2.discourse-cdn.com/flex034/user_avatar/discourse.myriadrf.org/mpb/32/1381_2.png) [@mpb](https://discourse.myriadrf.org/u/mpb)\
**Post date:** [6 November 2017 15:20 UTC](https://discourse.myriadrf.org/t/synchronize-two-limesdr/1714/27 "2017-11-06T15:20:40Z")

</div>

No, when I say “HW time” I mean the time inside the LimeSDR.

My TX signal is timestamped in absolute time, each sample has a “time of release” associated with it. So I need the LimeSDR to have absolute time. My OS knows the time (thanks to NTP), but I cannot just tell the LimeSDR “initialise time to X” because the latency of that command over USB is not predictable. [1]

That’s why I need the PPS input. I hope my previous question makes more sense now.

Cheers  
mpb

[1]  
Maybe that assumption is wrong, in which case I’d be curious to know how to set the HW time without a PPS signal.

---

<div class="post-metadata">

**Author:** ![mpb](https://yyz2.discourse-cdn.com/flex034/user_avatar/discourse.myriadrf.org/mpb/32/1381_2.png) [@mpb](https://discourse.myriadrf.org/u/mpb)\
**Post date:** [28 November 2017 04:14 UTC](https://discourse.myriadrf.org/t/synchronize-two-limesdr/1714/28 "2017-11-28T04:14:49Z")

</div>

Hi again,

I’ve looked at the LimeSuite sources, and see that neither `ILimeSDRStreaming::Streamer::SetHardwareTimestamp` nor `ILimeSDRStreaming::Streamer::GetHardwareTimestamp` do any interaction with the device.

In the HDL, I cannot find any mention of a hardware time counter, neither do I understand what happens on the FPGA with the timestamp given in the `counter` field of `lime::FPGA_DataPacket`.

Before I continue reading: is there a hardware time unit in the FPGA? Where is it?

---

<div class="post-metadata">

**Author:** ![IgnasJ](https://avatars.discourse-cdn.com/v4/letter/i/278dde/32.png) [@IgnasJ](https://discourse.myriadrf.org/u/IgnasJ)\
**Post date:** [28 November 2017 07:37 UTC](https://discourse.myriadrf.org/t/synchronize-two-limesdr/1714/29 "2017-11-28T07:37:00Z")

</div>

Hello,

> [@mpb](#):
>
> I’ve looked at the LimeSuite sources, and see that neither ILimeSDRStreaming::Streamer::SetHardwareTimestamp nor ILimeSDRStreaming::Streamer::GetHardwareTimestamp do any interaction with the device.

SetHardwareTimestamp - sets timestamp offset in software. It does not actually change timestamp in HW.  
GetHardwareTimestamp - returns last received HW timestamp + offset (if set).

> [@mpb](#):
>
> neither do I understand what happens on the FPGA with the timestamp given in the counter field of lime::FPGA\_DataPacket.

In Rx packet, it is the value of HW timestamp at the time when the first sample in the packet was received.  
In Tx packet, samples in packet are sent to LMS chip only when specified timestamp value is reached. (synchronization to timestamp has to be enabled).

---

<div class="post-metadata">

**Author:** ![mpb](https://yyz2.discourse-cdn.com/flex034/user_avatar/discourse.myriadrf.org/mpb/32/1381_2.png) [@mpb](https://discourse.myriadrf.org/u/mpb)\
**Post date:** [28 November 2017 09:23 UTC](https://discourse.myriadrf.org/t/synchronize-two-limesdr/1714/30 "2017-11-28T09:23:21Z")

</div>

Thanks for your answer!

> [@IgnasJ](#):
>
> In Rx packet, it is the value of HW timestamp at the time when the first sample in the packet was received.  
> In Tx packet, samples in packet are sent to LMS chip only when specified timestamp value is reached. (synchronization to timestamp has to be enabled).

Where is this HW timestamp counter in the HDL design?  
Can I reset this counter or set it to a given value at the exact instant a GPIO goes high?

---

<div class="post-metadata">

**Author:** ![IgnasJ](https://avatars.discourse-cdn.com/v4/letter/i/278dde/32.png) [@IgnasJ](https://discourse.myriadrf.org/u/IgnasJ)\
**Post date:** [28 November 2017 15:04 UTC](https://discourse.myriadrf.org/t/synchronize-two-limesdr/1714/31 "2017-11-28T15:04:32Z")

</div>

> [@mpb](#):
>
> Where is this HW timestamp counter in the HDL design?

> <https://github.com/myriadrf/LimeSDR-USB_GW/blob/master/src/rx_packets/rx_pct_data_v2.vhd#L520>

> [@mpb](#):
>
> Can I reset this counter or set it to a given value at the exact instant a GPIO goes high?

No, there is no such functionality, Currently, timestamp can be reset to 0 only from software (it is done before starting streaming)

---

<div class="post-metadata">

**Author:** ![mpb](https://yyz2.discourse-cdn.com/flex034/user_avatar/discourse.myriadrf.org/mpb/32/1381_2.png) [@mpb](https://discourse.myriadrf.org/u/mpb)\
**Post date:** [1 December 2017 03:09 UTC](https://discourse.myriadrf.org/t/synchronize-two-limesdr/1714/32 "2017-12-01T03:09:12Z")

</div>

Thanks for the pointers! This counter lives in the RX path, I don’t understand how it’s influencing the TX path yet.

`lpmcnt_q` is only going to `pct_smplnr`, which only goes to `outfifo_data`, then straight through the `rx_path` to the `lms7_trx_top.bdf`, which I cannot parse easily without installing Quartus.

Am I missing a path from the counter to the TX path?

---

<div class="post-metadata">

**Author:** ![VytautasB](https://avatars.discourse-cdn.com/v4/letter/v/ccd318/32.png) [@VytautasB](https://discourse.myriadrf.org/u/VytautasB)\
**Post date:** [1 December 2017 10:59 UTC](https://discourse.myriadrf.org/t/synchronize-two-limesdr/1714/33 "2017-12-01T10:59:23Z")

</div>

> [@mpb](#):
>
> Thanks for the pointers! This counter lives in the RX path, I don’t understand how it’s influencing the TX path yet.

Actually there are two counters.

The first one counter that you have already found is included in RX transfer packets which are sent to PC trough USB.

You can track down wrrxfifo\_wr signal which comes from rx\_path and you will found similar counter in TX path (called - sample\_nr\_cnt\_mimo). This counter is incremented when RX sample is captured and used to synchronize TX samples with RX samples. TX sample synchronization is done in tx\_pct\_data\_mimo\_v3 module.

---

<div class="post-metadata">

**Author:** ![mpb](https://yyz2.discourse-cdn.com/flex034/user_avatar/discourse.myriadrf.org/mpb/32/1381_2.png) [@mpb](https://discourse.myriadrf.org/u/mpb)\
**Post date:** [25 December 2017 09:35 UTC](https://discourse.myriadrf.org/t/synchronize-two-limesdr/1714/34 "2017-12-25T09:35:45Z")

</div>

Thank you @VytautasB for the pointers. Why was this design with two counters used? Is this needed for some application? I was expected to find only one counter that represents time, I’m probably not seeing the whole picture.

For my application, I need the following two behaviours:

- The TX counter can be synchronised to an external event (1PPS rising edge) and counts all the time, even in case the TX stream is interrupted. The TX stream starts when the TX timestamp matches the counter.
- There is a continuous TX stream with timestamps, and from time to time an RX burst is done, and the RX packets can be aligned to the TX stream thanks to its timestamps.

Both are easy to achieve if there is a single always-running counter used for the timestamps.

What would be the easiest way to achieve these two behaviours?

---

<div class="post-metadata">

**Author:** ![ogun](https://avatars.discourse-cdn.com/v4/letter/o/ce7236/32.png) [@ogun](https://discourse.myriadrf.org/u/ogun)\
**Post date:** [27 December 2017 11:51 UTC](https://discourse.myriadrf.org/t/synchronize-two-limesdr/1714/35 "2017-12-27T11:51:18Z")

</div>

Is it possible to connect two limeSDRs to same SBC and use one of them to extend rx bandwidth and use one for transmit signals on gnuradio osmocom source by connecting their clocks together only?

---

<div class="post-metadata">

**Author:** ![cmichal](https://avatars.discourse-cdn.com/v4/letter/c/8e7dd6/32.png) [@cmichal](https://discourse.myriadrf.org/u/cmichal)\
**Post date:** [12 March 2018 23:40 UTC](https://discourse.myriadrf.org/t/synchronize-two-limesdr/1714/36 "2018-03-12T23:40:46Z")

</div>

Not sure how lively this thread is, but I was wanting something along the lines of the PPS reset of the timestamps discussed above, and did make a small change to the FPGA to do it. On the top level of the FPGA source (lms7\_trx\_top.bdf) I disconnected lte\_clr\_smpl\_nr from the rx\_path\_top block. Then added an OR gate. The output of the OR goes to the clr\_smpl\_nr input of rx\_path\_top, the inputs of the OR come from lte\_clr\_smpl\_nr and from FPGA\_GPIO[0]. So now I can reset the timestamps with a pulse on GPIO0. If you don’t want to use it, you can set that GPIO as an output (low), but if you do want to use it, set that GPIO as an input. Seems to work great.

In trying to get the rbf onto the board though I noticed that function in LimeSuite doesn’t seem to work. LimeUtil does though.

Code-wise, its true you need to look for places where a timestamp is less than a previous timestamp, but that’s actually way easier than the rf switch solution mentioned above (which I did also implement and made work - but what a pain - since gain changes with frequency quite badly in the range I’m using, and then you have to look at the magnitude of every sample to figure out where the pulse starts or stops).

In my application there is no tx happening when the timestamp is reset, though I start transmitting shortly afterwards and things seem to work correctly. I’m not sure how nice things would be if you were in the middle of transmitting when the timestamps suddenly reset.

---

<div class="post-metadata">

**Author:** ![KarlL](https://avatars.discourse-cdn.com/v4/letter/k/dfb087/32.png) [@KarlL](https://discourse.myriadrf.org/u/KarlL)\
**Post date:** [22 March 2018 16:35 UTC](https://discourse.myriadrf.org/t/synchronize-two-limesdr/1714/37 "2018-03-22T16:35:10Z")

</div>

Does it work well? And how does it work exactly?

The 2 limes are receiving, then something (maybe one of the Limes) that is plugged to GPIO 0 of the 2 limes resets the counter on both at basically the same time (the same signal would be sent to both through a T), and so the next group of samples that is digitized will have a timestamp close to zero (probably not zero else it would mean the counter was reset at the beginning of a block of samples). From now on you can easily compare the samples from both Limes thanks to their shared timestamp.

Is this how it works?

@IgnasJ Any thoughts on this? Also if synchronization works well like this, and as it seems the changes at the FPGA level are minimum, could such changes be added to the official repo? Though thinking of it this would make 1 less available GPIO…

@cmichal Do you have a fork that others can use to test this? Or some detailed explanations for a complete FPGA novice on how to modify it?

A _LimeSDR Made Simple_ on the FPGA, how it works, what it does, how it’s linked to others things, would actually be really interesting.

---

<div class="post-metadata">

**Author:** ![cmichal](https://avatars.discourse-cdn.com/v4/letter/c/8e7dd6/32.png) [@cmichal](https://discourse.myriadrf.org/u/cmichal)\
**Post date:** [22 March 2018 18:03 UTC](https://discourse.myriadrf.org/t/synchronize-two-limesdr/1714/38 "2018-03-22T18:03:51Z")

</div>

What you describe is basically right -toggling the GPIO high then low will reset the timestamp counter, and you detect it by seeing that the most recent timestamp is lower than the one before. I think if you see a timestamp of 0 you shouldn’t trust it, because it probably gets held at zero as long as the GPIO is high, so look at the timestamp of the next packet.

I think the only source file that’s changed is the top level bdf. I’ve put mine at [http://www.phas.ubc.ca/~michal/lms7\_trx\_top.bdf](http://www.phas.ubc.ca/~michal/lms7_trx_top.bdf), and the binary at: [http://www.phas.ubc.ca/~michal/LimeSDR-USB\_lms7\_trx\_HW\_1.4.rbf](http://www.phas.ubc.ca/~michal/LimeSDR-USB_lms7_trx_HW_1.4.rbf)

If you open the .qsf, then browse the .bdf and search for the signal names I mentioned above, it should be easy to spot the or gate I added. The diff of the bdf seems very large, but I’m using a newer version of Quartus and it seems to have moved things around.

---

<div class="post-metadata">

**Author:** ![KarlL](https://avatars.discourse-cdn.com/v4/letter/k/dfb087/32.png) [@KarlL](https://discourse.myriadrf.org/u/KarlL)\
**Post date:** [23 March 2018 00:32 UTC](https://discourse.myriadrf.org/t/synchronize-two-limesdr/1714/39 "2018-03-23T00:32:14Z")

</div>

Thanks a lot, I’ll take a look at it.

---

<div class="post-metadata">

**Author:** ![mleech](https://avatars.discourse-cdn.com/v4/letter/m/34f0e0/32.png) [@mleech](https://discourse.myriadrf.org/u/mleech)\
**Post date:** [24 March 2018 00:07 UTC](https://discourse.myriadrf.org/t/synchronize-two-limesdr/1714/40 "2018-03-24T00:07:23Z")

</div>

So, I didn’t realize that LimeSDR supported timestamps AT ALL, let alone had any mechanisms  
that reset them. This is good news for people wanting to do coherence across multiple devices.

Quite apart from the external-trigger mechanism (1PPS, etc), there’s also, in USB3, the ITP packet which is broadcast to all devices. The Isochronous Timestamp Packet includes a 27-bit sequence number.

So, one might be able to setup a scenario were the host says to all the attached Lime boards “hey guys, when you see ITP sequence number , start streaming”.

---

<div class="post-metadata">

**Author:** ![andrewback](https://avatars.discourse-cdn.com/v4/letter/a/22d042/32.png) [@andrewback](https://discourse.myriadrf.org/u/andrewback)\
**Post date:** [27 March 2018 08:46 UTC](https://discourse.myriadrf.org/t/synchronize-two-limesdr/1714/41 "2018-03-27T08:46:57Z")

</div>

That is an interesting idea. @Zack, curious as to your thoughts on how hard this might be to implement?

[Previous page](https://discourse.myriadrf.org/t/synchronize-two-limesdr/1714.md?page=1)

[Next page](https://discourse.myriadrf.org/t/synchronize-two-limesdr/1714.md?page=3)