Synchronize two LimeSDR

Zero · 19 September 2017 13:39

Improved kludge: use the PPS to drive an RF switch. Run a carrier out the second TX, through the switch, and into the second RX. When the PPS occurs you see the tone in the receiver; you can find the leading edge and correlate the PPS with buffer sample number.

mitch · 23 October 2017 22:57

Hi @Zero did you guys implement the external 10MHz reference in the LimeSDR ? We could really use that. We are using OAI and need external clock synchronization.

If not, what would it take to make the Lime accept the external clock from OAI? I can make the change on the OAI side but have less knowledge of the LimeSDR code.

Thanks,
Mitch

Zero · 23 October 2017 23:46

The external reference input already exists, if all you need is frequency coherence. What’s missing is an external time sync input.

I didn’t implement it, but don’t think it would be such a big deal. My thinking is that time syncronized sample reception/transmission could be achieved using the real time tracking counters in the RX/TX burst control hardware. Set the software up to send at some time (say, 1e20 ns) into the future - much to long from the present to happen due to elapsed time. Then, the external signal occurs (brought in via one of the GPIO lines) and sets the real time counters to 1e20 - and there goes/comes your burst.

Alternatively you could just zero the counters with the external input, and transmit at some time after zero - but there’s a race condition. The counter is zeroed when the stream is opened, and starts running - it’s possible you would get to your transmit time before you received your sync input.

I’d be happy to hack something together, but I’m not in a position to test it - my burst-driven code still doesn’t work, and I still don’t know why. (You don’t know of any documentation that explains the ‘C’ api doing burst scheduling, by any chance?)

mpb · 5 November 2017 12:25

Hi all,

I’m currently refactoring the output logic of ODR-DabMod, a Digital Audio Broadcasting modulator, which has separate UHD and SoapySDR blocks. The intention is to make all features available to both USRPs and the LimeSDR. These include a digital predistortion prototype (which needs feedback after power amplifier, implemented with continuous TX and occasional RX bursts, with relative timestamps), and the ability to sync the transmission to a specific GPS-derived time (for single-frequency networks, where several units are transmitting from different remote sites). The latter also needs an external disciplined REFCLK, which I understood is supported.

Long story short: I’m also interested in the ability to set the HW time on the PPS rising edge.

My code assumes system time is synced using NTP, and the second change (in system time) is close to the PPS edge. To summarise: software waits until the second changes (+ margin, say 100ms), and tells the device “on the next PPS edge set your HW time to current_second+1”.

If I understand the discussion correctly, the LimeSDR doesn’t support anything like that for now. Is anybody working on this already? Could I help in any way (FPGA/SW dev)?

Cheers
mpb

andrewback · 6 November 2017 11:35

So you need a PPS input on the LimeSDR that can be used to set O/S time on the host? There is a GPIO header and API to access this, but I’m not certain that this is what you want.

github.com

myriadrf/LimeSuite/blob/master/src/examples/gpio_example.cpp

/**
    @file   gpio_example.cpp
    @author Lime Microsystems (www.limemicro.com)
    @brief  GPIO example
 */
#include <lime/LimeSuite.h>
#include <iostream>
#include <chrono>
#include <thread>

//Device structure, should be initialize to NULL
lms_device_t* device = NULL;

int error()
{
    if (device != NULL)
        LMS_Close(device);
    exit(-1);
}

This file has been truncated. show original

Adding in @Zack as he is best positioned to comment.

mpb · 6 November 2017 15:20

No, when I say “HW time” I mean the time inside the LimeSDR.

My TX signal is timestamped in absolute time, each sample has a “time of release” associated with it. So I need the LimeSDR to have absolute time. My OS knows the time (thanks to NTP), but I cannot just tell the LimeSDR “initialise time to X” because the latency of that command over USB is not predictable. [1]

That’s why I need the PPS input. I hope my previous question makes more sense now.

Cheers
mpb

[1]
Maybe that assumption is wrong, in which case I’d be curious to know how to set the HW time without a PPS signal.

mpb · 28 November 2017 04:14

Hi again,

I’ve looked at the LimeSuite sources, and see that neither ILimeSDRStreaming::Streamer::SetHardwareTimestamp nor ILimeSDRStreaming::Streamer::GetHardwareTimestamp do any interaction with the device.

In the HDL, I cannot find any mention of a hardware time counter, neither do I understand what happens on the FPGA with the timestamp given in the counter field of lime::FPGA_DataPacket.

Before I continue reading: is there a hardware time unit in the FPGA? Where is it?

IgnasJ · 28 November 2017 07:37

Hello,

SetHardwareTimestamp - sets timestamp offset in software. It does not actually change timestamp in HW.
GetHardwareTimestamp - returns last received HW timestamp + offset (if set).

In Rx packet, it is the value of HW timestamp at the time when the first sample in the packet was received.
In Tx packet, samples in packet are sent to LMS chip only when specified timestamp value is reached. (synchronization to timestamp has to be enabled).

mpb · 28 November 2017 09:23

Thanks for your answer!

Where is this HW timestamp counter in the HDL design?
Can I reset this counter or set it to a given value at the exact instant a GPIO goes high?

IgnasJ · 28 November 2017 15:04

github.com

myriadrf/LimeSDR-USB_GW/blob/master/src/rx_packets/rx_pct_data_v2.vhd#L520


      
            port map(
                  clk             => clk, 
                  reset_n         => reset_n,
                  data_in         => diq0,
                  data_in_valid   => infifo_rd_sig,
                  sample_width    => sample_width_reg1, 
                  data_out        => cmprsd_data,
                  data_out_valid  => cmprsd_data_valid     
                  );
                  
          		cnt_inst : LPM_COUNTER
          	GENERIC MAP (
          		lpm_direction 		=> "UP",
          		lpm_port_updown 	=> "PORT_UNUSED",
          		lpm_type 			=> "LPM_COUNTER",
          		lpm_width 			=> 64
          	)
          	PORT MAP (
          		aclr 		=> clr_smpl_nr,
          		clock 	=> clk,
          		cnt_en 	=> lpmcnt_cnten,

No, there is no such functionality, Currently, timestamp can be reset to 0 only from software (it is done before starting streaming)

mpb · 1 December 2017 03:09

Thanks for the pointers! This counter lives in the RX path, I don’t understand how it’s influencing the TX path yet.

lpmcnt_q is only going to pct_smplnr, which only goes to outfifo_data, then straight through the rx_path to the lms7_trx_top.bdf, which I cannot parse easily without installing Quartus.

Am I missing a path from the counter to the TX path?

VytautasB · 1 December 2017 10:59

Actually there are two counters.

The first one counter that you have already found is included in RX transfer packets which are sent to PC trough USB.

You can track down wrrxfifo_wr signal which comes from rx_path and you will found similar counter in TX path (called - sample_nr_cnt_mimo). This counter is incremented when RX sample is captured and used to synchronize TX samples with RX samples. TX sample synchronization is done in tx_pct_data_mimo_v3 module.

mpb · 25 December 2017 09:35

Thank you @VytautasB for the pointers. Why was this design with two counters used? Is this needed for some application? I was expected to find only one counter that represents time, I’m probably not seeing the whole picture.

For my application, I need the following two behaviours:

The TX counter can be synchronised to an external event (1PPS rising edge) and counts all the time, even in case the TX stream is interrupted. The TX stream starts when the TX timestamp matches the counter.
There is a continuous TX stream with timestamps, and from time to time an RX burst is done, and the RX packets can be aligned to the TX stream thanks to its timestamps.

Both are easy to achieve if there is a single always-running counter used for the timestamps.

What would be the easiest way to achieve these two behaviours?

ogun · 27 December 2017 11:51

Is it possible to connect two limeSDRs to same SBC and use one of them to extend rx bandwidth and use one for transmit signals on gnuradio osmocom source by connecting their clocks together only?

cmichal · 12 March 2018 23:40

Not sure how lively this thread is, but I was wanting something along the lines of the PPS reset of the timestamps discussed above, and did make a small change to the FPGA to do it. On the top level of the FPGA source (lms7_trx_top.bdf) I disconnected lte_clr_smpl_nr from the rx_path_top block. Then added an OR gate. The output of the OR goes to the clr_smpl_nr input of rx_path_top, the inputs of the OR come from lte_clr_smpl_nr and from FPGA_GPIO[0]. So now I can reset the timestamps with a pulse on GPIO0. If you don’t want to use it, you can set that GPIO as an output (low), but if you do want to use it, set that GPIO as an input. Seems to work great.

In trying to get the rbf onto the board though I noticed that function in LimeSuite doesn’t seem to work. LimeUtil does though.

Code-wise, its true you need to look for places where a timestamp is less than a previous timestamp, but that’s actually way easier than the rf switch solution mentioned above (which I did also implement and made work - but what a pain - since gain changes with frequency quite badly in the range I’m using, and then you have to look at the magnitude of every sample to figure out where the pulse starts or stops).

In my application there is no tx happening when the timestamp is reset, though I start transmitting shortly afterwards and things seem to work correctly. I’m not sure how nice things would be if you were in the middle of transmitting when the timestamps suddenly reset.

KarlL · 22 March 2018 16:35

Does it work well? And how does it work exactly?

The 2 limes are receiving, then something (maybe one of the Limes) that is plugged to GPIO 0 of the 2 limes resets the counter on both at basically the same time (the same signal would be sent to both through a T), and so the next group of samples that is digitized will have a timestamp close to zero (probably not zero else it would mean the counter was reset at the beginning of a block of samples). From now on you can easily compare the samples from both Limes thanks to their shared timestamp.

Is this how it works?

@IgnasJ Any thoughts on this? Also if synchronization works well like this, and as it seems the changes at the FPGA level are minimum, could such changes be added to the official repo? Though thinking of it this would make 1 less available GPIO…

@cmichal Do you have a fork that others can use to test this? Or some detailed explanations for a complete FPGA novice on how to modify it?

A LimeSDR Made Simple on the FPGA, how it works, what it does, how it’s linked to others things, would actually be really interesting.

cmichal · 22 March 2018 18:03

What you describe is basically right -toggling the GPIO high then low will reset the timestamp counter, and you detect it by seeing that the most recent timestamp is lower than the one before. I think if you see a timestamp of 0 you shouldn’t trust it, because it probably gets held at zero as long as the GPIO is high, so look at the timestamp of the next packet.

I think the only source file that’s changed is the top level bdf. I’ve put mine at http://www.phas.ubc.ca/~michal/lms7_trx_top.bdf, and the binary at: http://www.phas.ubc.ca/~michal/LimeSDR-USB_lms7_trx_HW_1.4.rbf

If you open the .qsf, then browse the .bdf and search for the signal names I mentioned above, it should be easy to spot the or gate I added. The diff of the bdf seems very large, but I’m using a newer version of Quartus and it seems to have moved things around.

KarlL · 23 March 2018 00:32

Thanks a lot, I’ll take a look at it.

mleech · 24 March 2018 00:07

So, I didn’t realize that LimeSDR supported timestamps AT ALL, let alone had any mechanisms
that reset them. This is good news for people wanting to do coherence across multiple devices.

Quite apart from the external-trigger mechanism (1PPS, etc), there’s also, in USB3, the ITP packet which is broadcast to all devices. The Isochronous Timestamp Packet includes a 27-bit sequence number.

So, one might be able to setup a scenario were the host says to all the attached Lime boards “hey guys, when you see ITP sequence number , start streaming”.

andrewback · 27 March 2018 08:46

That is an interesting idea. @Zack, curious as to your thoughts on how hard this might be to implement?