Synchronize two LimeSDR

I’m using @cmichal modified gateware and it works great. I should have access to a mini so I was thinking of testing your modification too (from LimeSDR-Mini-PPS-Sync-GW)

From what I understand, your modification seems to be working similarly to the one for the LimeSDR (provided that you use the correct buffer sizes, 1360 in sc12, 1020 in sc16). The only problem you have is when you try to send something in the future, the 1st packet is sent at once while the others are sent at the right time?

Thanks.

@KarlL Correct, the modifications are the same as those made for the LimeSDR by @cmichal above. EGPIO0 is used as the signal source, and take care to make sure the pulse remains high for at least one packet duration - nothing bad happens if you keep it high for a few so I’d just make it 200us or so.

I was experiencing problems with TX’ing in the future as you said, however in the latest commit of my software (that uses the LimeSuite API), this has been fixed. I managed to fix the problem by stopping the stream and then starting it again in between long periods of ‘silence’.

Let me know how you get on. I’ve had a few days off but today I’m going to have a look at getting the counter to increment on every sample in the mini in order to double the timing accuracy. Considering the LimeMini is only a SISO device you would have thought it would have done this anyway, but I guess it has been left as before for consistency.

Thanks @mc955. I indeed make sure my pulse stays high long enough so that I don’t miss it. I usually see it’s high one block, sometimes 2. I’m using an Arduino to generate that pulse, and the timing is quite constant, or at least constant enough.

Just to clarify things. Is the issue with the transmission in the future only in your version of the gateware, or also in the official one? If it’s in the official one, we should mention it to the team. It would be more convenient to just be able to leave the stream open. Also stopping then starting the stream will probably take some time.

I’ll try to test your GW soon and get back at you then.

@KarlL The modifications I made to the gateware were very minor and only affected the timestamp reported by the SDR in RX. I think if anything is at fault here, it is most likely the Lime suite library as I haven’t made any changes to the TX path in the VHDL. When I was initially testing to just get sending at a certain sample working I was using the stock gateware and the latest commit of the LMS API, however I sent the packets more frequently so didn’t notice any issues. It was only when I moved to using the PPS to send once per second that I saw problems.

@cmichal I’ve had a good look at the code you posted and it seems to make sense. I am assuming that here inst0_fifo_wrreq is a signal that goes high when a new sample is put into the FIFO?

The other thing is I can’t quite see how you arrived at this expression for selecting the mode:
my_mode <= "00" when (mimo_en_sync = '0' and ddr_en_sync = '1') else
"01" when ((mimo_en_sync = '1' or trxiqpulse_sync = '1') and ch_en_sync = "11") else
"10" when ((mimo_en_sync = '1' or trxiqpulse_sync = '1') and (ch_en_sync(0) xor ch_en_sync(1)) = '1' ) else
"11";

I think this may just be down to the confusing/poor comments and lack of documentation for the VHDL but if you’d be able to clarify a bit more the difference between SDR/DDR/TRXIQ Pulse that would be really useful.

Thanks,
Matt

EDIT: I’ve had a good look at the VHDL and I think I agree with what you’ve done. I believe the table at line 43 in txiq.vhd is incorrect as it contradicts there own code, so I have opened a github issue for it.

May I ask how you deduced how many clock cycles there are per increment of the counter in each of the modes?

I think I copied/adapted that from some other source file. I spent a few hours combing through a few places in the vhdl where those various modes affected things - and looked at things like under what conditions the timestamp counter was bit shifted. My recollection was that the comments documenting things weren’t entirely self consistent (or at least didn’t seem so to me).

I think you’ve got the right idea about inst0_fifo_wrreq - it marks a sample arriving from the rf chip.

@cmichal Surely for your code to be correct inst0_fifo_wrreq would need to mark an increment in the sample counter as opposed to a sample arriving from the chip?

For example here:
if my_mode ="00" and since_wrreq(0) = '1' then
flag_capture_q <= '1' & smpl_nr_cnt(62 downto 1 ) & '1';

When my_mode="00" your comments say that we are in “SISO DDR - here there is one clock tick per sample, but the counter increments every second time”. Now the above if statement increments flag_capture_q by one if there has been once clock cycle since inst0_fifo_wrreq was high. However in SISO DDR you said that a sample is put into the FIFO on every clock cycle, which would suggest that since_wrreq should always be zero.

You can make similar arguments for each of the other modes, but I think it all looks correct if the signal inst0_fifo_wrreq in fact marks an increment in the sample counter.

I might be completely misunderstanding what is going on here, but I’d be interested to hear what you think.

EDIT: That being said I have just made the modifications and it does seem to be working, I get odd time stamps being reported and the results seem believable when timing the number of samples between the 1Hz signal.

Samples since last PPS = 30720000
Samples since last PPS = 30720000
Samples since last PPS = 30720000
Samples since last PPS = 30719999
Samples since last PPS = 30720000
Samples since last PPS = 30720000
Samples since last PPS = 30719999
Samples since last PPS = 30720000
Samples since last PPS = 30720000
Samples since last PPS = 30719999
Samples since last PPS = 30720000
Samples since last PPS = 30720000
Samples since last PPS = 30719999
Samples since last PPS = 30720000
Samples since last PPS = 30720000
Samples since last PPS = 30719999
Samples since last PPS = 30720000
Samples since last PPS = 30719999
Samples since last PPS = 30720001
Samples since last PPS = 30719999
Samples since last PPS = 30720000
Samples since last PPS = 30719999
Samples since last PPS = 30720000
Samples since last PPS = 30720000
Samples since last PPS = 30719999
Samples since last PPS = 30720000
Samples since last PPS = 30720000
Samples since last PPS = 30720000
Samples since last PPS = 30719999

Previously I would just see alot of 30720000 and the occasional 30720002 or 30719998.

woops - sorry. I last looked at that seriously about 6 months ago. But of course you’re right. inst0_fifo_wrreq marks loading samples into the fifo, coinciding with sample counter increments. My since_wrreq counts how many samples have arrived since then.

Can someone comment on the status of PPS to sync both baseband and LO? I am looking to sync 2 limesdr’s for 4 x coherent receive. I have been digging around and cant figure out what the hardware mod is (for sure) and how to use this functionality. I presume I put a sma to a gpio pin.

For the LimeSDR you can get the modified gateware from that answer above.

You need to use the GPIO pin 0. When it’s low, the timestamps returned by the Lime are normal, but when it’s high the timestamp stops changing and it’s high bit becomes 1 (if you use SoapySDR, the timestamp will be negative). Once you put it low again, the timestamps continue as if nothing ever happened (so you don’t lose track of time).

So to synchronize your 2 devices, you would put both GPIO[0] low, then high at the same time, then back low. At first both devices will have normal timestamps, then at one point the high bit of the timestamp will become 1. The first timestamp for each device with a high bit of 1 correspond to the same time.

Thanks. So… if I fed a pps signal to GPIO1 it would continually reset the timestamps every second? DOes this sync the pll’s or just make sure I can appropriately align the samples in post-processing? I am trying to avoid doing the realignment in post so I can concentrate on adjusting phase after startup and after changing LO. I want a deterministic phase relationship. It can be off…as long as it is always off by the same number of samples.

Changing GPIO[0] will just allow you to sync the devices in post processing. Depending on how long you run your devices and how stable they are, you should be able to determine that a specific timestamp on device A correspond to a specific timestamp on device B. It will be precise up to a certain number of samples, as the FPGA gives one timestamp to the 1st sample of a group of samples (in 12 bit mode, one block is 1360 samples, so 1360 samples in mono channel and 680 samples in dual channels).

Thanks. I am convinced enough to buy 2 and fiddle. I will be back when I can’t figure out how to read the stamp in gnuradio. Earlier if I don’t make it that far.

I don’t think you will be able to do the synchronisation using GNU radio, this mod requires you to use the LMS C++ API directly to retrieve the PPS index that has been stashed inside the packet header.

You should be able to synchronise completely independent SDRs to within a singe baseband sample (so T = 1/30.72MS/s = 33ns) not just to the nearest group. It requires further gateware mods from the post you linked earlier (I believe it has been mentioned in this thread though), but it can be done.

In each of the sample streams from the separate SDRs you should be able to identify the sample corresponding to the 0 to 1 transition of GPIO[0], and you can then align them in post processing. The sample rates will differ by a few Hz if your SDRs are independent (not sharing an external clock) but that can be dealt with relatively easily.

Hi there,

Did you succeed in creating your 4 channel coherent receiver?
I would like to do the same and it would be helpful if you could share some of your experiences

Thanks in advance