Dear All,
I come with good news, for the ones, who want to use the XTRX on the RPi5,
and share with you this post as sort of Manual and Lessons Learnt,
of how I was able to get my unit(s) to work.
Note that I use the pineboards (former pineberrypi) HatMPCIe (my version is 2024/V1), as miniPCIe interface. I got a multitude of other adapter combinations first to PCIx1 and then to mPCIe,
however I’ve been using the pineboards products from the beginning and would recommend all their products - esp. since this is a native RPi5 PCIe x1 FPC to miniPCIe adapter board.
So far even the PCI provided power through the FPC cable seems sufficient for first limeTRX applications.
General Info:
The RPi5, has next to the external FPC PCIe connection also its ethernet controller connected to the PCI domains.
If something is not right with the XTRXs/RPi5’s PCI connection you will only see this ethernet device and its bridge controller on the PCI system.
You can check this using
lspci -vvv -nn
and/or
dmesg | grep pci
Your XTRX should show up as something like this using lspci
0000:01:00.0 Multimedia audio controller [0401]: Xilinx Corporation Device [10ee:7023] (rev 01)
with corresponding dmesg entries under
pci 0000:01:00.0: [10ee:7023] type 00 class 0x040100
pci 0000:01:00.0: ...
Note that if the XTRX is not detected by the RPi5, the ethernet controller occupies this 0000:01 (domain:bus) identifier, and if both are active, the ethernet device/bridge is listed under the next domain (0001:01).
1. Hardware Issue (for LimeSDR XTRX v 1.2): Check R114
If lspci doesnt list your XTRX, check first if R114 (0 Ohm) is soldered in place.
If R114 is not fitted, you have to solder it in place (or short it) as per the official recommendation above.
This is required to connect the XTRXs’ PRST line to the foreseen miniPCIe Pin 22, as designed, and thus allow a deterministic PCI protocol behavior.
As its a 0201 package, this is not trivial to do for the inexperienced user, and requires some prior soldering experience and/or proper equipment.
If you succeeded you should then be able to “see” the XTRX as
0000:01:00.0 Multimedia audio controller [0401]: Xilinx Corporation Device [10ee:7023] (rev 01)
using lspci.
Then you can continue with the next step.
Alternatively: Also the LEDs (at least my gateware version 1.11, shipped in July '24) may indicate proper PCI communication, settling with continuous behavior after their initial boot sequence.
| R114 | LED 1 | LED 2 |
|---|---|---|
| NF | Green 1 Hz | off |
| 0 Ohm (short) | Green 1 Hz | Green 3 Hz |
However, this LED behavior may only be indicative, as being undocumented, and may also appear in other (failure) scenarios.
2. Build LimeSuiteNG
Note: For this exercise I used the latest clean OS from the Raspberry Pi Imager:
Debian GNU/Linux 12 (bookworm)
Linux 6.6.31+rpt-rpi-2712 Debian 1:6.6.31-1+rpt1 (2024-05-29) aarch64 GNU/Linux
2.1 Install dependencies for LimeSuitNG
I installed the dependencies extracting the information from the documentation and the install_dependencies.sh, adapting for the RPi OS and Kernel.
For clarity, I list them step by step below.
Install dependencies:
sudo apt-get install build-essential
sudo apt-get install cmake
sudo apt-get install libusb-1.0-0-dev
sudo apt-get install libwxgtk3.2-dev
sudo apt-get install libsoapysdr-dev
sudo apt-get install "linux-headers-$(uname -r)"
Note that for me, the kernel headers were already installed. Even though they are not inline with the one from the install_dependencies.sh script the build config will account for them correctly.
Following the discussion above, I opted to install and use DKMS as well, as it seemed more sophisticated to me.
(optional/recommended) Install DKMS:
sudo apt-get install dkms
2.2. Build LimeSuiteNG
As per the discussion above I pulled from the (default) develop branch, to account for the latest continuous developments.
Clone from git:
git clone https://github.com/myriadrf/LimeSuiteNG
or explicitly pulling from develop branch (in case head is moved to stable at one point)
git clone https://github.com/myriadrf/LimeSuiteNG --branch develop
From now on also the stable branch is updated to use LimePCI instead of the LitePCI driver.
Note that any documentation referring to LitePCI can be considered deprecated.
Configure the built with:
$ cmake -B build
Even though its output may report missing required kernel headers (I reckon because the make config scripts report only for information and still parse for the kernel versions inline with the install_dependencies.sh), we can continue. The build would fail in the next stage if those headers are really missing.
So, we can ignore this as long as it also outputs you something like this:
...
-- ######################################################
-- ## LimeSuiteNG enabled features
-- ######################################################
--
* HEADERS, The limesuiteng headers
* LIBRARY, The limesuiteng library
* USB_FTDI, USB support for FTDI
* USB_FX3, USB support for Cypress FX3
* LIMEPCIE, PCIe support
* LIMEPCIE_KERNEL_MODULE, Build Linux LimePCIe kernel module
* DKMS, DKMS support for lime PCIe kernel module
* EXAMPLES, LimeSuite library API examples
* CLI, LimeSuite command line interface tools
* GUI, GUI Application for LimeSuite
* AMARISOFT_PLUGIN, LimeSuite Amarisoft integration plugin
* DESKTOP, LimeSuiteNG freedesktop integration
* SOAPYSDR, SoapySDR bindings for LimeSuiteNG
* UDEV_RULES, Install Linux udev rules
…
-- Configuring done
-- Generating done
-- Build files have been written to: /<removed, where ever you pulled from git>/LimeSuiteNG/build
Then we can compile from the build directory.
cd build && make -j4
Which should output an extended version of:
[ 1%] Building Linux kernel module in dir: /<removed, where ever you pulled from git>/LimeSuiteNG/build/src/comms/PCIe/linux-kernel-module
...
[100%] Linking CXX executable ../bin/limeGUI
[100%] Built target limeGUI
Next we need to deploy the installation to the system
sudo make install
sudo ldconfig
and from here on you should be able to verify already that lspci recognizes the limepci kernel module:
0000:01:00.0 Multimedia audio controller [0401]: Xilinx Corporation Device [10ee:7023] (rev 01)
Subsystem: Xilinx Corporation Device [10ee:0007]
Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Region 0: Memory at 1b00000000 (32-bit, non-prefetchable) [virtual] [size=1M]
Capabilities: <access denied>
Kernel modules: limepcie
The limeDevice command wont find the XTRX yet, and after reboot dmesg will give a hint whats the reason. See next step.
Update: The latest limesuiteNG/limepci will account for this and step 2.3 may not be necessary anymore (c.f., post below).
2.3 Force RPi5 to use 32 bit Addressing (temporarily):
Update: The latest limesuiteNG/limepci will account for this and step 2.3 may not be necessary anymore (c.f., post below).
As the shipped gateware still uses the 32 bit addressing scheme,
dmesg will indicate such issue:
…
[ 2.680344] limepcie 0000:01:00.0: try_set_dma_bitmask(32): dma_map_single error @ va:0000000090196e28 pa:107804000 bus:ffffffffffffffff
[ 2.680350] limepcie 0000:01:00.0: Failed to set DMA mask 32bit.
[ 2.680354] limepcie 0000:01:00.0: try_set_dma_bitmask(64): test buffer va:0000000090196e28 pa:107804000 bus:1107804000
[ 2.680359] limepcie 0000:01:00.0: using 64bit dma
[ 2.690287] limepcie 0000:01:00.0: arch assigned 64-bit MSI address 0xffffffe000 but device only supports 32 bits
[ 2.696613] limepcie 0000:01:00.0: Failed to enable MSI
[ 2.696623] limepcie 0000:01:00.0: Probe fail:
[ 2.696629] limepcie: probe of 0000:01:00.0 failed with error -1
...
Following @ricardas (see post below), this applies to all gateware versions < 1.13 being bound to this 32 bit addressing.
To resolve this, I recalled that @sergey was already early reporting progress in solving such error, with using the following dtoverlay:
dtoverlay=pciex1-compat-pi5,no-mip
So add the above dtoverlay line to your RPi’s /boot/firmware/config.txt,
reboot, and the device should be created this time:
...
[ 2.658822] limepcie 0000:01:00.0: try_set_dma_bitmask(32): dma_map_single error @ va:00000000d2114dee pa:1006a4000 bus:ffffffffffffffff
[ 2.658838] limepcie 0000:01:00.0: Failed to set DMA mask 32bit.
[ 2.658843] limepcie 0000:01:00.0: try_set_dma_bitmask(64): test buffer va:00000000d2114dee pa:1006a4000 bus:11006a4000
[ 2.658848] limepcie 0000:01:00.0: using 64bit dma
[ 2.663273] limepcie 0000:01:00.0: 1 MSI IRQs allocated.
[ 2.663295] limepcie 0000:01:00.0: DMA channels: 1
[ 2.664357] limepcie 0000:01:00.0: Creating /dev/limepcie0/trx0
[ 2.671991] limepcie 0000:01:00.0: Creating /dev/limepcie0/control0
Now also shown limeDevice --full
Found 1 device(s) :
0: LimeSDR XTRX, media=PCIe, addr=/dev/limepcie0, serial=000000008f6ac878
Current XTRX gateware does not support 64bit DMA addressing. RF data streaming might not work. Please update gateware.
Expansion name : UNSUPPORTED
Firmware version : 4
Gateware version : 1
Gateware revision : 11
Gateware target board : LimeSDR XTRX
Hardware version : 2
Protocol version : 1
Serial number : <removed>
SPI slave devices :
FPGA
LMS7002M
Memory devices :
FPGA/FLASH
GPS Lock:
GPS - Undefined
Glonass - Undefined
Galileo - Undefined
Beidou - Undefined
But note that there is still some addressing errors to be expected and in dmesg reported.
You’ll also come across a nice hint from @ricardas in the lime-tools that we still have to update the gateware to be able to switch (back) to and use RPi5’s 64 bit addressing, in order to achieve proper function.
However at this point we can make use of the most basic lime-tools.
Most importantly limeFLASH is usable and required in the next step.
So the dtoverlay is just a temporary workaround to enable the next step,
and we will have to remove the dtoverlay after switching to 64 bit addressing.
2.4 Upgrade Gateware and use RPi5’s 64 bit addressing
Depending on which gateware version you run, you have to flash either the combined or the user image following the instructions here and/or here.
Following instructions of @ricardas, I flashed this image [LimeSDR-XTRX_GW/bitstream/combined_flash_programming_file.bin at master · myriadrf/LimeSDR-XTRX_GW · GitHub] in its August 2024 commit.
However, please check what is applicable in the future when you read this as yours may be different - I do not take any responsibility on what you flash.
For my gateware (1.11) being <1.13, I needed to flash the combined image, using:
limeFLASH combined_flash_programming_file.bin
The flashing will take a short while, but shows you the live progress resulting in:
Current XTRX gateware does not support 64bit DMA addressing. RF data streaming might not work. Please update gateware.
File size : 3889296 bytes.
[100] bytes written 3889296/3889296
[100] bytes written 3889296/3889296
Programming completed.
I cannot stress enough that you should triple check what you downloaded and be 100% sure on what you are going to flash to which target.
It is your responsibility to verify your downloaded gateware is not corrupted and applicable for your target.
If you flash something wrong or corrupted you’ll need to fallback to a JTAG programming interface to recover your device functionality, just like I have to do now with one of my XTRXs, because I was rushing…but thats another story.
The new gateware will be used after powercycling the XTRX.
Important not to forget:
Remove or comment the dtoverlay line of step 2.3, as we want the RPi5 to use its native 64 bit addressing again, and reboot to effect.
After reboot, for me all works well so far.
limeDevice --full
Found 1 device(s) :
0: LimeSDR XTRX, media=PCIe, addr=/dev/limepcie0, serial=000000008f6ac878
Expansion name : UNSUPPORTED
Firmware version : 4
Gateware version : 1
Gateware revision : 17
Gateware target board : LimeSDR XTRX
Hardware version : 2
Protocol version : 1
Serial number : <removed>
SPI slave devices :
FPGA
LMS7002M
Memory devices :
FPGA/FLASH
FPGA/gold-image
FPGA/user-image
GPS Lock:
GPS - Undefined
Glonass - Undefined
Galileo - Undefined
Beidou - Undefined
Success - and dont rush!
A first feedback on the new LimeSuiteNG capabilities:
@ricardas , I appreciate the limeConfig/limeTRX combo of commands, which are a real added value compared to the old limeSuite - thanks!
Also the limeGUI surprises with little tabs and settings - but I reckon XTRX will still be able to be configured like used to in the signal processings heaven of old LimeSuite.
Lets see if I can utilize limeConfig/limeTRX or their source code to the extend I need, instead of adapting my existing code to the changed API.