Limesdr xtrx not found

masefi · 6 June 2024 18:29

I have recently got a limesdr xtrx board and found a raspberry pi5 hat that converts the mpcie to pcie (https://www.amazon.ca/dp/B0CWD54QCR?psc=1&ref=ppx_yo2ov_dt_b_product_details)

After installing the limeSuiteNG on the pi, it finds the device as Multimedia audio controller: Xilinx corporation Device 7023 (rev 01) and I cant seem to get the xtrx to work. any idea how I can fix this issue?

ricardas · 6 June 2024 20:10

On Raspberry Pi, currently you need to add “dtoverlay=pcie-32bit-dma” to it’s “config.txt”.

masefi · 6 June 2024 22:34

Thanks, seems to still have the same difficulties connecting to it for some reasons! After editing the config file, I am still facing the same issue. tried pcie-32bit-dma-pi5 as well, but no luck!

ricardas · 7 June 2024 06:23

Can you show your system log “sudo dmesg | grep litepcie”?

masefi · 7 June 2024 14:23

Unfortunately it does not give me any responses when I send this command!

ricardas · 7 June 2024 14:32

PCIe driver compilation requires linux-kernel-headers, on RaspbianOS the package is called differently, you should try “sudo apt-get install raspberrypi-kernel-headers”. And then recompile and install LimeSuiteNG.

When you run ‘cmake’ for LimeSuiteNG, it should show what features are enabled, like:

LimeSuiteNG enabled features:

HEADERS, The limesuiteng headers
LIBRARY, The limesuiteng library
LITE_PCIE, PCIe support
LITEPCIE_KERNEL_MODULE, Build Linux LitePCIe kernel module

masefi · 7 June 2024 15:27

Installed the kernel headers, now cant reinstall the limesuitng! Will work on this to see if I can find the problem, it is having difficulties installing the linux kernel module

47%] Building Linux kernel module in dir: /home/terrawave/LimeSuiteNG/build/src/comms/PCIe/linux-kernel-module
make[3]: *** /lib/modules/6.6.31-v8-16k+/build: No such file or directory. Stop.
make[2]: *** [src/comms/PCIe/linux-kernel-module/CMakeFiles/litepcie-kernel.dir/build.make:73: src/comms/PCIe/linux-kernel-module/litepcie.ko] Error 2
make[1]: *** [CMakeFiles/Makefile2:865: src/comms/PCIe/linux-kernel-module/CMakeFiles/litepcie-kernel.dir/all] Error 2
make: *** [Makefile:136: all] Error 2

ricardas · 7 June 2024 15:37

the script is attempting to build the module using directory: /lib/modules/`uname -r`/build, check if the /lib/modules/ contains a subdirectory which name matches output of “uname -r” command.

masefi · 7 June 2024 15:50

ok, got the kenel working and everyting is reinstalled

[ 3.302290] litepcie: loading out-of-tree module taints kernel.
[ 3.302330] litepcie: disagrees about version of symbol __pci_register_driver
[ 3.302333] litepcie: Unknown symbol __pci_register_driver (err -22)
[ 3.302340] litepcie: disagrees about version of symbol remap_pfn_range
[ 3.302342] litepcie: Unknown symbol remap_pfn_range (err -22)
[ 3.302352] litepcie: disagrees about version of symbol pci_unregister_driver
[ 3.302354] litepcie: Unknown symbol pci_unregister_driver (err -22)
[ 3.302401] litepcie: disagrees about version of symbol kmalloc_trace
[ 3.302403] litepcie: Unknown symbol kmalloc_trace (err -22)
[ 3.302414] litepcie: disagrees about version of symbol kmalloc_caches
[ 3.302416] litepcie: Unknown symbol kmalloc_caches (err -22)
[ 4.845816] litepcie: disagrees about version of symbol __pci_register_driver
[ 4.845826] litepcie: Unknown symbol __pci_register_driver (err -22)
[ 4.845835] litepcie: disagrees about version of symbol remap_pfn_range
[ 4.845837] litepcie: Unknown symbol remap_pfn_range (err -22)
[ 4.845850] litepcie: disagrees about version of symbol pci_unregister_driver
[ 4.845852] litepcie: Unknown symbol pci_unregister_driver (err -22)
[ 4.845904] litepcie: disagrees about version of symbol kmalloc_trace
[ 4.845906] litepcie: Unknown symbol kmalloc_trace (err -22)
[ 4.845920] litepcie: disagrees about version of symbol kmalloc_caches
[ 4.845922] litepcie: Unknown symbol kmalloc_caches (err -22)
[ 5.470826] litepcie: disagrees about version of symbol __pci_register_driver
[ 5.470834] litepcie: Unknown symbol __pci_register_driver (err -22)
[ 5.470841] litepcie: disagrees about version of symbol remap_pfn_range
[ 5.470844] litepcie: Unknown symbol remap_pfn_range (err -22)
[ 5.470854] litepcie: disagrees about version of symbol pci_unregister_driver
[ 5.470857] litepcie: Unknown symbol pci_unregister_driver (err -22)
[ 5.470906] litepcie: disagrees about version of symbol kmalloc_trace
[ 5.470907] litepcie: Unknown symbol kmalloc_trace (err -22)
[ 5.470919] litepcie: disagrees about version of symbol kmalloc_caches
[ 5.470920] litepcie: Unknown symbol kmalloc_caches (err -22)
[ 6.882823] litepcie: disagrees about version of symbol __pci_register_driver
[ 6.882832] litepcie: Unknown symbol __pci_register_driver (err -22)
[ 6.882840] litepcie: disagrees about version of symbol remap_pfn_range
[ 6.882842] litepcie: Unknown symbol remap_pfn_range (err -22)
[ 6.882854] litepcie: disagrees about version of symbol pci_unregister_driver
[ 6.882856] litepcie: Unknown symbol pci_unregister_driver (err -22)
[ 6.882908] litepcie: disagrees about version of symbol kmalloc_trace
[ 6.882910] litepcie: Unknown symbol kmalloc_trace (err -22)
[ 6.882922] litepcie: disagrees about version of symbol kmalloc_caches
[ 6.882924] litepcie: Unknown symbol kmalloc_caches (err -22)

ricardas · 7 June 2024 15:57

how did you solve your problem?

disagrees about version of symbol this indicates that the module was compiled with kernel headers that are different from the kernel that you are currently running. Do you have multiple linux kernels installed?

masefi · 7 June 2024 16:01

Created a symbolic link
sudo ln -s /usr/src/linux-headers-6.6.31+rpt-rpi-v8 /lib/modules/$(uname -r)/build

masefi · 7 June 2024 16:06

I fixed the kernel to be 16+ now and am rebuilding!

ricardas · 7 June 2024 16:07

That’s not good, you’re mixing different versions.
remove that symlink.
and try:

sudo apt-get install linux-headers-`uname -r`

masefi · 7 June 2024 17:53

I re-did everything on a new rspi5:

terrawave@raspberrypi:~ $ lspci
0000:00:00.0 PCI bridge: Broadcom Inc. and subsidiaries Device 2712 (rev 21)
0000:01:00.0 Multimedia audio controller: Xilinx Corporation Device 7023 (rev 01)
0001:00:00.0 PCI bridge: Broadcom Inc. and subsidiaries Device 2712 (rev 21)
0001:01:00.0 Ethernet controller: Device 1de4:0001
terrawave@raspberrypi:~ $ sudo dmesg | grep litepcie
[ 2.519850] litepcie: loading out-of-tree module taints kernel.
[ 2.520224] litepcie 0000:01:00.0: [Probing device]
[ 2.520264] litepcie 0000:01:00.0: enabling device (0000 → 0002)
[ 2.520287] litepcie 0000:01:00.0: BAR0 address=0x00000000aedb4f8b
[ 2.540005] litepcie 0000:01:00.0: Version
[ 2.540099] litepcie 0000:01:00.0: arch assigned 64-bit MSI address 0xffffffe000 but device only supports 32 bits
[ 2.553215] litepcie 0000:01:00.0: Failed to enable MSI
[ 2.558493] litepcie 0000:01:00.0: FreeIRQs 0
[ 2.558497] litepcie: probe of 0000:01:00.0 failed with error -5

ricardas · 7 June 2024 18:16

hmm, never seen issue like this, looks like some problem with “dtoverlay=pcie-32bit-dma”. I’ll look into it next week.

In the mean time, if you want, you could try modifying the module code, to use 64bit dma addressing. And remove the “dtoverlay=pcie-32bit-dma”.

github.com

myriadrf/LimeSuiteNG/blob/e47a3a8d49b1a3bee8afaf7240bf71de680c5124/src/comms/PCIe/linux-kernel-module/litepcie.c#L1561


      
              litepcie_writel(litepcie_dev, CSR_CTRL_RESET_ADDR, 1);
              msleep(10);
          #endif
              /* Show identifier */
              for (i = 0; i < 256; i++)
                  fpga_identifier[i] = litepcie_readl(litepcie_dev, CSR_IDENTIFIER_MEM_BASE + i * 4);
              dev_info(&dev->dev, "Version %s\n", fpga_identifier);
          
              pci_set_master(dev);
          
              ret = dma_set_mask_and_coherent(&dev->dev, DMA_BIT_MASK(32));
              if (ret)
              {
                  dev_err(&dev->dev, "Failed to set DMA mask 32bit\n");
                  goto fail1;
              };
          
              ret = AllocateIRQs(litepcie_dev);
              if (ret < 0)
                  goto fail2;

masefi · 7 June 2024 19:02

thanks, will give this a try as well.

masefi · 11 June 2024 00:10

did not work!

sergey · 16 June 2024 15:21

There is something that was confusing me for a moment. The build scripts seems to use /lib/modules, but the FindLinuxKernelHeaders.cmake uses /usr/src/linux-headers-${KERNEL_RELEASE} to find include/linux/user.h. This causes an issue on Raspberry Pi because of the way how headers are packaged there: they have a lot of headers in the common folder rather than a release version one (i.e. it is /usr/src/linux-headers-6.6.31+rpt-common-rpi which contains include/linux/user.h, and not the /usr/src/linux-headers-6.6.31+rpt-rpi-2712). The /lib/modules/uname -r/source points to the /usr/src/linux-headers-6.6.31+rpt-common-rpi.

Not sure if that causes some actual issues, but it at leads to a confusing message of CMake about missing required packages. Iv’e solved it by passing -DKERNELHEADERS_DIR=/lib/modules/uname -r/source to CMake.

@ricardas Are you aware of some plans to leverage DKMS? It seems that it could simplify some aspects of the kernel module, including making it easier to update the OS after the module has been installed. Maybe there are some non-obvious stoppers, or maybe something we can help you with?

sergey · 16 June 2024 15:29

I think the entire ret = ... and the if statement below it needs to be commented. Otherwise ret seems to be initialized to an error some lines above. It didn’t work for me though.

What worked for me to get past the litepcie 0000:01:00.0: arch assigned 64-bit MSI address 0xffffffe000 but device only supports 32 bits error is adding this to the config.txt:
dtoverlay=pciex1-compat-pi5,no-mip.From its documentation: Revert to the MSI target in the RC, instead of the MSI-MIP peripheral. Use if a) more than 8 interrupt vectors are required or b) the EP requires DMA and MSI addresses to be 32bit. The latter sounds related.

It allowed to get past this error, and allowed lineDevice to find the device. Unfortunately, the basicRX still didn’t work, as the litepcie_mmap() returned an error: litepcie 0000:01:00.0: mmap io_remap_pfn_range failed -22. Maybe some further work needed, or, maybe is related on some other quirk I have.

@masefi Does the lspci show XTRX after every reboot? For me it is super unreliable, it might take manu reboots to make XTRX listed there. And it only happens on RPi5, using XTRX on CM4 IO board works reliably. So doesn’t seem related on the XRTX itself.

masefi · 16 June 2024 16:02

Interesting, thanks for the detailed info Sergey, I will give this a try. For now RPi5 does not detect it as XTRX at all! I have to reboot multiple times to make it detectable as Multimedia audio controller: Xilinx corporation Device 7023 (rev 01). Sometimes the second light on the XTRX board stops blinking (the one furthest from the micro USB port) and either powers or stays on after boot. in either of these cases, it becomes completely undetectable. I have to reboot multiple times to make it detectable as the Xilinx Device
Is there a specific CM4 PCIe module that you use? the only reason I use RSPi5 is the PCIe option.