PCIe kernel driver does not bind to more than one LimeSDR PCIe board!

farhan296 · 26 March 2020 02:33

Hello,

I have two Limesdr PCIe boards. When I plug it into the motherboard only one of the board gets bound to the Kernel Driver: xillybus_pcie

0b:00.0 Unassigned class [ff00]: Altera Corporation Device ebeb (rev 01)
Subsystem: Altera Corporation Device ebeb
Flags: bus master, fast devsel, latency 0, IRQ 135
Memory at 2fffe00000 (64-bit, prefetchable) [size=128]
Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
Capabilities: [78] Power Management version 3
Capabilities: [80] Express Endpoint, MSI 00
Capabilities: [100] Virtual Channel
Kernel driver in use: xillybus_pcie
Kernel modules: xillybus_pcie, altera_cvp

0c:00.0 Unassigned class [ff00]: Altera Corporation Device ebeb (rev 01)
Subsystem: Altera Corporation Device ebeb
Flags: fast devsel, IRQ 16
Memory at 2fffd00000 (64-bit, prefetchable) [size=128]
Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [78] Power Management version 3
Capabilities: [80] Express Endpoint, MSI 00
Capabilities: [100] Virtual Channel
Kernel modules: xillybus_pcie, altera_cvp

Device Info:
driver=PCIEXillybus
hardware=LimeSDR-PCIe
boardSerialNumber=0x0
expansionName=UNKNOWN
firmwareVersion=7
gatewareVersion=2.11
hardwareVersion=3
protocolVersion=1

Can any one tell what the issue could be? Both board are working fine when individually plugged in.

farhan296 · 26 March 2020 02:35

@IgnasJ @andrewback @Zack Please help!

andrewback · 27 March 2020 11:28

I understand that only a single LimeSDR-PCIe is supported in a system.

farhan296 · 30 March 2020 16:31

Any plans for adding this support? Also if possible can you point out where and what needs to be modified in order to enumerate two PCIe LimeSDR boards? @andrewback

farhan296 · 31 March 2020 21:46

Also how can I change the PCIe device ID from 0xEBEB to 0xEB00. Thanks
@andrewback @IgnasJ @Zack

andrewback · 7 April 2020 10:29

Not that I’m aware of. The LimeSDR QPCIe was created for applications that need 4x4 MIMO.

Appears to be set in:

github.com

myriadrf/LimeSDR-PCIe_GW/blob/master/ip/pcie_core/pcie_c4_4x.v

// megafunction wizard: %IP Compiler for PCI Express v15.1%
// GENERATION: XML
// ============================================================
// Megafunction Name(s):
// ============================================================

//Legal Notice: (C)2017 Altera Corporation. All rights reserved.  Your
//use of Altera Corporation's design tools, logic functions and other
//software and tools, and its AMPP partner logic functions, and any
//output files any of the foregoing (including device programming or
//simulation files), and any associated documentation or information are
//expressly subject to the terms and conditions of the Altera Program
//License Subscription Agreement or other applicable license agreement,
//including, without limitation, that your use is for the sole purpose
//of programming logic devices manufactured by Altera and sold by Altera
//or its authorized distributors.  Please refer to the applicable
//agreement for further details.

// synthesis translate_off
`timescale 1ns / 1ps

This file has been truncated. show original

farhan296 · 7 April 2020 13:27

Thanks @andrewback. We are looking to implement a 8x8 system using LimeSDR PCIe, any idea of way forward since multiple boards are not supported.

Also can you share a guide on how to update the FPGA code of LimeSDR. Like if I modify the Device ID what are the steps I need to go through. Thanks.

andrewback · 8 April 2020 06:58

I’m afraid we don’t have guides to modifying the FPGA designs. Will discuss with colleagues and see if we can provide some suggestions.

Zack · 8 April 2020 07:23

Hi @farhan296,

You see only one board because endpoint names are the same for different boards. Hence you have to generate Xillybus with different endpoint names for second board, then replace it here:

github.com

myriadrf/LimeSDR-PCIe_GW/blob/70eb19c3f3676c344fa3aa4cf3e5d0df9ce9401f/src/pcie/pcie_top.vhd#L217


signal stream_rx_en_sync      : std_logic;
signal stream_rx_en_reg0      : std_logic;
signal stream_rx_en_reg1      : std_logic;
signal stream_rx_en_reg2      : std_logic;


signal EP01_1_rd_cnt          : unsigned(31 downto 0);


attribute noprune : boolean;
attribute noprune of EP01_1_rd_cnt : signal is true;


component xillybus
   port (
      clk_125                 : IN std_logic;
      clk_50                  : IN std_logic;
      reconfig_clk_locked     : IN std_logic;
      pcie_perstn             : IN std_logic;
      pcie_refclk             : IN std_logic;
      pcie_rx                 : IN std_logic_vector(3 DOWNTO 0);
      bus_clk                 : OUT std_logic;
      pcie_tx                 : OUT std_logic_vector(3 DOWNTO 0);
      quiesce                 : OUT std_logic;

and here:

github.com

myriadrf/LimeSDR-PCIe_GW/blob/70eb19c3f3676c344fa3aa4cf3e5d0df9ce9401f/src/pcie/pcie_top.vhd#L466


   );


  inst0_areset <= '0';
  inst0_inclk  <= '0' & inclk_125;
  clk_125      <= inst0_clk(0);
  clk_50       <= inst0_clk(1);
  
-- ----------------------------------------------------------------------------
-- Xillybus inst 
-- ----------------------------------------------------------------------------      
  xillybus_inst1 : xillybus
    port map (
      -- Ports related to /dev/xillybus_mem_8
      -- FPGA to CPU signals:
      user_r_mem_8_rden       => inst1_user_r_mem_8_rden,
      user_r_mem_8_empty      => inst1_user_r_mem_8_empty,
      user_r_mem_8_data       => inst1_user_r_mem_8_data,
      user_r_mem_8_eof        => inst1_user_r_mem_8_eof,
      user_r_mem_8_open       => inst1_user_r_mem_8_open,
      -- CPU to FPGA signals:
      user_w_mem_8_wren       => inst1_user_w_mem_8_wren,

The recompile the gateware. Check documnetation for more information:

farhan296 · 14 April 2020 14:37

@Zack Thank you for your response. Do I also have to update the Xillybus host system driver? I actually have a multicard support host system driver from Xillybus. In that case I assume I should not be changing any FPGA code other than the recommended change in the Device ID from 0xEBEB to 0xEB00. Any thoughts?

farhan296 · 16 April 2020 19:46

Also changes needs to be made here since now the endpoint would be different:

github.com

myriadrf/LimeSuite/blob/b14b34569bed8e6f9ee12b87a1a19799aff0443e/src/ConnectionXillybus/ConnectionXillybus.cpp#L34


#include <thread>
#include <chrono>


using namespace std;


using namespace lime;


const std::vector<ConnectionXillybus::EPConfig> ConnectionXillybus::deviceConfigs = {
#ifndef __unix__
    {
        "LimeSDR-PCIe",
        "\\\\.\\xillybus_read_8",
        "\\\\.\\xillybus_write_8",
        {"\\\\.\\xillybus_read_32", "\\\\.\\xillybus_read_32", "\\\\.\\xillybus_read_32"},
        {"\\\\.\\xillybus_write_32", "\\\\.\\xillybus_write_32", "\\\\.\\xillybus_write_32"}
    },
    {
        "LimeSDR-QPCIe",
        "\\\\.\\xillybus_control0_read_32",
        "\\\\.\\xillybus_control0_write_32",
        {"\\\\.\\xillybus_stream0_read_32", "\\\\.\\xillybus_stream1_read_32", "\\\\.\\xillybus_stream2_read_32"},