I don’t have RPi5 myself, so I can’t test it directly, but in any case that sounds like hardware level problem. I have looked over kernel logs produced by RPi5 when the device is detected, and when not. It doesn’t show any usefull information about what could be the problem.
RPi5 XTRX not found:
[ 0.387319] pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 0.811650] brcm-pcie 1000110000.pcie: link down
RPi5 XTRX found:
[ 0.389754] pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 0.495717] brcm-pcie 1000110000.pcie: link up, 5.0 GT/s PCIe x1 (!SSC)
So this requires hardware debugging tools to figure out if it is timing/power/clocking/signal integrity problem or something else is happening. That is beyond my expertise.
The RPi CM4 seems to work reliably with XTRX, the only difference in the logs compared to RPi5 is the indication of PCIe SSC usage. I don’t know if that is relevant.
[ 1.353255] brcm-pcie fd500000.pcie: link up, 5.0 GT/s PCIe x1 (SSC)