I do have a guess. The FPGA packs samples into packets of 1360 samples. If you’re using a buffer size of 128, several of those are coming from each packet. The timestamp only gets updated in the header of each packet. Are you receiving on both channels? If so, then the 5 block increment makes sense, since you get 5.3 of your blocks out of each 1360 sample packet.
I’d suggest trying a block size of 680 if you’re using two channels, or 1360 if just one.
I’m curious about what the sequence of timestamps looks like. Can you paste in a list of: last few positive timestamps before negative, then the list of 21 or 22 negative ones?