Is it possible to have the LimeSDR do to demodulation/Modulations on the FGPA?

“demodulation” and “modulation” could literally be anything at all e.g. 8PSK, 32768QAM, 65536QAM, AM, ASK, APSK, BSK, CPM, FM, FSK, GMSK, GFSK, MFSK, MSK, OOK, PM, PPM, PSK, QAM, QPSK, SC-FDE, SM, SSB, TCM, WDM …

So I would assume that your question is can you delete most of the existing FPGA code and replace it with your own to custom gateware and then transfer the fully, or partially, demodulated signal over USB to your own custom application. I honestly do not think that anyone can answer your question. It is like asking can you fit an elephant in a mini, and the answer is always going to be maybe, unless it is a tiny elephant or the closest living relative of the elephant - a Dassie.

Look at the current free space in the FPGA would be my first step.

You will hit a problem straight away, Intel who bought Altera in their infinite wisdom appears to have decided to delete nearly all Altera software older than 5 years old from their download servers, and this unfortunately includes "Altera Quartus prime 15.1 Lite Edition". What this means for you is that you will need to rewrite/migrate the current gateware that you want to keep into a later version of Intel Quartus that still supports the “Cyclone IV” FPGA chips, in addition to creating your own custom code!

.