How can I ground the SPI lines when there is no power on the slave board?

I seem to be having trouble with my board when it is not powered.
I have a TI ADC128S02 (datasheet) acting as a slave and communicating with another master board via SPI. There are buffers on the digital lines (Fairchild 74AC541MTC) between the master board and the slave.

My issue is that the master board could potentially be powered without the slave and the ADC being powered. If the master board is powered and sending SPI signals to the unpowered ADC128 then it will damage the ADC’s ESD protection circuitry and no longer work correctly.

I’m looking for a way to ground the SPI lines on the slave board when the slave board isn’t powered. Once the board is powered the SPI will be operating at 1MHz 5V CMOS logic.

The power analogue and digital power supplies to the ADC need to be isolated and are from different sources, which is to ensure no performance degradation. At the moment when the board is powered off but the SPI is still on this causes V_A = 0.8V and V_D = 1.8V. The datasheet of the ADC states that V_D < (V_A + 0.3V), because this limit is exceeded the device is then damaged.

I was hoping that when the buffer isn’t powered the SPI signal wouldn’t be seen on the ADC but this is not the case. I thought about using transistor inverters on the SPI lines but I’m worried these will cause signal delays and unnecessary increase in current consumption.

A reduced version of the schematic is included below.

The analogue circuitry of the ADC is powered by ARef. The digital side and the buffer are powered by a seperate 5V. The SPI_MISO line is an output from the ADC to the SPI master and is therefore reversed on the buffer. The other three SPI lines are all from the SPI master. None of the ADC inputs have been shown, nor the rest of the connections on the connector.

Any suggestions as to how I can ground the SPI lines when there is no power on the slave board would be greatly appreciated.