Hi,
I think this tutorial
answers your question regarding why the constellation looks so messy after the modulator
Blockquote
What we are seeing in this image is actually the ISI that we mentioned before. We get rid of ISI by using another filter at the receiver. Basically, what we’ve done is purposefully used a filter on the transmitter, the RRC filter, that creates the ISI. But when we convolve two RRC filters together, we get a raised cosine filter, which is a form of a Nyquist filter. So, knowing this property of the RRC filter, we can use another RRC filter at the receiver. Filtering is just a convolution here, so the output of the receive-side RRC filter is a raised cosine pulse shaped signal with minimized ISI. The other benefit is that absent effects of the channel, what we are also doing is using a matched filter at the receiver.
Blockquote
Which basically says that the “wrong” points on the constellation are actually INTENTIONAL Inter-Symbol-Interference (ISI) introduced by the modulator itself. That ISI is later filtered back on the receiver side using a matched filter. So this intentional ISI is actually better for the proper retrieval of the signal at the receiver side.
Hope this helps