I did a little research, and, yes, that's right. I was surprised at first to learn that they generally used water cooling instead of air cooling, but I learned that air cooling actually uses more water in many instances because air cooling at that scale requires so much more power consumption at the data center that the power plants have to ramp up generation significantly. The extra power consumption uses more water at the power plant than the data center would by just going with water-cooled towers.
So, yes, it's a real issue. They generally need to use drinking water, with added chemicals, for their processes to prevent corrosion and other maintenance issues. So that's two problems - they use the drinking water, and they can't dump back into it because of all the added chemicals.
Some places, like Sweden, have started pumping the data center's waste water back to the power plant for use there. And like you said, AWS (along with I think Google and Meta) have pledged to be "water positive" by 2030. Not sure I believe they'll achieve it, but at least they're working towards it.
In summary, I agree it's a legitimate concern, and the folks around Jackson, Madison, Brandon, etc. should all be asking questions about how/if their drinking water is being protected. Hopefully the Pearl River can be used in some way, without damaging it either.