That would only be true if we gave every single hurt feeling equal weight, but PoC in America have a long history of pretty blatant discrimination, specifically using the term "colored people", so I don't see much wrong with not using the phrase because they've asked you not to. It's not like we're entertaining every person that wants to be referred to as a "Hylian Deku scrub" or something.
"So much for the tolerant left!"
The paradox of tolerance requires intolerance of the intolerant. Nazis provide no meaningful addition to our political discourse - the only good Nazi is a dead Nazi.