That is exactly what the toxic, derisive, backwards, ugly concept of "privilege" is all about. No strawman about it.
The problem isn't one person's "privilege", the problem is the actual injustice done to the other person.
It's not a "privilege" if your skin color doesn't keep you from getting a mortgage because of bigotry -- it's an injustice and a violation of your rights if bigotry against your skin color results in you not getting a mortgage.