This implies the assumption that the person has no value of their own, but is only in their position by virtue of this privilege and are therefore not competent to comment.
This ties into the fundamental mysticism of the left, which is that poverty in individuals and nations is caused by some great oppressive conspiracy, and not by the decisions made by those individuals and nations.
In turn, that illusion arises from the idea that all individuals are perfect just as they are, which is what makes them equal. Any deviation from equality of outcome thus must be explained as oppression or conspiracy.
Tracing farther back, the root of that idea comes from the pretense of the individual. That pretense is: I am here, therefore I am equal, therefore whatever I do is good, and thus I extend that privilege to others — "equality" — to prevent oversight or criticism of my own actions.
Pretense arises from the self-importance of individuals. They assume that because they exist, there is importance to their existence. Further, since they are aware of nothing but their own wants, judgments, desires, feelings and impulses, they assign to these gestures a kind of objectivity and thus importance.
Realism destroys pretense. Under realism, we reverse the inversion accomplished by pretense: instead of the individual being the focus, the world is the focus. The individual is part of the world, not the other way around (which is how the pretentious individual perceives it, namely that the world is a manifestation of the individual's mind and emotions, a condition known as solipsism).
When a leftist says "Check your privilege" what they are actually saying is "Validate my pretense by joining me in the assumption that I am important for no reason."
Thus a significant and reasonable response to that leftist catch-phrase is to turn around and call them out on the one thing they really fear that you’ll notice: "Check your pretense."