Take that line of thought to its conclusion, and you end up claiming that paying workers anything at all hurts the poor. After all, if workers weren't paid, wouldn't prices decrease making things more affordable to the poor?

Of course, then the poor wouldn't make any money to pay for anything, but let's overlook that irrelevant detail.