I think it's a terms issue here. What he's saying is it sounds like "how dare you have food when other people don't!" That may not be what people mean, but it's the impression other people get.
I do think it gets confused too - for example, it would be perfectly reasonable to say "you have way more food than you need and other people don't have any at all, you need to give them your food." And sometimes that is the case.