"Hey, wait a minute, wouldn't this time and energy be better spent helping all disadvantaged people instead of one small group to the exclusion of others?"

..is a criticism, not a "shooting down". Being altruistic does not grant you a blanket exception from criticism.