I strongly disagree with that view. I don't have children of my own and while I would open the door at 3 am to take friends in (or heck a neighbor) while we get things sorted out taking on children is a major life disrupting experience.
So yes saying no would be selfish, assuming that any of your friends would have their lives completely disrupted just because you have children seems quite a bit more selfish.
The point is that throughout history, raising children are considered the point of life, not trophies that are otherwise a distraction from vacations and toys.
Well, no. Traditionally care for those kids goes to the extended family, not the friends. Sneering at friends who won't take in children is not something that you can support in terms of an argument of tradition.
So yes saying no would be selfish, assuming that any of your friends would have their lives completely disrupted just because you have children seems quite a bit more selfish.