I would be up for a points based system like you said.

I had a really difficult childhood; fatherless, impoverished schools, welfare, surrounded by crime (heroin addiction, prostitution, violence), I was quiet and bookish so I got abuse from all angles.

I had food most days and a home made of bricks so I had it better than most.

But if you're going to assume my life is easier because I have a certain skin tone I'm going to be annoyed, because my life was decidedly not easy in the early stages and I only got my "leg up" by being willing to throw myself into extremely uncomfortable situations.