Why does what I could have done with the money justify how I chose to spend the time? I see this meme occasionally nowadays, implying that it is my sad duty to become rich because of all the philanthropy I could engage in, and I don't trust it.
No, you mistake me. While I couldn't put Singer's name to it, I'm quite familiar with his argument. But this familiar argument is subtly modified in this case: not "if you're rich you're obligated to aid", but rather "if you have the chance to become rich, then you are obligated to become rich so that you can aid". Regardless of whether that's truly your goal, and regardless of the personal cost to yourself, which is why I don't like it.
I suspect it's often used as a rationalization for becoming rich. Either way, I don't trust it.