The same argument is often used against abortion (of children not conceived by rape). After all, who are you to decide to end a life you've willingly conceived?

There are even rare cases where women went digging through the trash of famous people they've had sex with to impregnate themselves. There are also plenty of stories of women sabotaging birth control to "baby trap" a man.

With our current laws, in many places, the male is forced to pay child support in this case (a failed attempt made the news recently: https://www.jacarandafm.com/news/entertainment/model-wants-s..., with more details here: https://en.m.wikipedia.org/wiki/Sperm_theft) or have other responsibilities not because of birth control being unreliable, but because of the malicious intent of others.

"If the baby is yours it's your problem" is too simplistic and causes legal issues in cases like these.