This has been my experience as well. In my case Ive never ever installed Facebook on my phone. It has to be whatsapp leaking/sharing the contacts list.

Or that contact allowing FB to scan their contact list. Remember, there are two endpoints here and if you did not allow it then is is more likely the other party did.

Or the contact. It'd make sense for Facebook to try to connect people from both sides of the relationship.