You could set up your MUA today so that anyone you've already emailed with gets put in a priority folder. No need to mess with end to end encryption just for that functionality.

Theoretically, any spammer could research you enough to send to mail purporting to be from your friends, but realistically no one would bother.