I used to struggle with this too. What helped me get over it is realizing that this is completely backwards. At every company I've worked for there has been a near-constant need for good people. When a former colleague who I know does good work reaches out to me about jobs at my company, I'm thrilled to have a conversation with them.

Even aside from any degree of friendship, this is nearly always a mutually beneficial situation. If it works out, I get a new teammate who I know is good (lower risk than hiring someone random). I probably also get some kind of kudos (generally even a bonus) from the company for the referral.