To me, this has always been a matter of my attention filter. If I'm in an environment where my attention is being pulled to various snatches of conversation, all of which have some tiny detail that I notice, my focus will invariably suffer.

In a coffeeshop, almost nobody says anything that triggers that filter, thereby letting me block out that noise more effectively.