How do you know nobody has a copy of your house key? I live in an apartment building and it's actually mandatory for the super to be able to enter my apartment in emergencies so he has a key.
I do not know that there is no one out there in the world who has a key to my house. Even if there was and someone found that key on a city street they would likely not be able to figure out which house it opened.
I know there is no legally mandated key to everyone's house that would allow anyone to enter whomever's house they please.