I keep my SSH key pairs on a smart card. It is very cheap to purchase a smart card and card reader (~£25 for a gemalto reader & a gemalto .Net smardcard) and if you're buying in medium to large quantities for a business the cost is even less.
This has some major security and convenience advantages over keeping keys in a file. Firstly, you can generate the keypair actually on the card so that the only device that ever has, and will ever have, the private key is the smartcard. Secondly, the key is secured by a pin and, by default, it will block the card after 3 incorrect pin attempts and after 3 incorrect attempts to unblock the card it will permanently erase the secure storage on the card. Also you can easily make use of your keypair on multiple computers, even untrusted machines, without compromising security. I keep my smartcard in my wallet, so I always have access to it where ever I go.
Alternatively, if you don't want an actual card you can get smartcard-like devices are are physically similar to a USB stick.
> it will block the card after 3 incorrect pin attempts and after 3 incorrect attempts to unblock the card it will permanently erase the secure storage on the card.
Why three? Can someone actually guess your pin in 4 tries? I hate the number 3 used for these things. Raise it to 20 and you'll have virtually the same amount of security, but the false positives will drop to 0.
The whole thing is even more complex that that. You can actually have different pins for different roles. For example, you could setup a different pin that can only be used for authentication with a specific keypair/certificate. Also a pin does not even have to be a pin as such. The admin pin on the card (i.e the role that can unblock the user pin) is actually a challenge-response as it is designed so that a user can phone up IT support or something and tell them the challenge from the card and IT support can give them a response that will then unblock their normal pin.
If someone gets hold if your smart card, they can enter a wrong pass phrase to remove your private key, right? So, you have to have multiple of these smart cards (that all share the same private key), some in your wallet, some in your house and some in a bank or vault somewhere. Otherwise, you can lose access to your private keys forever.
Depending on the cards and their configuration, you could generate the key outside (say, on a trusted machine at home) and upload it to the card. If the cards gets stolen, and you trust the security of the card to not leak the key, you could just get yourself a new one and upload the old key again.
The other philosophy is to generate the key on the RSA card, and never allow the card to export it. Then you consider the private key dispensable and will have to update all systems to accept a new card's private key.
I don't see a situation where you are unable to get console access as an acceptable situation. You don't need to be using a smart card to require local console access if things go bad. What about if the SSH server fails or someone applies a bad firewall rule that prevents SSH access?
You don't need multiple cards with the same private key. You can use multiple cards with different private keys, that are all authorized for your server(s). If you lose one card, just remove that card's public key from the authorized keys.
It does not matter if you lose your SSH private keys though as they are disposable and you can just generate new ones, it's not like they are being used to encrypt anything. To add your new public key to your servers you either get someone else who can SSH in and su to root to do it for you or you login locally (via kvm or similar). Larger deployments most likely have some centrally managed system for dealing with SSH keys.
The smartcard needs to support openpgp smart cards. You then create an RSA gpg key with three subkeys - signing, encryption, and authentication. You'll have to enable the advanced key-creation mode for the last one. That authentication subkey then becomes your ssh key.
There are plenty of tutorials online, but none are particularly good. I've been meaning to regenerate my keys, so maybe I'll take notes and try to write up a good one.
It will display the public key but it is in hex and uses Microsoft's PublicKeyBlob structure. OpenSSL will convert this to PEM for you though, which is what you want for SSH authentication. Something like:
Open this new file in a text editor and get rid of the begin and end public key lines and just smack it all on one line with "ssh-rsa " (note the space) in front of it. You can then add this to your server's authorized_keys file.
OpenSSH can use a pkcs11 library for authentication (-I option), which is also avaliable from Gemalto's website (these are generally specific to the smartcard), and on Windows there is a version of PuTTY called PuTTY SC that will also let you use the pkcs11 library.
I have done something similar -- I've run ssh-agent on my phone, then I ssh from my phone to my PC with the agent-forward (-A) option so that any ssh connections I do from my PC forwards the authentication requests to my phone. (This was with my Nokia N900 phone -- I'm working on setting up a similar thing for Android).
This has some major security and convenience advantages over keeping keys in a file. Firstly, you can generate the keypair actually on the card so that the only device that ever has, and will ever have, the private key is the smartcard. Secondly, the key is secured by a pin and, by default, it will block the card after 3 incorrect pin attempts and after 3 incorrect attempts to unblock the card it will permanently erase the secure storage on the card. Also you can easily make use of your keypair on multiple computers, even untrusted machines, without compromising security. I keep my smartcard in my wallet, so I always have access to it where ever I go.
Alternatively, if you don't want an actual card you can get smartcard-like devices are are physically similar to a USB stick.