I am of the same opinion. I added the library in my SSH config and now SSH will just use the yubikey, if it's plugged in. The only problem is that I haven't been able to find an agent supporting all of (gnome keyring for persistent passwords, ed25519 keys, the yubikey).