Look into the history of OpenSSL vulns -- many are in obscure TLS paths that application developers did not intend to be supporting.
I think Heartbleed's even an example? Apparently there's some TLS ping ("heartbeat") command in the protocol that I'm pretty sure no-one actually asked for or uses in their applications, but there it is in the code, and so you're vulnerable as an application author.
That's a different kind of thing—TLS is an extensible protocol, where its various extensions (i.e. code paths) are activated by patterns of data on the wire. libsodium is just a set of components—you can know at compile time exactly which functions in libsodium will or will not ever be called by your program, and can, if you like, use a linker that does LTO to prune the majority of the library out.
I think Heartbleed's even an example? Apparently there's some TLS ping ("heartbeat") command in the protocol that I'm pretty sure no-one actually asked for or uses in their applications, but there it is in the code, and so you're vulnerable as an application author.