I guess TLS has a dependency on the public key infrastructure (eg Let's Encrypt, or whoever issues wifey accepted certs). Which makes end to end encryption between users harder (most of this stuff is intended for server auth and encryption)?
But otherwise big +1 not to reimplement crypto when the are alternatives. Another option for secret key stuff might be ssh?
There is no requirement to use TLS with webPKI if you are making your own application (not the browser), you can use TLS with custom certificate mangement.
You still need to figure out how you handle trust and key authentication somehow, but that is true of all cryptographic protocols.
But otherwise big +1 not to reimplement crypto when the are alternatives. Another option for secret key stuff might be ssh?