I think clients are allowed to send whatever user agent they want - it's essentially a preference for how they would like the website rendered for them.
Also I believe it's technically a violation of the HTTP spec to serve different GET requests to different user agents.
If it would be a violation to change things based on user agent, but the purpose of the user agent to specify how you want things changed, then...
if it was purely about different rendering of the exact same content, then wouldn’t it be the browsers responsibility (and thus UA is unecessary)
I have to imagine browsers are allowed to extend the spec, and UAs are either in theory or in practice a way to communicate those extensions; but because site operators were only specifically designating content for the major browser(s), any compliant but unlisted browser would never get that content.. and suddenly everyone was calling themselves netscape