Because TeX documents are usually made by people who have something valuable to say, and the tool reflects that. On the other hand most of the web is designed to sell you stuff, and the actual content is not important. The tool, again, reflects that.

I don't think TeX output looks better than what Web browsers do if you enable all the high-quality text features like hyphenation and use modern OpenType fonts (in particular not the Core Fonts for the Web). The reason why the Web doesn't do this by default is backwards compatibility and performance: hyphenation is asymptotically more expensive.

TeX doesn't hyphenate every word, it only attempts hyphenation when it looks where to break a line. That said, I don't see why a document cannot be prehyphenated once on the server; I'd say this is the right thing to do. (Besides, in this case we don't have to rely solely on algorithms.)

It's not that CSS is still very far from professional typesetting; what amazes me is that at the same time it's much more complex. Look at all these possible 'display' values and box models! And you still cannot do a run-in header! Or automatic numeration! (You can do the latter natively with XSLT, but it's not cool.) By comparison TeX only has boxes, vertical and horizontal lists, glue, and a few lesser things like kerning. And it produces works of art.

A few things that make LaTeX docs look good:

    - quality justified paragraphs making use of inter-word spacing and hyphenation
    - font ligatures (tex uses quality fonts have special characters for sequences like fi fi fl ffi ffl ft etc)
    - good heuristics about where to cut pages (e.g. flexible spacing around heading)

Web browsers implement all of this nowadays. In fact, OpenType GSUB ligatures in particular go far beyond what TeX supports.

But that begs the question why web browsers don’t implement this. Is there a technical reason or did at some point some people decided that decent text rendering is irrelevant one should focus on other gimmicks?

The Web started as a tool for rendering text and occasional images pretty much as soon as it comes out of the wire. TeX on the other hand, was meant to be a typesetting package which compiles textual description of the document into a beautiful printable form. That compilation takes noticeable time even today, I imagine it was unacceptably slow for the web back before. Also in TeX you deal with paper sizes, in browsers you deal with adjustable viewports of unpredictable size.

I guess that the Web just started simple and evolved from that, and then nobody bothered to remake it for prettier text.

I remember an old publishing package, Ventura Publisher; the first time I saw it was 1990 or 91 maybe and then it was at v3. It was a very versatile publishing tool (all the perks of a professional typesetting package, tables, equations, floating illustrations; lots).

What's interesting is that it could separate the content and the styling information, so you could load the same content into two or more publications with different styles and get two different typesetting from the same source (e.g. one column small page for a book and two or three column large page for an article in a journal). And it was very easy to change the format or paper size: just change it and it would reflow the content accordingly. It didn't even took much time, as far as I can remember, and I'm talking about IBM PC 286 here.

So, truly, it's not a technical obstacle; web could have a much better typesetting engine working at high speed (and I'd say that it would be much better to imitate paged media instead of scrolling; scrolling is really inconvenient for reading). It's just that it grows wildly and in all directions at once; it tries to be a "semantic" storage, a rendering medium, and an application engine at the same time, and is not particularly good at any of this.

I don't think HTML-typeset math has to look bad. e.g. http://www.deeplearningbook.org/contents/linear_algebra.html Well this particular example is auto-generated and the code looks terrible, but it's not hopeless to get nice math without TeX.