I don't think it's really possible to do much better than the next-gen browser architecture (Servo, fully-fleshed-out Quantum) if you support the entire feature set of browsers (once typed CSSOM is a thing, anyway). You can certainly do better in constrained environments, though. For example, Leo Meyerovich is doing really neat things with data viz, where all the elements to be laid out have the same shape and you can take advantage of that to do things like GPU layout.
But if you're making your own UI kit, couldn't you just eschew CSS and the like? I was under the impression that part of the reason browser rendering is such a gnarly process is because of the reflow issues that CSS and HTML layout quirks/changes can cause. I would assume that you can implement a couple of layouts that prevent those sort of pitfalls and thus speed up rendering...
Please correct me (you know a lot, and I'm betting some of my assumptions are wrong).
Sure, you could get rid of CSS, but in favor of what? You probably need something just like flexbox, and it's not easy to beat an optimized implementation of CSS flexbox in terms of layout performance (especially if parallelized). You could eliminate the restyling step by not having cascading and selector matching, but that hurts productivity and maintainability, which is why you see frameworks like GTK+ moving toward CSS-like styling. There's no free lunch here...
