SGML itself has link process declarations, an additional type of declaration set that can appear in an SGML prolog next to DTDs and that can be used to remap elements (in SGML you can have multiple DTDs and LPDs, pipeline LPDs, and so on). sgmljs uses this and adds templating to capture attributes at call sites for passing these into templates as regular entities, allowing for parametric macro expansion. Basically, if you have eg

    <div bla=x>

in your main doc, you can make SGML expand it using

    <!DOCTYPE div SYSTEM [
      <!ENTITY bla SYSTEM>
    ]>
    <div>
      <p>Value of bla is &bla</p>
    </div>

honoring escaping/sanitizing etc. LPDs can apply rules in a context-dependent way using an automaton capturing much of core CSS.

Now, for arbitrary markup manipulation (XSLT is Turing-complete), don't tell the HN crowd that SGML has/had Scheme-based DSSSL (precursor of XSLT) ;) My opinion, having done large, nontrivial XSLT projects (including extracting the DTD grammar rules you see on the site from spec text) is that the more complex it gets, the more a general-purpose language with unit testing etc becomes a better choice over XSLT.

Edit: much luck with your browser project! Don't hesitate to use my code or ask questions (here or on StackOverflow tagged sgml)