Thanks, I think that's pretty much the issue. Because of the flexibility of the language (not just macros, but individual characters can change meaning using the \catcode mechanism), you can't have a fixed lexer. This makes the situation unlike the standard language setups I'm used to.