Idiomatic Clojure is also really slow compared to SBCL-compiled Common Lisp or one of the faster scheme implementations. The compiler's fairly basic and it relies on the JVM for optimisations, which isn't great at the kind of optimisations a compiler aware of higher level semantics can do. It is possible to achieve decent performance by using it as basically a Java wrapper, but that's much uglier than idiomatic code, compared to e.g. SBCL Common Lisp where idiomatic code can still be fairly performant.
Do you happen to know what the present state of SBCL optimization is? The old CMUCL manual had quite a lot of guidance on writing high performance code. I can't seem to find it though, mostly I remember the bits about what can and can't be open coded and how to give enough basic type hints to help the optimizer out (and using DISASSEMBLE to check that you really did). In fact I think you could even ask for warnings for probably inefficient constructs.
Obviously CMUCL was a mess in a lot of other ways though.
One great thing about sbcl is you can put (declare (optimize (speed 3))) at the top of a function and the compiler will give useful feedback when it can’t optimize something and suggest type hints to improve performance.
I’m not a optimization wizard, but I’ve been able to get some very impressive speed ups this way, in combination with some attention to how my code is written.
Tangentially, I think the future of optimization in CL and similar languages is run time optimization/compilation. For example, the goal should be to make (typical cases of) generic function dispatch as fast as inlined structure field access, even in the presence of the possibility of redefinition of the functions and classes. This is not easy, since these may get redefined while calls to functions using them are on the stack.
Since code is data, some kind of functional persistence might be the right path forward here. I lack the expertise to develop the idea further though. I don't much care about the exact behavior vis-a-vis functions on the stack, so long as it's well-defined.
