It's moot either way. There is dynamic dispatch based on use provided inputs. There will always be lookups, jumps, and cache misses regardless of the syntax or semantics.
Is it moot? With templates, the Visitor type is selected at compile time and needn't even be polymorphic. The member function call(s) can often be inlined. Pretty sure there is no dynamic dispatch or jumps in such a scenario. Cache misses, sure, but I don't think that's important in this case.
Almost always these systems operate on external inputs like source files or JSON blobs. You cannot compile time collapse the branching needed to parse that sort of thing.