I'm more inclined to believe M1 is fast because it's a modern design where every part of the architecture is under Apple's control.

In particular, the design uses a memory hub with the memory chips very close to the CPU core. And it has a massive L2 cache on top.