> Maybe there's some way to shut that off

Not using oh-my-zsh or the like, zsh has lots of hooks and omz uses a lot of this hooks for or around git repos, so CD-ing into a git repo entails loads of work.

Use vanilla zsh and configure as needed, and it'll be as zippy as everywhere else.