The only thing which takes time is debuginfod downloads.
Sounds like maybe you have reverse-debugging enabled? I mean target record or target record-full?
Anyway, I seldom do step-by-step. I typically work with dprintf.
I am curious does your project have large external dependencies or is it self-contained.
The only thing which takes time is debuginfod downloads.