I have a large rails app that was plagued with slow specs using factory_bot. Associations in factories are especially dangerous given how easy it is to build up big dependency chains. The single largest speedup was noting that nearly every test was in the context of a user and org, and creating a default_user and default_org fixture.
there's a profiler that can show you what to focus on, probably fprof here: https://test-prof.evilmartians.io/ (been a while and I don't remember exactly what I used)
(now maybe that's what you used to see what was causing the slowdown, but mentioning to for others to help them identify the bottlenecks.)
