This jives with my memory of that time as well. Apache just couldn't keep up with Nginx' async speed, and if you weren't having to deal with PHP (before FastCGI's adoption), there was no real reason to use Apache.
And post-FCGI's adoption, you didn't need to use Apache, so... why use it?
Also the Apache that was beaten was Apache 1, which was fork-only, and that was the whole reason Nginx was written in the first place.
Then Apache did Apache2 with mpm modules and badly missed the mark. After that Apache was doomed. No async support == dead. It was that simple.