I'm old enough to have collaborated on papers using troff (strictly speaking psroff) and sccs as a revision control system, long before latex became popular and PDF didn't yet exist. So, yes, the primary requirement is, as you say, for an ASCII-based markup system to enable this workflow.
However, latex is extremely good at flowing text while applying good style guidelines. Most of the time you know that extending a paragraph by a line won't make anything bad happen - no titles left at the bottom of a page, acceptable stretch of line spacing, and things like that. Of course you can only push this so far before you have to fiddle about moving figures to the right page, etc. This is probably unavoidable, although it would be better to have a bit more direct control over where floats end up. The key thing is that latex does mostly get the small stuff right, so you're not constantly yelling at your collaborators for breaking anything.
However, latex is extremely good at flowing text while applying good style guidelines. Most of the time you know that extending a paragraph by a line won't make anything bad happen - no titles left at the bottom of a page, acceptable stretch of line spacing, and things like that. Of course you can only push this so far before you have to fiddle about moving figures to the right page, etc. This is probably unavoidable, although it would be better to have a bit more direct control over where floats end up. The key thing is that latex does mostly get the small stuff right, so you're not constantly yelling at your collaborators for breaking anything.