The reference implementation does make that assumption. OpenBSD's ntp does too. phk, author of ntimed had written some blog posts exploring asymmetric paths, so I think ntimed doesn't assume symmetry, but assigns it higher probability. I don't remember what chrony does.

The thing is, it's rather difficult to determine the individual components of the path latency without an out of band reference clock. Especially if all of your network clocks are about the same round trip away.