No, you wouldn't have to assume that. All the alternative factors you mention can tear down a sound building, but none of them can help an unsound one survive 2000 years.
A building that lasts that long is robust almost by definition. So, as long as any buildings do survive, we have fertile soil for survivorship bias.
I don't think you are applying the statistical notion of survivorship bias completely accurately.
The original comment questioned the accuracy of the statement "they don't build them like they used to", which I took to refer to building standards 2000 years ago.
Consider the following two scenarios:
A. 2000 years ago, all buildings were shoddily engineered except the Pantheon.
B. 2000 years ago, 90% of buildings were well engineered, of which the Pantheon is one example. The rest were deliberately torn down for one reason or other.
Survivorship bias does not allow you in itself to distinguish between A and B, which is the interesting question. It just allows you to disprove C:
C. Since the Patheon still stands, all buildings 2000 years ago must have been brilliantly engineered.
A building that lasts that long is robust almost by definition. So, as long as any buildings do survive, we have fertile soil for survivorship bias.