Agreed! But if aliens visited Earth in the past, how far back could they visit and still conclude that we're intelligent?
2000 years? Absolutely, that's Roman Empire times and we would absolutely count as an intelligent civilization.
20,000 years? Maybe. This is before agriculture, but we'd still be animals making tools and even boats.
200,000 years? I'm not sure we would be distinguishable from modern apes.
So the window is around 200,000 years, which is about 0.004% of the age of the Earth, and 0.04% of the time Earth has had multicellular life. If we visit a planet that has multicellular life, what's the chance that we will visit exactly in the 200,000 year window when they have technology similar to ours?
Most likely we'll either visit too early (before civilization and even intelligence) or millions of years after the first civilization.