You're correct that we build upon the previous efforts of others, but I don't think that's a counterpoint to what I've said. For one thing, copyright doesn't prevent you from building upon previous intellectual and material gains made by civilization; it just prevents you from, for example, making a movie of Lord of the Flies without permission from the owner of the copyright.
Separately:
There's a legitimate discourse to be had about how long copyright protection should last. But I disagree totally with people who claim that it should be nonexistant.
But copyright can prevent you from accessing less-famous works. Some of these works are out-of-print, so you can't get them and the copyright owners are nowhere to be found. These lost copyrighted works are essentially lost.
IANAL. If a copyright holder doesn't effectively exist like in this scenario, how can they charge you with copyright infringement?
If a work is out of print and not findable, and the copyright holder is gone, how would you know that the work existed, and how would anybody go about printing said work again?
I also think that your statement implies that society is somehow entitled to peoples' copy written works at some point. I think it might be the prerogative of the copyright holder to discontinue (and possibly destroy) his or her work before the copyright's lifetime ended. What then?
Just because you can't find the copyright holder doesn't mean they don't exist. If your derived work is a big hit, some corporate lawyer somewhere might realized they acquired the rights two generations ago, and bankrupt you with an infringement suit. This is why projects go through the whole clearance-of-rights licensing nightmare, and whenever clearance fails, society loses because those projects never see the light of day.
"If a copyright holder doesn't effectively exist like in this scenario"
I was surmising a scenario where one did not exist to point out an issue with the parent post. Of course this situation would almost never occur in reality.
As a practical matter, an unenforced copyright is as good as no copyright. However, if citizens routinely feel they can violate the law because it is neither enforced nor harmful to do so, the law should be rewritten to restrict it to scenarios where it is meaningful.
Also, 'out of print' is not 'not findable'. When Google created their Google Books system they encountered many books they wanted to scan whose copyright holders could not be located. The same is true for many old video games: many people remember playing games in their childhood that would now only be available if someone posted a ROM because the original game cartridges or consoles are difficult to come by. However, because the copyright holder cannot be found, there can be no legal means to distribute said games for many decades.
Agreed, it's limbo really. Those who choose to 'pirate' 'abandonwares' assume the risk (however large or small) of being hit with a lawsuit. Though, I think that quantifying damages of a non-distributed copy written work might not be very easy in court, but lawyers are a crafty bunch. It seems like there's a trend to remarket old copywritten material where there's a market for it, especially with new cheap distribution models you see like with Steam or Amazon's Ebook store, for example. This could work to reduce the number of abandoned works and push them back into availability to the general public.
Separately:
There's a legitimate discourse to be had about how long copyright protection should last. But I disagree totally with people who claim that it should be nonexistant.