Indeed. And that totally sucks. Something needs to be done so that Internet Archive doesn't have to retroactively apply robots.txt or deletion requests (sans criminally illegal content).