Hacker News new | past | comments | ask | show | jobs | submit login

Roughly 44 million items are available via torrent (I maintain a catalog of IA items independent of IA, with each item's torrent file). Wayback data is not (to my knowledge). This is important, as IA then can act as a global metadata catalog for the items, with the underlying content being served up through an uncoordinated fleet of seeders. I think many might agree that the time has arrived for this data to live on globally distributed storage nodes.

It would be helpful if IA published Wayback data files over torrents, alongside cryptographic signatures of the files (for attestation and provenance purposes, as Wayback data has been used in legal proceedings and you would want that trust in the data maintained regardless of where the bits were retrieved from for hydrating the WARC client side).




Join us for AI Startup School this June 16-17 in San Francisco!

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: