Yeah, need to write the program and distributed in r/datahoarder
This should be next on my list since my current project depends on SD model and having data backed up gives me confidence that I can get rid of all their stuff if needed.
Hmm, you can just create a torrent out of it. Either as a single file (impractical, but you can just avoid downloading it fully), or chunk it into multiple files.
You don't even need to store it all at once on your computer: stream it and generate checksums on the fly. Then distribute the torrent, and seed sections at a time. It can also be distributed on IPFS.
I've seen a lot of torrents being used for distributing neural network (mostly stable diffusion forks).
This should be next on my list since my current project depends on SD model and having data backed up gives me confidence that I can get rid of all their stuff if needed.