I feel your pain, I haven't been in ML world directly for a few years now but I've done the same exercise multiple times.
The worst part is that block compression actually does not help if it doesn't do a significantly good job of compression AND decompression. My use case had to immediately deploy the models across a few nodes in a live environment at customer sites. Cloud wasn't an option for us and fiber was also unavailable many times.
The fastest transport protocol was someone's car and a workday of wages.
> The fastest transport protocol was someone's car and a workday of wages.
This is actually the entire premise of AWS Snowball: send someone a bunch of storage space, have them copy their data to that storage, then just ship the storage back with the data on it. It can be several orders of magnitude faster and easier than an internet transfer.
it would be totally cyberpunk to have a data cafe where you bring your hard drive to upload to the cloud and you'd pay by the terabyte/s. have all day? cheap. need to do it in 30 mins? pay up.
The worst part is that block compression actually does not help if it doesn't do a significantly good job of compression AND decompression. My use case had to immediately deploy the models across a few nodes in a live environment at customer sites. Cloud wasn't an option for us and fiber was also unavailable many times.
The fastest transport protocol was someone's car and a workday of wages.