Another benefit of this system is that the storage media is immune electromagnetic pulse. So, even in the case of a large-scale EMP event our cat pictures and network graphs will remain intact and recoverable. :)
Can anyone knowledgeable explain why they use Blu-Ray instead of tape backup? When would an organization use Blu-Ray, and when wouldn't then? Is this reflective of any kind of general trend?
1) As blu-ray is a consumer product its probably manufactured in greater quantities than backup grade tapes, and therefore can be had lower in price, and there are more vendors to source them from. Similarly the drives themselves are cheaper.
2) Tapes aren't random access, therefore it's slow and cumbersome to retrieve single files.
Re: 1) Looking at the prices on Newegg, 1.5TB(native) LTO5 tapes run about $30.00, so $20 per TB. Write-once BluRay looks to be about $32/TB, and rewriteable is $80/TB.
Re: 2) Although it would take an hour or so to search through an entire tape if all the files were written in a single session, normally backup software writes files in sessions containing a few GB. You can seek to a specific session in about 30 seconds average. So if you put about 4GB per session, it would take a max of approx. 30 seconds to read in that session to retrieve a file. So figure about a minute to retrieve any file. Also, LTO5 transfers about 140MB/sec, vs. 6x BDR disks (what I'm seeing for cheap right now) is only 27MB/sec.
3) IIRC most tape storage technolgies are closed source. FB has been heavily pushing their open compute project. Using BDs get them past a lot of licensing problems.
and LTO has way less licensing problems than Blu-ray and LTO-6 has a max of 2.5TB. LTO is also guaranteed to to read tapes from 2 generations earlier and write one generation earlier. Blu-ray has no such long term guarantee.
Not knowledgeable either, but one difference is durability. LTO is rated 15-30 years while the video notes the blu-ray discs FB uses are "certified for 50 years of operation" and some go up to 1,000(!).
Any media is as durable as the availability of media reader. I seriously doubt, in 15-20 or 50 years, you will have the media reader available to read either of the media, whether tape or BD.
How many 8" or 5.25" floppy disk drives are around today and usable with current server hardware? Anyone who is considering cold storage on media for more than 10 years lifespan is fooling themselves.
A few years ago I had a customer who had to perform data migration from DLT to LTO-3. That was over 24 month project and customer begging the vendor to provide a few old DLT drives to read. On top of that the provided DLT drives were so finicky communicating with modern server hardware that they barely kept up for a few hours before going offline.
> I seriously doubt, in 15-20 or 50 years, you will have the media reader available to read either of the media, whether tape or BD.
Today's Blu-Ray drives are perfectly capable of playing an audio CD from the 1980s, and compatible players probably will continue to be available for another 10-20 years even after everyone stops buying content on physical media.
I wouldn't be surprised if I can plug in my USB 3.0 CD/DVD/Blu-Ray combo drive into a USB 7.5 port in 2030 and use it to play an audio CD from 1980, provided that the disc is in good condition. That's a lifespan of 50 years!
Even in the digital age, well-designed and highly forwards-compatible forms of physical media can last a long time.
15 years isn't that long of a time. I have CDs from much earlier than 1999 for sure, and the media reader is sitting right in my living room. And I'm not even running a data center.
The cost of cartridges are comparable to optical disks. However, LTO drives are non-trivial in terms of both mechanical complexity and cost. I can toss an optical tray into the recycle bin without batting an eye. Not true with LTO, and I suspect their MTBF is similar (trade the quality of the LTO drive for the mass production and cheapness of the optical tray).
Enemy #1 in a large data center is power. When he mentioned the entire rack pulls 1,000W for the density suspected in this thread (~500TB to ~1PB), that piqued my interest. Even if you can't push 2-3PB per rack, the top stat you're crunching forever and always is watt-per-gigabyte. Space is important but if it breaks into the top spot you've done something else wrong along the way.
Enemy #27 for cold(ish) storage might be, say, pressure per sq. in. on your floor. I'd be interested to see a weight comparison of a fully-loaded LTO/cartridge rack and this design.
Perhaps not what you were looking for in an answer, but I'm interested to look over the schematics.
No comment on the access time and having only one drive means reading only one disc at once. This puts this storage solution in the realm of "I don't need this data within an hour" so I presume the application this system acts as redundancy for failed disks.
The Facebook open computer definition for "cold storage":
"where data is stored on disk but almost never read again, like legal data or backups of third copies of data — keep on increasing dramatically, there are huge demands for developing some form of cold storage system with the highest capacity and the lowest cost."
Are these Blu-ray disks rewritable? If so, wouldn't this be a great tech for Amazon's Glacier service? I believe they currently depend of costly tape drives to store data.
I believe Glacier is an all-hard-drive solution. Tapes are troublesome because they're not "online" so it's harder to verify data integrity and notice failures.
The drives are spun much slower than a regular drive (I've heard 300rpm) and so they can be crammed into much higher density racks without worrying about heat.
Maybe data is eventually moved to tapes for Glacier; I am privy to no special information.
Each LTO-6 tape cartridge stores 6TB of data. Each quadruple layer Blue-ray disk stores only 128GB. I am surprised that FB went with Optical Disk instead of Tape.
Optical jukeboxes/libraries for MO/DVD/CD used to be popular a decade or so ago and now considered antique. There is just not enough density with optical.
My guess:
1) Lower cost for drives (commodity drives).
2) The ability to (easily) seek to an arbitrary location on disk
3) Cost of BD burners are <$100; Cost per GB of disks is $0.025/GB. LTO-6 drives are $1000+, and cost per GB is around $0.012/GB. -- I think the cost works out actually, especially as the cost of blu-ray is still falling.
Why bother with rewriteable disks? This is cold storage meaning that they will likely never throw this data away. I'd suspect Amazon's Glacier runs on something similar.
Its okay for Facebook and other single entities for backups. However for Glacier, it would mean throwing away the disk whenever someone deletes their data. It would also means constant replacement of disks.
FB is required by law to store all its credit card transactions that every singal FB ads account has for 6 years. (So are other US based companies that process credit cards)
Facebook has built a novel implementation, but this isn't a new idea or product. Similar to how they used to buy servers and now build them to spec, they probably used to buy these and now design and build them.
After I saw this on James Hamilton's blog the other day I did a quick check of BD-R versus 4TB HDDs cost per GB. Using whatever I could find after about 5 minutes of looking around, I came up with $0.05/GB for both types of media.
If anyone can provide a link to BD media for sale at a price that make this economical I'd be interested to know.
I would think the media's longevity factors into the lifetime cost. If the optical media is certified for 50+ years, that essentially means they won't have a need to recopy data if for some reason they want to preserve it.. for a long time.. I don't know how long magnetic tape lasts, but I would imagine it would be a huge pain to need to keep backing up old tapes to new tapes every X years.
I talked to them at the Open Compute Summit and got the chance to look at it up close. They quoted me 100GB/disc and 10,000 discs in the cabinet for 1PB. They could have been rounding those numbers for simplicity, but that gets you in the neighborhood.
From the video: 24 magazines/rack * 36 cartridges/magazine * 12 discs/cartridge * 50 GB/disc = 518TB/rack. Based standard dual-layer discs. I cannot comment on storage of our traditional spinning-disk racks.
I might be interested in cloud backup service which would backup my "keep forever" stuff, put them to optical disc and then mail the discs to me when they fill up.
Could be also an add-on service to existing online backup business.
What about bit rot though? Some of my dvdr are no longer readable after 5 yrs, stored in very good conditions. Hdd backup, esp. with Zfs can provide very good bit rot protection.
Blu-rays are but a few years old. It's preposterous for any one to claim that they are "guaranteed" for 30 years, let alone 1,000 years. Not to mention that that guarantee most likely has an asterisk that limits damages to the cost of the original blu-ray disk in case it actually fails.
