Why don't people make backups anymore? I know that using Google Apps is much easier than running your own stuff in-house. It requires much less time and resources, but at the same time there are still best practices and disaster recovery plans that should always have something about backups in them. And just having them isn't good enough. They also need to be followed. Having worked at a place without these ( and attempting to get them implemented ) versus where I currently work that has them and follows them, I have to say, there are a lot fewer times where I feel like screaming here. Do it, and stop expecting 100% uptime and availability of all docs/important information from any service, internal or external. Nothing is 100% but there are many ways to make it less stressful.
100% agree.
We are developing service which tries to solve some of these problems, so I'll do a shameless plug here.
One way to address these kind of problems is to use cloudHQ to synchronize your Google Docs with SugarSync (clickable: http://cloudHQ.net/sugarsync). (We are releasing dropbox this weekend).
Here is our idea: there are many different cloud storage services and each of them is very good for a particular use case but not so good for other use cases. For example, Google Docs service is excellent for collaboration but not so good for storage. On the other hand, Dropbox or Sugarsync service is excellent for storage but not so for collaboration.
So if data is continuously synchronized between Google Docs and lets say SugarSync storage, then you can always fall back to SugarSync when Google Docs becomes unavailable and vice versa. In order words, you can use other service as a standby site in case of service unavailability, malicious or accidental deletion of files, etc.
One of the system admins here made a comment about how important backups are for a business. He said the only time he ever saw someone get fired immediately was when they screwed up the backup system.
He talked me through all the design considerations they had to do for their backup system for the datacenter here. It's a lot of work, but there is a lot of business value in having a working backup system.
That's what we're doing now. We have backups of all our servers, and user backups for all the laptops..
But Google Apps is a pain in the butt to backup, since we don't have direct file access to things. The best strategy I can see, if you want to stick with it, is to email yourself every file, every edit. That's pretty insanely tedious.
Actually, since SugarSync inaugurated the shameless plug portion of the thread, let it be said that Google Apps is hard to manually backup. That why we automated it at Backupify. http://www.backupify.com
Automatic daily Google Apps backups -- all of it, even Google Sites -- for just $3 per user per month. Unlimited storage. One-click restore for Google Docs and Gmail messages. And we're soon rolling out full-domain search, so you can query all your Gmail, Docs and Calendar accounts simultaneously.
We also backup your Twitter Feed, your Facebook Pages and your Flickr or Picasa photo albums, should the need arise.
Also, for the record, we have a 1-800 number that we actually answer, so in the event you have trouble with our service, we'll pick up the phone and talk you through the fix. We can't repair your Apps account, but we can darn well make sure you get to the data that's in it. We wouldn't charge you, otherwise.
Do you have any integration (or plans for integration) with Dropbox? I'd love to sign up to a service that simply dumps a copy of all my backed up files to a Dropbox folder somewhere, once a week.
That's on our roadmap, yes. In the future we'll be storage-agnostic, so you can use our S3 storage, your own S3 storage, Box.net, Dropbox, and a few other options we're mulling over.
The $4.99 per month plan, Backupify Pro 100, allows for 5 users on an Apps domain and 20GB of storage. That plan is geared towards small mom & pop operations, not the bigdog system e1ven was running.
Our Pro 500 plan is $19.99 per month and includes 10 Google Apps users and unlimited storage. That actually breaks down to less than $2 per user per month. Beyond that first ten, additional users are $3 per user per month. I usually just quote the $3 figure for simplicity's sake, especially since most admins using a paid Google Apps domain (Google Apps Premium) have more than 10 users anyway.
Aren't the Google Docs GDATA enabled? [1] If so then it should be pretty straightforward to write a script which would pull down copies of any file that had changed for local backup.
I understood that, he's going through the 'standard' Google escalation process for paying customers. Someone from Site Reliability Engineering (SRE) in the docs group is no doubt working on a ticket to track down broken GFS cell or maybe a PCR (that was code for changes that might take out a whole cluster or data center for periods of time). Its probably one of a few dozen they are looking at and given the layering the actual fix could take a couple of days to take hold, especially if the data in question is actually only available on tape at this point.
But he's an operations guy like me, and he probably wears both a belt and suspenders just in case the belt breaks. So in response to the complaint that emailing yourself files to insure you have copies is a pain, I suggested he script the moral equivalent using Google's provided APIs to the docs service.
How do you backup Google Documents? Can any of the popular backup utilities do that for you as part of your whole-company backup plan? That would be pretty cool.
I know you can use pop (and maybe imap) to pull email out of GMail. but can you recommend a product that can handle slurping and backing up all the mail for your domain?
You can also do it direct from the web UI. Just select all your documents from the Google Docs homescreen, click on the "Download..." link in the righthand column, and it will ask you what formats you want then generate a zip file to download.
I don't know of a command line script that uses OAuth (GDocsBackup has a command line mode but uses password auth.) Anyone?
Agree, I use backupify.com to backup my FB, Twitter, Gmail and Docs.
The thing about unreliable customer service is really bad, but you just can't relly on one company, even Google.
I agree. Using a third party service is reasonable. The important thing is having a backup plan.
Shameless plug: This is why I wrote my app, CloudPull. CloudPull backs up your Google Docs, Calendars, and Contacts to your Mac. If for any reason you are unable to access your Google account, CloudPull ensures that you have backups of your Google data on your Mac.
So your reply about atrocious customer service (i.e., a post about a PAYING customer being unable to get access to his file) is to say "well, you should have backed up"?
The real story here is that a paying customer was unable to get anything better than a free user.
FWIW, I must be lazy or something, but I don't think the user of a VERSIONED online document storage service should be reasonably be expected to also have his/her own backups. What the hell am I paying for then?
If I park my car and don't lock the doors, and come back to find it gone, is the story that I'm an idiot for leaving a car unlocked, or is the story that the neighborhood is unsafe?
They're both problems, but I can actually solve one of them myself. That doesn't excuse the $BAD_THING that happened, but it does offer me options for when it inevitably does.
The issue here is that he paid someone else to do it.
So in your analogy: You park your car at a garage with a guard, pay them to keep it safe and leave your car unlocked. And when you come back it's gone.
Yes he could have done something to make it less likely it, but he also explicitly paid someone else to handle it for him.
Personal responsibility can never be outsourced.
If one does not want to lose something, one makes an independent backup.
Google or whoever offers a probabilistically reliable system at a certain price. If one does not like the odds, pay a little more for another backup. Recurse. (Or just curse when data is lost.)
Do you backup your backups? And backup your backup backups? You have to draw the "good enough" line somewhere, and to most people, Google backing up their stuff is backed up enough.
Sure, and that's why we're moving off Google for hosted Apps.
I entirely agree- When things are mission critical to you, you need them in an environment where you can control it, back it up properly, and ensure it stays up.
We're not a priority to Google (Understandably.)
For things that really matter to your company, you should bring them in-house, so you can ENSURE they're backed up, and available when necessary. At least that way when things do fail, you can make fixing it a top priority, rather than a nice-to-have.
I don't follow. Why does the fact that you have to do your own backups mean that you can't use Google (or, conversely, why does using Google mean that you can't do your own backups)?
You certainly can pay to use Google Documents and do your own backups. It's just that doing so would defy the biggest appeal of the service. The appeal is that Google handles day-to-day operations for you.
There's more to "day-to-day operations" than backups.
The reality is that Google is much less likely to damage or lose your data than you are, but it's always good to keep extra copies around locally. This is important for many reasons, like access when either Google or your whole net connection is unavailable, and more flexibility with the data.
I think you are unfairly contrasting he flaws of something that exists against he perfection of something you want to exist.
In-house IT fails plenty of the time also, and plenty of organization cannot afford to fix some failures. One must consider the probabilities in all cases.
key, I think, is to make sure that more than one entity needs to screw it up. If it's really important, make that three entities. But the point is, if google hosts your email, make sure you have backups somewhere else.
Personally, I think this applies to yourself, too. As much as possible, make it difficult for you to overwrite your own backups before they expire. If possible, encrypt and outsource.
"You always backup" is an oversimplification. If you run a medium-size company, chances are, you don't back up stuff yourself. Instead, you pay someone else to do it. You trust them to do it right. This article raises a good question. Why can't you trust Google to do it right as well?
You should always have at least 2 different backups, one on site and one off site (disaster recovery) these are completely separate from "it's in the cloud so I don't have to worry about it." And this is why I personally dislike "the cloud" because too many people rely on it and don't have a proper plan in place. You can rely on it sure, but don't only rely on it.
A decent hoster or mail service provider takes care of that for you. They have at least 2 different backups, one on site and one off site. You don't take care of that yourself and the backup is entirely transparent to you. This is why service providers can make money: because they can take care of things for you. It is their raison d'être. Taking care of that yourself is as ridiculous as hosting the email servers yourself. It's not cost effective anymore. The only conclusion from this story is that Google is not a decent hoster or email service provider. The company using them can be excused for expecting better service and they are rightfully moving their business elsewhere.
And to address who does it, it doesn't matter. Someone in the company should be in charge of doing the backups. If no one else will, then take it upon yourself to do so. If you have a medium sized company, someone in your IT department should be doing it. And if you are a large company, there is simply no excuse.
This is precisely why I think it's insane that people trust their email - the nexus of their online presence, normally the key to every account and their primary communication tool for everything from conversations to account statements to a free service. It's not 1996 anymore, this stuff matters. I pay for Rackspace email. When I have a problem I can call a human and get an answer. As far as I'm concerned it's a bargain.
Google doesn't do human, it doesn't matter if it's a free gmail account, a serious-money adwords account or your entire business on apps, you're just a number if something goes wrong you're on your own, shouting at the unmoving monolith.
I've got Google Apps for Business, and when all of our mailboxes were unavailable I contacted Google at the number listed on their website, talked to a human, got it resolved in about 30 minutes.
I'm not sure that they followed the support instructions, the email excerpt said to reply if that wasn't the right solution but it just says they sent another support email.
It probably would have been better to call, I mean you are paying for 24/7 phone and email support.
I haven't had any problems with getting support through Google Apps for Business, I might be an exception though.
Not that it matters much, but I called them several times, and replied to the original email. But I think you're focusing too much on the specifics.
The problem is that your mail is crucially important to you, but rather less important to Google. In such a scenario, it makes sense to change your plan to something under your control, even if it's painful.
The point isn't so much whether or not Google comes out bad, it's whether you have someone to talk to that can actually fix the problem when something happens.
There are providers that can give you various guarantees, response times and that will make key personnel available to you (at a price). Google is not one of those providers.
This person was a paying customer, and did get human support. From the article:
We got back several emails explaining that they are very sorry,
but while they are working on retrieving the file, they can’t
promise any dates or times.
It's easy to miss that, as most of the article was a copy-paste of a useless email. Looks like after that, he did get the human support you're talking about.
Also it's not fair to single out just Google here. SalesForce, Microsoft and many others offer cloud based solutions. Any cloud solution has the risk of them screwing up something in their data center and losing your data. Stay local and in control if the data is so mission critical.
Similar argument could be used against staying local: are your engineers as capable at data retention as Google's? What if your in-house people screw it up? Will you have some remedy that isn't available to you if Google had screwed it up?
If your in-house people screw it up then it means you don't value the data very much. If you say you value the data but don't act like it, it means you don't value the data.
If you really, actually do value the data, then the best way to assure its safety is to take control of it. One company I worked had a practice of taking a set of offsite backups to an emergency facility, restoring the data and doing some sample processing. Every month they did this, without fail. That's caring about your data.
You avoided the question: what if something goes wrong with your in-house process? Is there anything you can do vis-à-vis your employee that you can't do to Google? Is that worth the price of keeping that person on your payroll, drawing benefits, driving up your costs? Would you fire them? Worth the employment claims risk?
These are not binary decisions. People vastly overestimate the value of doing things in-house. Most of the candidates for those "in-house" data management jobs couldn't cut it at Google. Why not leverage Google's HR and recruitment, training, and expertise in infrastructure operations to your benefit, esp. when you consider:
(1) that your own hires offer you no better protection; and
I didn't mean to avoid that question. I think I answered it implicitly: if you care about it then take control of it.
There's a lot that you can do about your employee that you can't to with Google. You can educate them, align their motivations with yours, etc. You can review the process and make sure it doesn't depend on one person. The list goes on.
Of course there are times to outsource. Times when practicality makes it necessary, when money makes it desirable, and so on. I do not take issue with the very idea. Sometimes it's the right choice. But sometimes it's the wrong choice.
Cool. It sounds like we draw that line in different places. I think your comments about aligning motivations is universal, and applies regardless of whether or not a services is sourced. My position is the illusion of "control" by assigning tasks to full time employees is almost never worthwhile in light of the enormous costs and risks of putting someone on your payroll. When a disaster strikes and you're caught empty-handed, you'll either be firing an employee, or firing a vendor. It's a lot easier and is less risky and it costs less to fire and replace a vendor.
But if a problem occurs in-house, I feel like it's under my control!
I'd rather have a 1% chance of losing my valuable data if I feel in control than a 0.1% chance of losing it while feeling like there is nothing I can do about it.
You're missing the point. The problem is not that the service is free (especially since as a free service, Google's uptime probably leaves all their competitors in the dust).
The problem is just having your email stored outside your private network.
It makes a lot of sense for startups and small companies, less so when they start growing and they can afford a staff of sysadmins.
This attitude drives me nuts. Why would anyone want a staff of sysadmins? You're almost certainly buying (at a high capital outlay) the B-team when you can rent the A-team for next to nothing.
There's nothing magical about your "private network". These days corporate networks are chock full of holes and ripe with hypocritical security priorities.
Use Google apps for business ($50/year/user). Hire one of the many companies that archive Google Apps data.
It is total folly to think that you'll buy better security by paying for your own infrastructure and a small army to run it. That's like claiming the only safe automobile is one that you and your friends build.
As I said in my comment, this is never going to fly for any company past a certain size. There is just too much confidential and legally sensitive stuff going through your email on a daily basis. Especially if you want to go public at some point (or even be acquired).
As a lawyer, I respectfully scoff at this response. The exchange servers running the biggest companies in the world are sourced to third party providers. I love how people will put their "legally sensitive" stuff in the hands of a $60K/year sysadmin, but not in the hands of a multi-billion dollar infrastructure operations specialist that also happens to manage the world's best index of Internet content. I really struggle to follow the logic.
It's the application. Gmail is a hosting service for your email, but it's also a fabulous intuitive email webapp. I am sure many companies/people would purchase an internally hosted gmail appliance (heck, maybe a whole google apps appliance), if they could. I think that's one reason that many of us end up in this situation.
I pay $1 per mailbox per month. The current price is $2 per month per mailbox, still a deal. Rackspace Email (formerly MailTrust) is incredibly reliable, backed up daily, good spam filters, and excellent 24/7/365 support. In the past ~3 years I've only experienced a single downtime incident of about 10 minutes.
When your hosting company goes down, you're basically stuck waiting for them.
The only way to ensure that you never have downtime is to not use the Internet, and keep all your servers connected to a nuclear reactor, all inside a mountain. That covers power and not being flooded or hit by a tornado, anyway.
Even then you'd better be careful not to trip over any of the power cables or type "rm -rf /" as root.
For mission critical things, you want to reduce the points of failure, and ensure that when things DO go wrong, you have a reasonable escalation path.
It's now been over 8 hours since it went down, and no fix from Google yet. 4+ days on the missing file.
If I was running in house, I could have entirely restored the mail server from tape by now. I could have swapped over to a hot-spare in a few minutes. I could have failed over to our backup internet service. I have a lot of options.
With Google, my option is to wait.. And hope my business doesn't lose too much money while Google gets around to fixing it.
"""So what are we doing differently? Simple. Amazon serves as “cold storage” where everyone’s valuable photos go to live in safety. Our own storage clusters are now “hot storage” for photos that need to be served up fast and furious to the millions of unique visitors we get every day. That’s a bit of an oversimplification of our architecture, as you can imagine, but it’s mostly accurate."""
You can always maintain a hot-backup, fail-over of your site on your own servers -- perhaps with reduced functionality until the scalable cloud services come back online. For a mission-critical site, this would seem to be a reasonable tradeoff.
Here's a guide to using Google Apps or any other outsourced email provider successfully:
1. Use a resilient external DNS system
2. Configure local mail clients to save messages for "offline browsing" or equivalent.
3. Use local mail client to check mail at least once a day
When email goes down, point the DNS to a pre-configured mail server. You have all your archives and shouldn't be missing any mail since delivery failures will try again not too much later.
As for the Google Docs issue, ensure that users are exporting vital docs frequently, preferably when they're done making a change. You should actually edit locally and use git. You can set it up to automatically upload changes to gdocs on push.
Okay e1ven, call this number 1.800.571.4984. Ask for Ryan. Or, if your prefer, email ryan@backupify.com or tweet @ryanatbackupify. He's our inside sales guy. Backupify can't get your lost file back, but if it happens again, we can make sure its restored in minutes, not days (or never). 30-day trial of our premium product, for you or anybody else who reads this post.
http://www.backupify.com/business/500
in other words: if google goes down, you can do other work meanwhile or go to lunch, if your system goes down, you have to spend your time/money to fix it.
If Google got things fixed over a lunch I would entirely agree with you. The problem is they take days to get started, don't give you much feedback about what the problem is, and I've never actually had a bug ticket fixed. So, yes you can keep on doing work, but you also will keep on not getting emails/business.
Dropbox has an advantage here, in that all files are stored local (mirrored, really) in user's computers, and they are backed up. It doesn't scale to shared editing, though.
I'm turning more and more to plaintext/markdown on Dropbox.
Dropbox-style mirroring is still susceptible to the hypothetical "service problem killed/corrupted a file, the mistake was happily sync'd down to all mirrors."
In short: if you're reliant on the cloud service for your backups (as distinct from your sync'ing), you're at risk.
Dropbox stores different versions of files so even if files are corrupted/deleted and this change is sync'd up to them, you can still restore your files.
Of course, simplistic mirroring solutions will have these problems.
However, the right mirroring solution for syncing information (documents, presentations, etc.), should provide auditing, providence of created documents, and rewind for all actions.
At which point you're using technology even more complicate than dropbox and you have to trust that it works just as you have to trust that dropbox works. Not to mention that anything that can write to primary store (i.e. the "rewind for all actions" in your hypothetical solution) is just as likely to corrupt that store as dropbox is.
For backing up files that need to be on Google Docs, I use GDocBackup (http://gs.fhtino.it/gdocbackup) and have it download my Google Docs to a Dropbox folder.
That's where we've moved all our Shared files. I don't love the pricing models, or the privacy issues, or the security model, so we're looking for something else.
AeroFS is the best off-site solution so far, but it's still very beta.
In case anyone cares to hear a counterpoint, I run a nearly 20,000 seat Apps domain and the service and transparency we receive from Google is leaps & bounds better than from any vendor of similar size. Perhaps my happiness makes sense given how much we're paying them compared to, say, e1ven's company.
I think a lot of this is not about security or service level. It's psychological, it's about control/trust. When it's under your control and you can fix things yourself, you psychologically feel more safe. If you rely on someone else and they say "we're working on it" but don't give you an estimate, you feel a lack of control. It may be that doing things yourself has worse uptime/performance/etc than relying on someone else, but the psychological effect of that loss of control when something bad happens is huge.
I would be afraid that in a dispute Google could do a PayPal and lock you out of your data. I would also be concerned that Google would provide law enforcement access to your data without your knowledge, consent or a warrant.
Mixed reviews on the Google Apps marketplace for backupify: https://www.google.com/enterprise/marketplace/viewReviews?pr... -- 27 users take 21 days to backup? Might well be the Google Apps infrastructure that's not up to the task. Looking at my company's Google apps domain, we probably have several million threads and are close to a terabyte of data. I imagine backing that up via IMAP could take a while.
Those are fair criticisms of Backupify to date. We've recently totally reworked our throttling mechanisms, to our initial backup times are decreasing rather rapidly. But, to your point, we are at the mercy of how fast Google can give us the data through their API. That said, given Google's general uptime, the likelihood of Google losing a file before we can back it up is pretty remote. Not zero chance, but really unlikely.
It's a big deal to us, not a big deal to them. We'd pay extra for a tighter SLA if that option were available, but as it is, on outage hurts us a LOT, and them not very much.
That would seem to be prescription for bringing it in-house, despite how little I want to do that.
Find a small local IT company to use. It's been my experience that they are very responsive and do really care and many of them are very high-quality (old Unix guys). Google are just so large and so far away. They can't be everything to everybody (try as they might).
Personally, I use Google Docs every day, and I back up all of my docs on a regular basis (easy to do). Same with my GMail: use POP3 to keep a backup copy locally.
I would not consider running my own services and not back up my data. If you use GMail and/or Docs for free or as a paid customer, still, why would you not make local backups?
No person or organization should put itself in a position where it depends only on an external source for either security or backups. Whatever else might or might not be "mission critical", I think this inherent is.
Come back and tell me that relying on using Google docs while keeping local backups is problematic or awkward. That would be a fair complaint.
Toshiba isn't going to help me if my drive goes down either. Whether they send a nice or not nice email isn't the point.
This sounds great, but in practice is impossible. Are you personally as an individual going to take care of 100% of your own backups? Are you going to drive the tape over to the other facility that you own, make regular checks that you can restore, etc? No.
Ultimately you are going to rely on someone else and something else. Be it your IT guy, the company that hosts your backups, the power company, the company that makes the machines you keep your backups on, etc. You can distribute this so that you rely on lots of someone else's and any one or more of them can fail, but often you'll still have one point of failure somewhere (the guy who monitors the distributed backups for example).
I mean each organization needs to have in-organization backups and each individual needs to back-up their machine.
Yes, my IT guy. I need to have an institutional relation with my backup guy, not just a paying relation. The IT guy could send the data to three different online backup services but me having a personal and institutional relation with him makes sure that whatever scheme he uses will remain reliable as other things change.
The point is that every organization has at least one mission, one task that it is worth doing within the organization rather than offloading to another entity. And backup is always going to be close to that mission.
I worked for a company that switched to Google Business a few years ago. The one thing I'll say about the experience is that administering the back end totally sucks.
The UI in a lot of places makes no sense. For instance, there are settings for when you set up an email list that are labeled so obtusely that you're not really sure of what you're selecting.
Another thing that totally bit us once was when dealing with document ownership in Google Docs -- a lady in marketing had created a bunch of documents, then later left the company. After scouring the back end for about an hour (literally), I concluded that there was NO WAY for the admin user to reassign the owner of the document to a different person. I ended up having to log in as that person (luckily we kept people's accounts active for 6-12 months after they left) and switching the owner that way. Totally absurd. (If I'm wrong about that, please do tell.)
As far as customer service goes, it's definitely a minus. They don't make it easy to tell you who to call or email when you do have a problem, and once you get there, the time before your problem is resolved can vary wildly (from hours to days). That really sucks when you have a salesman breathing down your neck because he can't use his email.
We've moved from Google Docs to Dropbox, at least as a placeholder; We're still trying to find a better long-term solution. AeroFS is leading the pack, once it is out of beta.
For Email, we plan to, but it's a bit more complicated.
I'll need to imapsync each account, so it's going to be messy, so I'll need a week or two to plan it out, and build/test a new Zimbra server.
This is precisely the reason why I don't use Google Docs for any mission-critical documents. Store them in Dropbox or some other shared, automatically-backed-up system where you can easily retrieve a copy from somewhere.
Always have redundancy too. We keep most of our files in Dropbox. But we all use Time Machine too. We also use version control for a lot of our work.
This might seem like overkill, but each has a specific use and purpose with the added bonus of providing us with backup redundancy.
I think this is the problem with all cloud services. The liability costs involved in dealing with worst case scenarios are potentially so huge, the majority of providers have terms and conditions which relinquish all responsibility when bad things happen.
The only real solution is backup, and the only backup that can truly be relied upon is one you make yourself; in which case the point in making use of a cloud service in the first place is pretty much reduced.
>We’ve been running flawlessly for 6 months, paying them $50/user to avoid handling it ourselves.
if you open Excel and put some typical numbers into it (like commodity hardware cost, networks, electricity, data center mortgage, basic maintenance employees, 40% margin, etc...) you'd see that $50/user/month is just basic usage - timeslice to run provided software on provided hardware. There is no room in that number for a real person dealing with your specific issue. An elephant can't squeeze into a needle eye, even if he promised it to you in writing and accepted the money - just look at the elephant and at the needle with your own eyes.
