Recently there was an interesting blog post over at ZDNet entitled “Where Are the Affordable Enterprise Online Backups?” I made a comment to it – to the effect that bandwidth and RTO (Recovery Time Objective) are incompatible.  That caused some other comments to be made.  I decided to write a bit longer post concerning the entire phenomenon on this blog.

First of all, the term “enterprise online backup” is a bit misleading.  What the author of the post was complaining about was that there are no unlimited storage options with associated flat-rate pricing of the sort in the consumer space.  The reason tends to be that pesky term “enterprise” and what it means to people.  There’s an old joke that when someone mentions “enterprise” that means “hold on to your wallets.”  It’s actually a bit more complex than that.

The characteristics of enterprise-level online backup can be categorized as follows:

• Infrastructure.
• Quickly protecting data.
• Recovering from disaster.
• Security and privacy.
• Reliability and availability.
• Manageability.
• Trust and monitoring.
• Alternative disaster recovery methods.

For more on this you can search “cloud” on this blog and find lots of information concerning these; there’s also a webinar and an associated white paper that goes into this in much more detail and describes each one.  What I’d note at a high level is that this is a good bit more complex than getting back that the latest Lady Gaga MP3 that you lost when you spilled beer on your laptop.

Dan Goldberg had a good comment on the original ZDNet blog post that there was a tendency of unlimited online vendors to bait and switch with regard to upload speeds – to offer unlimited backup storage at a flat rate but then to throttle you down if you upload too much.  He’s right, of course.  He also notes that you can get many online vendors to ship you a disk at an extra cost.  Of course, what you do with that disk belies the “enterprise” nature of the discussion.  Enterprises, even in 100 person organizations, have multiple servers, desktops, and laptops – what you do with that data you get back and how quickly you can bring your business back is at the heart of your recovery strategy.

My favorite part of one comment was the following Freudian slip:

I am curious though why the prices are higher for businesses. Amazon S3 pricing page does not distinguish businesses and individuals. And many web backup providers sue Amazon s3 anyway.

As we saw with Carbonite, there really is quite a bit of suing going on (note: I realize that the intended term was “use” – just found this humorous given the nature of our industry.)

The fundamental truth of the matter is that consumer-based online backup is based upon the fact that the change rate of data on a consumer PC is relatively low and that a profit can be made in an arbitrage of the monthly fee with the underlying cost of the bandwidth and storage.  Read the terms and conditions and the SLAs of these services – if you lose data, you don’t have many remedies.  Small and medium businesses tend to have higher data change rates and larger data volumes – thus the attempt to create an attractive flat rate structure in which a profit can be made is much more difficult.

And note that I say this as someone who works in a company that charges $0.49 per gigabyte per month – and does so quite intentionally because companies such as Mozy Business, Carbonite Business, Barracuda Networks, and others charge $0.50.  The margins are awfully skinny regardless of the $0.01 difference in pricing – if you’re making sure that your customer can recover, and recover quickly.

One last note.  When we talk about “alternative disaster recovery methods” above, note that this can mean anything from rotational archiving to single tenant private cloud vaulting – none of which should have a monthly fee attached.  Of course, there’s no free lunch – there’s a capital and operational expenditure associated with everything.

After the SAN backup fiasco in Virgina, I started thinking about how how far we’ve gotten from the concept of what backup is really all about. Imagine the following scenario from a college football game this weekend:

 [In the middle of a nationally televised football game a linebacker blitzes and catches the quarterback on the blind side; the quarterback's leg snapping is audible to the sideline microphones.]

Trainer: Coach, your first-string quarterback, Tom, has a broken leg.  You need to get your backup in there.

Coach: Okay.  Let’t get Tom in there.

Trainer: I don’t think that you understood what I said.  Tom’s leg is broken.  You need to get your backup ready and put him in the game.

Coach: I understood.  Tom will take over.

Trainer: How will Tom take over?  Tom’s leg is broken.

Coach: Look – I know you don’t understand it, but I made a decision at the beginning of the season to optimize our backup strategy.  In order to have the most effective backup strategy, I decided that it would take the least amount of time to recover if our first-string quarterback and the backup quarterback were the same person.  That optimizes our RPO (Recovery Point Objective) and our RTO (Recovery Time Objective) because the first-string quarterback is our best quarterback and is already playing the game.  It’s brilliant!

Trainer: But the entire idea of a backup is that we have an independent and complete whole-bodied player that is able to go into the game.

Coach: Yes, but it’s inefficient to have to deal with two different quarterbacks – I’ve increased our efficiency by 250%!  So get Tom ready to go back into the game.

Trainer: Okay, you’re the coach, I’ll do my best.

[Tom is carried by teammates back onto the field.  The quarterback lines up in a shotgun formation, takes the snap, and gets ready to throw to his wide receiver.  The pocket breaks down and the quarterback is once again sacked.  After the play, the quarterback doesn't get back up.]

Trainer: Coach, Tom now has a concussion.  We have to get your backup in there – and your backup can’t be Tom.

Coach: I don’t understand.  The odds of two injuries occurring are astronomical.  Now we don’t have a backup!  What do we do?

SAN backup using the SAN as the backup storage is every bit as stupid as this.  SAN snapshots used in place of SAN backup is even more ridiculous.

SAN replication seems more logical – in this case the quarterback and backup quarterback would be a clones with identical memories that are continuously replicated to each other.  The problem here is that when the first-string quarterback goes down with a concussion, the backup quarterback catches that concussion as well.

This is what occurred in Virginia – and the reason that it took so many days of checking data integrity.  Smart people did something stupid – they designed a SAN backup system with no SAN backup.

SAN backup, or the lack of SAN backup, is in the news.  The failure of the EMC DMX-3 SAN for the Virginia Department of Motor Vehicles has made not only the technology press but the Washington Post, the Boston Globe, and others. No wonder.  The lack of a SAN backup strategy left 26 of 83 of Virginia’s state agencies down.

From Sam Nixon, Virginia’s Chief Information Officer

A piece of equipment went down that is meant to never go down, went down. [...]

The recovery time has been unacceptable.

From 2001: A Space Oddessey

Interviewer: HAL, you have an enormous responsibility on this mission, in many ways perhaps the greatest responsibility of any single mission element. You’re the brain, and central nervous system of the ship, and your responsibilities include watching over the men in hibernation. Does this ever cause you any lack of confidence?

HAL: Let me put it this way, Mr. Amor. The 9000 series is the most reliable computer ever made. No 9000 computer has ever made a mistake or distorted information. We are all, by any practical definition of the words, foolproof and incapable of error.

 There are echoes of the arrogance and hubris that lay at the heart of Kubrick’s 2001 – the belief that humans can build infallable solutions and systems.  Human’s can’t do that – and never will be able to do it.  The more important any technology, the more reliant we are on the artifacts of that technology, the more important it is that we think through the implications of failure.

The trouble is when intelligent people begin believing the hype regarding SAN reliability and availabilty.  SANs go down.  Period.  SAN data becomes corrupted.  Period.  SAN snapshots, SAN replication, and SAN redundancy are all techniques to manage reliability and availability.  They are not techniques to manage true SAN data protection – SAN backup does that.

The initial blame in this case was a memory card failure in the caching subsystem of the SAN coupled with a problem with a fail-over to another SAN.  The true trouble is that once the failure occurred, they couldn’t be sure that the replication system they had in place hadn’t corrupted the data.  This is the heart of the limits of replication, versus SAN backup, that plague the attempts by some vendors to tout their SAN technology as a substitute for SAN backup.  It isn’t.  And the people of Virginia are now living with the consequences.

I turned 50 this year.  It’s not a particularly pleasant experience – but of course is vastly more preferable than the alternative of not turning 50.  The advantage of turning 50 and having been in the technology industry for more than a quarter of a century is that the evolution of technology is so fast that you can gain a real appreciation for the old adage that “history repeats itself.”  Of course, as the wag once said “History repeats itself because nobody listens.”

When Data Domain went public in its very successful 2007 IPO (even 2007 seems like decades ago, doesn’t it?) one often overlook aspect of the company’s SEC filings was that it noted that secondary storage (typically associated with backup) was only one aspect of the company’s direction.  It was clear that the company had designs even then on eventually moving to primary storage deduplication.  When EMC outbid NetApp to purchase Data Domain, it wasn’t to increase the company’s market share in secondary storage deduplication – it was about leveraging a technology across EMC’s and NetApp’s storage assets across the board.

Dell’s acquisition of Ocarina in late July was part of Dell’s strategy toward building out its own storage portfolio.  At the time of the acquisition, Dell announced that it intended to be the leader in primary storage deduplication by 2012.  Ocarina’s technology isn’t based on the more typical hash-based deduplication technology; it instead is focused on data such as images, videos, and office documents that typically only deduplicate effectively in secondary storage environments when redundant backups of data allow higher data reduction ratios.  Dell has already stated that Ocarina’s technology will be leveraged across its iSCSI SAN assets (its acquired EqualLogic product set); it’s highly probable that it will be embedded within the NAS clustering software acquired from Exanet in February and I’d be surprised if Dell doesn’t pursue embedding it into its DAS (Direct Attached Storage) offerings supported directly but in its server offerings.

Dancing with the One that Brought You: Backup and Deduplication

Where does all of this leave the technology sector that originally led in the introduction of deduplication?  It seems increasingly obvious that most buyers are growing skeptical of paying large amounts of money for secondary deduplication devices and then paying additional large sums for backup, archiving, and disaster recovery software and services and then having to integrate the two by adding a backup server and then manage and monitor the collection.  There are buyers who have budgets, the vast data stores, and the desire for years of near-line retention who will pay for state-of-the-art secondary deduplication devices; however, this isn’t where most buyers will be focused.  As deduplication becomes increasingly more available both within backup software packages and within backup appliances, buyers are increasingly making a rational choice to focus on total cost of ownership and return on investment rather than on pushing the limits of technology.

This is where history repeating itself becomes so obvious.  The history of technology companies consist of those technology “pioneers” (you know, the ones with all the arrows in their back) and the “fast followers” who bring the technology to the market in an affordable and easy-to-use fashion.  [Geoffrey Moore wrote a nice book on this phenomenon called "Cross the Chasm." - recommended reading]  Technology companies take advantage of technology pioneers by realizing high profits; however, the pace of technology is so fast that it’s difficult to do this for very long.  [As a side note, it's interesting Data Domain is now owned by a company that is sometimes called "Excess Margin Corporation" by not only its competitors but its customers as well.]  Share of wallet, which is a metric associated with the percentage of overall information technology spend that a vendor can claim, is a wonderful thing to strive for from the perspective of a vendor – but can have pretty dire consequences for fast-growing businesses being strangled by their IT costs

What is less obvious and perhaps more interesting is what the impact of primary storage deduplication will be on data protection. That will be the subject of my next blog post.

[Preface: This is an excerpt to the 7 Shortcuts to Losing Your Data (and Probably Your Job) white paper and the 7 Shortcuts to Losing Your Data (and Probably Your Job) webinar; click on the respective links to get the white paper or to get access to a pre-recorded webinar (or sign up for a live webinar.)]

Take Recovery for Granted

Shortcut to Losing Your Data

All of the previous shortcuts we’ve discussed were derived from the tables of statistics depicted previously which illustrated the reasons for data loss. However, this shortcut – concerning losing your data by taking recovery for granted – applies to all of those reasons for data loss. This one is pretty simple – simply assume recovery will work.

Taking Another Path

[Preface: This is an excerpt to the 7 Shortcuts to Losing Your Data (and Probably Your Job) white paper and the 7 Shortcuts to Losing Your Data (and Probably Your Job) webinar; click on the respective links to get the white paper or to get access to a pre-recorded webinar (or sign up for a live webinar.)]

Play the Odds on Disasters

Shortcut to Losing Your Data

Taking Another Path

Why don’t people walk around outside in thunderstorms? The odds of getting struck by lightning are pretty low. The National Weather Service estimates that the odds in any given year are 1 in 500,000. The reason of course is that the consequences of being struck by lightning are very high. The odds of death are 1 in 10; the odds of disability approach 9 in 10.

The odds of data loss due to a natural disaster are relatively low; however, the consequences are severe. In order to safeguard your data, you need to have a disaster recovery plan for your environment. A major part of that disaster recovery plan is protecting your data. There are two basic schemes for this: tape-based rotational archiving and electronic-based replication of data to an off-premise site.

We advise looking at vendors that support an integrated D2D2x approach whereby you can use disk, tape, or electronic replication concurrently to optimize your overall spending in support of true disaster recovery.

[Preface: This is an excerpt to the 7 Shortcuts to Losing Your Data (and Probably Your Job) white paper and the 7 Shortcuts to Losing Your Data (and Probably Your Job) webinar; click on the respective links to get the white paper or to get access to a pre-recorded webinar (or sign up for a live webinar.)]

Pay No Mind to Computer Viruses

Shortcut to Losing Your Data

Computer viruses range from the annoying to those that threaten not only the systems of your organization but your organization’s reputation as well. The easiest way to lose your data with respect to computer viruses is to not install a firewall and anti-virus software. In addition, make sure that all of your systems operate using Windows – not just your PCs but your servers and your backup servers as well. That way you insure 100% infection when a virus occurs.

Taking Another Path

In order to protect your data you will of course have a firewall and install anti-virus software. From a backup perspective, the important thing here is to operate your backup and disaster recovery software on a non-Windows platform.

Vendors ship their backup software on Windows platforms for one reason – they can make the most money with the least expense because Windows is so ubiquitous. If you take a step back and think about it, however, it just doesn’t make sense to run your “protection” software on the same operating system that is relentlessly under attack by malicious people.

[Preface: This is an excerpt to the 7 Shortcuts to Losing Your Data (and Probably Your Job) white paper and the 7 Shortcuts to Losing Your Data (and Probably Your Job) webinar; click on the respective links to get the white paper or to get access to a pre-recorded webinar (or sign up for a live webinar.)]

Rely Upon the Honesty of Others

Shortcut to Losing Your Data

Theft is a another cause of data loss. Theft manifests itself either via a “data spill” in which data is not lost but instead made available to third-parties for whom the data wasn’t intended or in outright destruction. For the purposes of this paper, we’re going to limit our discussion to outright destruction of data.

The destruction of data is rarely performed by a relatively disinterested “hacker”; instead, it is most often performed by a disgruntled employee or ex-employee. It is incredibly difficult to prevent; although precautions should be and most often are taken particularly around the involuntary termination of employees.

Taking Another Path

The first step to avoid malicious destruction is to create policies which make your primary data more difficult to destroy. These include strict policies and procedures associated with not only involuntary but voluntary termination as well and on taking steps to secure your environment from external access.

From the perspective of data protection, theft is largely indistinguishable from human error in terms of the tools and techniques that must be used to protect your data – the only difference between the two is motive and motive isn’t really a factor in terms of this type of logical failure. Automation and retention again are the most important strategies for ensuring that you can survive this type of threat.

[Preface: This is an excerpt to the 7 Shortcuts to Losing Your Data (and Probably Your Job) white paper and the 7 Shortcuts to Losing Your Data (and Probably Your Job) webinar; click on the respective links to get the white paper or to get access to a pre-recorded webinar (or sign up for a live webinar.)]

Disregard Software Corruption

Shortcut to Losing Your Data

Software corruption is the third leading cause of data loss. Anyone who has lived through a BSoD (Blue Screen of Death) in Windows understands the concept. Of course, software corruption is caused not only by software defects but through the chaining of errors in systems as well. It’s important to ignore software corruption in order to increase your odds of losing your data.

Taking Another Path

Software corruption, like human error, is another type of logical (as opposed to physical, or hardware) failure. The primary differentiation with respect to data loss is that software corruption can occur and remain undetected for days, weeks, months, or years. Thus automation for strict adherence to policy and retention are incredibly important techniques for protecting your data against software corruption.

[Preface: This is an excerpt to the 7 Shortcuts to Losing Your Data (and Probably Your Job) white paper and the 7 Shortcuts to Losing Your Data (and Probably Your Job) webinar; click on the respective links to get the white paper or to get access to a pre-recorded webinar (or sign up for a live webinar.)]

Trust Your Fellow Coworkers to Follow Policy

Shortcut to Losing Your Data

Human error is the second leading cause of data loss. Human error ranges from accidental deletion of files and records to ignoring policies regarding data to rebooting systems without proper shutdown procedures. Blind belief and trust in your fellow coworkers to not only follow policy but to not make any mistakes at all are fundamental to using this shortcut to its fullest potential in losing your data.

Taking Another Path

There are two fundamental reasons for human error: ignorance and arrogance. Attempting to change human nature is the height of arrogance. People have a tendency to be incredibly poor at following policy. Thus specifying that all “important” data will be stored only on centralized corporate servers and storage tends to fail as soon as a C-level executive loses the data on their notebook. But even when people try their best to follow policy, accidents such as file and record deletion will occur.

The best defenses against human error are automation and retention. Automation allows policies and procedures to be created and automatically executed. Retention allows recovery of data even when the data loss isn’t noticed for some period of time.

Retention is one of the fundamental differentiations between backup and simple high availability (which is typically achieved with some type of replication) – high availability handles hardware failure well but does a poor job of handling logical failures such as those caused by human error – because logical failure is simply replicated in highly available systems. Of course, protecting against hardware failure using high availability and against all types of failure using backup is a common technique for protecting data and systems.

Previously, we described why D2D is such an important component of protecting your system. When we discuss any type of logical failure, including human error, another important concept is to protect your data using a superset of D2D which is called D2D2x (Disk-to-Disk-to-Any.) D2D2x simply means that you have longer-term strategies for backups to either on-premise rotational archiving media (disk or tape – although tape has the risks we’ve discussed in previous posts) or to a private or public cloud.