Backup and Security – The 3-2-1 Rule of Backups and Blackhats

I had a wonderful telephone call with a prospective buyer today who was very concerned with potential worst-case security scenarios. After getting all of the easy questions out of the way, we began working through blackhat scenarios. The worst possibility we came up with concerned a blackhat disgruntled employee that had physical and logical access to all of the data centers of a company. This hypothetical employee knew all usernames and passwords of all systems and gained access to the data center where she erased all of the data (and snapshots) from the company’s servers, SANs, and NAS devices. She then logged onto the on premise backup appliances where she erased all of the backups. Finally, she remote logged into all other data centers and deleted the replicated backup from the original data center. As she was leaving, she poured gasoline on the floor, lit and threw a match, and ran away.

The question the prospective buyer asked me was whether Unitrends could handle this scenario without the loss of any backup data. It was a great question!

The answer I gave was rooted in a core backup best practice: the 3–2–1 rule of backups. The 3–2–1 backup rule specifies that when you back something up that you have:

At least three copies,
in at least two different formats,
with at least one of those copies off site.

Three different copies specifies copies in three distinct places. This doesn’t mean three copies on a single backup appliance (or in the same backup media pool). This isn’t about retention, it’s about redundancy.

Two different formats means at least two different methods to store data are used. This could be to the backup appliance and then via archive to a different device. Or it could be to the backup appliance and then via replication to a different device.

Finally, keeping one copy of data off-site protects information from natural disasters and against most human-induced mistakes.

So in this worst-case scenario, how do you architect a solution that is resilient despite the best malicious efforts of an insider? You have to move to a model of rotational archiving using on-line and off-line media. Traditionally, that is done with rotational tape; in recent years rotational removable disk drives have also been used.

What’s the problem with this? It takes more labor because you have to take the archive media off-line manually. But it’s the safest technique out there — because that archive media can then be physically transported, and put under lock and key, at another location.

Of course, what most companies do is use replication to get their backup data off-site —   and are then careful to use different usernames and passwords between sites and to not allow employees to have access to more than one site’s usernames and passwords. For most companies this qualifies as “safe enough”. But after you’ve been through a bad situation or two, you become paranoid. This paranoia often inspires comically meticulous behavior that helps keep you safe — and helps you sleep at night.

Have backup and security stories to share? I’d love to hear from you.