Unitrends Enterprise Backup Supports Global Deduplication
Unitrends Enterprise Backup, the software-only product that I’ve discussed in prior posts and which will be released this month, will offer global deduplication. I realize that one way to announce this would be to “brand” it as “honest global deduplication” or “bona fide global deduplication” or even “really, really, true global deduplication.” But I think it’s more valuable to discuss what it is that we’re going to offer and compare and contrast that with other approaches.
People use the term “global deduplication” to mean different things. So what I’m going to do in this post is describe what has been done in order to make this occur and what it really means for users.
First, what’s the benefit to users? The benefit is that you can achieve a higher data reduction ratio. How high? No one in the world can tell you the specific data reduction ratio for your data. Heck – they can’t even tell you the specific data reduction ratio for compression (which is much more predictable) for your data. The reason is that until you’ve gone through and identified how much data is duplicated, you just don’t know. But conceptually, global deduplication originally was conceived as a way to allow higher deduplication rates across multiple sets of “servers” or “appliances” (let’s just call them nodes) with processors, memory, and storage – and of course it makes sense that if you have 4 “nodes” that your best deduplication ratio will be achieved if you have only 1 block of data among those 4 nodes versus 4 blocks of data.
What I see out there is that there’s an arms race to state that people have amazingly high deduplication ratios. It used to be that vendors would promise 20:1. Then came 40:1. And 50:1. Today, I saw someone claiming 80:1. I’ve always wondered if there’s a “gullibility factor” that the marketing people are using to come up with these numbers – in order words, you can’t say a gazillion to one because no one will believe you but 97:1 is okay. 🙂
The fact is first that any form of deduplication works better when you have more redundant data. So what’s started happening is vendors are taking as a starting premise that you get a very high deduplication ratio through the act of not doing full-only backups. That drives a lot of these numbers higher.
In fact, getting 20:1 on an incremental forever with a specific synthetic full creation policy of a week or two is pretty good – it means you have a lot of redundant data.
But what about the “secret sauce” of global deduplication? Well – it comes down to how that secret sauce is implemented. There are three ways I’ve seen people claim “global deduplication”:
- Complete deduplication devices with associated processor, memory, and storage that deduplicate across those devices.
- Using an LVM (Logical Volume Manager) to take disparate physical storage devices and tie those into a single “logical volume” – and deduplicating within that logical volume.
- Redefining the term “global deduplication” to mean “non-jobs-based” deduplication. This comes about because there are vendors who only deduplicate within a single backup job rather than across backup jobs.
Companies such as Data Domain and Exagrid with deduplication devices typically will perform global deduplication defined as #1 above. Backup software vendors supporting LVMs (and optimizations to LVMs) will sometimes perform global deduplication as defined as #2 above. And only a few niche-oriented point solutions will define global deduplication as defined as #3.
Which approach gets the best data reduction ratios? In my experience, it’s the approach used with #1 above – although the specific algorithms used in this type of process are critical. What gets the worst? Obviously, it’s #3 above.
So with all of this exposition, what’s Unitrends offering. We’re offering an LVM-based approach with our software-only offering to perform global deduplication across disparate physical storage devices.
Got any thoughts on global deduplication or anything else associated with backup and data protection? Would love to hear from you.
