Follow the Clues: Why did this recovery fail?

Over the past 30 years, IT pros have come to us for solutions to issues they can’t solve. This is based on a true story. So as not to embarrass a customer we have masked their true identity, but the rest of the facts are accurate. If your IT systems gave you the same clues could you deduce the cause?

The Setting

The IT department for a large entertainment complex supported multiple critical business operations, including SQL Server databases, file shares, and customer loyalty systems at their primary, on-premises data center. They used best practices to protect these applications and data. In addition to meeting the basics of data protection with a firewall and virus scanner that they diligently updated with the scanner’s latest library of known viruses is constantly being kept up to date.) Every employee with access to the corporate network had been trained to identify and avoid dangerous downloads.

They used Veeam to back up their primary data center. They stored backups both locally and replicated over a WAN to a remote building on the property. Backups were performed regularly and appeared to be successful. They periodically ran tests to ensure servers would recover in the event of a downtime incident.

One day employees from multiple departments called the Service Desk because their workstations were unresponsive, and they couldn’t access their business applications. The servers appeared to be in operation with no alarms coming from the hardware. Attempts by admins to restart the applications failed so recovery procedures were initiated. The emergency recovery did not fix a single app and they were left in near total shutdown.

So, what happened? We gave you a few clues. Would it help to tell you that a few minutes later screens demanding a ransomware payment began to appear on screens all over the operation?

What Happened?

Yes – they had been hit with a ransomware attack. Even though employees had been trained,  someone had clicked on a link that initiated the infection. Business data files and folders were encrypted and inaccessible. The worst part is that this form of ransomware actively sought out the backup infrastructure and encrypted it as well. Once the backup application is encrypted backup files are worthless. Don’t believe Veeam can be a target, check out this string from Veeam’s own Community Forum.

The enterprise had to close their doors for five days over a busy holiday weekend at a cost of hundreds of thousands of dollars in lost revenue, and recovery costs.

How to Avoid this Recovery Failure Scenario

This recovery failure brought down the entire organization as the backup infrastructure was completely compromised. The only real way to recover from this is taking steps to avoid an infection entirely. A few steps every organization should take:

• Phishing simulation – You must continuously educate your employees that they and the business is constantly under attack. Many companies are turning to products to educate employees by conducting simulated phishing attacks and security awareness training. Consider a product such as Bullphish ID by ID Agent.
• Deploy Linux-based backup appliances – To avoid this and other Windows ransomware issues Unitrends backup and recovery appliances are delivered in hardened Linux.
• Utilize cloud storage – Get your backups way offsite and physically disconnected from your production environment with Unitrends Cloud.
• Gain the benefit of machine-learning based automated ransomware detection – Unitrends appliances can quickly and automatically identify ransomware activity as part of every backup. Upon detection email and dashboard alerts are sent immediately to administrators, and all suspected backups are flagged with icons to prevent recoveries using infected files. Admins can recover faster using easily identified uninfected backups.

Ransomware is just one of the causes of failed recoveries. To see a checklist of other common issues that can cause recoveries to fail check the link. Continue to follow our blog as we present other challenges to successful recoveries and learn how best to avoid being the victim.