UniSuper Google Cloud Outage Reveals Need for Keen Backup and DR Planning
Earlier this month, Google Cloud disclosed that it accidentally deleted the account of its customer UniSuper, a $125 billion Australian pension fund. According to a joint statement from Google Cloud and UniSuper, the deletion occurred due to an “inadvertent misconfiguration.” As a result, more than 600,000 pension fund users were unable to access their accounts for over a week.
UniSuper had geographical redundancy built into its Google Cloud environment, according to the statement. However, the misconfiguration caused fund data in both locations to be deleted.
The fund was able restore data and services via an unnamed third-party backup provider. However, it wasn’t easy. Restoring UniSuper’s Google Cloud instance required “an incredible amount of focus, effort, and partnership between our teams to enable an extensive recovery of all the core systems,” according to the statement.
Key takeaways:
- Cloud infrastructure as a service (IaaS) does not eliminate the need for backup.
- Geo-redundancy does not eliminate the need for backup.
- There’s a reason public cloud providers like Google and Amazon recommend third-party data protection in their user agreements. Don’t put all your eggs in one basket.
- As IaaS adoption continues to increase, many organizations will require IaaS-specific backup and disaster recovery planning expertise.
So, what happened?
According to the statement, “the disruption arose from an unprecedented sequence of events whereby an inadvertent misconfiguration during provisioning of UniSuper’s Private Cloud services ultimately resulted in the deletion of UniSuper’s Private Cloud subscription.”
As noted above, UniSuper had data redundancy in two geographies, but because the fund’s subscription was terminated, data was deleted across both. The joint statement indicated that this was an isolated occurrence and Google Cloud has taken measures to ensure it can’t happen again.
3-2-1 backup in IaaS
The 3-2-1 data backup strategy is a well-known and reliable method to protect against data loss. The principle is straightforward: maintain three copies of your data, store two copies on different types of media, and keep one in a geographically disparate location for disaster recovery.
The concept has been around since the days of tape backups. Back then, it meant production data, onsite disk or tape backup, and offsite tape for disaster recovery. In an Infrastructure as a Service (IaaS) environment, the 3-2-1 backup strategy maintains these core principles.
However, the strategy can be implemented in different ways, and the approach taken can have a significant impact on your ability to meet recovery time objectives (RTO). This is evident in the UniSuper situation:
According to the statement, UniSuper had three copies of data:
- The live data within UniSuper’s Google Cloud environment.
- The secondary redundant copy also in Google Cloud.
- The third-party backup that they recovered from.
It also had two different “media” (in this case providers: Google Cloud and the third-party backup) and one that is “offsite” (the third-party backup as it was outside of Google Cloud).
For many organizations, this might be an adequate solution; especially if the backup provider offers business continuity capabilities that enable fast recovery of mission-critical apps and services. So, what went wrong?
Well, restoring a large data set to a complex operating environment (virtual servers, cloud storage, databases, etc) can be very time consuming. So, organizations with extremely limited tolerance for application downtime (like say, a super fund with $125bn under management) most likely require a solution that provides both redundancy and availability.
Some organizations accomplish this by replicating workloads to a second IaaS provider. In other words, if the primary data and applications are in Google Cloud, the secondary (failover) copy would reside in AWS or Microsoft Azure. Third-party backup would remain to provide fast file and folder restores, ransomware resilience, compliance, and so on.
The devil is in the details.
DR Expertise Needed
Remember, this is a fund with hundreds of billions of dollars under management. If UniSuper could benefit from additional DR expertise, the average SMB could too. That’s why disaster recovery as a service (DRaaS), is a profitable offering for so many managed services providers (MSPs).
We believe that DRaaS represents a huge opportunity for Cove partners. That’s why we added and continue to update Cove Continuity, our virtualization-based business continuity capabilities. Most recently, we added Standby Image for VMware ESXi, a more efficient way for MSPs to deliver DRaaS to VMware users.
We also developed a free hands-on DR Masterclass to help MSPs on their journey to delivering DRaaS with Cove. If you are interested in attending a DR masterclass, click here to register.
Andrew Burton is Product Marketing Manager for Cove Data Protection at N‑able
If you are interested in learning more about Cove’s approach to cyber resilience, please don’t hesitate to schedule a demo.
To FIND OUT MORE about Cove Data Protection visit www.n-able.com/products/cove-data-protection Or simply start a FREE TRIAL at www.n-able.com/products/cove-data-protection/trial
© N‑able Solutions ULC and N‑able Technologies Ltd. All rights reserved.
This document is provided for informational purposes only and should not be relied upon as legal advice. N‑able makes no warranty, express or implied, or assumes any legal liability or responsibility for the accuracy, completeness, or usefulness of any information contained herein.
The N-ABLE, N-CENTRAL, and other N‑able trademarks and logos are the exclusive property of N‑able Solutions ULC and N‑able Technologies Ltd. and may be common law marks, are registered, or are pending registration with the U.S. Patent and Trademark Office and with other countries. All other trademarks mentioned herein are used for identification purposes only and are trademarks (and may be registered trademarks) of their respective companies.