Protect File Servers with Azure for Rapid Recovery

Cyber-attacks have become all too common in recent year, as criminals have found them to be low risk / high reward endeavors.    Mitigating the risk and disruption from these attacks is a team effort across IT security, IT infrastructure, Applications, and front-line support teams. 

Even with thoughtful security policies and tools in place, organizations need the capability to rapidly recover from future CryptoLocker / WannaCry style ransomware attacks which have yet to be developed or discovered.

Azure File Sync with Azure Backup provides a non-disruptive means to synchronize the contents of Windows-based file servers to Azure, and rapidly restore access without the need for complicated tape recovery, or the complete replication of data to on premises file servers – in our example access was restored to a file server in less than one hour.

If you want to understand the value of Azure File Sync with Azure backup to help rapidly recover from a ransomware attack (or already have Azure File Sync deployed) jump ahead to the section titled Recovering an Encrypted File Server with Azure Backup / Azure File Sync.

A Typical Ransomware Event

Many cyber-attacks in recent years have common themes and elements (not that any cyber attack should be consider ordinary), which include the exploitation of credentials to access files and encrypt them to secure a ransom.  Often the automation will attempt to limit access to backups (by attempting to clear Volume Shadow Copy snapshots), interfere with backup processing.  Attack code can linger inside a network for weeks or months, anticipating the expiration of backups to create more pressure on organizations to pay up for decryption of their data.

Typical File Recovery Processes Often Fail

For organizations unwilling to bend to the will of these criminal attackers (there are no guarantees a ransom payment equals recovery), file servers must be rebuilt and secured before data can be recovered.  When data recovery begins using typical, long term retention of backup tapes (if they exist) the recovery process can take weeks or months of recalling offsite tapes, loading them into tape drives, cataloging, restoring, and verifying their contents.

Often inadequate backup frequency / long term rotation, bad tapes, missing tapes, poor storage, misaligned tape drive heads, or any number of challenges can cause recovery efforts to fail, leaving data loss a common result after weeks of sleepless nights for administrators and unproductive employees (not to mention disgruntled customers or constituents!).

Why Azure File Sync with Azure Backup is Different

Azure File Sync Azure provides the capability to synchronize file hierarchies between Window Server and Azure File Shares, including the ability to tier/recall requested files.  This allows existing Windows Server to function much like a storage gateway to a large Azure File share. 

Existing file servers can use Azure File Sync to copy files to seamlessly Azure, and ultimately use Azure Backup to create periodic snapshots of the files which can be rapidly restored.

Should a file server be destroyed (or encrypted through ransomware), directory structures, NTFS permissions, and file contents can be restored to the Azure File Share within minutes from a snapshot for resync to the impacted server.  The File server can be replaced with a lower capacity VM (to help meet DR capacity limits) or the infected hardware can be wiped (sanitized / reloaded /  patched / secured) and the file hierarchy reload using Azure File Sync to provide file system “dial tone” – users will be able to see directories and files.  As files are accessed (become “hot”), they will be downloaded to the file server and cached on the local server as needed.  If all files are required locally, they can be downloaded using a simple set of commands or options in the Azure console.

Azure File Sync with Azure Backup Overview

Azure File Sync using an agent that can be easily installed on current versions of Windows Server (2012 R2 / 2016 / 2019) and non-disruptively replicates directory structures and files from NTFS volumes to Azure File Shares.  These Azure File Shares support backup via snapshots using Azure Backup… up to 200 separate disk snapshots that can be used to rapidly restore each backed up Azure File Share.

Installing and Configure Azure File Sync

For organizations that already have access to Azure, setting up Azure File Sync for an existing file server or file server cluster is a straightforward, well documented process consisting of a few high-level steps:

1. Create an Azure File Share in Azure
2. Create / configure a Storage Sync Service in Azure
3. Install and configure the Azure File Sync agent on the local file server
4. Associate the local file server to the Sync Service and configure settings
5. Creating Recovery Services Vault / Configuring Azure Backup

With proper access to Azure and the local file server, the entire manual configuration process can be completed in about an hour as well as automated with scripting for large scale deployments.

The following steps are intended to show the process to setup protection for a single on premises file server, and later a sample process to restore file access to a rebuilt server.

Creating Azure File Share

Creating a Standard Azure File Share (now referred to as “Transaction Optimize”), is well documented in the Azure Documentation here:  https://docs.microsoft.com/en-us/azure/storage/files/storage-how-to-create-file-share.  For our purposes the Azure File Share will merely act as a repository for file data and will not be accesses as a file share directly. 

This file repository can be up to 100TB per folder hierarchy, with the ability to assign a lower quota if desired.  For details on how to create large Azure File Shares, you can go here:  https://docs.microsoft.com/en-us/azure/storage/files/storage-files-how-to-create-large-file-share.

File servers with more than 100TB of file data on a single volume might necessitate the creation of multiple Azure File Shares to contain all the data (synchronizing separate directory trees to different Azure File Shares).

Once you have completed creating your share, it will be listed in the Azure portal, like those below:

Creating (File) Storage Sync Service

Details of the process create a Storage Sync Service and connect it to an Azure File Share can found here:  https://docs.microsoft.com/en-us/azure/storage/files/storage-sync-files-deployment-guide?tabs=azure-portal#deploy-the-storage-sync-service

To deploy a Storage Sync Service, go to the Azure portal, click Create a resource and then search for Azure File Sync. In the search results, select Azure File Sync, and then select Create to open the Deploy Storage Sync tab.

 

Create Sync Group and  Cloud Endpoint

A sync group defines the sync topology for a set of files, and points to the Azure File Share previously created. 

Creating the Sync Group and Cloud Endpoint is easy, and documented here:  https://docs.microsoft.com/en-us/azure/storage/files/storage-sync-files-deployment-guide?tabs=azure-portal#create-a-sync-group-and-a-cloud-endpoint.

To create a sync group, in the Azure portal, go to your Storage Sync Service, and then select + Sync group:

In the pane that opens, enter the following information to create a sync group with a cloud endpoint:

• Sync group name: The name of the sync group to be created. This name must be unique within the Storage Sync Service, but can be any name that is logical for you.
• Subscription: The subscription where you deployed the Storage Sync Service in Deploy the Storage Sync Service.
• Storage account: If you choose Select storage account, another pane appears in which you can select the storage account that has the Azure file share that you want to sync with.
• Azure file share: The name of the Azure file share with which you want to sync (created earlier).

Installing / Configuring the Azure File Sync Agent on a Local File Server

Moving out of the Azure portal to the file server itself, there are a few prerequisites and considerations for the installation of the File Sync Agent on Windows Server-based file servers, which are documented here:  https://docs.microsoft.com/en-us/azure/storage/files/storage-sync-files-planning.

The actual process of downloading, installing, and configuring the agent to connect to your subscription and Sync Service is detailed here:  https://docs.microsoft.com/en-us/azure/storage/files/storage-sync-files-deployment-guide?tabs=azure-portal#prepare-windows-server-to-use-with-azure-file-sync

The agent can be quickly downloaded from here: https://www.microsoft.com/en-us/download/details.aspx?id=57159, and many Azure savvy server administrators will navigate the process of connecting a file server to their previously configured subscription / Storage Sync Service quickly.  Once the initial installation is completed, the server registration process will begin, which will require the administrator to use their Azure administrative credentials to locate the Azure resources, which should be available on the following screen:

Associating (File) Storage Server to Sync Service (Creating a Server Endpoint)

The previous step connected the file server to Azure, now we need to define the set of files we want to protect by connecting Azure to a path on an NTFS-bad volume.

To add a server endpoint, go to the newly created sync group and then select Add server endpoint.

In the Add server endpoint pane, enter the following information to create a server endpoint:

• Registered server: The name of the server or cluster where you want to create the server endpoint.
• Path: The Windows Server path to be synced as part of the sync group.
• Cloud Tiering: A switch to enable or disable cloud tiering. With cloud tiering, infrequently used or accessed files can be tiered to Azure Files.
• Volume Free Space: The amount of free space to reserve on the volume on which the server endpoint is located. For example, if volume free space is set to 50% on a volume that has a single server endpoint, roughly half the amount of data is tiered to Azure Files. Regardless of whether cloud tiering is enabled, your Azure file share always has a complete copy of the data in the sync group.

To add the server endpoint, select Create.

Your files will now be kept in sync across your Azure file share and Windows Server.

It is important to mention that the speed of replication to Azure is dependent upon Internet bandwidth as well as spare processing capacity on the file server.  Completing replication of large file servers can take several days, and will be impacted by file change churn (changed files will need to be re-replicated).  For estimation purposes it is expected that a single file server with adequate resources can replicate files to an Azure File Share at a rate of about 500GB / day (1TB ever 2 days).

Creating Recovery Services Vault / Configuring Azure Backup

Configuring backup for the Azure File Share repository takes only a few more minutes.

After a few short steps your on-premises file server will be protected.  Those steps include:

1. Creating a Recovery Services Vault
2. Configuring the backup of the Azure File Share
3. Setting up your backup schedule

A walk through of this process can be found here:  https://docs.microsoft.com/en-us/azure/backup/backup-azure-files.

Note that Azure File Sync supports up to 200 separate snapshots – taking only daily backups would means that approximately six months of rapidly restorable snapshots would be available for recovery purposes in addition to existing heritage backup / archiving process that might be in use in your data center.

Recovering an Encrypted File Server with Azure Backup / Azure File Sync

After a file server has been compromised, you will want to leverage prevailing industry best practices to secure, sanitize, and rebuild your environment.  Once your core infrastructure has been secured and cleaned (including Active Directory, DNS, and other foundation services), recovering data and restoring end user access will likely be a priority. 

Whether you choose to simply wipe out encrypted files on impacted servers or build new servers / VMs Azure File Sync with Azure Backup can be used to rapidly provide access to file server directories and data.  Steps to restore access to file-based data will generally following those outlined below:

1. Purge / clean file server
2. Delete files on Azure File Share – DO NOT DELETE Azure File Share!
3. Restore desired backup from Recovery Services Vault to Azure File Share
4. Configure Azure File Sync with Tiering enabled as per previous steps to sync folder hierarchy

Depending on your actual recovery process, the above steps can restore file access can restore user access in a matter of minutes!

Purge / Clean File Server – Duration Variable

Wiping out the “infection” can mean creating an entirely new file server, or simply wiping out share directories and files (depending on the extent of the issue).  In any case, eradicating the offending code and its impact will likely need to be handled completely before a file recovery, and is not addressed in this discussion.  Consult with qualified security resources and guidance to ensure your environment is ready for file recovery.

Delete Files on Azure File Share – Duration 10 minutes

While it may not be necessary to delete all the files from the Azure File Share (as they may not all be encrypted / infected) it may be necessary, and Azure Backup has the ability to restore the entire share (assuming you have properly configured backups!). 

In any case DO NOT DELETE THE AZURE FILE SHARE ITSELF – the Azure Backup data is linked to the share and will be lost.  Deleting individual files / directories or wiping the entire server may be thoughtfully pursued actions. The simplest way to delete the files can be from the infected server (if it is online), and the deletes will be propagated to the Azure File Share repository.  Before Deletion:

After deletion:

With File Sync enabled, these deletions will propagate to the Azure File Share (if you use this method, verify that the File Share is empty).

You can also delete files from within the Azure console, but the web interface does not support deleting sub-directories which contain files / sub-directories – making this laborious.   Connecting to the share (mapping a drive) from another host – even a VM in Azure will allow for simple / rapid deletion of the directory hierarchy.  Select “Connect” from the Azure File page in your storage account:

“Connect” will provide instructions on how to connect to that specific share from a Windows (or other) host:

Once connected, you should be able to delete all files on the Azure File share, with the exception of the system share information:

Restore Desired Backup to Azure File Share – Duration 5 minutes

The restore process is well documented here:  https://docs.microsoft.com/en-us/azure/backup/backup-azure-files#restore-from-backup-of-azure-file-share. Navigate to the Recovery Services Vault used to backup the Azure File Share, and select “Restore Share”:

Select on the desired last backup (before the introduction of malware into your environment) and click OK. Select the restore location – in this case you will overwrite contents in original location – and select OK:

Then select Restore. You can check the status of the restore job within the Recover Services Vault by navigating to Monitoring -> Backup Jobs:

The total recovery time will depend on the volume of data, but as it is already on disk in Azure, it will significantly faster and more reliable than manual, tape-based restore – often completing in a matter of minutes for smaller shares.

The Azure File Share is now repopulated with “clean” files:

Configure File Sync on Rebuilt Server – Variable (5 minutes or more)

Resynchronization to online / configured file servers will happen once the Azure File Sync agent is loaded and configured.  While not discussed in depth earlier, it is important to consider the value and impact of enabling cloud tiering during a recovery scenario.  Cloud tiering in Azure File Sync allows the file system hierarchy to sync down to file servers along with file “stubs” without all the associated data – reducing the time to file system “dial tone”.  All the files will appear to be on the local server when users request them but will only be recalled when needed – reducing significantly bandwidth necessary for a full, local restore.

In this example, a laptop with a small SSD was pre-installed with Windows Server 2019 and used to recover a much larger file server, with Cloud Tiering enabled.  Once the File Sync agent was configured, the entire file hierarchy was available on the system in minutes – note the size on disk used:

At this point, shares can be recreated and pointed at the appropriate directories, providing access to uninfected files.

Without tiering enabled, all data will be pushed down to the server, delaying access and utilizing bandwidth.

NOTE THAT the share size and number of files will impact the overall recovery time and sync of files (handles) to the target file server. Really big shares (with millions of files) could take quite some time for all files to sync down and be represented.

Summary

Pre-integrating Azure File Sync with Azure Backup in your on-premises file server infrastructure can significantly reduce recovery time from ransomware attacks.

The recovery example above restored access to a file server’s data in a matter of minutes without tying up heritage on-premises backup and recovery infrastructure – freeing it to be used for other recovery/restore efforts in parallel.

Next Steps

Deploy Azure File Sync with Azure Back and test it for yourself, developing and augmenting your existing recovery and ransomware mitigation strategies.