Backup and Restore

Overview

TOS Aurora uses a Point in Time Recovery (PITR) backup methodology, which substantially reduces the time and disk space required to create a complete daily backup and allows you to restore your data to a specified point in time in a single step. You can define a backup policy, in which you specify the frequency of automated backups that will run without the need for administrator maintenance. This is the recommended method of handling TOS Aurora backups. In addition, on-demand backups can be made, unconnected to your policy.

All backup and restore functions apply equally to central clusters and remote collectors. The backup will always apply to the deployment type as specified in the modules parameter when the install command was run. A backup taken from one kind of cluster must be restored to the same type. For example, a central cluster must only be restored from a central cluster backup and a remote collector must only be restored from a remote collector backup. Otherwise; data may be lost or corrupted.

Zero Touch Automated Backups

Once you create a policy, TOS Aurora will take care of everything else. This mechanism does not have the concepts of full backup (e.g. weekly, monthly) and sequential backup (e.g. daily) because every backup is an image, made up of pointers to saved 'blocks' of your original and modified data. The image file is effectively a full backup and running the restore commands on it will rebuild your database to the appropriate point in time from the saved blocks .

Reduced Disk Space

Backups, which use this concept, duplicate all your original data in a new location one time only. After the initial backup, only changed blocks are added to a steadily growing pool, and pointers in the backup file refer to these blocks. Subsequent backups have no need to refer to blocks that have since been replaced, while prior backups refer only to the blocks needed to reconstruct the data in a restore operation, and not those that changed subsequent to the backup execution.

Cleanup

Once initiated, the backup policy runs a cleanup function in the background, in which blocks no longer referred to by any backups are deleted. This situation will occur when backup files reach their expiry time and a block referred to only by this backup has since been replaced with another.

VM Snapshots

We do not support using VM snapshots as a backup mechanism for TOS data and using this method may lead to undesired results. We strongly recommend using the TOS CLI backup/restore commands only.

What Gets Backed up

Included: All TOS data, TOS configurations, data for Tufin extensions (formerly Tufin Marketplace apps), certificates that you have imported.

Not included: TOS software, backup policy, Tufin auto-generated certificates and certificates for communicating between central and remote collector clusters.

Following restore, remote collector clusters must be reconnected manually to the central cluster, as explained in Remote Collectors . This will automatically recreate the certificates required.

Backup File Name and Location

By default, the name and location of backup files are fixed, with all backups being placed in the TOS backup directory /opt/tufin/data/volumes/minio/velerok8s/backups on the primary data node.

  • On-demand backups are named automatically in the format DD-Month-HH-MM. For example, "5-november-2019-19-32" for November 5th 2019 at 19:32

  • Scheduled backups initiated from your backup policy are named in the format tufin-scheduler-yyyymmddhhmmss. For example, "tufin-scheduler-20191110000045" for November 10th 2019 at 00:00:45 (45 seconds past midnight).

All backup files are created with a time-to-live parameter. When the time to live expires, the backup will be destroyed automatically and it's disk space released. If no value is specified, the default value of 720 hours will be used. Backup files cannot be set to never expire, but the maximum value is 2,147,483,647 (245 thousand years), which should be enough for most users!

There are two approaches to store backups to a safe environment - Local (default) and External. Local backups are placed in the TOS backup directory on the primary data node. External backups are stored outside of the cluster. To change the default location to external, use sudo tos backup storage set and set the location to external . This command changes the location of the backup file according to the selected provider. To change the backup storage location back to the primary data node, you can use the same command - sudo tos backup storage set - and set the location to local.

Local Backup Storage

Local backups reside only on the primary data node, regardless of the cluster configuration. Disaster recovery planning dictates that you should separate these backups from the cluster. We recommend that you export your backups, which saves all accumulated backups to a single file. Then, you can copy this file to a safe, remote location.

External Backup Storage

External backups reside in a location outside of the cluster, according to the selected provider. TOS Aurora supports these external backup providers:

  • AWS

  • Azure

  • GCP

  • Minio - You can also deploy Minio on prem and use it externally. See Minio official documentation

  • Network attached storage (NAS) using the network file system (NFS) protocol. For PGA, if you are using NFS your backup server needs to be running NFS 4. From PHF1.0.0 and later, if you are running NFS 3 on your backup server it will not work because of a security vulnerability. If you want to ignore the security vulnerability to enable NFS 3, you need to run the following commands on all TOS servers that are using TufinOS 4.20 and later:systemctl unmask rpcbind.socket rpcbind.servicesystemctl unmask rpcbind.socket rpcbind.service systemctl start rpcbind.socket rpcbind.servicesystemctl start rpcbind.socket rpcbind.service systemctl enable rpcbind.socket rpcbind.servicesystemctl enable rpcbind.socket rpcbind.service  

Read/write permissions must be granted to the external backup location.

The external provider configuration details can be provided to TOS either through a configuration file, or inputted manually into the CLI when setting the external storage. See Provider Details, for the full list of fields.

When you change the backup storage to external, all new backups will be placed in the specified location. If you have previously run the backup command with the backup location set to local, the existing backups will remain on the primary data node.

You must confirm that there is sufficient network throughput to the external backup storage bucket; otherwise, backup and restore will take a long time to complete.

Scheduled Backup Procedure

We recommend the following best practice for backing up regularly:

  1. Decide on your backup location: local to create the backup on your cluster or external to backup directly to a remote location. Use command - sudo tos backup storage set.

    If the backup location is local: When the /opt partition becomes 90% full, TOS will stop making automatic backups and core services will be stopped. We recommend configuring TOS monitoring to send notifications if too much storage is being consumed.

  2. Create a backup policy using sudo tos backup policy set to get TOS to automatically backup your data regularly to the backup location.

  3. If the backup location is local, we recommend that you export your backups regularly to a remote location.

    1. Run sudo tos backup export to save all your backups from the TOS backup directory as a single .gzip file.

    2. Transfer the exported .gzip file to a remote location.

    3. Make sure you have the location of your backups safely documented and accessible, including credentials needed to access them, for recovery when needed.

  4. To restore your data, see Restore Procedure.

One-Time Backup Procedure

We recommend the following procedure for one-time, one-off backups.

  1. If you have not set a backup location, use command - sudo tos backup storage set: local to create the backup on your cluster or external to backup directly to a remote location.

  2. Create the backup using sudo tos backup create.

  3. Monitor the backup status using sudo tos backup status, which shows the status of backups in progress and sudo tos backup list, which lists all completed backups. Wait until completion before continuing.

  4. When sudo tos backup storage set location is local:

    1. Run sudo tos backup export regularly to save all your backups from the TOS backup directory as a single .gzip file. If there are other backups present, they will be included as well.

    2. Transfer the exported .gzip file to a remote location after creation.

    3. Make sure you have the location of your backups safely documented and accessible, including credentials needed to access them, for recovery when needed.

  5. To restore your data, see Restore Procedure.

Restore Procedure

This procedure is for restoring data from either one-time or scheduled TOS Aurora backups.

Backups can only be restored to a TOS deployment with the same release and version from which the backup was made. For example, a backup of R24-2 PHF1.1.0 can only be restored to an R24-2 PHF1.1.0 deployment.

Important. After you restore TOS to a different or changed production environment, such as a new appliance or VM, or an in-place machine following an operating system upgrade, you must upload the license file. If you are restoring to a lab environment, there is no need to upload the license file. TOS will be fully functional and in 'lab mode' . TOS will then automatically shut down after 30 days.

If the cluster was backed up with a different primary VIP, after the restore, change the primary VIP using tos cluster primary-vip set.

Backup Restore Commands

If your system runs on TufinOS, all TOS commands must be run using sudo, because user root is not available on this operating system. Before running any of these commands, switch the current directory to /opt.

[<ADMIN> ~]# cd /opt
cd /opt

All commands can be run with any of the following optional parameters:

  • --help - provides explanation of the command and its parameters without executing
  • --debug - executes the command and displays system information