Disaster Recovery

Overview

Tufin Orchestration Suite (TOS) disaster recovery (DR) can be implemented to create redundancy across sites. It is a manual failover process based on automatic backup/restore functionality. When the switchover between the two clusters is made, the latest backup is used to restore data to the new active cluster.

DR is meant for redundancy, it is not a high availability (HA) solution. HA must be deployed where all data nodes are located on the same site, therefore if you want both HA and site redundancy, you must deploy HA separately on each cluster.

Since it is highly likely that there will be a time difference between the last backup and the switchover, data loss is expected.

DR Architecture

For DR, you need two Tufin Orchestration Suite (TOS) clusters connected to the same external backup storage. Each cluster has it's own primary and syslog VIPs or domains and the VIPS / domains used by users and devices will always be those of the initial active cluster. DR is initiated on the cluster designated to be the active cluster; the second cluster then becomes standby by default.

Important: DR is not supported for TOS Aurora on cloud deployments (AWS/Azure/GCP).

Prerequisites

  • Confirm that all required ports are open, see Central Cluster Ports.

  • Two clusters with the same configuration:

    • If you configure HA on one cluster, you must configure HA also on the second cluster.

    • Both clusters should have the same modules (SecureTrack only or SecureTrack and SecureChange).

    • Both clusters should have the same hardware configuration.

  • Both clusters must be connected to the same backup storage bucket.

  • IP connectivity must be allowed between the two clusters; Network Address Translation (NAT) is not supported.

  • Each cluster must be setup with its own primary and syslog VIP addresses or domains.

Setup and Initialization

Initializing Disaster Recovery

Run tos dr init on the cluster that will initially be designated the active cluster.

Description

Initializes DR on a cluster and makes it the active cluster. The backup destination must be set to external storage before running. The command also sets the backup policy in the same way as sudo tos backup policy set.

Syntax

[<ADMIN> ~]$ sudo tos dr init [-n <NAME>][--rate <RATE>][--hour <HOUR>][--minute <MINUTE>]
sudo tos dr init -n <NAME>

Parameters

Parameter

Description

Required/Optional

Possible Values

-n or --name

Not used. For future use.

N/A

N/A

--rate

Backup frequency (in hours).

Optional

24 (default), 12, 8, 6

--hour

Hour when the first daily backup occurs. Format is HH.

Optional

Default: 00

--minute

Minute of the hour when the first daily backup occurs. Format is mm.

Optional

Default: 00

Example

$ sudo tos dr init 
[Jul 18 10:38:34]  INFO DR initialization finished successfully

Generating an Authentication Token

Run this command on the cluster that will be designated as the active cluster.

Description

Generates a unique token that is used to authenticate the connection between the two clusters.

Syntax

[<ADMIN> ~]$ sudo tos dr generate-token
sudo tos dr generate-token

Example

$ sudo tos dr generate-token
[Jul 18 10:38:45]  INFO Please save the token and use it when running the connect command from remote peer
Token: z1obGYNhdcb85rsDI7IrygGfMP5rHFq50iygPcEWxnE=

Connecting the Clusters

Run this command on the cluster that will be designated as the standby cluster.

Description

Connects the standby cluster with the active cluster. After the connection is complete, most of the services in the standby cluster will be shutdown.

Syntax

[<ADMIN> ~]$ sudo tos dr connect [-n <name>][-p <IP_ADDRESS>][-t <TOKEN>]
sudo tos dr connect [name][-p <IP_ADDRESS>][-t <TOKEN>]

Parameters

Parameter

Description

Required/Optional

Possible Values

-n or --name

Name of the Disaster Recover (DR) cluster.

Optional

Default: hostname of the primary node

This parameter is required if you changed the name when running sudo tos dr init.

-p or --peer-cluster

IP address of peer cluster primary node.

 Required

 

-t or --token

Token generated

Required

Value generated from tos dr generate-token

Example

$ sudo tos dr connect -p 192.168.32.23 -t Zbk7lwH4Qu7rINz8DvuwjQgJgpWjcsgtxKe3h90=
[Jul 18 10:39:07]  INFO Local cluster state is "StandBy"

Show the Status of the Clusters

Description

Display DR status for active and standby peers. The standby cluster can view the backups taken on the active cluster .

Syntax

[<ADMIN> ~]$ sudo tos dr status
sudo tos dr status

Example

$ sudo tos dr status 
DR configuration: 
  Backup policy rate: 24H
  Hour: 0:0h

Last valid backup: 2022-07-26 06:10:40 +0000 UTC

Local cluster status:
  Name: local
  State: Active, Ready
  Last Update: 2022-07-26 09:33:45 +0300 IDT

Peer cluster status:  
  Name: remote
  State: StandBy, Ready
  Last Update: 2022-07-26 13:53:22 +0300 IDT

Switch Clusters

Run this command on the standby cluster to switch the two clusters - the standby cluster becomes the active cluster.

During the switch process, if the active cluster is still running, it is notified that a switch is in progress and all the services of that cluster will be shut down. The data on the new active cluster will be automatically restored from the most recent backup. You can optionally specify a different backup to use.

After the switch, you must make the necessary DNS changes to ensure that requests sent to the VIPs of the previous active cluster are redirected to the new active cluster.

Description

Switches activity between peer clusters.

Syntax

[<ADMIN> ~]$ sudo tos dr switch [-b <BACKUP_NAME>]
sudo tos dr switch [-b <BACKUP_NAME>]

Parameters

Parameter

Description

Required/Optional

Possible Values

<BACKUP_NAME>

Backup to be used on switch.

Optional

Default: latest completed backup will be used

Example

$ sudo tos dr switch
[Jul 18 10:46:55]  INFO Switching current peer to Active
[Jul 18 10:46:55]  INFO Setting peer to "Switch" mode

Disable Disaster Recovery

Description

Disables DR on the cluster.

Syntax

[<ADMIN> ~]$ sudo tos dr disable
sudo tos dr disable