High Availability Maintenance

Replace a Data Node

Limitations

• If the existing node is not healthy, you must use the --force flag. This will allow the command to be run but will not delete the TOS data.

• The new node cannot already be a node in the cluster.

• The primary data node cannot be replaced using this command. You must first elect a different data node for the role, using sudo tos cluster set-primary.

Prerequisites

• Prepare a new machine.

• Install TOS on the new machine

• If failover occurred:

  • Run sudo tos cluster node list. This will show you which node has failed.

  • If the failed node was the primary data node, select a new primary data node from one of the remaining data nodes by running sudo tos cluster node set-primary on the selected node.

Procedure

1. On the primary data node:

   [primary data node]# sudo tos cluster node replace <OLD_NODE> [--force] 

   sudo tos cluster node replace <OLD_NODE> [--force]

   Parameters

   Parameter	Description	Required/Optional
   <OLD NODE>	Name of the node you want to remove, as seen in the output of sudo tos cluster node list.	Required
   --force	Only use this flag when the node you want to remove is not in a healthy state.	Optional

   On completion, one or two new command strings will appear in the output. The second command string will appear only if the original node was in a healthy state, i.e.  -force was not used.

2. Copy/paste the first string and run it on the new node.

3. If a second string appeared in the output from the command that you ran on the primary data node, copy/paste this string and run it on the old node.

4. Verify that high availability has been restored by running the commands:

   • sudo tos cluster node list

   • sudo tos status

Disable High Availability

1. On the primary data node:

   [<ADMIN> ~]$ sudo tos cluster ha disable [--force]

   sudo tos cluster ha disable

   --force should be used only if the cluster status is not healthy - see sudo tos status.

   On completion replication of data to the additional data nodes will cease.

1. Verify that HA is no longer active by running sudo tos status on the primary data node.

2. Data nodes other than the primary data node will contribute very little to the running of the cluster. We recommend removing them using sudo tos cluster node remove.