Upgrade TOS Aurora

If you monitor Stonesoft, Check Point or Fortinet devices, upgrade only to R23-1 PHF2.1.0 or later, due to a bug discovered in the upgrade process to earlier R23-1 releases.

Overview

This procedure is for upgrading any supported release to R23-1 and it is identical for all platforms and operating systems. For all other installation paths such as upgrading from TOS Classic to TOS Aurora and clean install, see the appropriate procedure in the menu. All clusters must be running the same TOS version.

Before starting, you should make a backup and export it outside of the cluster in case you need to roll back. After the upgrade completes successfully, you should make a new backup as previous backups made on one product version cannot be restored to another.

How Should I Upgrade My Deployment?

Worker Nodes

Only the primary data node needs to be upgraded. It will automatically upgrade TOS on all other worker and data nodes in the same cluster. The TOS CLI will be upgraded on the other nodes when you next run a TOS CLI command on them.

Remote Clusters

All clusters need to be running the same TOS version. Therefore, make sure to upgrade the primary data node in both the central cluster and remote clusters. Upgrade the central cluster first.

High Availability (HA)

If you are upgrading a high availability deployment, you are going to need to prepare the other data nodes before upgrading TOS. This will require logging into them separately in a different session.

Disaster Recovery (DR)

If you have disaster recovery, first upgrade the active deployment and then upgrade the standby deployment.

Prerequisites

TOS Compatibility and Upgrade Paths

Make sure your current version can be upgraded directly to this version of TOS Aurora - see TOS Aurora Upgrade Paths.
If you are running on a non-TufinOS operating system, make sure it is supported for this TOS release.
If you are running on the TufinOS operating system, make sure your current OS version will support the new version of TOS Aurora - see TufinOS Compatibility. Either way, we always recommend upgrading TufinOS to the latest version.

Port and Services

If your deployment incorporates remote clusters and you are upgrading from a release lower than R23-1, be aware that an additional port 9090 is now required for successful running of TOS - see remote collector ports.

Downloads

Download the TOS R23-1 PHF2.1.0 installation package from the Download Center.

The downloaded files are in .tgz format <FILENAME>.tgz.

Required Steps Before Starting

Run the command tos status. In the output, make sure system status is "OK", all nodes are "healthy" and under "Disk usage" /opt is not more than 70%. If any of these conditions are not met, the upgrade will fail.
Make sure you have at least 25 GB free on the primary data node in the /tmp directory.
If you monitor devices managed by a management device/domain that does not have a dedicated license because it inherits its license status from its monitored devices/domains e.g. FMC, FMG, Panorama, make sure all such monitored devices/domains are licensed or removed. Failure to do this will cause the management device/domain to be unlicensed after the upgrade.
If you are upgrading a remote collector cluster:
- Do not start the upgrade until the upgrade to the central cluster has completed.
- It must run it under the same release as the central cluster.
- Be aware that the remote cluster will be reconnected automatically to the central cluster.
Your license must be activated before starting the upgrade, otherwise the procedure will abort.

Select Admin > Licenses. The License window appears.

Activated:

Not Activated:

If the license is not activated, follow the instructions in Activate License.
Make a backup of the installation file that was used for your current TOS Aurora installation - /opt/tos/tos.tar - to a directory outside of /opt/tos This is necessary in case there is a need to roll back.
Transfer the run file to the primary data node to directory /opt/tufin/data.
If you use automated provisioning and you are upgrading from R21-3 or higher, make sure there are no queued provisioning tasks. You can check this using the waiting_tasks API.
Verify that the DNS server can resolve its own address using a reverse lookup
Run:
[<ADMIN> ~]$ sudo nslookup <DNS SERVER IP>
sudo nslookup <DNS SERVER IP>
If the name of the server is displayed, the DNS server can resolve its own address using a reverse lookup.

Example Output
```
[<ADMIN> ~]$ sudo nslookup  1.2.3.4
4.3.2.1.in-addr.arpa	name=EXAMPLE.company.com
```
If the name of the server is not displayed, set it using a reverse lookup entry at /etc/hosts.

Example Output where 1.2.3.4 4.3.2.1.in-addr-arpa was added.
```
[<ADMIN> ~]$ sudo cat /etc/hosts 
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost6 localhost6.localdomain6
1.2.3.4 admin.company.com admin
2.3.4.5 5.4.3.2.in-addr-arpa
```
See the Important Installation Information in the Release Notes

Upgrade Procedure

Read and understand Prerequisites before you start.

Log in to the primary data node using SSH as user tufin-admin or another user with sudo or root privileges.
Check your current version by running the following command:
```
[<ADMIN> ~]# tos version
```
tos version
Check that your cluster status is healthy.

If you are upgrading from TOS Aurora R21-2 or earlier:
1. Run the following command on the primary data node:
```
[<ADMIN> ~]# gravity status
```
  gravity status
  Example output below:
  
  #gravity status
  Cluster status: active
  Application: TOS, version 0.20.1-pga-final
  Join …oken: 188u988d48e2
  Last completed operation:
  * operation_install (3b2438ba-5b7e-4279-ae2c-ad1042a3549d)
  started: Thu Jan 30 10:58 UTC (5 hours ago)
  completed: Thu Jan 30 10:58 UTC (5 hours ago)
  Cluster endpoints:
  * Authentication gateway:
  - 10.100.14.102:32009
  * Cluster management URL:
  - https://10.100.10.102:32109
  Cluster nodes: tufin-orchestration-suite
  Masters:
  * tufinos (10.100.14.102, tos)
  Status: healthy
2. In the output from the command, check that:
  - Cluster status is Active
  - Status is healthy
  If these conditions are true, proceed with the upgrade, otherwise contact Tufin support.

If you are upgrading from TOS Aurora R21-3 or later,

Run the following command on the primary data node:

[<ADMIN> ~]# systemctl status k3s

systemctl status k3s

Example Output

[primary data node]# systemctl status k3s
[root@TufinOS ~]# systemctl status k3s
Redirecting to /bin/systemctl status k3s.service
● k3s.service - Aurora Kubernetes
   Loaded: loaded (/etc/systemd/system/k3s.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2021-08-24 17:14:38 IDT; 1 day 18h ago
     Docs: https://k3s.io
  Process: 1241 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
  Process: 1226 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
 Main PID: 1250 (k3s-server)
    Tasks: 1042
   Memory: 2.3G

In the output under the line k3s.service - Aurora Kubernetes, check that two lines appear - Loaded... and Active... similar to the example above. If they appear, continue with the next step, otherwise contact Tufin Support for assistance.

Make sure all users are logged out from the browser.
Make a one-time backup.
After your backup has completed, continue by running commands:
```
[<ADMIN> ~]# tmux new-session -s upgrade
```
tmux new-session -s upgrade
Extract the TOS run file from its archive.
```
[<ADMIN> ~]$ tar -xvzf tos-xxxx-xxxxxxxx-final-xxxx.run.tgz
```
tar -xvzf tos-xxxx-xxxxxxxx-final-xxxx.run.tgz
Run the following commands:
```
[<ADMIN> ~]# cd /opt/tufin/data/
```
cd /opt/tufin/data/
```
[<ADMIN> ~]# sh <rls>.run
```
sh <rls>.run
where

<rls> is the name of the file extracted in prerequisites.
If upgrading from R22-2 PGA.0.0 or R22-1 PHF3.x and your syslog VIP is set up with transport TCP, import the syslog certificate.
```
[<ADMIN> ~]# sudo tos certificate import --type syslog --ca=<CA-PATH> --cert=<CERT-PATH> --key=<KEY-PATH> --skip-cli-validation
```
sudo tos certificate import --type syslog --ca=<CA-PATH> --cert=<CERT-PATH> --key=<KEY-PATH> --skip-cli-validation
Make sure your TCP syslogs are sent over TLS.
Upgrade TOS:
```
[<ADMIN> ~]# tos update /opt/tos/tos.tar
```
tos update /opt/tos/tos.tar
where

<rls> is the name of the file extracted in prerequisites.
Reconnect worker nodes (only if upgrading from R21-2 or earlier).

If you are upgrading from release R21-3 or later, skip this step.

If you have worker nodes, they will be disconnected. The upgrade procedure will detect the nodes and ask you if you want to reconnect them now or later. Select one of the following:
- Stop the upgrade and add the nodes now.
  
  After adding the nodes, run the tos update command again.
```
[<ADMIN> ~]# tos update /opt/tos/tos.tar
```
  tos update /opt/tos/tos.tar
  If all nodes have been added, the upgrade will continue from where it left off, otherwise the upgrade procedure will again ask you if you want to reconnect them now or later.
- Continue with the upgrade without adding nodes.
  
  The upgrade procedure will continue. You can add the worker nodes if and when required.
On completion, a confirmation message appears.
Verify.

Check again the tos version as described in upgrade procedure step 2 above. Make sure that the version displayed is the one to which you intended to update.
```
[<ADMIN> ~]# tos version
```
tos version
Check again the cluster status. This time there is only one option - that for R21-3 and later - as described in upgrade procedure step 3 above.
```
[<ADMIN> ~]# systemctl status k3s
```
systemctl status k3s
Make a new backup.

Before allowing users to start work, make a new one-time backup. This is necessary because the data schemas have been modified and any backups made before the upgrade can no longer be restored to the new version of the product. See Backup Procedure.
Update Tufin extensions (formerly Tufin Marketplace apps) and Tufin Professional Services solutions (only if upgrading from R21-2 or earlier)

If you had apps installed and you did not upload the latest install files to the primary data node as explained in Prerequisites, they must be upgraded manually to work with the new TOS release.
- Download the latest versions at extensions.tufin.com or marketplace.tufin.com/my-apps/, or get the upgrade packages directly from your PS representative. and place them on the primary data node.
- For files of type .run, reinstall as described in the appropriate app KC.
- For files of type .tar, run the following command on the primary data node
```
[<ADMIN> ~]# tos apps upgrade <path.tar>
```
  tos apps upgrade <path.tar>
  where <path.tar> is the full path of the file
If you have FortiManager devices in SecureTrack, add a SAN signed certificate to each device.
If your deployment incorporates remote clusters and you have upgraded from a release lower than R23-1, make sure port 9090 is open - see remote collector ports.
Make sure users clear their browser cache.