Update BIOS and iDRAC Firmware

Overview

This procedure is relevant only for T-1220 and T-820 appliances running TufinOS 4.70. If you do not have that version, update TufinOS first. You need to update both worker nodes and data nodes. Worker nodes should be updated first.

Downloads

Download two update files from the Dell website. Use only these files and ignore any messages about them not being the latest versions.

• iDRAC 7.20.60.50

• BIOS: 1.18.1

Worker nodes

1. Verify that the appliance version is T-1220 or T-820.

   Run the command:

   dmidecode -s chassis-version

   dmidecode -s chassis-version
2. Update the iDRAC and BIOS drivers.

   Follow the official Dell instructions for updating iDRAC9. The video on the page explains how to install both update files in one action.

Data node (non-HA)

1. Verify that the appliance version is T-1220 or T-820.

   Run the command:

   dmidecode -s chassis-version

   dmidecode -s chassis-version
2. Backup your TOS data. This needs to be performed for all primary data nodes - the central cluster and remote collector clusters.

   If you are going to perform this procedure over multiple maintenance periods, create a new backup each time.

   1. Create the backup using tos backup create:

   [<ADMIN> ~]$ sudo tos backup create

   sudo tos backup create

   Example output:

   [<ADMIN> ~]$ sudo tos backup create
   [Aug 23 16:18:42]  INFO Running backup
   Backup status can be monitored with "tos backup status"

   2. You can check the backup creation status using tos backup status, which shows the status of backups in progress. Wait until completion before continuing.

   [<ADMIN> ~]$ sudo tos backup status

   sudo tos backup status

   Example output:

   [<ADMIN> ~]$ sudo tos backup status
   Found active backup "23-august-2021-16-18"

   3. Run the following command to display the list of backups saved on the node:

      [<ADMIN> ~]$ sudo tos backup list

      sudo tos backup list

   Example output:

   [<ADMIN> ~]$ sudo tos backup list
   ["23-august-2021-16-18"]
   Started: "2021-08-23 13:18:43 +0000 UTC"
   Completed: "N/A"
   Modules: "ST, SC"
   HA mode: "false"
   TOS release: "21.2 (PGA.0.0) Final"
   TOS build: "21.2.2100-210722164631509"
   Expiration Date: "2021-09-22 13:18:43 +0000 UTC"
   Status: "Completed"

   4. Check that your backup file appears in the list, and that the status is "Completed".

   5. If your backup files are saved locally:

      1. Run sudo tos backup export to save your backup file from a TOS backup directory as a single .gzip file. If there are other backups present, they will be included as well.

      2. Transfer the exported .gzip file to a safe, remote location.

         Make sure you have the location of your backups safely documented and accessible, including credentials needed to access them, for recovery when needed.

      After the backup is exported, we recommend verifying that the file contents can be viewed by running the following command:

      [Target location]$ tar tzvf <filename>

      tar tzvf <file name>

3. [<ADMIN> ~]$ sudo tos stop 

   sudo tos stop

   Parameters

   Parameter	Description	Required/Optional
   [-y] or --confirm	Suppresses warning messages and confirmation prompts.	Optional
   --force	If this parameter is included, no warning messages appear before files are deleted.	Optional
   --app-only (From R25-1 PHF1.0.0)	Stops application services only.	Optional

4. Update the iDRAC and BIOS drivers.

   Follow the official Dell instructions for updating iDRAC9. The video on the page explains how to install both update files in one action.

5. Start TOS

   [<ADMIN> ~]$ sudo tos start

   sudo tos start

Data nodes (HA)

BIOS and iDRAC need to be updated on each data node in the cluster, without stopping TOS. To prevent data corruption, you will first drain each node before updating BIOS and iDRAC.

1. Verify that the appliance version is T-1220 or T-820.

   Run the command:

   dmidecode -s chassis-version

   dmidecode -s chassis-version
2. Backup your TOS data. This needs to be performed for all primary data nodes - the central cluster and remote collector clusters.

   If you are going to perform this procedure over multiple maintenance periods, create a new backup each time.

   1. Create the backup using tos backup create:

   [<ADMIN> ~]$ sudo tos backup create

   sudo tos backup create

   Example output:

   [<ADMIN> ~]$ sudo tos backup create
   [Aug 23 16:18:42]  INFO Running backup
   Backup status can be monitored with "tos backup status"

   2. You can check the backup creation status using tos backup status, which shows the status of backups in progress. Wait until completion before continuing.

   [<ADMIN> ~]$ sudo tos backup status

   sudo tos backup status

   Example output:

   [<ADMIN> ~]$ sudo tos backup status
   Found active backup "23-august-2021-16-18"

   3. Run the following command to display the list of backups saved on the node:

      [<ADMIN> ~]$ sudo tos backup list

      sudo tos backup list

   Example output:

   [<ADMIN> ~]$ sudo tos backup list
   ["23-august-2021-16-18"]
   Started: "2021-08-23 13:18:43 +0000 UTC"
   Completed: "N/A"
   Modules: "ST, SC"
   HA mode: "false"
   TOS release: "21.2 (PGA.0.0) Final"
   TOS build: "21.2.2100-210722164631509"
   Expiration Date: "2021-09-22 13:18:43 +0000 UTC"
   Status: "Completed"

   4. Check that your backup file appears in the list, and that the status is "Completed".

   5. If your backup files are saved locally:

      1. Run sudo tos backup export to save your backup file from a TOS backup directory as a single .gzip file. If there are other backups present, they will be included as well.

      2. Transfer the exported .gzip file to a safe, remote location.

         Make sure you have the location of your backups safely documented and accessible, including credentials needed to access them, for recovery when needed.

      After the backup is exported, we recommend verifying that the file contents can be viewed by running the following command:

      [Target location]$ tar tzvf <filename>

      tar tzvf <file name>
3. Drain one of the non-primary data nodes.

   You will perform the following procedure on each node you plan to update:

   1. Check the cluster health.

      1. On the primary data node, check the following status.

         [<ADMIN> ~]$ sudo systemctl status k3s

         sudo systemctl status k3s

      In the output under the line k3s.service - Aurora Kubernetes, two lines should appear - Loaded... and Active... similar to the example below. If they appear, continue with the next step, otherwise contact Tufin Support for assistance.

      Example output:

      [<ADMIN> ~]$ sudo systemctl status k3s
       [root@TufinOS ~]# systemctl status k3s
       Redirecting to /bin/systemctl status k3s.service
       ● k3s.service - Aurora Kubernetes
          Loaded: loaded (/etc/systemd/system/k3s.service; enabled; vendor preset: disabled)
          Active: active (running) since Tue 2021-08-24 17:14:38 IDT; 1 day 18h ago
            Docs: https://k3s.io
         Process: 1241 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
         Process: 1226 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
        Main PID: 1250 (k3s-server)
           Tasks: 1042
          Memory: 2.3G

      2. On the primary data node, check the TOS status.

         [<ADMIN> ~]$ sudo tos status

         sudo tos status

      3. In the output, check if the System Status is Ok and all the items listed under Components appear as ok. If this is not the case, contact Tufin Support.

      Example output:

      [<ADMIN> ~]$ tos status         
      [Mar 28 13:42:09]  INFO Checking cluster health status           
      TOS Aurora
      Tos Version: 24.2 (PRC1.1.0)
      
      System Status: "Ok"
                  
      Cluster Status:
         Status: "Ok"
         Mode: "High Availability"
      
      Nodes
        Nodes:
        - ["node1"]
          Type: "Primary"
          Status: "Ok"
          Disk usage:
          - ["/opt"]
            Status: "Ok"
            Usage: 32%
        - ["node3"]
          Type: "Ha Data Node"
          Status: "Ok"
          Disk usage:
          - ["/opt"]
            Status: "Ok"
            Usage: 11%
        - ["node2"]
          Type: "Ha Data Node"
          Status: "Ok"
          Disk usage:
          - ["/opt"]
            Status: "Ok"
            Usage: 11%
      
      registry
        Expiration ETA: 819 days
        Status: "Ok"
      
      Infra
      Databases:
      - ["cassandra"]
        Status: "Ok"
      - ["kafka"]
        Status: "Ok"
      - ["mongodb"]
        Status: "Ok"
      - ["mongodb_sc"]
        Status: "Ok"
      - ["ongDb"]
        Status: "Ok"
      - ["postgres"]
        Status: "Ok"
      - [postgres_sc"]
        Status: "Ok"
      
      Application
      Application Services Status OK
      Running services 54/54
      
        Backup Storage:
        Location: "Local
      s3:http://minio.default.svc:9000/velerok8s/restic/default "
        Status: "Ok"
        Latest Backup: 2024-03-23 05:00:34 +0000 UTC			

   2. On the node on which you will work, run the command:

      [<ADMIN> ~]$ kubectl get node 

      kubectl get node

      A list of nodes which currently exist in the cluster appears.

   3. In the output, note the name of the data node on which you want to run maintenance.

   4. Run the command:

      [<ADMIN> ~]$ kubectl drain <NODENAME> --delete-emptydir-data --ignore-daemonsets   

      kubectl drain <NODENAME> --delete-emptydir-data --ignore-daemonsets

      where <NODENAME> is the name of the data node on which you want to run the maintenance.

4. Update the iDRAC and BIOS drivers.

   Follow the official Dell instructions for updating iDRAC9. The video on the page explains how to install both update files in one action.

5. Reinstate the data node.

   1. When the maintenance is completed, power up and log in again to the data node.

   2. Run the command:

      [<ADMIN> ~]$ kubectl uncordon <NODENAME>

      kubectl uncordon <NODENAME>

   3. 1. On the primary data node, check the following status.

         [<ADMIN> ~]$ sudo systemctl status k3s

         sudo systemctl status k3s

      In the output under the line k3s.service - Aurora Kubernetes, two lines should appear - Loaded... and Active... similar to the example below. If they appear, continue with the next step, otherwise contact Tufin Support for assistance.

      Example output:

      [<ADMIN> ~]$ sudo systemctl status k3s
       [root@TufinOS ~]# systemctl status k3s
       Redirecting to /bin/systemctl status k3s.service
       ● k3s.service - Aurora Kubernetes
          Loaded: loaded (/etc/systemd/system/k3s.service; enabled; vendor preset: disabled)
          Active: active (running) since Tue 2021-08-24 17:14:38 IDT; 1 day 18h ago
            Docs: https://k3s.io
         Process: 1241 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
         Process: 1226 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
        Main PID: 1250 (k3s-server)
           Tasks: 1042
          Memory: 2.3G

      2. On the primary data node, check the TOS status.

         [<ADMIN> ~]$ sudo tos status

         sudo tos status

      3. In the output, check if the System Status is Ok and all the items listed under Components appear as ok. If this is not the case, contact Tufin Support.

      Example output:

      [<ADMIN> ~]$ tos status         
      [Mar 28 13:42:09]  INFO Checking cluster health status           
      TOS Aurora
      Tos Version: 24.2 (PRC1.1.0)
      
      System Status: "Ok"
                  
      Cluster Status:
         Status: "Ok"
         Mode: "High Availability"
      
      Nodes
        Nodes:
        - ["node1"]
          Type: "Primary"
          Status: "Ok"
          Disk usage:
          - ["/opt"]
            Status: "Ok"
            Usage: 32%
        - ["node3"]
          Type: "Ha Data Node"
          Status: "Ok"
          Disk usage:
          - ["/opt"]
            Status: "Ok"
            Usage: 11%
        - ["node2"]
          Type: "Ha Data Node"
          Status: "Ok"
          Disk usage:
          - ["/opt"]
            Status: "Ok"
            Usage: 11%
      
      registry
        Expiration ETA: 819 days
        Status: "Ok"
      
      Infra
      Databases:
      - ["cassandra"]
        Status: "Ok"
      - ["kafka"]
        Status: "Ok"
      - ["mongodb"]
        Status: "Ok"
      - ["mongodb_sc"]
        Status: "Ok"
      - ["ongDb"]
        Status: "Ok"
      - ["postgres"]
        Status: "Ok"
      - [postgres_sc"]
        Status: "Ok"
      
      Application
      Application Services Status OK
      Running services 54/54
      
        Backup Storage:
        Location: "Local
      s3:http://minio.default.svc:9000/velerok8s/restic/default "
        Status: "Ok"
        Latest Backup: 2024-03-23 05:00:34 +0000 UTC			

      4. On the primary data node, check the TOS status.

         [<ADMIN> ~]$ sudo tos status

         sudo tos status

      5. In the output, check if the System Status is Ok and all the items listed under Components appear as Ok. If this is not the case, contact Tufin Support.

      Example output for a central cluster data node:

      [<ADMIN> ~]$ tos status         
      [Mar 28 13:42:09]  INFO Checking cluster health status           
      TOS Aurora
      Tos Version: 24.2 (PRC1.1.0)
      
      System Status: "Ok"
                  
      Cluster Status:
         Status: "Ok"
         Mode: "Multi Node"
      
      Nodes
        Nodes:
        - ["node1"]
          Type: "Primary"
          Status: "Ok"
          Disk usage:
          - ["/opt"]
            Status: "Ok"
            Usage: 19%
        - ["node3"]
          Type: "Worker Node"
          Status: "Ok"
          Disk usage:
          - ["/opt"]
            Status: "Ok"
            Usage: 4%
      
      registry
        Expiration ETA: 819 days
        Status: "Ok"
      
      Infra
      Databases:
      - ["cassandra"]
        Status: "Ok"
      - ["kafka"]
        Status: "Ok"
      - ["mongodb"]
        Status: "Ok"
      - ["mongodb_sc"]
        Status: "Ok"
      - ["ongDb"]
        Status: "Ok"
      - ["postgres"]
        Status: "Ok"
      - ["postgres_sc"]
        Status: "Ok"
      
      Application
      Application Services Status OK
      Running services 50/50
      
      Remote Clusters
      Number Of Remote Clusters: 2
        - ["RC"]
           Connectivity Status:: "OK:"
        - ["RC2"]
           Connectivity Status:: "OK"
      
        Backup Storage:
        Location: "Local
      s3:http://minio.default.svc:9000/velerok8s/restic/default "
        Status: "Ok"
        Latest Backup: 2024-03-23 05:00:34 +0000 UTC			

      Example output for a remote collector cluster data node:

      [<ADMIN> ~]$ tos status         
      [Mar 28 13:42:09]  INFO Checking cluster health status           
      TOS Aurora
      Tos Version: 24.2 (PRC1.0.0)
      
      System Status: "Ok"
                  
      Cluster Status:
         Status: "Ok"
         Mode: "Single Node"
      
      Nodes
        Nodes:
        - ["node2"]
          Type: "Primary"
          Status: "Ok"
          Disk usage:
          - ["/opt"]
            Status: "Ok"
            Usage: 19%
        
      registry
        Expiration ETA: 819 days
        Status: "Ok"
      
      Infra
      Databases:
      - ["mongodb"]
        Status: "Ok"
      - ["postgres"]
        Status: "Ok"
      
      Application
      Application Services Status OK
      Running services 16/16
      
        Backup Storage:
        Location: "Local
      s3:http://minio.default.svc:9000/velerok8s/restic/default "
        Status: "Ok"
        Latest Backup: 2024-03-23 05:00:34 +0000 UTC			

6. Repeat steps 3-6 for the second non-primary data node in the HA cluster.

7. Transfer the primary role to one of the updated data nodes.

   1. On the node you want to assume the role of primary data node:

      [Data Node]$ sudo tos cluster node set-primary

      sudo tos cluster node set-primary

   2. Verify that the node roles are as you intended, by running sudo tos cluster node list on any data node.

   3. We strongly recommend performing a one-time backup and exporting it immediately, as all prior backups are deleted from the original primary data node.

8. Repeat steps 3-6 for the third node (previously the primary node) in the HA cluster.

9. Optional: Set the original primary data node as primary.

   1. On the node you want to assume the role of primary data node:

      [Data Node]$ sudo tos cluster node set-primary

      sudo tos cluster node set-primary

   2. Verify that the node roles are as you intended, by running sudo tos cluster node list on any data node.

   3. We strongly recommend performing a one-time backup and exporting it immediately, as all prior backups are deleted from the original primary data node.