Update BIOS and iDRAC Firmware

Overview

This procedure is relevant only for T-1220 and T-820 appliances running TufinOS 4.50. If you do not have that version, update TufinOS first. You need to update both worker nodes and data nodes. Worker nodes should be updated first.

Worker Nodes

1. Verify that the appliance version is T-1220 or T-820.

   Run the command:

   dmidecode -s chassis-version

   dmidecode -s chassis-version
2. Upgrade iDRAC.

   1. Download the iDRAC 7.20.10.05 upgrade file.
   2. From the iDRAC screen, click the Maintenance menu.

      

   3. Select System Update.
   4. From the System Update > Manual Update screen:

      Location Type: Local

      Update Path: Select the file that you downloaded at the start of this procedure.

   5.   In Upgrade Details, select the file you downloaded.

      

   6. Click Install.
   7. Reboot the system.

   You can also see Dell’s guide on updating Firmware and BIOS.

3. Upgrade BIOS.

   1. Download the BIOS 1.162 upgrade file.
   2. From the iDRAC screen, click the Maintenance menu.

      

   3. Select System Update.
   4. From the System Update > Manual Update screen:

      Location Type: Local

      Update Path: Select the file that you downloaded at the start of this procedure.

   5. In Upgrade Details, select the file you downloaded.

      

   6. Click Install Next Reboot.

      You can also see Dell’s guide on updating Firmware and BIOS.

Single Data Node Deployment

1. Verify that the appliance version is T-1220 or T-820.

   Run the command:

   dmidecode -s chassis-version

   dmidecode -s chassis-version
2. Backup your TOS data. This needs to be performed for all primary data nodes - the central cluster and remote collector clusters.

   If you are going to perform this procedure over multiple maintenance periods, create a new backup each time.

   1. Create the backup using tos backup create:

   [<ADMIN> ~]$ sudo tos backup create

   sudo tos backup create

   Example output:

   [<ADMIN> ~]$ sudo tos backup create
   [Aug 23 16:18:42]  INFO Running backup
   Backup status can be monitored with "tos backup status"

   2. You can check the backup creation status using tos backup status, which shows the status of backups in progress. Wait until completion before continuing.

   [<ADMIN> ~]$ sudo tos backup status

   sudo tos backup status

   Example output:

   [<ADMIN> ~]$ sudo tos backup status
   Found active backup "23-august-2021-16-18"

   3. Run the following command to display the list of backups saved on the node:

      [<ADMIN> ~]$ sudo tos backup list

      sudo tos backup list

   Example output:

   [<ADMIN> ~]$ sudo tos backup list
   ["23-august-2021-16-18"]
   Started: "2021-08-23 13:18:43 +0000 UTC"
   Completed: "N/A"
   Modules: "ST, SC"
   HA mode: "false"
   TOS release: "21.2 (PGA.0.0) Final"
   TOS build: "21.2.2100-210722164631509"
   Expiration Date: "2021-09-22 13:18:43 +0000 UTC"
   Status: "Completed"

   4. Check that your backup file appears in the list, and that the status is "Completed".

   5. If your backup files are saved locally:

      1. Run sudo tos backup export to save your backup file from a TOS backup directory as a single .gzip file. If there are other backups present, they will be included as well.

      2. Transfer the exported .gzip file to a safe, remote location.

         Make sure you have the location of your backups safely documented and accessible, including credentials needed to access them, for recovery when needed.

      After the backup is exported, we recommend verifying that the file contents can be viewed by running the following command:

      [Target location]$ tar tzvf <filename>

      tar tzvf <file name>
3. Stop TOS.

   Run the command:

   [<ADMIN> ~]$sudo tos stop

   sudo tos stop
4. Upgrade iDRAC.

   1. Download the iDRAC 7.20.10.05 upgrade file.
   2. From the iDRAC screen, click the Maintenance menu.

      

   3. Select System Update.
   4. From the System Update > Manual Update screen:

      Location Type: Local

      Update Path: Select the file that you downloaded at the start of this procedure.

   5.   In Upgrade Details, select the file you downloaded.

      

   6. Click Install.
   7. Reboot the system.

   You can also see Dell’s guide on updating Firmware and BIOS.

5. Upgrade BIOS.

   1. Download the BIOS 1.162 upgrade file.
   2. From the iDRAC screen, click the Maintenance menu.

      

   3. Select System Update.
   4. From the System Update > Manual Update screen:

      Location Type: Local

      Update Path: Select the file that you downloaded at the start of this procedure.

   5. In Upgrade Details, select the file you downloaded.

      

   6. Click Install Next Reboot.

      You can also see Dell’s guide on updating Firmware and BIOS.

6. Start TOS.

   Run the command:

   [<ADMIN> ~]$ sudo tos start

   sudo tos start

High Availability Deployment

BIOS and iDRAC need to be updated on each node in the cluster, without stopping TOS. To prevent data corruption, you will first drain each node before updating BIOS and iDRAC.

1. Verify that the appliance version is T-1220 or T-820.

   Run the command:

   dmidecode -s chassis-version

   dmidecode -s chassis-version
2. Backup your TOS data. This needs to be performed for all primary data nodes - the central cluster and remote collector clusters.

   If you are going to perform this procedure over multiple maintenance periods, create a new backup each time.

   1. Create the backup using tos backup create:

   [<ADMIN> ~]$ sudo tos backup create

   sudo tos backup create

   Example output:

   [<ADMIN> ~]$ sudo tos backup create
   [Aug 23 16:18:42]  INFO Running backup
   Backup status can be monitored with "tos backup status"

   2. You can check the backup creation status using tos backup status, which shows the status of backups in progress. Wait until completion before continuing.

   [<ADMIN> ~]$ sudo tos backup status

   sudo tos backup status

   Example output:

   [<ADMIN> ~]$ sudo tos backup status
   Found active backup "23-august-2021-16-18"

   3. Run the following command to display the list of backups saved on the node:

      [<ADMIN> ~]$ sudo tos backup list

      sudo tos backup list

   Example output:

   [<ADMIN> ~]$ sudo tos backup list
   ["23-august-2021-16-18"]
   Started: "2021-08-23 13:18:43 +0000 UTC"
   Completed: "N/A"
   Modules: "ST, SC"
   HA mode: "false"
   TOS release: "21.2 (PGA.0.0) Final"
   TOS build: "21.2.2100-210722164631509"
   Expiration Date: "2021-09-22 13:18:43 +0000 UTC"
   Status: "Completed"

   4. Check that your backup file appears in the list, and that the status is "Completed".

   5. If your backup files are saved locally:

      1. Run sudo tos backup export to save your backup file from a TOS backup directory as a single .gzip file. If there are other backups present, they will be included as well.

      2. Transfer the exported .gzip file to a safe, remote location.

         Make sure you have the location of your backups safely documented and accessible, including credentials needed to access them, for recovery when needed.

      After the backup is exported, we recommend verifying that the file contents can be viewed by running the following command:

      [Target location]$ tar tzvf <filename>

      tar tzvf <file name>
3. Drain one of the non-primary data nodes.

   You will perform the following procedure on each node you plan to update:

   1. Check the cluster health.

      1. On the primary data node, check the following status.

         [<ADMIN> ~]$ sudo systemctl status k3s

         sudo systemctl status k3s

      In the output under the line k3s.service - Aurora Kubernetes, two lines should appear - Loaded... and Active... similar to the example below. If they appear, continue with the next step, otherwise contact Tufin Support for assistance.

      Example output:

      [<ADMIN> ~]$ sudo systemctl status k3s
       [root@TufinOS ~]# systemctl status k3s
       Redirecting to /bin/systemctl status k3s.service
       ● k3s.service - Aurora Kubernetes
          Loaded: loaded (/etc/systemd/system/k3s.service; enabled; vendor preset: disabled)
          Active: active (running) since Tue 2021-08-24 17:14:38 IDT; 1 day 18h ago
            Docs: https://k3s.io
         Process: 1241 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
         Process: 1226 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
        Main PID: 1250 (k3s-server)
           Tasks: 1042
          Memory: 2.3G

      2. On the primary data node, check the TOS status.

         [<ADMIN> ~]$ sudo tos status

         sudo tos status

      3. In the output, check if the System Status is Ok and all the items listed under Components appear as ok. If this is not the case, contact Tufin Support.

      Example output:

      [<ADMIN> ~]$ tos status         
      [Mar 28 13:42:09]  INFO Checking cluster health status           
      TOS Aurora
      Tos Version: 24.2 (PRC1.1.0)
      
      System Status: "Ok"
                  
      Cluster Status:
         Status: "Ok"
         Mode: "High Availability"
      
      Nodes
        Nodes:
        - ["node1"]
          Type: "Primary"
          Status: "Ok"
          Disk usage:
          - ["/opt"]
            Status: "Ok"
            Usage: 32%
        - ["node3"]
          Type: "Ha Data Node"
          Status: "Ok"
          Disk usage:
          - ["/opt"]
            Status: "Ok"
            Usage: 11%
        - ["node2"]
          Type: "Ha Data Node"
          Status: "Ok"
          Disk usage:
          - ["/opt"]
            Status: "Ok"
            Usage: 11%
      
      registry
        Expiration ETA: 819 days
        Status: "Ok"
      
      Infra
      Databases:
      - ["cassandra"]
        Status: "Ok"
      - ["kafka"]
        Status: "Ok"
      - ["mongodb"]
        Status: "Ok"
      - ["mongodb_sc"]
        Status: "Ok"
      - ["ongDb"]
        Status: "Ok"
      - ["postgres"]
        Status: "Ok"
      - [postgres_sc"]
        Status: "Ok"
      
      Application
      Application Services Status OK
      Running services 54/54
      
        Backup Storage:
        Location: "Local
      s3:http://minio.default.svc:9000/velerok8s/restic/default "
        Status: "Ok"
        Latest Backup: 2024-03-23 05:00:34 +0000 UTC			

   2. On the node on which you will work, run the command:

      [<ADMIN> ~]$ kubectl get node 

      kubectl get node

      A list of nodes which currently exist in the cluster appears.

   3. In the output, note the name of the data node on which you want to run maintenance.

   4. Run the command:

      [<ADMIN> ~]$ kubectl drain <NODENAME> --delete-emptydir-data --ignore-daemonsets   

      kubectl drain <NODENAME> --delete-emptydir-data --ignore-daemonsets

      where <NODENAME> is the name of the data node on which you want to run the maintenance.

4. Upgrade iDRAC.

   1. Download the iDRAC 7.20.10.05 upgrade file.
   2. From the iDRAC screen, click the Maintenance menu.

      

   3. Select System Update.
   4. From the System Update > Manual Update screen:

      Location Type: Local

      Update Path: Select the file that you downloaded at the start of this procedure.

   5.   In Upgrade Details, select the file you downloaded.

      

   6. Click Install.
   7. Reboot the system.

   You can also see Dell’s guide on updating Firmware and BIOS.

5. Upgrade BIOS.

   1. Download the BIOS 1.162 upgrade file.
   2. From the iDRAC screen, click the Maintenance menu.

      

   3. Select System Update.
   4. From the System Update > Manual Update screen:

      Location Type: Local

      Update Path: Select the file that you downloaded at the start of this procedure.

   5. In Upgrade Details, select the file you downloaded.

      

   6. Click Install Next Reboot.

      You can also see Dell’s guide on updating Firmware and BIOS.

6. Reinstate the data node.

   1. When the maintenance is completed, power up and log in again to the data node.

   2. Run the command:

      [<ADMIN> ~]$ kubectl uncordon <NODENAME>

      kubectl uncordon <NODENAME>

   3. 1. On the primary data node, check the following status.

         [<ADMIN> ~]$ sudo systemctl status k3s

         sudo systemctl status k3s

      In the output under the line k3s.service - Aurora Kubernetes, two lines should appear - Loaded... and Active... similar to the example below. If they appear, continue with the next step, otherwise contact Tufin Support for assistance.

      Example output:

      [<ADMIN> ~]$ sudo systemctl status k3s
       [root@TufinOS ~]# systemctl status k3s
       Redirecting to /bin/systemctl status k3s.service
       ● k3s.service - Aurora Kubernetes
          Loaded: loaded (/etc/systemd/system/k3s.service; enabled; vendor preset: disabled)
          Active: active (running) since Tue 2021-08-24 17:14:38 IDT; 1 day 18h ago
            Docs: https://k3s.io
         Process: 1241 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
         Process: 1226 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
        Main PID: 1250 (k3s-server)
           Tasks: 1042
          Memory: 2.3G

      2. On the primary data node, check the TOS status.

         [<ADMIN> ~]$ sudo tos status

         sudo tos status

      3. In the output, check if the System Status is Ok and all the items listed under Components appear as ok. If this is not the case, contact Tufin Support.

      Example output:

      [<ADMIN> ~]$ tos status         
      [Mar 28 13:42:09]  INFO Checking cluster health status           
      TOS Aurora
      Tos Version: 24.2 (PRC1.1.0)
      
      System Status: "Ok"
                  
      Cluster Status:
         Status: "Ok"
         Mode: "High Availability"
      
      Nodes
        Nodes:
        - ["node1"]
          Type: "Primary"
          Status: "Ok"
          Disk usage:
          - ["/opt"]
            Status: "Ok"
            Usage: 32%
        - ["node3"]
          Type: "Ha Data Node"
          Status: "Ok"
          Disk usage:
          - ["/opt"]
            Status: "Ok"
            Usage: 11%
        - ["node2"]
          Type: "Ha Data Node"
          Status: "Ok"
          Disk usage:
          - ["/opt"]
            Status: "Ok"
            Usage: 11%
      
      registry
        Expiration ETA: 819 days
        Status: "Ok"
      
      Infra
      Databases:
      - ["cassandra"]
        Status: "Ok"
      - ["kafka"]
        Status: "Ok"
      - ["mongodb"]
        Status: "Ok"
      - ["mongodb_sc"]
        Status: "Ok"
      - ["ongDb"]
        Status: "Ok"
      - ["postgres"]
        Status: "Ok"
      - [postgres_sc"]
        Status: "Ok"
      
      Application
      Application Services Status OK
      Running services 54/54
      
        Backup Storage:
        Location: "Local
      s3:http://minio.default.svc:9000/velerok8s/restic/default "
        Status: "Ok"
        Latest Backup: 2024-03-23 05:00:34 +0000 UTC			

      1. On the primary data node, check the TOS status.

         [<ADMIN> ~]$ tos status

         tos status

      2. In the output, check if the System Status is Ok and all the items listed under Components appear as Ok. If this is not the case, contact Tufin Support.

      Example output for a central cluster data node:

      [<ADMIN> ~]$ tos status         
      [Mar 28 13:42:09]  INFO Checking cluster health status           
      TOS Aurora
      Tos Version: 24.2 (PRC1.1.0)
      
      System Status: "Ok"
                  
      Cluster Status:
         Status: "Ok"
         Mode: "Multi Node"
      
      Nodes
        Nodes:
        - ["node1"]
          Type: "Primary"
          Status: "Ok"
          Disk usage:
          - ["/opt"]
            Status: "Ok"
            Usage: 19%
        - ["node3"]
          Type: "Worker Node"
          Status: "Ok"
          Disk usage:
          - ["/opt"]
            Status: "Ok"
            Usage: 4%
      
      registry
        Expiration ETA: 819 days
        Status: "Ok"
      
      Infra
      Databases:
      - ["cassandra"]
        Status: "Ok"
      - ["kafka"]
        Status: "Ok"
      - ["mongodb"]
        Status: "Ok"
      - ["mongodb_sc"]
        Status: "Ok"
      - ["ongDb"]
        Status: "Ok"
      - ["postgres"]
        Status: "Ok"
      - ["postgres_sc"]
        Status: "Ok"
      
      Application
      Application Services Status OK
      Running services 50/50
      
      Remote Clusters
      Number Of Remote Clusters: 2
        - ["RC"]
           Connectivity Status:: "OK:"
        - ["RC2"]
           Connectivity Status:: "OK"
      
        Backup Storage:
        Location: "Local
      s3:http://minio.default.svc:9000/velerok8s/restic/default "
        Status: "Ok"
        Latest Backup: 2024-03-23 05:00:34 +0000 UTC			

      Example output for a remote cluster data node:

      [<ADMIN> ~]$ tos status         
      [Mar 28 13:42:09]  INFO Checking cluster health status           
      TOS Aurora
      Tos Version: 24.2 (PRC1.0.0)
      
      System Status: "Ok"
                  
      Cluster Status:
         Status: "Ok"
         Mode: "Single Node"
      
      Nodes
        Nodes:
        - ["node2"]
          Type: "Primary"
          Status: "Ok"
          Disk usage:
          - ["/opt"]
            Status: "Ok"
            Usage: 19%
        
      registry
        Expiration ETA: 819 days
        Status: "Ok"
      
      Infra
      Databases:
      - ["mongodb"]
        Status: "Ok"
      - ["postgres"]
        Status: "Ok"
      
      Application
      Application Services Status OK
      Running services 16/16
      
        Backup Storage:
        Location: "Local
      s3:http://minio.default.svc:9000/velerok8s/restic/default "
        Status: "Ok"
        Latest Backup: 2024-03-23 05:00:34 +0000 UTC			

7. Repeat steps 3-6 for the second non-primary data node in the HA cluster.

8. Transfer the primary role to one of the updated data nodes.

   1. On the node you want to assume the role of primary data node:

      [Data Node]$ sudo tos cluster node set-primary

      sudo tos cluster node set-primary

   2. Verify that the node roles are as you intended, by running sudo tos cluster node list on any data node.

   3. We strongly recommend performing a one-time backup and exporting it immediately, as all prior backups are deleted from the original primary data node.

9. Repeat steps 3-6 for the third node (previously the primary node) in the HA cluster.

10. Optional: Set the original primary data node as primary.

    1. On the node you want to assume the role of primary data node:

       [Data Node]$ sudo tos cluster node set-primary

       sudo tos cluster node set-primary

    2. Verify that the node roles are as you intended, by running sudo tos cluster node list on any data node.

    3. We strongly recommend performing a one-time backup and exporting it immediately, as all prior backups are deleted from the original primary data node.