Move etcd - New AWS Instance

The etcd database should be on a separate volume to improve the stability of TOS Aurora and reduce latency. Moving the etcd database to a separate volume ensures that the kubernetes database has access to all the resources required to ensure an optimal TOS performance. We recommend performing this procedure after you install the operating system and before you install TOS.

This procedure is only required for data nodes running: Rocky Linux 8 or RHEL 8.

This procedure must be performed by an experienced Linux administrator with knowledge of network configuration.

Preliminary Preparations

  1. Switch to the root user.

    [<ADMIN> ~]$ sudo su -
    sudo su -

  2. Install the rsync RPM.

    [<ADMIN> ~]$ dnf install rsync
    dnf install rsync

Mount The etcd Database to A separate Volume

    1. In the AWS instance navigation pane, go to Volumes pane, and click Create Volume.

    2. Configure the following settings:

      • Volume type: SSD gp3

      • Size: Allocate a disk size of at least 128 GB

      • IOPS: 7500

      • Availability Zone: Same availability zone as instance

      • Throughput: 250 MBps

      • Snapshot ID: Keep the default value

      • Encryption: If the volume is encrypted, it can only be attached to an instance type that supports Amazon EBS encryption.

    3. Click Create Volume.

    4. In the navigation pane, go to Volumes pane, and click Actions > Attach Volume.

    5. In Instance, enter the ID of the instance or select the instance from the list of options.

    6. In Device name, select an available device name from the Recommended for data volumes section of the list.

      7. The device name is used by Amazon EC2. The block device driver for the instance might assign a different device name when mounting the volume.

    7. Connect to the instance and proceed to the next step.

    1. Log into the data node as the root user.

    2. Verify that the new volume s recognized by the the OS.

      [<ADMIN> ~]# lsblk
      lsblk
      [<ADMIN> ~]# ls -l /dev/nvme*
      ls -l /dev/nvme*

      Compare the output with the name of the volume you created in the previous step, and verify that the index number in the volume name is the latest number. In most cases this will be 1. For example, if this is the only volume the output will be nvme1n1. If an older volume was previously created, the index number will be nvme2n1.

    3. Create a variable with the block device path of the new volume.

      [<ADMIN> ~]# BLOCK_DEV="/dev/nvme<>"
      BLOCK_DEV="/dev/nvme<>"

      where <> represents the index number of the new volume.

    4. Generate a UUID for the block device of the new volume.

      [<ADMIN> ~]# BLOCK_UUID="$(uuidgen)"
      BLOCK_UUID="$(uuidgen)"

    5. Create a primary partition on the new volume.

      [<ADMIN> ~]# parted -s -a optimal ${BLOCK_DEV} mklabel msdos -- mkpart primary ext4 1MiB 100%
      parted -s -a optimal ${BLOCK_DEV} mklabel msdos -- mkpart primary ext4 1MiB 100%

    6. Verify that the partition was created.

      [<ADMIN> ~]# parted -s ${BLOCK_DEV} print
      parted -s ${BLOCK_DEV} print

    7. Format the partition as ext4.

      [<ADMIN> ~]# mkfs.ext4 -L ETCD -U ${BLOCK_UUID} ${BLOCK_DEV}p1
      mkfs.ext4 -L ETCD -U ${BLOCK_UUID} ${BLOCK_DEV}p1

    8. Verify that the partition has been formatted with the UUID and the etcd label (output should return the partition with UUID and an ETCD label).

      [<ADMIN> ~]# blkid | grep "$BLOCK_UUID"
      blkid | grep "$BLOCK_UUID"

    9. Create the mount point of the etcd database.

      [<ADMIN> ~]# mkdir -p /var/lib/rancher/k3s/server/db
      mkdir -p /var/lib/rancher/k3s/server/db

    10. Set the partition to mount upon operating system startup.

      [<ADMIN> ~]# echo "UUID=${BLOCK_UUID} /var/lib/rancher/k3s/server/db ext4 defaults 0 0" >> /etc/fstab
      echo "UUID=${BLOCK_UUID} /var/lib/rancher/k3s/server/db ext4 defaults 0 0" >> /etc/fstab

    11. Load the changes to the filesystem.

      [<ADMIN> ~]# systemctl daemon-reload
      systemctl daemon-reload

    12. Mount the partition that was added to /etc/fstab.

      [<ADMIN> ~]# mount /var/lib/rancher/k3s/server/db
      mount /var/lib/rancher/k3s/server/db

      If the output is not empty, stop the procedure. The etcd volume cannot be mounted. Review what was missed in the previous steps.

    13. Verify the partition has been mounted (the output should return the block device and mount point).

      [<ADMIN> ~]# mount | grep "/var/lib/rancher/k3s/server/db"
      mount | grep "/var/lib/rancher/k3s/server/db"

      If the output is empty, stop the procedure. The etcd volume is not mounted. Review what was missed in the previous steps.

  3. See Clean Install of TOS Aurora on AWS

    1. On the primary data node, check the TOS status.

      [<ADMIN> ~]$ tos status
      tos status

    2. In the output, check if the System Status is Ok and all the items listed under Components appear as Ok. If this is not the case, contact Tufin Support.

      3. Example output for a central cluster data node:

      [<ADMIN> ~]$ tos status         
[Mar 28 13:42:09]  INFO Checking cluster health status           
TOS Aurora
Tos Version: 24.2 (PRC1.1.0)

System Status: "Ok"
            
Cluster Status:
   Status: "Ok"
   Mode: "Multi Node"

Nodes
  Nodes:
  - ["node1"]
    Type: "Primary"
    Status: "Ok"
    Disk usage:
    - ["/opt"]
      Status: "Ok"
      Usage: 19%
  - ["node3"]
    Type: "Worker Node"
    Status: "Ok"
    Disk usage:
    - ["/opt"]
      Status: "Ok"
      Usage: 4%

registry
  Expiration ETA: 819 days
  Status: "Ok"

Infra
Databases:
- ["cassandra"]
  Status: "Ok"
- ["kafka"]
  Status: "Ok"
- ["mongodb"]
  Status: "Ok"
- ["mongodb_sc"]
  Status: "Ok"
- ["ongDb"]
  Status: "Ok"
- ["postgres"]
  Status: "Ok"
- ["postgres_sc"]
  Status: "Ok"

Application
Application Services Status OK
Running services 50/50

Remote Clusters
Number Of Remote Clusters: 2
  - ["RC"]
     Connectivity Status:: "OK:"
  - ["RC2"]
     Connectivity Status:: "OK"

  Backup Storage:
  Location: "Local
s3:http://minio.default.svc:9000/velerok8s/restic/default "
  Status: "Ok"
  Latest Backup: 2024-03-23 05:00:34 +0000 UTC

      Example output for a remote cluster data node:

      [<ADMIN> ~]$ tos status         
[Mar 28 13:42:09]  INFO Checking cluster health status           
TOS Aurora
Tos Version: 24.2 (PRC1.0.0)

System Status: "Ok"
            
Cluster Status:
   Status: "Ok"
   Mode: "Single Node"

Nodes
  Nodes:
  - ["node2"]
    Type: "Primary"
    Status: "Ok"
    Disk usage:
    - ["/opt"]
      Status: "Ok"
      Usage: 19%
  
registry
  Expiration ETA: 819 days
  Status: "Ok"

Infra
Databases:
- ["mongodb"]
  Status: "Ok"
- ["postgres"]
  Status: "Ok"

Application
Application Services Status OK
Running services 16/16

  Backup Storage:
  Location: "Local
s3:http://minio.default.svc:9000/velerok8s/restic/default "
  Status: "Ok"
  Latest Backup: 2024-03-23 05:00:34 +0000 UTC