Configuring Strata Cloud Manager Syslogs

Overview

This topic explains how to configure log forwarding from the Strata Logging Service in the Strata Cloud Manager (SCM) tenant to send encrypted syslogs to TOS.

This procedure can be performed using a custom PKI or a public CA.

Custom PKI procedure

This configuration uses TLS with a custom PKI, an OCSP responder, and a syslog VIP in TOS.

The process includes:

1. Creating certificates

2. Configuring an OCSP responder

3. Importing certificates into TOS

4. Configuring the syslog VIP

5. Configuring the Strata Logging Service

6. Adding the Strata CA to TOS

During this procedure you will be asked to verify the results of the commands you ran. If you are unable to verify them successfully, contact Support.

Public certificate procedure

This procedure requires that your syslog VIP be configured with a public CA.

1. Configuring the Strata Logging Service

2. Adding the Strata CA to TOS

For additional information about configuring log forwarding in Strata Cloud Manager, see the Palo Alto Strata documentation:

• Launch Strata Logging Service

• Forward Logs to a Syslog Server

Prerequisites

• TOS R25-2 PHF3 and later

• Palo Alto Strata Cloud Manager with the Strata Logging Service

• A TOS VIP for syslog traffic

• External IP address reachable from Strata Cloud Manager (for OCSP and syslog)

Configure SCM syslogs using a custom PKI

If you are using a public CA, skip to Configure the Strata Logging Service.

Create certificates

1. On the TOS Primary data node, create a local PKI directory structure (based on openSSL)

export PKI="/root/certificates/pki"
mkdir -p $PKI/{certs,private,ocsp,newcerts}
chmod 700 $PKI/private
touch $PKI/index.txt
echo 1000 > $PKI/serial

2. Create the OpenSSL configuration.

cat > $PKI/openssl.cnf <<'EOF'
[ ca ]
default_ca = CA_default
[ CA_default ]
dir               = /root/certificates/pki
database          = $dir/index.txt
new_certs_dir     = $dir/newcerts
certificate       = $dir/certs/ca.cert.pem
serial            = $dir/serial
private_key       = $dir/private/ca.key.pem
default_md        = sha256
policy            = policy_loose
email_in_dn       = no
copy_extensions   = copy
default_days      = 365
[ policy_loose ]
commonName        = supplied
[ server_cert ]
basicConstraints = CA:FALSE
keyUsage = digitalSignature,keyEncipherment
extendedKeyUsage = serverAuth
subjectAltName = IP:<syslog-external-ip>
authorityInfoAccess = @aia
[ ocsp ]
basicConstraints = CA:FALSE
keyUsage = digitalSignature
extendedKeyUsage = OCSPSigning
[ aia ]
OCSP;URI.0 = http://<tos-external-ip>:2560
EOF

Replace:

• <syslog-external-ip> with the SecureTrack syslog VIP.

• <tos-external-ip> with the external TOS IP that need to be reachable from the SCM tenant on port 2560.

3. Create the root certificate authority (CA).

   The CA key is encrypted with a passphrase. It is used for signing and not actually imported into TOS.

   1. Run the following command:

      cd $PKI

   2. Generate an encrypted CA key.

      openssl genrsa -aes256 -out private/ca.key.pem 4096

      chmod 600 private/ca.key.pem

   3. Create the CA certificate. Enter the CA passphrase when prompted.

      openssl req -x509 -new -sha256 -days 3650 \
        -key private/ca.key.pem \
        -out certs/ca.cert.pem \
        -subj "/CN=Tufin-Root-CA"

   4. Verify that the certificate was created.

      openssl x509 -in certs/ca.cert.pem -noout -subject -issuer

   The output should show.

   subject=CN = Tufin-Root-CA
   issuer=CN = Tufin-Root-CA

4. Create a dedicated certificate for the OCSP responder. This certificate is signed by your root CA and is used by the OCSP service to sign responses.

   1. Run the following command:

      cd $PKI

   2. Generate an OCSP key.

      openssl genrsa -out private/ocsp.key.pem 4096

      chmod 600 private/ocsp.key.pem

   3. Create the certificate signing request (CSR).

      openssl req -new \
        -key private/ocsp.key.pem \
        -out ocsp/ocsp.csr.pem \
        -subj "/CN=Tufin-OCSP"

   4. Sign the OCSP key with the CA certificate. Enter the CA passphrase when prompted.

      openssl ca -config openssl.cnf \
        -extensions ocsp \
        -in ocsp/ocsp.csr.pem \
        -out ocsp/ocsp.cert.pem \
        -batch

   5. Verify that the OCSP certificate was created properly.

      openssl x509 -in ocsp/ocsp.cert.pem -noout -text | grep "OCSP Signing"

   The output should show: OCSP Signing

   6. If you receive the following error, run the following command and repeat steps d and e.

      sed -i '/Tufin-OCSP/d' index.txt

5. Create the server certificate.

   The server key is not encrypted. TOS requires that all key be unencrypted.

   1. Run the following command:

      cd $PKI

   2. Generate the server key.

      openssl genrsa -out private/server.key.pem 2048

      chmod 600 private/server.key.pem

   3. Create the certificate signing request (CSR).

      openssl req -new \
        -key private/server.key.pem \
        -out certs/server.csr.pem \
        -subj "/CN=<syslog-external-ip>"

   Replace syslog-external-IP with the SecureTrack syslog VIP.

   4. Sign the CSR with the CA certificate. Enter the CA passphrase when prompted.

      openssl ca -config openssl.cnf \
        -extensions server_cert \
        -in certs/server.csr.pem \
        -out certs/server.crt.pem \
        -batch

   5. Verify that the OCSP Authority Information Access (AIA) is available.

      openssl x509 -in certs/server.crt.pem -noout -text | \
        grep -A2 "Authority Information Access"

   The output should show: CSP - URI:http://<tos-external-ip>:2560.

   6. Verify that a SAN certificate is available.

      openssl x509 -in certs/server.crt.pem -noout -text | \
        grep -A2 "Subject Alternative Name"

   The output should show: IP Address:<syslog-external-ip>

Set up the OCSP responder

1. On the TOS primary data node, create the systemd service.

   cat > /etc/systemd/system/ocsp.service <<'EOF'
   [Unit]
   Description=OCSP Responder for Secure Syslog
   After=network.target
   [Service]
   ExecStart=/usr/bin/openssl ocsp \
     -index /root/certificates/pki/index.txt \
     -port 2560 \
     -rsigner /root/certificates/pki/ocsp/ocsp.cert.pem \
     -rkey /root/certificates/pki/private/ocsp.key.pem \
     -CA /root/certificates/pki/certs/ca.cert.pem \
     -text
   Restart=always
   RestartSec=10
   [Install]
   WantedBy=multi-user.target
   EOF

2. Start the OCSP service.

   systemctl daemon-reload
   systemctl enable --now ocsp
   systemctl status ocsp

3. Verify that the OCSP service is listening.

   ss -lntp | grep 2560

The output should show: LISTEN on port 2560

4. Test the OCSP locally.

   cd $PKI
   openssl ocsp \
     -issuer certs/ca.cert.pem \
     -cert certs/server.crt.pem \
     -url http://127.0.0.1:2560 \
     -VAfile ocsp/ocsp.cert.pem \
     -CAfile certs/ca.cert.pem \
     -text

5. Expected output:

   • Response is verify OK.

   • certs/server.crt.pem: good

Import certificates into TOS

1. On the primary data node, go to /root/certificates/pki.

2. Import the certificates into TOS.

   tos certificate import --type syslog \
     --ca   certs/ca.cert.pem \
     --cert certs/server.crt.pem \
     --key  private/server.key.pem

3. Verify that a secret key was created.

   kubectl get secret | grep syslog-agent-nginx-secret

4. Configure the syslog VIP.

   tos cluster syslog-vip add <syslog-external-ip> --transport tcp

5. Verify the service is running.

   kubectl get svc | grep syslog-agent-tcp

6. Verify the pod is running.

   kubectl get pod | grep syslog-agent

Verify OCSP, certificates and Syslog configuration

Verify the following:

• OCSP service running and listening on 2560

• OCSP accessible from external: <tos-external-ip>

• Certificates imported to TOS

• Syslog VIP configured for TCP

• NGINX container running (5/5)

Configure Strata Logging Service (Part 1)

Configuring the Strata Logging Service requires performing actions in both the SCM tenant and the TOS primary data node.

1. Log in to Strata Cloud Manager.

2. Create a syslog profile.

3. Configure the following settings:

   • Name: tufin-syslog
   • Syslog Server: <syslog-external-ip>
   • Port: 6514
   • Facility: LOG_USER

4. Under Server Authentication, upload the server certificate: /root/certificates/pki/certs/ca.crt.pem

   This can be either a public CA or the custom CA certificate you created in the previous section.

5. Download the Strata CA Cert chain, and then continue to the next section.

Add the certificates to TOS

Custom PKI

1. Send the Strata CA cert chain to TOS.

   scp ca-chain.cert root@<TUFIN_IP>:/root/strata-ca.cert

2. On the primary data node go to /root.

3. Extract the root CA.

   kubectl get secret syslog-agent-nginx-secret -o jsonpath='{.data.ca\.crt}' | base64 -d > current-ca.crt

4. Combine the root CA with the Strata CA cert chain.

   cat current-ca.crt strata-ca.cert > combined-ca.crt

5. Verify that both certificates are on the machine.

   grep -c "BEGIN CERTIFICATE" combined-ca.crt

6. Update the NGINX secret key.

   kubectl create secret generic syslog-agent-nginx-secret \
       --from-file=tls.crt=/root/certificates/pki/certs/server.crt.pem \
       --from-file=tls.key=/root/certificates/pki/private/server.key.pem \
       --from-file=ca.crt=/root/combined-ca.crt \
       --dry-run=client -o yaml | kubectl apply -f -

7. Restart syslog-agent-service.

kubectl rollout restart deploy/syslog-agent-service

8. Wait for the pod to be ready.

   kubectl get pod -w | grep syslog-agent

Public CA

1. Send the Strata CA cert chain to TOS.

   scp ca-chain.cert root@<TUFIN_IP>:/root/strata-ca.cert

2. On the primary data node go to /root.

3. Update the NGINX secret key.

   kubectl create secret generic syslog-agent-nginx-secret \
     --from-file=ca.crt=/root/strata-ca.cert \
     --dry-run=client -o yaml | kubectl apply -f -

4. Restart syslog-agent-service.

kubectl rollout restart deploy/syslog-agent-service

5. Wait for the pod to be ready.

   kubectl get pod -w | grep syslog-agent

Configure Strata Logging Service (Part 2)

1. Go back to the SCM tenant, and test the connection in the syslog profile you created.

   If you created a custom CA and the test fails, verify that the OCSP responder can be accessed by Strata.

2. Enter the Profile Token.

   You will need the Profile Token when adding the SCM tenant to TOS.

3. Select the log types to forward:

   • firewall_traffic
   • system
   • config
   TOS supports CEF and LEEF syslog formats.

4. Save your changes to the SCM tenant.

Verify TOS is receiving syslog traffic

1. On the TOS primary data node, check that you are receiving encrypted traffic.

   tcpdump -nvi any port 6514 -c 10

You should see encrypted TLS traffic from Strata Cloud Manager..

2. Check that you are receiving decrypted traffic.

   tos config set -p network.debug=true -s syslog-agent -c

   kubectl exec -it deploy/syslog-agent-service -c netshoot -- \
    tcpdump -i any port 10514 -A -nn

You should see decrypted syslog messages forwarded to TOS

You should see file sizes increasing, files being processed and then being deleted.

3. Check NGINX logs.

kubectl exec -it deploy/syslog-agent-service -c nginx -- \
  tail -50 /var/log/nginx/error.log

4. Verify that the following messages are not displayed:

• client sent no required SSL certificate

• SSL_shutdown() failed

• certificate verify failed

Troubleshooting

Issue: 'Test Connection' Fails in Strata

• Test OCSP connection from an external location:

  curl -v http://<tos-external-ip>:2560

• Check the OCSP service

  systemctl status ocsp

• Check the server certificate.

  openssl x509 -in /root/certificates/pki/certs/server.crt.pem -noout -text | grep -A2 "OCSP"

Output must show: OCSP - URI:http://<tos-external-ip>:2560

Issue: No Traffic on Port 10514

There is traffic on port 6514, but no traffic on port 10514.

Cause: Strata CA not added to NGINX.

Solutions:

• Verify combined CA has both certificates

kubectl get secret syslog-agent-nginx-secret -o jsonpath='{.data.ca\.crt}' | \
  base64 -d | grep -c "BEGIN CERTIFICATE"

Output should show two certificates. If you see only one certificate, repeat the steps in Add the Strata CA.

• Check NGINX logs

  kubectl exec -it deploy/syslog-agent-service -c nginx -- \
    tail -100 /var/log/nginx/error.log | grep -i "certificate"

If the output shows client sent no required SSL certificate, repeat the steps in Add the Strata CA.

Issue: OCSP Service Fails to Start

• Check key permissions.

  ls -l /root/certificates/pki/private/ocsp.key.pem

Output should show: -rw------- (600)

If the output is different, run the following command:

chmod 600 /root/certificates/pki/private/ocsp.key.pem

• Check the service logs.

  journalctl -u ocsp.service -n 50

Issue: Syslog Files Stay at 0 Bytes

Cause: NGINX rejecting connections.

Solution:

• Check for NGINX errors.

  kubectl exec -it deploy/syslog-agent-service -c nginx -- \
    tail -50 /var/log/nginx/error.log

• Check for SSLor certificate errors. The error message indicates what needs to be fixed.