Secrets Are No Fun—Unless You Compare Them in Kubernetes

Kubernetes secrets are frequently used to securely store sensitive data, including passwords, API credentials, and certificates. In some cases, it may be necessary to compare secrets across several namespaces, clusters, or even two sets deployed by different systems.

In this article, we will look at easy and effective methods for analyzing two sets of Kubernetes secrets.

Problem in the wild

I can't claim that comparing Kubernetes secrets is part of the usual routine for DevOps or cloud engineers; however, it may be necessary for the following reasons:

Maintaining consistency across environments

• Teams may need to make a replica of the current environment for testing, including all configurations and secrets. In these cases, it's important to ensure that the two sets of secrets are equivalent in terms of structure and values.
• To avoid any structural difference during the deployment of an application to different environments (such as development, staging, and production), it's important to maintain consistency in underlying secret structures (not values). Otherwise, any differences could lead to security vulnerabilities or application failures (e.g., due to missing attributes).

Maintaining consistency across provisioners

• This use case frequently happens when two secret delivery systems are tested concurrently or when you migrate from one to the other (for example, from SealedSecrets to ExternalSecrets). To verify the secret provisioning mechanism, it is necessary to compare two sets of secrets that were deployed by different systems and represent the same K8S objects.

Versioning and updates

• During upgrades, you may need to ensure that the updated versions of secrets are properly propagated across the application. Comparing secrets helps to verify that no obsolete or stale secrets remain, reducing the risk of using invalid or expired credentials in the system.

Audit and security

• Secret comparison can also play a role in security audits, checking that sensitive information, such as passwords or API keys, hasn't been mistakenly exposed, updated, or leaked across environments. For example, production environments must always have an individual set of secrets.

To deal with any of these scenarios, a structural (keys and formats) or combination (structure and values) comparison of Kubernetes secrets may be required. Let's see how to do it.

How to compare Kubernetes secrets

To compare Kubernetes secrets, we need to perform the steps listed below, which we will implement as a script further:

1. Retrieve a list of Kubernetes secrets with the first prefix (e.g.: kvendingoldo1) using the Kubernetes API (kubectl).
2. Retrieve a list of Kubernetes secrets with the second prefix (e.g.: alextest) using the Kubernetes API (kubectl).
3. Iterate through the first list, checking each secret:
   • Attempt to find a matching secret in the second list.
   • If found, compare both structure and values using tools like jq.
4. Repeat the process for the second list to ensure completeness.
5. Generate a report in the terminal, highlighting:
   • Missing secrets
   • Missing fields within secrets
   • Differences in values

The basic implementation of the algorithm is something like this:

#!/bin/bash


#
# inputs
#
SECRET1_PREFIX="${1}"
SECRET1_NS="${2}"
SECRET2_PREFIX="${3}"
SECRET2_NS="${4}"


#
# ANSI color codes
#
COLOR_GREEN="\033[0;32m"
COLOR_YELLOW="\033[1;33m"
COLOR_RED="\033[0;31m"
COLOR_RESET="\033[0m"


function get_secrets_with_prefix() {
 local prefix="${1}"
 kubectl get secrets -n="${SECRET1_NS}" -o json | jq -r ".items[] | select(.metadata.name | startswith(\"${prefix}\")) | .metadata.name"
}


function mask_value() {
 local value=${1}
 local length=${#value}
 local mask_length=$((length / 2))
 local visible_part=${value:2:mask_length}
 echo "${visible_part}******"
}

function compare_secrets() {
 local secret1=${1}
 local secret2=${2}


 # Get keys (structure) for both secrets
 keys1=$(kubectl get secret -n="${SECRET1_NS}" "${secret1}" -o json | jq -r '.data | keys[]')
 keys2=$(kubectl get secret -n="${SECRET2_NS}" "${secret2}" -o json | jq -r '.data | keys[]')


 # Find common keys between the two secrets
 common_keys=$(echo -e "${keys1}\n${keys2}" | sort | uniq -d)


 if [ -z "${common_keys}" ]; then
   echo -e "No matching keys between ${secret1} and ${secret2}"
   return
 fi


 # Compare values of each common key
 for key in ${common_keys}; do
   value1=$(kubectl get secret -n="${SECRET1_NS}" "${secret1}" -o json | jq -r ".data[\"${key}\"]" | base64 --decode)
   value2=$(kubectl get secret -n="${SECRET2_NS}" "${secret2}" -o json | jq -r ".data[\"${key}\"]" | base64 --decode)


   if [ "${value1}" != "${value2}" ]; then
     echo -e "${COLOR_YELLOW}Difference found in key '${key}':${COLOR_RESET}"
     echo -e "  - ${secret1}: $(mask_value ${value1})"
     echo -e "  - ${secret2}: $(mask_value ${value2})"
   else
     echo -e "${COLOR_GREEN}Key '${key}' is identical in both secrets.${COLOR_RESET}"
   fi
 done


 # Check for keys that are missing in one of the secrets
 for key in ${keys1}; do
   if ! echo -e "${keys2}" | grep -q "${key}"; then
     echo -e "${COLOR_RED}Key '${key}' is missing in ${secret2}.${COLOR_RESET}"
   fi
 done


 for key in ${keys2}; do
   if ! echo -e "${keys1}" | grep -q "${key}"; then
     echo -e "${COLOR_RED}Key '${key}' is missing in ${secret1}.${COLOR_RESET}"
   fi
 done
}


function main() {
 secrets1=$(get_secrets_with_prefix "${SECRET1_PREFIX}")
 secrets2=$(get_secrets_with_prefix "${SECRET2_PREFIX}")


 for s1 in ${secrets1}; do
   suffix=${s1#$SECRET1_PREFIX}       # Get the suffix by removing the prefix
   s2="${SECRET2_PREFIX}${suffix}"    # Construct the corresponding secret in the second namespace


   # Check if the corresponding secret exists
   if echo -e "${secrets2}" | grep -q "${s2}"; then
     echo -e "Comparing ${s1} and ${s2}:"
     compare_secrets "${s1}" "${s2}"
     echo -e
   else
     echo -e "No corresponding secret for ${s1} in ${SECRET2_PREFIX}."
   fi
 done
}


main "${@}"

As you can see, the basic script supports different Kubernetes clusters, but it’s not difficult to implement.Let's do it with easy modifications:

1. Add KUBECONFIG variables for each secret set

   KUBECONFIG1="${1}"
   SECRET1_PREFIX="${2}"
   SECRET1_NS="${3}"
   
   KUBECONFIG2="${4}"
   SECRET2_PREFIX="${5}"
   SECRET2_NS="${6}"
   

   
   

2. Add kubeconfig variable into get_secrets_with_prefix function

   function get_secrets_with_prefix() {
     local kubeconfig=${1}
     local prefix=${2}
     local namespace=${3}
     KUBECONFIG=${kubeconfig} kubectl get secrets -n=${namespace} -o json | jq -r ".items[] | select(.metadata.name | startswith(\"${prefix}\")) | .metadata.name"
   }
   

   
   

3. Add kubeconfig1 and kubeconfig2 variables to compare_secrets

As a result, you will get the following script that can work with different clusters:

#!/bin/bash


#
# inputs
#
KUBECONFIG1="${1}"
SECRET1_PREFIX="${2}"
SECRET1_NS="${3}"
KUBECONFIG2="${4}"
SECRET2_PREFIX="${5}"
SECRET2_NS="${6}"


#
# ANSI color codes
#
COLOR_GREEN="\033[0;32m"
COLOR_YELLOW="\033[1;33m"
COLOR_RED="\033[0;31m"
COLOR_RESET="\033[0m"


function mask_value() {
 local value=${1}
 local length=${#value}
 local mask_length=$((length / 2))
 local visible_part=${value:2:mask_length}
 echo "${visible_part}******"
}


function get_secrets_with_prefix() {
 local kubeconfig="${1}"
 local prefix="${2}"
 local namespace="${3}"
 KUBECONFIG=${kubeconfig} kubectl get secrets -n=${namespace} -o json | jq -r ".items[] | select(.metadata.name | startswith(\"${prefix}\")) | .metadata.name"
}


function compare_secrets() {
 local kubeconfig1="${1}"
 local kubeconfig2="${2}"
 local secret1=""${3}"
 local secret2="${4}"


 # Get keys (structure) for both secrets
 keys1=$(KUBECONFIG=${kubeconfig1} kubectl get secret -n="${SECRET1_NS}" "${secret1}" -o json | jq -r '.data | keys[]')
 keys2=$(KUBECONFIG=${kubeconfig2} kubectl get secret -n="${SECRET2_NS}" "${secret2}" -o json | jq -r '.data | keys[]')


 # Find common keys between the two secrets
 common_keys=$(echo -e "${keys1}\n${keys2}" | sort | uniq -d)


 if [ -z "${common_keys}" ]; then
   echo -e "No matching keys between ${secret1} and ${secret2}"
   return
 fi


 # Compare values of each common key
 for key in ${common_keys}; do
   value1=$(KUBECONFIG=${kubeconfig1} kubectl get secret -n="${SECRET1_NS}" "${secret1}" -o json | jq -r ".data[\"${key}\"]" | base64 --decode)
   value2=$(KUBECONFIG=${kubeconfig2} kubectl get secret -n="${SECRET2_NS}" "${secret2}" -o json | jq -r ".data[\"${key}\"]" | base64 --decode)


   if [ "${value1}" != "${value2}" ]; then
     echo -e "${COLOR_YELLOW}Difference found in key '${key}':${COLOR_RESET}"
     echo -e "  - ${secret1}: $(mask_value ${value1})"
     echo -e "  - ${secret2}: $(mask_value ${value2})"
   else
     echo -e "${COLOR_GREEN}Key '${key}' is identical in both secrets.${COLOR_RESET}"
   fi
 done


 # Check for keys that are missing in one of the secrets
 for key in ${keys1}; do
   if ! echo -e "${keys2}" | grep -q "${key}"; then
     echo -e "${COLOR_RED}Key '${key}' is missing in ${secret2}.${COLOR_RESET}"
   fi
 done


 for key in ${keys2}; do
   if ! echo -e "${keys1}" | grep -q "${key}"; then
     echo -e "${COLOR_RED}Key '${key}' is missing in ${secret1}.${COLOR_RESET}"
   fi
 done
}


function main() {
 secrets1=$(get_secrets_with_prefix "${KUBECONFIG1}" "${SECRET1_PREFIX}" "${SECRET1_NS}")
 secrets2=$(get_secrets_with_prefix "${KUBECONFIG2}" "${SECRET2_PREFIX}" "${SECRET2_NS}")


 for s1 in ${secrets1}; do
   suffix=${s1#$SECRET1_PREFIX}       # Get the suffix by removing the prefix
   s2="${SECRET2_PREFIX}${suffix}"    # Construct the corresponding secret in the second namespace


   # Check if the corresponding secret exists
   if echo -e "${secrets2}" | grep -q "${s2}"; then
     echo -e "Comparing ${s1} in Cluster1 and ${s2} in Cluster2:"
     compare_secrets "${KUBECONFIG1}" "${KUBECONFIG2}" "${s1}" "${s2}"
     echo -e
   else
     echo -e "No corresponding secret for ${s1} in Cluster2 (${SECRET2_PREFIX})."
   fi
 done
}


main "${@}"

Testing the script

To test the script, let's generate test data using another bash script. The script source code:

#!/bin/bash


#
# inputs
#
SECRET1_PREFIX="${1}"
SECRET1_NS="${2}"
SECRET2_PREFIX="${3}"
SECRET2_NS="${4}"


NUM_SECRETS=5  # Number of secrets to generate
NUM_KEYS=4     # Number of keys per secret


function generate_random_value() {
 head -c 8 /dev/urandom | base64
}

function create_test_secrets() {
 local prefix=${1}
 local namespace=${2}


 for i in $(seq 1 $NUM_SECRETS); do
   secret_name="${prefix}test-secret-${i}"
   data=""


   if (( i % 3 == 1 )); then
     # Generate identical values
     data="  key1: $(echo -n 'fixedValue' | base64)\n  key2: $(echo -n 'fixedValue' | base64)\n  key3: $(echo -n 'fixedValue' | base64)"
   elif (( i % 3 == 2 )); then
     # Generate different values
     data="  key1: $(generate_random_value)\n  key2: $(generate_random_value)\n  key3: $(generate_random_value)"
   else
     # Generate different keys
     data="  keyA: $(generate_random_value)\n  keyB: $(generate_random_value)"
   fi


   echo "Creating secret ${secret_name} in namespace ${namespace}"
   echo -e "apiVersion: v1\nkind: Secret\nmetadata:\n  name: ${secret_name}\n  namespace: ${namespace}\ntype: Opaque\ndata:\n${data}" | kubectl apply -f -
 done
}


function main() {
 create_test_secrets "${SECRET1_PREFIX}" "${SECRET1_NS}"
 create_test_secrets "${SECRET2_PREFIX}" "${SECRET2_NS}"
}


main "${@}"

To run it, do the following commands:

• $ bash generate_test_data.sh kvendingoldo test-secrets-demo alextest test-secrets-demo
• $ bash compare_secrets.sh kvendingoldo test-secrets-demo alextest test-secrets-demo

Following that, you will see the output log, which shows that some secrets are identical while others have different values:

Conclusion

Comparing Kubernetes secrets is essential to maintaining consistency, security, and versioning in your application secrets. The approach described in this post is an excellent solution for rapid manual activities, which CloudOps/DevOps teams may need. You may need to modify the script a little bit for your task, but this should be easy thanks to the Bash language.

Feel free to experiment, and keep your secrets safeguarded!