In this article, we will look into common ways to secure secrets in a Kubernetes application and how to manage them in a GitOps workflow based on ArgoCD with the help of Sops.
The problem is the following: your application depends on some secrets that you need to store securely and make available to your running application.
You can address this requirement in two ways:
This second solution has a clear advantage: you can provide your own GPG key and you don’t need to rely on a cloud provider or any external tools. If your goal is a multi-cloud strategy, it’s the way to go.
If you are using ArgoCD to deploy our Kubernetes objects you may wonder how to integrate Sops with ArgoCD. Let’s see what the ArgoCD documentation has to say.
ArgoCD documentation makes it quite clear:
Argo CD is un-opinionated about how secrets are managed. There’s many ways to do it and there’s no one-size-fits-all solution.
Basically, you are left to your own devices to make sops work with ArgoCD. In this article, we will take a look at how we can implement secret handling in an elegant, non-breaking way.
Let’s recap the tools we will use:
Helm Secrets is essentially a wrapper for Helm that encrypt and decrypt secrets on the fly for you. While no longer under heavy development, it’s still working really well.
But the problem is that ArgoCD doesn’t know this plugin as it only comes with the basic Helm binary built-in. Let’s address this now.
We will create a new ArgoCD docker image which contains exactly what we need:
The Dockerfile will look like this:
FROM argoproj/argocd:v1.7.6
ARG SOPS_VERSION="v3.6.1"
ARG HELM_SECRETS_VERSION="2.0.2"
ARG SOPS_PGP_FP="141B69EE206943BA9A64E691A00C9B1A7DCB6D07"
ENV SOPS_PGP_FP=${SOPS_PGP_FP}
USER root
COPY helm-wrapper.sh /usr/local/bin/
RUN apt-get update && \
apt-get install -y \
curl \
gpg && \
apt-get clean && \
rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* && \
curl -o /usr/local/bin/sops -L https://github.com/mozilla/sops/releases/download/${SOPS_VERSION}/sops-${SOPS_VERSION}.linux && \
chmod +x /usr/local/bin/sops && \
cd /usr/local/bin && \
mv helm helm.bin && \
mv helm2 helm2.bin && \
mv helm-wrapper.sh helm && \
ln helm helm2 && \
chmod +x helm helm2
# helm secrets plugin should be installed as user argocd or it won't be found
USER argocd
RUN /usr/local/bin/helm.bin plugin install https://github.com/zendesk/helm-secrets --version ${HELM_SECRETS_VERSION}
ENV HELM_PLUGINS="/home/argocd/.local/share/helm/plugins/"
A few things to note:
This wrapper script will look after the GPG key (you can mount it as a secret volume for example) and if found will import it. Then it will replace every call to helm with calls to helm secrets . Well, that’s in theory because Helm secrets does not support all Helm commands and is quite talkative so we need to alter the output for certain commands.
#! /bin/sh
GPG_KEY='/home/argocd/gpg/gpg.asc'
if [ -f ${GPG_KEY} ]
then
gpg --quiet --import ${GPG_KEY}
fi
# helm secrets only supports a few helm commands
if [ $1 = "template" ] || [ $1 = "install" ] || [ $1 = "upgrade" ] || [ $1 = "lint" ] || [ $1 = "diff" ]
then
# Helm secrets add some useless outputs to every commands including template, namely
# 'remove: <secret-path>.dec' for every decoded secrets.
# As argocd use helm template output to compute the resources to apply, these outputs
# will cause a parsing error from argocd, so we need to remove them.
# We cannot use exec here as we need to pipe the output so we call helm in a subprocess and
# handle the return code ourselves.
out=$(helm.bin secrets $@)
code=$?
if [ $code -eq 0 ]; then
# printf insted of echo here because we really don't want any backslash character processing
printf '%s\n' "$out" | sed -E "/^removed '.+\.dec'$/d"
exit 0
else
exit $code
fi
else
# helm.bin is the original helm binary
exec helm.bin $@
fi
Our ArgoCD image now understands Helm secrets without any additional configuration! Cool.
Our test application is a Helm chart with encrypted secrets. Please note the
secrets.yaml
file which is supposed to contain sensitive data.testapp
├── Chart.yaml
├── charts
├── secrets.yaml
├── templates
│ ├── NOTES.txt
│ ├── _helpers.tpl
│ ├── deployment.yaml
│ ├── hpa.yaml
│ ├── ingress.yaml
│ ├── service.yaml
│ └── serviceaccount.yaml
└── values.yaml
We encode the secrets with sops using our private key, the same one that is looked after by our Helm wrapper script:
sops -i --encrypt testapp/secrets.yaml
Then we push this Helm chart in our Git repository. The only thing left to do is to create the corresponding Application CRD to watch this new repository:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: testapp
namespace: argocd
spec:
project: default
source:
repoURL: [email protected]:my/repo/charts.git
targetRevision: master
path: charts/testapp
helm:
releaseName: testapp
valueFiles:
- "secrets.yaml"
destination:
server: https://kubernetes.default.svc
namespace: testapp
syncPolicy:
automated: {}
syncOptions:
- CreateNamespace=true
After
kubectl apply -f
this, you should see your new application appears in the ArgoCD dashboard. Secrets have been decrypted under the hood using the provided GPG key and the app is working properly.From an ArgoCD standpoint, the Helm wrapper appears as the built-in Helm binary so any GUI functionalities related to Helm are still working as usual.
To be exhaustive, let’s mention a simpler solution to our problem. We could have used an ArgoCD plugin. A plugin responsibility is to output some YAML that ArgoCD will then send to the Kubernetes API.
To make this work, you will still need a custom ArgoCD Dockerfile but you will not replace the Helm binary, only adding sops and Helm secrets. Then you will declare the plugin. The example below is an extract of the
values.yaml
file using the ArgoCD Helm chart:server:
config:
configManagementPlugins: |
- name: helmSecrets
init:
command: ["gpg"]
args: ["--import", "/home/argocd/gpg/gpg.asc"] # is mounted as a kube secret
generate:
command: ["/bin/sh", "-c"]
args: ["helm secrets template $HELM_OPTS $RELEASE_NAME ."]
It is the same philosophy: we import the GPG key when initializing the plugin and then we call helm secrets template with parameters from the environment to generates the expected YAML objects.
To use the plugin in an Application, do it like this:
piVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: testapp
namespace: argocd
spec:
project: default
source:
repoURL: [email protected]:my/repo/charts.git
targetRevision: master
path: charts/testapp
plugin:
name: helmSecrets
env:
- name: HELM_OPTS
value: "secrets.yaml"
- name: RELEASE_NAME
value: "testapp"
destination:
server: https://kubernetes.default.svc
namespace: testapp
syncPolicy:
automated: {}
syncOptions:
- CreateNamespace=true
You should get the same result as with our previous solution but with one notable exception: ArgoCD cannot recognize your plugin is in fact Helm in disguise so any GUI functionalities related to Helm will not be available, like seeing the values and parameters.
For this reason, our initial solution, albeit a little more complicated, is clearly superior.
With some quick adjustments, you can make your secrets handling tools work with ArgoCD. Even if ArgoCD doesn’t handle secrets by itself, it is really well thought since you can integrate your own tools quite easily.
Also published on: https://medium.com/faun/handling-kubernetes-secrets-with-argocd-and-sops-650df91de173