Deploying Sendy On Kubernetes to Reduce Newsletter Costs 100x - Part 1

Sendy is an amazing application. It's basically software that allows you to utilize Amazon SES to send newsletters to your members. It is also much, much cheaper than the alternatives. Sendy costs $69 once off (at the time of writing), and it should save you the cost multiple times over, assuming you have some subscribers to send newsletters to!

Why Sendy

Here are the cost benefits over alternatives like Campaign Monitor and MailChimp:

I was originally using Mailchimp for my old WordPress blog. However, I knew that when I hit 2,000 subscribers, I would enter the next pricing tier, which didn't make financial sense yet. So, when I migrated my blog to ghost, I switched to SendGrid, which I used for transactional emails. I also needed to migrate/delete members who had signed up to my Ghost blog into my SendGrid email list instead.

So after some research, I found a Zapier integration that would connect my Ghost blog to SendGrid and add/update/delete members as required. The only issue is that this was for version 2 (which is now deprecated), and version 3 of the integration only had some basic API functions (bummer).

So I contacted the SendGrid support team to inquire if it was possible to get the integration between Ghost and SendGrid updated. They said they would add it to their list, which means it will happen on some unspecified day in the future. This left me with two choices: keep manually exporting my contact list from Ghost and adding it to my Newsletter email list or figure out some other solution. So that is how I stumbled onto Sendy because after scanning the Zapier integration list, I saw that, yes, Sendy was listed!

Getting Started

The normal way to install Sendy is to spin up a VPS (I recommend DigitalOcean) and follow the installation instructions provided on the Sendy website. This works great, but I like using Kubernetes, so I wanted to figure out a way to get it working on my clusters. Luckily, DigitalOcean also lets you build Kubernetes Clusters!

To start, get a domain (one where you could perhaps use newsletters for a lot of your different projects). For example, something like projects.com could be cool because you could have 'newsletters.projects.com.’ You can even charge other users to send newsletters if you want to build a newsletter provider service.

You can now purchase your Sendy license or upgrade from a stable release version.

In your Sendy dashboard, set your domain as the licensed one. If you want to use staging and production clusters, it will also let you deploy on sub-domains for your licensed domain.

If you need to change the domain for your license code later, you can do so at Change Licensed Domain and insert your license key:

Now let's set up your database. I decided to use a helm chart of MariaDB. You can also use a managed database from DigitalOcean or another provider.

If you have installed the self-managed version of MariaDB (This link is the replicated version - Galera) on your cluster, you can also install a visual interface tool such as PhpMyAdmin.

Once installed, you can either open it up to the outside using a NodePort configuration or via a LoadBalancer (ensure you enabled HTTPS on your database), or you can port forward the database on localhost and connect using your browser. To port forward, you can do:

kubectl port-forward --namespace databases svc/phpmyadmin 8888:80

This should result in something like:

Forwarding from 127.0.0.1:8888 -> 8080
Forwarding from [::1]:8888 -> 8080

Now you can go to your browser and in a new tab, type the following in the address bar:

http://localhost:8888/

You should now see the login page:

When you are logged in successfully, you should see a section that lets you create a new user. Create a user with the name "sendy" and a secure password of your choosing. You will also see a checkbox to create a database table for this user. Enable this as the following image showcases:

You should store these details in a secure location as we will be using them soon.

After some research, I found that a developer called bubbajames had created a docker file so that you could deploy Sendy using the docker platform. This was great for people that used docker on their VPS, but I wanted to go one step further. Having that docker file was immensely useful for my plan. You will see why shortly.

You can download the entire project folder from my repo. Inside you will see the docker folder. Open this in your editor and navigate to the /includes folder. Inside you should see the config.php file. Here, edit the following and insert the database, user, and domain details you created for Sendy. You can enter HTTPS instead of HTTP for the domain section.

The issue with the docker image is that it was created using version 5.2 of Sendy (He now updated it to 5.2.3) and hasn't been updated yet (at the time of writing).

I like to keep my software updated to avail all the fancy features. It's free for the incremental updates until the next major version, so head over to Sendy and download the latest zip file if you haven't already (You will need to enter your license code).

Once you have the latest zip, you can go ahead and change the following files in your docker folder (That you downloaded from my repo or Sendy).

So in the artifacts folder, change the file name from 5.2 to 5.2.3

You can replace the old sendy-5.2.zip file with the one you downloaded called sendy-5.2.3.zip.

In the root of the docker folder is a version file, inside change the value from 5.2 to 5.2.3

Inside the Dockerfile you can change:

ARG SENDY_VER=5.2
ARG ARTIFACT_DIR=5.2

To:

ARG SENDY_VER=5.2.3
ARG ARTIFACT_DIR=5.2.3

Sendy Replication On Kubernetes

I am not replicating Sendy on my staging and production cluster. This is because Sendy uses cron jobs, so I would need to remove the cron job file, create a Kubernetes CronJob that would run, and send commands to my Sendy pods.

All of this is done to avoid having multiple pods duplicating actions because of the individual cron jobs running inside them. I opted for just having one instance running to avoid this. In the future, I might spend some time building a more elegant solution, or if this is something that interests you, please open a pull request at SendyKube.

Project Structure

Everything from this point on you should do for both the staging and for the production environments separately.

Ensure you have a SQL database ready that works with Sendy. MariaDB is a good one. Ensure you create the credentials that are needed. Alternatively, you can use a managed database with DigitalOcean.

Our project folder structure will look like the following:

sendy
  /production <- see repo for full files
  /staging <- see repo for full files
  /docker
    cron
    Dockerfile
    version
    /artifacts
      docker-entrypoint.sh
      /5.2.3
        sendy-5.2.3.zip
        /includes
          config.php
          update.php

You can see the entire file list in the repository.

For the secrets files, we should encrypt the details. This lets us deploy the files to our repository, to have one source of truth. So for this, you should install and configure sealed secrets for both your staging and production clusters. Of course, you could also ignore this step if you wanted to deploy normal secrets (encoded in base64, not encrypted).

If you decide to use encryption, you should install sealed secrets using the helm chart here: SealedSecrets Once installed, be sure to check the correct resources are running by using the following command:

kubectl get all -n kube-system

This should show you resources in the kube-system namespace where you installed the SealedSecrets helm chart (assuming you kept the default settings):

pod/sealed-secrets-5c88886466-4rnng    1/1     Running   0          26d
service/sealed-secrets   ClusterIP   10.245.241.51   <none>        8080/TCP                 28d
deployment.apps/sealed-secrets    1/1     1            1           28d
replicaset.apps/sealed-secrets-5c88886466    1         1         1       28d

Now that we have confirmed that sealed secrets is installed, you need to download the cert.pem file for both your production and staging clusters. To get the public encryption key from a cluster, you can:

First, expose the kubeseal service to localhost:

kubectl port-forward service/sealed-secrets -n kube-system 8081:8080

Then you can call the endpoint to get the file:

curl localhost:8081/v1/cert.pem

In some commands below, I will be using these cert.pem files to encrypt some secret files. So it will look for them in a certain location. You can also change the command to suit your own cert.pem file locations. The folder structure where I store my .pem files is :

./secrets/sealed-secrets/secrets/staging/cert.pem
./secrets/sealed-secrets/secrets/production/cert.pem
./newsletters/sendy

Now we need to set some key/value pairs. These values will be used inside the cluster to access a Sendy Docker file that we will be creating. It will be stored in our GitLab image repository.

For the variables, you first need to encode them in base64 format. This is the standardized encoding that Kubernetes uses. You can encode each password or value using the following command:

echo -n 'superpower' | base64

Which would result in:

c3VwZXJwb3dlcg==

If you need to decode to remember what they are, you can run:

echo 'c3VwZXJwb3dlcg==' | base64 -D

Now you need to set up the permissions for your docker registry in Gitlab. This will let us store the newly created Dockerfile there during the deployment stage.

For this, we need to navigate to the following section within your repo:

Once here, you should be able to create a deploy token with the following details:

Now you need to gather all the base64 values for your user within Gitlab; it is good practice to create a new user if you are using the Administrator account (ensure the new user has been added with the relevant permissions to the Sendy repo - ensure you have also created a repo called 'sendy').

When you have collected all the base64 values, you can now add them to your docker-config.yml file for staging and production (should be unique per different cluster). Ensure the URL path matches the URL of your sendy repo in GitLab.

{
    "auths": {
        "https://gitlab.domain.com:5050/path/to/sendy-repo":{
            "username":"eXV5YXhhTk1ibVRGQ2oK",
            "password":"RUVHdGFCRkpQSDh6ZW4K",
            "email":"bVBoelJIeWJEcnZHYXQK=",
            "auth":"NDQ0NTQ1NDU0NTRzZGZkZHJ2R2F0Cg=="
        }
    }
}

We will now encode this docker configuration file and add it to a secrets file so that our cluster can use it. You can do this by using the following command from the root of your project folder:

cat ./production/.production-docker-config.json | base64

Which results in something like this:

ewogICAgYXV0aHM6IHsKICAgICAgICPodHRwczovL2dpdGxhYi5kb21haW4uY29tOjUwNTAvcHJvamVjdHMvbmV3c2xldHRlcnMvc2VuZHk6ewogICAgICAgICAgICB1c2VybmFtZTplWFY1WVhoaFRrMWliVlJHUTJvSywKICAgICAgICAgICAgcGFzc3dvcmQ6UlVWSGRHRkNSa3BRU0RoNlpXNEssCiAgICAgICAgICAgIGVtYWlsOmJWQm9lbEpJZVdKRWNuWkhZWFFLPSwKICAgICAgICAgICAgYXV0aDpORFEwTlRRMU5EVTBOVFJ6Wkdaa1pISjJSMkYwQ2c9PQogICAgICAgIH0KICAgIH0KfQo=

Now we can add it to our secrets file (production-registry-credentials.yml):

apiVersion: v1
kind: Secret
metadata:
  name: registry-credentials
  namespace: newsletters
type: kubernetes.io/dockerconfigjson
data:
  .dockerconfigjson: ewogICAgYXV0aHM6IHsKICAgICAgICPodHRwczovL2dpdGxhYi5kb21haW4uY29tOjUwNTAvcHJvamVjdHMvbmV3c2xldHRlcnMvc2VuZHk6ewogICAgICAgICAgICB1c2VybmFtZTplWFY1WVhoaFRrMWliVlJHUTJvSywKICAgICAgICAgICAgcGFzc3dvcmQ6UlVWSGRHRkNSa3BRU0RoNlpXNEssCiAgICAgICAgICAgIGVtYWlsOmJWQm9lbEpJZVdKRWNuWkhZWFFLPSwKICAgICAgICAgICAgYXV0aDpORFEwTlRRMU5EVTBOVFJ6Wkdaa1pISjJSMkYwQ2c9PQogICAgICAgIH0KICAgIH0KfQo=

This file is also added to the .gitignore so that it will not be committed online.

We will now turn this secret (which has the private details encoded) into a SealedSecret, which becomes encrypted.

So we run this command from the root of the project folder. Be sure to specify where your cert.pem file is located (ensure it is the correct one for either the staging or the production cluster)

kubeseal < ./production/production-registry-credentials.yml --cert ../../secrets/sealed-secrets/secrets/production/cert.pem -o yaml > ./production/production-registry-credentials-encrypted.yml

The result should look something like:

apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  creationTimestamp: null
  name: registry-credentials
  namespace: newsletters
spec:
  encryptedData:
    .dockerconfigjson: <large-encrypted-string-here>
  template:
    metadata:
      creationTimestamp: null
      name: registry-credentials
      namespace: newsletters
    type: kubernetes.io/dockerconfigjson

Now that file is ready to be used in the deployment. Next, we move onto some of the other files. Be sure to replace "domain" with whatever your project name is (without the extension, i.e .com, .net, etc.). You will also replace it in the file names. These are just the production files; you need to do the same for the staging files. You can see them in the repo.

newsletters-domain-ingress-route.yml

This file is used to route traffic that hits our Traefik LoadBalancer to the correct application within our cluster. My cluster uses Traefik for a LoadBalancer, but you can also use nginx or others; you need to replace my Ingress files with yours.

In your domain provider, ensure to set the LoadBalancers provided IP A records for staging and production (newsletters.staging.domain.com and newsletters.domain.com). You can follow the Traefik instructions here.

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  namespace: newsletters
  name: newsletters-domain-ingress-route
annotations:
  kubernetes.io/ingress.class: "traefik"
  cert-manager.io/issuer: newsletters-domain-issuer
  traefik.ingress.kubernetes.io/router.entrypoints: web
  traefik.frontend.redirect.entryPoint: https
spec:
  entryPoints:
    - web
  routes:
    - match: Host(`newsletters.domain.com`)
      middlewares:
        - name: https-only
      kind: Rule
      services:
        - name: newsletters-production-domain
          namespace: newsletters
          port: 80

newsletters-domain-secure-ingress-route.yml

Here is the secure version of the ingress file.

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  namespace: newsletters
  name: newsletters-domain-secure-ingress-route
annotations:
  kubernetes.io/ingress.class: "traefik"
  cert-manager.io/issuer: newsletters-domain-issuer
  traefik.ingress.kubernetes.io/router.entrypoints: websecure
  traefik.frontend.redirect.entryPoint: https
spec:
  entryPoints:
    - websecure
  routes:
    - match: Host(`newsletters.domain.com`)
      kind: Rule
      services:
        - name: newsletters-production-domain
          namespace: newsletters
          port: 80
  tls:
    domains:                      
    - main: newsletters.domain.com
    options:
      namespace: newsletters
    secretName: newsletters-domain-com-tls

newsletters-domain-issuer.yml

This file is used to issue the SSL cert so that we can use HTTPS for our Sendy integration. I used the CertManager helm chart for this; there are other alternatives if you wish to use those (Traefik itself has a built-in letsencrypt certificate manager - I just liked the extra control from CertManager).

For the email section in the file, put in where you would like to get the email notifications from letsencrypt. For example, I have a catch-all set up in my email for my domain so that [email protected] will send it to my email. You could also create another address for your domain’s email; basically, it's up to you.

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: newsletters-domain-issuer
  namespace: newsletters
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: [email protected]
    privateKeySecretRef:
      name: newsletters-domain-com-tls-letsencrypt
    solvers:
    - http01:
        ingress:
          class: traefik

newsletters-domain-solver.yml

This file is the solver for the issuer file.

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: newsletters-domain-com
  namespace: newsletters
spec:
  secretName: newsletters-domain-com-tls
  issuerRef:
    name: newsletters-domain-issuer
    kind: Issuer
  commonName: newsletters.domain.com
  dnsNames:
  - newsletters.domain.com

newsletters-domain-middleware.yml

This file is used to create a middleware object that can intercept traffic and basically stick something in the middle. For example, I use this one to redirect HTTP traffic to HTTPS. Some other types you can use are authentication middlewares that ask for a password.

apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: https-only
  namespace: newsletters
spec:
  redirectScheme:
    scheme: https
    permanent: true

newsletters-domain-service.yml

This file is used to create the service which manages our application.

apiVersion: v1
kind: Service
metadata:
  namespace: newsletters
  name: 'newsletters-production-domain'
spec:
  type: ClusterIP
  ports:
    - protocol: TCP
      name: http
      port: 80
      targetPort: 80
    - protocol: TCP
      name: https
      port: 443
      targetPort: 80
  selector:
    app: 'newsletters-production-domain'

Thanks for following along with part one. Due to the text size limit, you will have to wait for part 2! Be sure to check the SendyKube Repo or https://blog.jotcode.com/deploying-sendy-on-kubernetes/