Monitor Your Kubernetes Cluster Events With EventRouter, Golang, and Kafka

You are certainly familiar with Kubernetes events, especially when you investigate a dysfunction in your cluster using the infamous kubectl describe command or the event API resource. It’s a goldmine of information.

$ kubectl get events

15m         Warning   FailedCreate                                                                                                      replicaset/ml-pipeline-visualizationserver-865c7865bc    

Error creating: pods "ml-pipeline-visualizationserver-865c7865bc-" is forbidden: error looking up service account default/default-editor: serviceaccount "default-editor" not found

However useful this information is, it is only temporary. The retention period is generally between 5 minutes and 30 days. You may want to keep this precious information for auditing purposes or ulterior diagnosis in more durable and efficient storage like Kafka. You can then react to certain events by having a tool (e.g. Argo Events) or your own applications subscribing to a Kafka topic.

In this article, I will show you how to build such a pipeline for processing and storing Kubernetes cluster events.

What We Want to Build

We will build a whole Kubernetes events processing chain. The main components are:

Eventrouter from HeptioLab, a Kubernetes event handler that consolidates all the cluster events into various sinks among which a Kafka topic.
Strimzi operator as an easy way to manage Kafka brokers in Kubernetes.
A custom Go binary to distribute the events in their target Kafka topics.

Why do we want to distribute the events into different topics? Let’s say for example that in each namespace of your cluster you have Kubernetes assets related to a specific customer. You clearly want to segregate the customer events from each other before using them.

All the configurations, source code, and detailed setup instructions may be found in this GitHub repository

Create your Kafka Brokers and Topic

I chose to deploy Kafka in Kubernetes with Strimzi. To keep it short, it’s a Kafka operator for creating and updating Kafka brokers or topics. You can find detailed instructions for installing the operator in the official documentation.

We first create a new Kafka cluster:

apiVersion: kafka.strimzi.io/v1beta1
kind: Kafka
metadata:
  name: kube-events
spec:
  entityOperator:
    topicOperator: {}
    userOperator: {}
  kafka:
    config:
      default.replication.factor: 3
      log.message.format.version: "2.6"
      offsets.topic.replication.factor: 3
      transaction.state.log.min.isr: 2
      transaction.state.log.replication.factor: 3
    listeners:
    - name: plain
      port: 9092
      tls: false
      type: internal
    - name: tls
      port: 9093
      tls: true
      type: internal
    replicas: 3
    storage:
      type: jbod
      volumes:
      - deleteClaim: false
        id: 0
        size: 10Gi
        type: persistent-claim
    version: 2.6.0
  zookeeper:
    replicas: 3
    storage:
      deleteClaim: false
      size: 10Gi
      type: persistent-claim

Then we create the Kafka topic to ingest our events:

apiVersion: kafka.strimzi.io/v1beta1
kind: KafkaTopic
metadata:
  name: cluster-events
spec:
  config:
    retention.ms: 7200000
    segment.bytes: 1073741824
  partitions: 1
  replicas: 1

Setup EventRouter

Follow through the official instructions from EventRouter but a simple kubectl apply should be enough. We need to edit the router configuration to indicate our Kafka endpoint and the topic to use:

apiVersion: v1
data:
  config.json: |-
    {
      "sink": "kafka",
      "kafkaBrokers": "kube-events-kafka-bootstrap.kube-events.svc.cluster.local:9092",
      "kafkaTopic": "cluster-events"
    }
kind: ConfigMap
metadata:
  name: eventrouter-cm

Verify the Setup is Working

Our cluster-events Kafka's topic should now receive all the events. The simplest way to check it is the case is to run a consumer on the topic. To do so, we use for convenience one of our Kafka broker pods which already has all the utilities required. You should see events flowing:

kubectl -n kube-events exec kube-events-kafka-0 -- bin/kafka-console-consumer.sh \
  --bootstrap-server kube-events-kafka-bootstrap:9092 \
  --topic kube-events \
  --from-beginning
{"verb":"ADDED","event":{...}}
{"verb":"ADDED","event":{...}}
...

Write the Golang Consumer

We now want to distribute our Kubernetes events to multiple topics depending on the namespace they originate from. We will write a Golang consumer and producer to implement this logic:

The consumer part of the program listens for incoming cluster event on the cluster-events topic
The producer section write into a Kafka topic matching the namespace of the event

There’s no explicit need to create the new topics if the proper options are configured for Kafka (they are by default) as Kafka will create topics for you implicitly. It’s a really cool feature from the Kafka client API.

Have a look at the full code here: https://github.com/esys/kube-events-kafka/blob/master/events-fanout/cmd/main.go

There are of course more performant options, depending on the projected volume of events and complexity of the fanout logic. For a more robust implementation, a consumer making use of Spark Structured Streaming would be a great choice.

Deploying the Consumer

After building and pushing our binary into a Docker image, we wrap it into a proper Kubernetes deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: events-fanout
  name: events-fanout
spec:
  replicas: 1
  selector:
    matchLabels:
      app: events-fanout
  template:
    metadata:
      labels:
        app: events-fanout
    spec:
      containers:
        - image: emmsys/events-fanout:latest
          name: events-fanout
          command: [ "./events-fanout"]
          args:
            - -logLevel=info
          env:
            - name: ENDPOINT
              value: kube-events-kafka-bootstrap:9092
            - name: TOPIC
              value: cluster-events

Check the Destination Topics are Created

By now, new topics should have been created:

kubectl -n kube-events get kafkatopics.kafka.strimzi.io -o name

kafkatopic.kafka.strimzi.io/cluster-events
kafkatopic.kafka.strimzi.io/kube-system
kafkatopic.kafka.strimzi.io/default
kafkatopic.kafka.strimzi.io/kafka
kafkatopic.kafka.strimzi.io/kube-events

You will find your events neatly stored into these topics according to their namespace.

Takeaway

The ability to access historical Kubernetes event logs provides you with a better understanding of the state of your Kubernetes system than kubectl alone would allow you. More importantly, it’s a gateway to automating your cluster or application operations by reacting to events and as such building reliable, reactive, and smart software.

Alsopublished behind a paywall at https://codeburst.io/watch-your-kubernetes-cluster-events-with-eventrouter-and-kafka-f1221bea3d5e