Using Kafka & Zookeeper Offsets

Kafka version 0.9v and above provide the capability to store the topic offsets on the broker directly instead of relying on the Zookeeper.

The decision on whether to store the offset in Kafka or Zookeeper is dependent on both the Kafka broker version and the version of the client driver. You can refer to the following table for the behavior

Offset Storage - ZookeeperData Model

Data Model

Zookeeper data more is very similar to a file directory system. Each node can have none or many children. Unlike the file system , Zookeeper nodes can have data associated with the nodes. These data can range from configuration data, status details , timestamps etc which help Zookeeper do what it does best.

Each node is called a 'znode' in Zookeeper parlance.

For the offsets, the data is stored under the following path :

Retrieving Offsets From Zookeeper

Step#1 : Connect to Zookeeper Shell

The '.sh' script is located under the /bin folder of the Zookeeper installation.

Step#2 : Execute the following command to retrieve the offset metadata

Offset Storage - Kafka

Offsets in Kafka are stored as messages in a separate topic named '__consumer_offsets' . Each consumer commits a message into the topic at periodic intervals. The message contains the metadata related to the current offset, the consumer group , partition number, topic associated with the offset and other useful information.

Reading Offsets From Kafka

Since __consumer_offsets is a just like any other topic, its possible to consume the message off. Before we do that we need make this topic visible to the consumers since this is an internal KAFKA topic and is not visible to the consumers by default. In order to make the topic visible, execute the following command.

Structure of message in __consumer_offset topic