MQTT (formerly known as MQ Telemetry Transport) was created in 1991 by Andy Stanford-Clark (IBM) and Arlen Nipper (Eurotech) in order to connect oil pipelines over unreliable, satellite networks. It is a machine-to-machine (M2M)/”Internet of Things” connectivity protocol.

It was designed as an extremely lightweight publish/subscribe messaging transport. It is mostly useful for establishing connections with remote locations where a small code footprint is required and/or network bandwidth is at a premium. The protocol is data-centric, unlike for example HTTP, which is document centric.

It is based on the Pub/Sub messaging pattern where clients instead of focusing on who the receiver of the message is, they focus on categorizing the messages in topics of interest. Messages are sent to an intermediate, a message broker where interested clients subscribe to.

The message structure is compact and the overhead of the binary header is just 2 bytes.

Every client has to be connected to the Message Broker to interact with the system, which sometimes makes the usage of the protocol not very efficient in cases of one message duration connections. Connecting and disconnecting to the broker has a resource cost which, in certain cases, can make the usage of the protocol as efficient as other protocol alternatives. Connection utilization is important in MQTT.

At the moment the most widely used version of the protocol is MQTT v3.1.1 which is implemented by many brokers. Version 3.1.1 came out on November 7, 2014.

On April 3, 2019, MQTT v5.0 became an official OASIS standard bringing a lot of new features. Some of them are:

Shared SubscriptionsSession ExpirationMessage Expiration

For the details of this version, you can refer to this link.

MQTT and Message Queues

The protocol is not directly related to message queues. Although some versions of the protocol can implement components that have similar behavior to queues, this is not one of the protocols described capabilities. This is usually a confusing subject. Many sources label MQTT as Message Queueing Telemetry Transport which is not correct. “MQ” refers to the MQ Series, an IBM product developed to support MQ telemetry transport. When the protocol was created, it was named after the IBM product.

In message queues, messages are stored in a queue until a consumer gets the message. In MQTT, messages that arrive in a topic without any subscribers are not kept in queues (unless a persistent session is enabled where all QOS1 & QOS2 messages, that are not yet confirmed or missed, are stored until consumption).

Messages in queues can be consumed by one client. In MQTT a message can have multiple subscribers. Message Queuing and Pub/Sub Messaging are two completely different messaging patterns.

The biggest difference with queues is that they need to be explicitly created and they also must be named. This is not the case with MQTT, where topics are flexible and created on the fly under a tree-like structure.

MQTT can provide a type of message queues under certain circumstances. This can happen when a persistent session is present between a client and the broker.

If a client that has a persistent session is disconnected from the broker and at some point reconnects, the following information is available immediately:

All the subscriptions of a clientAll QOS1 & QOS2 flow messages that have not been confirmedAll new QOS1 & QOS2 messages that the client missedAll QOS2 messages received from the client and are not yet completely acknowledged

More information regarding persistent sessions can be found here.

Protocol Characteristics

Simplicity

By design, the MQTT protocol focuses on simplicity and mainly exposes 5 ways of interaction.

Connect — Client waits for a connection to be established with the message broker.Disconnect — Broker waits for the MQTT client to finish any work, which needs to be done for the TCP/IP session to disconnect.Subscribe — Client Requests to subscribe to one or more topics.Unsubscribe — Client Requests to unsubscribe from one or more topics.Publish — Client or Broker, publish a message to a topic.

Multiple Service Levels (QoS Quality of Service)

Message delivery can be achieved in three different levels by the protocol which can be very convenient depending on the use case. QoS can be different for subscribing and publishing. Below we can see how the different levels work. NOTE: Entities can either refer to any client or the broker.

QOS0

The message will be published or received at most once (fire and forget).

QOS1

The message will be published or received at least once,meaning that the sender stored the message and will probably repeat the procedure if he does not receive an acknowledgment.

QOS2

The message will be published or received exactly one timeand will include an extra verification step for the communicated message.

What is really worth making crystal clear, is that the involved entities in each communication scenario are three, the publisher, the broker, and the subscriber.

Publisher and subscriber are not designed to use the same type of service(QoS). Each client communicates only with the intermediate, the broker. Clients are not destined to know the existence of one another.

Let’s analyze an example.

Say we have a smart thermometer that connects to the message broker and publishes its temperature in a topic every 10 seconds in a “fire and forget” type of service. An app for example on the other side has subscribed to the temperature topic, in an “at least once” type of service, and displays the received temperature on the screen.

In this example, the smart thermometer has a very small network bandwidth footprint since it is publishing in a fire and forget manner. In most cases, temperatures do not tend to drastically change in the time frame of 10 seconds so QOS0 seems to be the most appropriate type to use.

The app, on the other hand, should display any differences in temperature, so it should receive any messages at least once.

This summarizes our design and brings us to the conclusion that involved entities, always depending on the case, should not use the same type of service.

One To ∞ Message Distribution & Topic Structure

Each client that publishes a message on the message broker, by design, has no awareness that anyone other than the broker has received the message. Messages could be identified as interests or topics of discussion that anyone interested, could receive knowledge of.

What I find particularly funny is that when a client subscribes to a topic, it has no knowledge, whether or not, the particular topic actually exists or has (LWT) or will have any data in it.

This reminds me of the Schrödinger’s cat thought experiment, but instead of the cat, we are referring to the data that exist or will exist in a topic.

The only way to find out about them is to subscribe to a topic.

Topics that exist or will exist can have multiple or no subscribers at all.

MQTT topic structure is often mentioned as the topic tree. It is a tree-like structure that is split into levels. Topics defined by clients are UTF-8 strings that are processed by the broker to filter messages. Each topic consists of one or more topic levels. Topic levels are separated by forward-slashes “/”.

When subscribing to a topic, there are specific characters that can be used as wildcards. MQTT supports two types of wildcards. The single-level wildcard “+” and the multi-level wildcard “#”.

Let’s take the following topic tree as an example.

If we wanted to subscribe to messages regarding the humidity of the kitchen, all we would have to do is subscribe to:

home/kitchen/generic/humidity

What if we wanted to subscribe to the temperature of all the available rooms? We would use the single level wildcard character.

home/+/generic/temperature

Last but not least, where would we subscribe to, if we wanted to listen to all messages regarding the kitchen and everything that is in there? This time we would use the multi-level wildcard character.

home/kitchen/#

There are some best practices regarding topic structure that we will discuss later on.

Compact Packet

One of MQTT’s main strengths is the small overhead it adds to communications. Its packet headers are compact and have a binary structure.
They consist of 3 parts at most.

Fixed header, present in all MQTT Control PacketsVariable header, present in some MQTT Control PacketsPayload, present in some MQTT Control Packets

The Fixed header is always present in each MQTT packet. It is two bytes in size. The detailed structure of the fixed header is given below.

The Variable header is not present in all MQTT control packets. Its structure is different for different MQTT requests.

Payload is the actual data which is going to be sent. It’s not present in all the MQTT control packets. The payload of the message is limited to 268,435,456 bytes.

The length of the actual topic string is at most 65536 bytes. This is a limit imposed by the MQTT spec. It is also worth noting that the topic is encoded in UTF-8, so there might be less than 65536 characters available.

The MQTT spec details can be found here.

Connection loss awareness

The broker is always aware of all the connections that clients establish or terminate. There are also certain features that utilize this event.

Last Will and Testament (LWT), where every connected client describes his will, that upon disconnection a message will be published on a specific topic.Persistent Sessions that we referred to earlier

Simple but not limiting authentication

By design, MQTT offers a very simple authentication mechanism.

The CONNECT Packet contains Username and Password fields.

Implementations can choose how to make use of the content of these fields. They may provide their own authentication mechanism, use an external authentication system such as LDAP [RFC4511] or OAuth [RFC6749] tokens, or leverage operating system authentication mechanisms. Implementations passing authentication data in clear text, obfuscating such data elements or requiring no authentication data should be aware, that this can give rise to Man-in-the-Middle and replay attacks.

TLS [RFC5246] can provide encryption of data sent over the network. There are valid TLS cipher suites that include a NULL encryption algorithm that does not encrypt data. To ensure privacy, Clients and Servers should avoid these cipher suites.

An application might independently encrypt the contents of its Application Messages. This could provide privacy of the Application Message both over the network and at rest. This would not provide privacy for other properties of the Application Message such as Topic Name.

Client and Server implementations can provide encrypted storage for data at rest such as Application Messages that are stored as part of a Session.

The use of VPNs to connect Clients and Servers can provide privacy of data across the section of the network covered by a VPN.

Every broker that exists in the market has its own modules to handle authentication but generally they can be extended further to support authentication mechanisms such as those mentioned above. These implementations are usually custom, although some brokers on the market have some modules already implemented.

Best Practices

Although there are no restrictions on how Apps using the protocol are implemented, there are some practices that are widely adopted.

Topic structure

Topics are meant to categorize information into groups, in terms of interest. The topic structure should resemble a tree structure and should make possible the utilization of wildcards for subscriptions. Information regarding a specific topic should provide the ability to be retrieved by multiple entities.

To achieve that, consistency between topic layers should be maintained.Do not use a leading forward slash since it introduces another unused topic layer in the tree with a zero character that is not used. This can cause unnecessary confusion.

Never use spaces in a topic, UTF-8 has many different white space types and can cause trouble in debugging. Characters like these should be always avoided.

Never subscribe to “#”, the client might not be able to process the load and might also put some unnecessary strain on the broker. If you need, for some reason let’s say store all messages on a database, you should implement some kind of plugin on the broker to achieve this.

Specific topics and Extensibility should always be on the designer's mind since your topic structure should be able to extend. Topic name conventions should be maintained across all topics in order for your system to be able to adapt to new topic addition in the future without breaking the tree structure.

Use a unique identifier on topics that refer to specific entities. You should be able to distinguish each entity’s information or group it by a single level wildcard.

Proper Service Levels

As we already mentioned, service levels can be different from client to broker & broker to client. Always take note that QOS2 service level is expensive and should be used only when it is absolutely necessary. QOS0 should be used only on non-sensitive data as message loss can occur. QOS1 should be used when receiving duplicate data will not cause any trouble.

Connection Utilization

In MQTT, establishing a connection is not a cheap task. Always avoid connections that will be utilized for a single information transaction. Maintain connection or send data to batches if possible. Single transaction connections almost ruin all the pros of the fast and compound nature of the protocol.

Security

Security should never be taken lightly and TLS over TCP should always be used along with all the best practices that come along. Your connected entities will most likely be on a remote location meaning that your broker should be visible to the world. If your entities can execute commands or take orders from messages, then it’s not only a matter of security but also a matter of safety since sophisticated attackers can perform man-in-the-middle attacks at least.

This is a matter of great importance, not only on MQTT related implementations but, generally on IoT technologies, we all can remember what the Mirai-bot attacks taught us in 2016.

Brokers on the market

There are several brokers out there that implement the MQTT protocol. Here are some of them that are my personal favorites.

VerneMQ — Is particularly interesting due to it’s scaling capabilities and provided MQTT features.

KubeEdge — (Mosquitto on Kubernetes) — having a broker “natively” on kubernetes.

Mosca-Mosquitto — I actually started out experimenting some years ago using Mosca. Mosca and Mosquitto are most likely the best brokers to learn, test and debug.

Conclusions

MQTT is a PUB/SUB communication protocol that enables light, fast and effective communication between entities. It has many features such as the multiple service levels, persistent sessions, last will & testament and many more. It implements a unique tree topic structure that enables flexible management of information. Last but not least, there are several brokers and clients out there can cope with the scaling demands of the present.