MQTT Broker

MQTT is a lightweight protocol that supports the Internet of Things (IoT). This article explains the functionality of its central hub known as the MQTT broker, compares its various implementations, and reviews its use cases, features, and best practices.

An MQTT broker is an intermediary entity that enables MQTT clients to communicate. Specifically, an MQTT broker receives messages published by clients, filters the messages by topic, and distributes them to subscribers.

Using MQTT brokers to enable the publish/subscribe (pub/sub) communication model helps make MQTT a highly efficient and scalable protocol.

In this article, we'll review MQTT brokers in-depth. To make sure you have a complete picture, we’ll start with an overview of MQTT and then dive into the specific role brokers play in communications.

What is MQTT?

MQTT is an open-source pub/sub communication protocol created by Andy Stanford-Clark and Arlen Nipper. It is most commonly run over TCP/IP stack, but there are MQTT implementations that use other protocols (e.g. Bluetooth). Since the protocol handles more than “Message Queuing” and “Telemetry Transport”, the term MQTT is commonly used as the protocol’s name instead of being thought of as an acronym for a longer name.

MQTT is efficient in its utilization of bandwidth and system resources. As a result, it is popular in use cases ranging from low-powered embedded systems to cloud-native microservices.

The pub/sub model is at the core of MQTT communication. Information is organized by topics. Client devices can publish data to a topic to send messages to any subscribers. Similarly, clients can subscribe to a topic to be notified when a message is published.

‍

MQTT Use-cases

MQTT is popular in IoT and IIoT (Industrial Internet of Things) because it can be used across platforms while consuming minimal bandwidth. IoT devices usually have limited resources. Because MQTT implementations are more lightweight and efficient than other communication architectures (for example, an HTTP-based RESTful API), MQTT is often a wise choice for IoT. Some of the most common MQTT use cases in IoT and IIoT are:

• Collecting data from sensor nodes and publishing it to the server.
• Publishing mission-critical data directly from a sensor node to a user device.
• Configuring IoT and IIoT devices remotely.
• Sending configuration data from a single web platform or smartphone app to all devices at once (thanks to the MQTT topics which allow wildcards).
• Instantly pushing OTA(over-the-air) updates to all the devices in the network.

‍

In addition to IoT and IIoT use cases, MQTT has also found its way into traditional software products and large-scale distributed systems. Most notably, Facebook uses MQTT for its Messenger application. Similarly, many open-source messaging applications, like Chat-App, use MQTT. As the MQTT community continues to grow, more and more open-source and closed-source projects using the protocol are being released.

MQTT Broker

In MQTT protocol there are two major entities: a broker and clients. What exactly are MQTT brokers and MQTT clients? How do they differ? Let’s take a look...

MQTT Broker vs MQTT Clients

An MQTT broker is a central software entity in the MQTT architecture. It acts just like a real estate broker which first does background checks on the parties involved and then after making sure that the relevant rules are enforced, the broker initiates a transaction.

An MQTT broker does the same thing but instead of monetary transactions, MQTT brokers handle message transactions. Specifically, here’s how MQTT brokers facilitate transactions between MQTT clients:

1. Allow devices (a.k.a. “client devices” or simply “clients”) to make a connection request
2. Authenticate the devices based on the connection information shared by the connecting device(s)
3. Once authenticated, make sure that the device can send/receive messages to/from other devices securely using Transport Layer Security (TLS) encryption (as one option)
4. Stores messages within the server so that they could be re-sent in the case of unwanted connection loss, on client-connect, on client-disconnect, etc.

As an MQTT broker allows devices (clients) to communicate in a decoupled way, the complete architecture can be scaled very easily without even affecting existing client devices.

Because the MQTT broker is a central entity and does all of the heavy lifting, the client devices only have to do minimal processing with minimal bandwidth.

Now let’s take a closer look at how those client devices work. Clients can be almost any “smart” device including smartphones, web apps, sensor nodes, actuators, or even smartwatches. Clients can:

1. Connect to a broker with a username and password
2. Subscribe to a topic (listen for messages published to that topic)
3. Publish to a topic (send a message(s) to that topic)

{{banner-3="/design/banners"}}

Types of MQTT Brokers

Generally, there are two types of brokers:

1. Managed Brokers
2. Self-Hosted Brokers

Managed Brokers

Managed brokers don’t require you to set up anything on your server to enable MQTT communication. Managed broker services let you use their hosted brokers for your system. AWS IoT Core is a good example of a managed MQTT Broker.

Self-Hosted Brokers

As the name implies, self-hosted MQTT brokers require you to install the broker on your own VPS or server with a static IP. The installation process is not difficult but managing, securing, and scaling the brokers requires in-depth knowledge of the system. There are several open-source implementations of MQTT brokers including mosquitto and hivemq.

List of Popular MQTT Brokers

There are plenty of viable managed and self-hosted MQTT brokers available. Here is an overview of some of the most popular options.

The above list mentions the most popular self-hosted and managed MQTT brokers with each broker having its own pros and cons. Some brokers are suitable for small-scale projects while others are made for large-scale applications. For example, brokers like VerneMQ and similar are made specifically for easy scalability of the system.

MQTT Message Types

Regardless of the type of broker, MQTT message types are similar. Here, we’ll take a look at the most common MQTT message types: connect messages, disconnect messages, and publish messages.

When a client initiates a connection with a broker a connect message is used. The connect message will include the following data:

clientID, cleanSession, username, password, lastWillTopic, lastWillQOS, lastWillMessage, lastWillRetain and keepAlive.

The first two variables, clientID and cleanSession, are required for all connection requests while all other variables are optional. The clientID should be a unique ID for each client.

A disconnect message is used when the client tries to disconnect a TCP/IP Session. If a disconnect message is set, a predefined message is sent before the complete disconnection.

Lastly, we have the generic publish message. Any client can publish a message to a topic so that the subscribed clients can be notified.

{{banner-1="/design/banners"}}

MQTT QoS

QoS or Quality of Service is a predefined agreement between publisher and subscriber of a topic. QoS is used in MQTT to set the message delivery guarantee levels. There are 3 QoS levels in MQTT:

1. QoS 0 (At most once)
2. QoS 1 (At least once)
3. QoS 2 (Exactly once)

Some brokers skip QoS 2 implementation.

QoS 0 - At most Once

QoS 1 - At least once

The QoS1 level requires the broker to guarantee that the message is sent at least one time to the client using an acknowledgment packet named PUBACK.

QoS 2 - At most once

The QoS 2 level guarantees that the broker will send the published message only once. It uses multilevel acknowledgment as shown in the figure above. PUBREC is the first acknowledgment from the broker, PUBREL is the acknowledgment from the publishing client that PUBREC is received while PUBCOMP is the second and final acknowledgment from the broker. This multilevel acknowledgment makes QoS 2 the slowest of the above-mentioned QoS levels. Because of the overhead, most of the managed broker services, including AWS IoT Core MQTT,  skip QoS level 2.

LWT and Keep-Alive in MQTT Brokers

Keep-Alive

MQTT Keep-Alive is the maximum time interval defined by the client for keeping the client-broker connection open.

In the MQTT Specification, we have the following definition of Keep-Alive

"The Keep-Alive[...] is the maximum time interval that is permitted to elapse between the point at which the Client finishes transmitting one Control Packet and the point it starts sending the next. It is the responsibility of the Client to ensure that the interval between Control Packets being sent does not exceed the Keep Alive value. In the absence of sending any other Control Packets, the Client MUST send a PINGREQ Packet."

LWT - Last Will and Testament

LWT message is a simple PUBLISH message stored in the broker and defined by the client. If a client disconnects from the broker, maybe due to a connection problem, the broker publishes the LWT messages to all the clients subscribed to the disconnecting device. LWT also allows the use of QoS which can be set in LWT based on the criticality of the application.‍

MQTT Broker Security

MQTT broker security depends on the implementation. MQTT brokers generally provide TLS encryption for secure communication between clients and brokers. The default secure MQTT broker ports are 8883 for MQTT and 443 for MQTT over WebSockets.

MQTT Broker Scalability

Like security, MQTT broker scalability depends on the implementation. MQTT brokers can be scaled vertically or horizontally. For example, the Mosquitto broker provides horizontal scalability options. Horizontal scalability is hard to achieve as it requires manual configuration and in-depth knowledge of networks.

There are also several self-hosted and managed brokers which are built around the unique selling proposition (USP) of scalability. If we take VerneMQ broker as an example, it provides both vertical and horizontal scalability options while providing high availability as a feature. VerneMQ also provides easy scalability configuration templates to make sure that the system can be scaled without affecting the clients.

MQTT Broker Challenges and Best Practices

How to select an MQTT broker?

Should I go with a Managed Broker or a Self-Hosted solution?

The answer to this question generally depends on the use case you are trying to implement. If you are looking to develop a prototype or a proof-of-concept (POC) quickly and don’t want to spend time managing the infrastructure and securing the connections, a managed broker service makes sense. Just a few clicks and you have a ready-to-use broker.

However, managed brokers have their drawbacks as well. For example, you will have to pay for the number of data packets transferred (the pricing model varies from service provider to service provider). You can’t control and configure most of the settings of the broker and can only tweak settings that the vendor allows. The managed broker provider usually puts a cap on data transfer and packets/second or packets/minutes which can slow down the communication.

So is it a good idea to use a self-hosted broker instead? It depends. If you have tried a  managed broker and the cost or control limitations created a problem, then a self-hosted solution may be the right choice. With a self-hosted solution, you can scale the system at your own accord, configure everything as you wish, and can put your rules in place easily.

Should I use traditional MQTT or MQTT-SN?

MQTT-SN was created to enable very low-power use and be placed at remote locations where traditional networks like 4G-LTE, NB-IoT, and WiFi, are not present. In such conditions, the devices should:

• Use minimal power to keep them running over a long periods (usually several years).
• Make use of a communication protocol, like LoRA or Zigbee, that consumes very little power and can transmit over long distances.

MQTT-SN divides the connection message into different parts, has short topic names, predefined topics, and other optimizations to keep the bandwidth usage and message size as little as possible.

The above picture shows a typical MQTT-SN system in which an MQTT-SN gateway is used. An MQTT-SN gateway is an entity that takes the message from a network like LoRA or Zigbee, parses it, and sends it to a traditional MQTT Broker using a WiFi or cellular network.

In short, if sensor nodes need to be placed at very distant remote places where traditional network connectivity is not available, MQTT-SN is the way to go.

How To Set Up An MQTT broker (implementation)

Local setup

Installing and Configuring MQTT Broker on Raspberry Pi

1. Connect Raspberry Pi to your WiFi Router
2. Open the terminal on Main Raspberry Pi and execute the following commands
3. sudo nano /etc/hostssome text
   1. Change rasberrypi   to mainrasberrypi
   2. Press CTRL+O then ENTER and then CTRL+X
4. sudo reboot
5. Once the raspberry pi reboots run the following commands
6. sudo apt install ufw
7. sudo ufw enable
8. sudo apt install mosquitto mosquitto-clients
9. sudo systemctl enable mosquitto
10.  sudo ufw allow 1883

Once done, open two terminal, and in the first terminal enter the following subscribe command:

mosquitto_sub -t outTopic -v -h  localhost

While in the second terminal execute the following publish command:

mosquitto_pub -d -t outTopic -m "Hello World" -h localhost

You will see “Hello World” in the subscription terminal.

Cloud-based MQTT Broker

Installing and Configuring MQTT Broker a VPS

1. Open the terminal on your Ubuntu instance and execute the following commands
2. sudo apt install ufw
3. sudo ufw enable
4. sudo apt install mosquitto mosquitto-clients
5. sudo systemctl enable mosquitto
6. sudo ufw allow 1883
7. sudo ufw allow 8883

Now you will need to attach a static IP address to your instance. In the case of AWS, you will need to attach an elastic IP address to your EC2 instance. Once done, you will need to open the ports 1883 and 8883 from your VPS provider platform to allow MQTT clients to connect to the recently installed MQTT Broker.

Once done, open the terminal of your VPS (or ssh into your VPS), and enter the following subscribe command

mosquitto_sub -t ScentStick/a/b -v -h  localhost

While on any other device you can publish command after installing mosquitto-clients

mosquitto_pub -d -t ScentStick/a/b -m "Hello World" -h [IP-ADDRESS OF VPS]

You will see “Hello World” in the subscription terminal.

If you don’t want to install mosquitto-clients in your client device you can use MQTTLens.

MQTT Lens is a Chrome Extension the link to it is given below.

https://chrome.google.com/webstore/detail/mqttlens/hemojaaeigabkbcookmlgmdigohjobjm?hl=en

• Install and open it
• Configure it based on the details of your MQTT broker installed on your VPS.

‍

‍

Then publish to the topic ScentStick/a/b and you will see the published message on your VPS terminal

{{banner-sre="/design/banners"}}

Conclusion

In a nutshell, MQTT is now a standard in IoT. There are countless MQTT broker and client implementations available in different languages like Go, C++, Python, and even NodeJS. MQTT has large and very active communities and has very well-written tutorials and technical documents. This makes for a simple learning curve for programmers and software architects. There are a small handful of concepts related to MQTT brokers and clients that can be learned in a very short time.

Further reading

How MQTT and MQTT Brokers work?

MQTT-SN - Detailed Overview

QoS Levels

Keep-Alive Structure

Last Will and Testament - Details

MQTT Real-life use-cases analysis