Skip to content

MTM-64962 Rework MQTT Service overview for GA #3852

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
scmi-c8y merged 12 commits into from
Nov 25, 2025
Merged

MTM-64962 Rework MQTT Service overview for GA #3852

Show file tree
Hide file tree
Changes from 10 commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
99e1dd7
MTM-64962 initial rework of the introduction
scmi-c8y Oct 24, 2025
31aedfe
MTM-64962 commit work in progress
scmi-c8y Oct 24, 2025
17169ca
MTM-64962 new architecture diagram
scmi-c8y Oct 24, 2025
2c75520
MTM-64962 first draft for review
scmi-c8y Oct 27, 2025
341832f
Apply suggestions from code review
scmi-c8y Oct 28, 2025
5ecd905
MTM-64962 add a disclaimer to the preview block
scmi-c8y Oct 30, 2025
211e9c1
MTM-64962 Review comments from @rthrippleton
scmi-c8y Nov 19, 2025
828e7b2
MTM-64962 Fix inconsistent font size
scmi-c8y Nov 19, 2025
89f5353
MTM-64962 Review comments from @schmiel-cumulocity
scmi-c8y Nov 19, 2025
9df840a
MTM-64962 Review comments from @bbyreddy
scmi-c8y Nov 19, 2025
e803fd4
MTM-64962 Review comments from @rohitjoshi-c8y
scmi-c8y Nov 24, 2025
6164ddc
MTM-64962 Very minor typo
scmi-c8y Nov 25, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
92 changes: 62 additions & 30 deletions content/device-integration/mqtt-service-bundle/overview.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,43 +4,75 @@ layout: redirect
title: Overview
---

### Architecture {#architecture}
The MQTT Service works together with the Messaging Service to quickly and securely integrate MQTT devices with the {{< product-c8y-iot >}} platform.
Devices that already understand the {{< product-c8y-iot >}} domain model can use the [Core MQTT](/device-integration/mqtt/) protocols (SmartREST and JSON-over-MQTT) to communicate directly with {{< product-c8y-iot >}}.
Alternatively, devices can send and receive messages with arbitrary payloads on arbitrary MQTT topics.
For these _generic_ devices, the tenant is responsible for converting between the device's protocol and the {{< product-c8y-iot >}} domain model.
This conversion can be implemented in a microservice running inside the platform, or in an external client application.

The MQTT Service should be regarded as an MQTT _endpoint_ rather than a full MQTT _broker_.
It is optimized for the _IoT device integration_ use case, which has some highly asymmetric properties:
* A large number (up to tens of millions) of simultaneously connected devices publishing messages into the {{< product-c8y-iot >}} platform
* A large number (up to tens of millions) of unique MQTT topics
* A high aggregate throughput (up to millions per second) of unique messages published into the {{< product-c8y-iot >}} platform
* A small number of high-throughput message consumers within the {{< product-c8y-iot >}} platform
* Individual devices have a smaller message throughput (up to hundreds per second)
* Smaller aggregate message throughput from the {{< product-c8y-iot >}} platform to devices (up to thousands per second)
* No direct peer-to-peer device communication
* Devices are geographically distributed, often with limited physical security
* The {{< product-c8y-iot >}} platform is deployed on secure, managed, highly available infrastructure

Other, more symmetric _server-to-server_ or _application integration_ use cases may exceed the [limits](/service-terms/quotas/#mqtt-service) enforced by the MQTT Service.
For optimal performance, these use cases should be implemented using a more traditional publish/subscribe architecture with direct connections to the Messaging Service.

### Key features {#key-features}

The MQTT Service works together with the Messaging Service to provide a framework for highly customizable and flexible MQTT message processing solutions.
The diagram below illustrates how a message flows, starting from the device, through the Messaging Service,
then to a user-provided microservice where it is converted to the {{< product-c8y-iot >}} JSON format and
delivered to {{< product-c8y-iot >}} using the standard REST API.
#### Protocols and clients {#protocols-clients}

![MQTT Service send](/images/mqtt-service/mqtt-service-send.svg)
| | |
|---------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------|
| Connection protocols | TCP only. |
| MQTT protocol versions | 3.11 and 5.0. See [MQTT protocol implementation](#implementation) for more details. |
| Generic MQTT device protocols | MQTT devices can publish and subscribe arbitrary payloads on arbitrary MQTT topics. |
| {{< product-c8y-iot >}} Core MQTT protocols | **Preview** support for SmartREST 1.0, SmartREST 2.0 and JSON-over-MQTT protocols. See [TBD](#implementation) for more details. |
| Apache Pulsar | Microservices and external clients [connect directly to the Messaging Service](#pulsar-client) to convert between device protocols and the {{< product-c8y-iot >}} domain model.<br>Messaging Service clients are also responsible for registering devices as {{< product-c8y-iot >}} Managed Objects if required. |
| Thin Edge | Out of the box support in <a href="https://thin-edge.io/" target="_blank" rel="noopener noreferrer">Thin Edge</a> for generic and Core MQTT protocols on the same device. |

All MQTT messages published to the MQTT Service are forwarded to the Messaging Service, where they are persisted, waiting to be consumed.
A custom microservice or [Streaming Analytics app](/streaming-analytics/epl-apps/#using-cumulocity-mqtt-service) that understands the topic and payload structure can consume the MQTT messages, and then translate and push them into {{< product-c8y-iot >}}.
#### Security and isolation {#security-isolation}

Similarly, a custom microservice or Streaming Analytics app can send messages to devices, as shown in the diagram below.
In this case, the user-provided microservice receives messages from {{< product-c8y-iot >}} through a Notifications 2.0 subscription.
These messages are mapped to the payload structure used by the MQTT devices, then published to MQTT topics.
| | |
|--------------------|-------------------------------------------------------------------------------------------------------------------------------------|
| Multi tenancy | MQTT devices from multiple tenants connect to the same endpoint. Each tenant's connections, topics and messages are fully isolated. |
| Device isolation | MQTT devices within a tenant cannot communicate directly with each other. Each device effectively has its own private topic space. |
| Client access | Microservices and external applications connecting to the Messaging Service have access to all topics and messages within a tenant. |
| Bi-directional TLS | Certificate trust anchors are managed within {{< product-c8y-iot >}}. Certificates are tightly bound to individual MQTT devices. |

![MQTT Service push](/images/mqtt-service/mqtt-service-push.svg)
#### Performance and scaling {#performance-scaling}

| | |
|--------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------|
| Horizontal scaling | The MQTT Service can be scaled independently from the {{< product-c8y-iot >}} core. |
| Benchmark results | TBC |
| Limits and quotas | Per-tenant and per-client [limits and quotas](/service-terms/quotas/#mqtt-service) ensure service stability and prevent "noisy neighbour" problems. |


### Architecture {#architecture}

As with MQTT messages published by devices, messages published from a microservice will be forwarded to the Messaging Service, where they can be consumed by MQTT devices subscribed to the relevant topics.
The diagram below illustrates the MQTT Service data flows within a tenant.

Custom microservices may use the [Java client](/device-integration/mqtt-service#java-client) to publish to or consume from MQTT topics. They can use the [Microservice SDK](/microservice-sdk/java) to push data into {{< product-c8y-iot >}}.
All messages published by MQTT devices are forwarded to the Messaging Service, where they are persisted until they are consumed.
{{< product-c8y-iot >}} domain model messages published to the MQTT topics used by the Core MQTT protocols are consumed directly by the {{< product-c8y-iot >}} core.
Messages published to other MQTT topics are consumed by microservices and/or external clients that are responsible for mapping the messages to the {{< product-c8y-iot >}} domain model.
Similarly, the {{< product-c8y-iot >}} core and clients can publish messages to the Messaging Service that will be consumed by the MQTT Service and forwarded to devices.

### MQTT Service compared to Core MQTT {#mqtt-service-vs-cumulocity-iot-mqtt}
<p align="center" width="100%">
<img width="80%" src="/images/mqtt-service/mqtt-service-architecture.svg" alt="MQTT Service architecture">
</p>

The table below presents a basic comparison between the {{< product-c8y-iot >}} Core MQTT functionality and that of the MQTT Service.
#### Device isolation {#device-isolation}

| | Core MQTT | MQTT Service |
|:-----------------------------|:--------------------------------------------------------|:------------------------------------------------------------------------------------|
| QoS | 0, 1, 2 | 0, 1 |
| Clean session | Starting with clean session is recommended | Starting with clean session is required |
| Retained flag | Not supported | Not supported |
| Last will | Supported | Supported |
| MQTT 5.0 features | Not supported | MQTT 5.0 clients can connect. Partial support for MQTT 5.0 features |
| Authentication | Basic and TLS device certificates | Basic and TLS device certificates |
| Scalability | Horizontal | Currently a single instance. Horizontal scaling will be available in the GA release |
| Topic format | Determined by the SmartREST 2.0 protocol | Unrestricted. SmartREST topic names are reserved and cannot currently be used |
| Payload | Determined by the SmartREST 2.0 protocol | Unrestricted. The maximum message size is 128 KiB including all headers |
| Extensibility | Limited by SmartREST 2.0 custom templates | Streaming Analytics apps or custom mapping microservices can support arbitrary MQTT-based protocols |
| Message processors/consumers | Built-in message processor for each SmartREST 2.0 topic | Streaming Analytics apps or custom mapping microservices can support multiple processors for a topic |
| JSON via MQTT | Limited feature set | Streaming Analytics apps or custom mapping microservices can support arbitrary JSON payloads |
Because of the _device isolation_ feature, there is no interaction between topics with the same name used by different clients.
Effectively, every device has its own private topic space that can only be accessed by that device.
This can be seen in the diagram where _device 1_ and _device N_ are both publishing and subscribing on _topic A_.
Because devices are isolated, _device 1_ cannot see any of the messages published by _device N_, and vice-versa.
However, a microservice or external application client [connecting directly to the Messaging Service](#pulsar-client) has full access to the topics used by all devices, and can forward messages between clients if required.
39 changes: 13 additions & 26 deletions content/device-integration/mqtt-service.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,35 +9,22 @@ sector:
{{< c8y-admon-preview >}}
This feature is in **Public Preview**.
That is, it is not yet generally available and may be subject to change in the future.

The documentation describes the expected state of the MQTT Service when it becomes Generally Available.
Some behaviour may not yet be exactly as described in the documentation.
{{< /c8y-admon-preview >}}

{{< c8y-admon-req >}}
To work with the MQTT Service, the following requirements must be met:
* The {{< product-c8y-iot >}} Messaging Service must be deployed in your {{< product-c8y-iot >}} environment.
* The {{< product-c8y-iot >}} MQTT Service must be deployed in your {{< product-c8y-iot >}} environment.
* Your tenant must be subscribed to the mqtt-service microservice.
This may have been done automatically, depending on how your {{< product-c8y-iot >}} environment was configured.
To check the subscription, open the Administration application and navigate to **Ecosystem** > **Microservices**.
If you do not see the mqtt-service microservice listed, contact [product support](/additional-resources/contacting-support/) (for public environments) or your {{< product-c8y-iot >}} administrator (for dedicated environments) to request the subscription for your tenant.
To use the MQTT Service your tenant must be subscribed to the `Mqtt-service` microservice.
This may have been done automatically, depending on how your {{< product-c8y-iot >}} environment was configured.
To check the subscription, open the Administration application and navigate to **Ecosystem** > **Microservices**.
If you do not see the `Mqtt-service` microservice listed, contact [product support](/additional-resources/contacting-support/) (for public environments) or your {{< product-c8y-iot >}} administrator (for dedicated environments) to request the subscription for your tenant.
{{< /c8y-admon-req >}}

The MQTT Service is a new MQTT endpoint implementation for {{< product-c8y-iot >}} that provides the following benefits:

* Sending and receiving arbitrary payloads on any MQTT topic.
Note that the topics used by the {{< product-c8y-iot >}} [Core MQTT](/device-integration/mqtt/) implementation currently cannot be used with the MQTT Service.
* User-provided microservices can send and receive messages on MQTT topics, and map messages to and from the {{< product-c8y-iot >}} data model.
The typical use case for such a microservice is to map between MQTT device payloads, and the {{< product-c8y-iot >}} REST and Notifications 2.0 APIs.
* Multi-tenancy support.
A single endpoint serves multiple tenants and tenants are completely isolated from each other.
* Bi-directional TLS support.
All MQTT traffic is encrypted and clients can authenticate using X.509 certificates.

The MQTT Service does not replace the existing [Core MQTT](/device-integration/mqtt/) capability of {{< product-c8y-iot >}} that supports sending device data already in the {{< product-c8y-iot >}} domain model directly into the platform.
The new capability provided by the MQTT Service allows for easier integration of MQTT devices that cannot use the {{< product-c8y-iot >}} domain model.
It also supports more flexible communication patterns between devices, applications, and the {{< product-c8y-iot >}} platform, controlled by user-provided microservices.

**Device isolation** is the default behaviour for the MQTT Service. Each MQTT client has its own private topic space and cannot directly receive messages from other clients, enhancing security and isolation between devices.
The MQTT Service provides a single, unified endpoint for integrating MQTT devices with the {{< product-c8y-iot >}} platform.
It is not a general-purpose MQTT broker.
It is optimized for scenarios where there are a large number of connected devices, and a high aggregate throughput of messages from devices into the platform.

This documentation does not describe the basics of MQTT communication.
If you are unfamiliar with MQTT, we recommend you to consult one of the numerous introductions on the internet.
Some references can be found on the <a href="https://mqtt.org/mqtt-specification/" target="_blank">MQTT website</a>.
This documentation is aimed at developers who want to integrate MQTT devices with {{< product-c8y-iot >}}, or to build clients that communicate with their devices.
It does not describe the basics of MQTT communication.
If you are unfamiliar with MQTT, there are numerous introductions available, starting with the <a href="https://mqtt.org/mqtt-specification/" target="_blank" rel="noopener noreferrer">MQTT website</a>.
Loading