Add Features to Align with Entity Topic Structure of MTConnect.NET Agent

[pacssys]

We are currently working with the MTConnect CPP Agent version 2.4.0.7, and are attempting to publish data via MQTT to an IoT Gateway, where the gateway is currently consuming MQTT data published by clients using the MTConnect.NET agent. However, the .NET agent publishes data with different payload structure and topic structure, which is preventing us from using the CPP Agent to publish data to this IoT Gateway. We would prefer not to switch to the .NET agent as that would force us to redo many of our build processes and re-test existing functionality, so I am opening this issue to request a few additions to the functionality offered by the CPP Agent.

The topic structure of the MTConnect.NET Agent that we want to emulate is:

• ObservationTopic/<DataItemId>, where ObservationTopic is a configurable parameter like CurrentTopic, ...
  • For example, if ObservationTopic was MTConnect/Devices/[device]/Observations, each data item would be published on the topic MTConnect/Devices/[device]/Observations/<DataItemId>
• See "Entity Topic Structure" in https://github.com/TrakHound/MTConnect.NET/tree/master/libraries/MTConnect.NET-MQTT for a more detailed description -- the .NET agent offers both a "Flat" format and "Hierarchy" format for Observation topics. We are asking for the flat format.
  • The legacy MQTT sink in the CPP agent seemed to have functionality that published an Observation topic with a Hierarchy format. We need the flat format because the AWS IoT Core limits MQTT topics to 8 topics/7 forward slashes.

The payload structure for each data item that is used by the .NET Agent when publishing Observations is

{"dataItemId":"L2avail","name":"avail","type":"AVAILABILITY","timestamp":"2023-02-07T20:02:26.8978653Z","result":"AVAILABLE"}

There have been some previous inquiries into having some of these features in the CPP Agent -- see the following:

• Query Regarding FlatMode Configuration in MTConnect CPP Agent
• Unified MQTT Topic and Payload structure

@wsobel Would it be possible to add this topic structure and payload format into the CPP Agent?

[wsobel]

We deprecated the mqtt sink that is used per data item topics. We were having issues with broker performance, and it was dropping data. Maybe with Hive it will not be an issue.

This could also be added as a third-party plugin as well. I'm not sure what the .Net agent is doing. The MQTT support is not normative at this point; we need to make it normative in the future. There is support for Sparkplug B that is available from a third party.

We should discuss your requirements.

[pacssys]

To shed a bit more light on our requirements, we are exploring multiple options to resolve the format differences between what the CPP agent publishes and what our Gateway is expecting. There are two main avenues we are exploring:

• (Here) Making changes in this cppagent repo to support the topic and payload structure we are expecting. You've mentioned that you ran into broker performance issues and dropping data. We have existing systems which use the .NET agent to publish on per data item topics without issues with broker performance, so I'm hopeful this won't be an issue, but based on what you've said we may run into performance issues there.
  • Ideally, we would be able to publish on per data item topics for each observation as described above.
• Making changes in our Gateway to support the existing payload structure. @Pranay265 is currently working on exploring what changes would be needed on that front.
  • I'm not extremely familiar with Sparkplug B, but from some initial reading it seems that it defines a specific topic structure that is different from the topic structure we are expecting. Due to our existing infrastructure that consumes the published data, it doesn't seem feasible for us to use that -- ideally we would either convert the incoming data into the structure we want, or the data would already be in that format.

In either case, there is something I missed in the initial feature request:
Adding the "durable" buffer that the .NET agent has -- adding the capability to retain data when the agent restarts or loses connection with the broker. Upon reconnection, we would want to send all unsent data to the broker. This is something we would like to have eventually, but the foremost priority is resolving the difference in topic/payload structure.

[wsobel]

There is a third option to create a plugin for the agent that will publish in the same format as the .NET agent. The agent has an extensible architecture and you can create a new MQTT sink that has the behavior you want. It does require some C++, but I can advise.

[pacssys]

Thanks for your comments and sorry for the delayed response @wsobel -- we have been having internal discussions on the path we want to take.
We are still talking about how we want to approach this issue, but I wanted to gain some clarification regarding the third option to create a plugin to the agent.

To my understanding, creating a plugin to the agent would essentially be adding another sink to the agent (in addition to the rest_sink and mqtt_sink that currently exist) with the behavior we desire.
Is my understanding here correct?

[wsobel]

Exactly. The sink has an interface and a fairly simple API. There is also a contract with the Agent to get various pieces of info like the observations and assets. The other items are posted. There is a timed callback mechanism that is used in the mqtt and the rest sinks.

The sink can then be used as a DLL or shared lib and dynamically loaded. This can all be orchestrated with Conan.

If you would like I can walk you through an example and maybe create some educational material in the process.

Best

• Will

[pacssys]

That would be great! What would be easiest for you to use to walk through an example (Teams, this forum, ...)?