The MIMIC MQTT Protocol Module is an optional facility that simulates the Internet of Things (IoT) Message Queue Telemetry Transport (MQTT) standard version 3.1.1 as detailed on the OASIS site and ISO , as well as limited support for the new MQTT 5.0 . TLS v1.3 , 1.2 and 1.1 is supported.
Each MIMIC agent instance configured with MQTT acts as a publisher client (eg. a sensor ) that connects to the broker (system under test) and publishes the configured messages for third-party subscriber clients to consume. In addition, it can act as a subscriber client, eg. for a control channel, or to do end-to-end latency testing.
MIMIC supports both scenarios of a different IP address per simulated device instance (agent), as in a LAN at the edge, or same IP address for any number of them, as behind a NAT device.
Any type of payload is supported, whether ASCII, JSON or binary, whether stateless or stateful. For example, we have implemented the widely used Sparkplug , SmartREST , LWM2M-MQTT and TR50 standards, as well as a variety of ad-hoc schemes.
Compatibility with various on-site and cloud-based brokers is documented in the section below.
MQTT support is made available in MIMIC as an optional dynamically loadable module. Starting with MIMIC 16.00, you can use the Protocol Wizard to install the MQTT module. If you prefer to enable MQTT by hand, you need to do the following:
INFO - MQTT : Loaded protocol from < path-to-DLL > INFO - MQTT v16.00
Once MQTT is loaded, any agent instance configured to support the MQTT protocol will be able to publish MQTT messages to a MQTT broker.
If the MQTT module is enabled, then Agent->Add, Agent->Configure and Agent->Paste dialogs will display MQTT as an additional checkbox in the Advanced pane along with the SNMP protocols. On selecting the checkbox a new MQTT pane will appear.
This MQTT configuration pane lets the user configure the parameters for a MQTT session:
This mandatory parameter specifies the MQTT configuration file, to determine what MQTT packet is generated by the corresponding sensor. You will not be able to start the agent unless this parameter is set.
The configuration file is detailed below. You can edit configuration files directly.
This mandatory parameter specifies the address of the broker server.
This optional parameter specifies the broker port to use. The default port is 1883.
By default, the client identifier field in the CONNECT message will be a string like mimic/mqtt/00001, the last number indicating the agent instance number. You can override this string with this configurable. If the string ends with #, then this is meant to be a prefix, and the client identifier sent to the server will replace the # with the agent instance number.
This optional parameter specifies the username to use.
This optional parameter specifies the username to use.
This optional parameter specifies the MQTT version to use. You can set this to 3.1.1 or 5.0. By default, version 3.1.1 is used.
This optional boolean parameter determines whether TLS security is enabled for this simulated sensor. The default is false.
If the sensor is configured to Use TLS security as above, then this parameter specifies the TLS configuration to use. If no configuration file specified, the agent's TLS client will not have a default certification. To use the sample certification included in MIMIC installation, you need to load the tls.cfg file.
NOTE: if configuring a large number of agents at different IP addresses, and if SNMP access to the device is not required, there is a performance benefit in configuring non-privileged ports for the SNMP agent (reference 5377). After MIMIC 20.00, if SNMP is not needed, it's best to disable the SNMP protocol.
A few new commands and some enhanced old commands can be used from the MIMICShell to control the MQTT functionality. Here is a synopsis:
This command lets the user gather the self-defining list of arguments required and their particulars. The parameters are detailed above. A sample exchange for this command would be:
mimicsh> mimic protocol msg MQTT get args {{filename} {Config File} {file} {scripts/mqtt {{*.cfg {MQTT config files} {edit yes} {new yes}}} - both} {mandatory} {}} {{broker} {Broker Address} {string} {} {mandatory} {}} {{port} {Port} {integer} {} {optional} {1883}} {{clientid} {Client ID} {string} {} {optional} {}} {{username} {Username(blank for all-access)} {string} {} {optional} {}} {{password} {Password(blank for all-access)} {string} {} {optional} {}} {{version} {Version} {string} {} {optional} {}} {{is_tls} {Use TLS} {boolean} {} {optional} {false}} {{tls_conf_filename} {TLS Config (Optional)} {file} {config/mqtt {{*.cfg {TLS config files} {edit yes} {new yes}}} - both} {optional} {}}
This command lets the user look at the protocols currently configured on the agent. A sample exchange for this command would be:
mimicsh> mimic agent get protocol snmpv1,snmpv2c,MQTT
This command lets the user change the protocol setting for an agent. A sample exchange for this command would be:
mimicsh> mimic agent get protocol snmpv1 mimicsh> mimic agent set protocol snmpv1,MQTT mimicsh> mimic agent get protocol snmpv1,MQTT
This command lets the user get the current argument settings. A sample exchange for this command would be:
mimicsh> mimic agent protocol msg MQTT get config {filename=mqtt1.cfg} {broker=2001:a18:1:10:d6be:d9ff:fecb:589f} {port=1883} {clientid=} {username=} {password=} {version=} {is_tls=false} {tls_conf_filename=}
This command lets the user change the current argument settings of all MQTT sessions for an agent. A sample exchange for this command would be:
mimicsh> mimic agent protocol msg MQTT get config {filename=mqtt1.cfg} {broker=2001:a18:1:10:d6be:d9ff:fecb:589f} {port=1883} {clientid=} {username=} {password=} {version=} {is_tls=false} {tls_conf_filename=} mimicsh> mimic agent protocol msg MQTT set config \{broker=192.9.200.71\} mimicsh> mimic agent protocol msg MQTT get config {filename=mqtt1.cfg} {broker=192.9.200.71} {port=1883} {clientid=} {username=} {password=} {version=} {is_tls=false} {tls_conf_filename=} mimicsh> mimic agent protocol msg MQTT set config \{filename=mqtt2.cfg\} mimicsh> mimic agent protocol msg MQTT get config {filename=mqtt2.cfg} {broker=192.9.200.71} {port=1883} {clientid=} {username=} {password=} {version=} {is_tls=false} {tls_conf_filename=}
This command lets the user change the MQTT tracing configuration for an agent. A sample exchange would be:
mimicsh> mimic agent assign 1 mimicsh> mimic agent protocol msg MQTT get trace 0 mimicsh> mimic agent protocol msg MQTT set trace 1 mimicsh> mimic agent protocol msg MQTT get trace 1and the log would show:
INFO 01/05.09:42:35 - agent 1 trace enabled for MQTT INFO 01/05.09:42:51 - agent 1 loaded device INFO 01/05.09:42:51 - MQTT[AGT=1] - sent CONNECT (30 bytes) INFO 01/05.09:42:51 - MQTT[AGT=1]: client started INFO 01/05.09:42:51 - MQTT[AGT=1] - rcvd CONNACK (4 bytes) INFO 01/05.09:43:00 - MQTT[AGT=1] - sent PUBLISH (119 bytes) INFO 01/05.09:43:00 - MQTT[AGT=1] - rcvd PUBACK (4 bytes) ...
Returns MQTT statistics information:
In order, the statistic values are:
A sample exchange for these commands would be:
mimicsh> mimic protocol msg MQTT get stats_hdr {{connect} {Connect}} {{disconnect} {Disconnect}} {{published} {Published}} {{pktSnt} {PktSent}} {{pktRcv} {PktRcvd}} {{pktDisc} {PktDisc}} {{abort} {Abort}} {{CONNECTsent} {CONNECTsent}} {{CONNACKrcvd} {CONNACKrcvd}} {{CONNACKerr} {CONNACKerr}} {{PINGREQsent} {PINGREQsent}} {{PINGRESPrcvd} {PINGRESPrcvd}} {{PUBLISHsent} {PUBLISHsent}} {{PUBLISHsentbytes} {PUBLISHsentbytes}} {{PUBACKrcvd} {PUBACKrcvd}} {{PUBRECrcvd} {PUBRECrcvd}} {{PUBRELsent} {PUBRELsent}} {{PUBCOMPrcvd} {PUBCOMPrcvd}} {{PUBLISHrcvd} {PUBLISHrcvd}} {{PUBLISHrcvdbytes} {PUBLISHrcvdbytes}} {{PUBACKsent} {PUBACKsent}} {{PUBRECsent} {PUBRECsent}} {{PUBRELrcvd} {PUBRELrcvd}} {{PUBCOMPsent} {PUBCOMPsent}} {{SUBSCRIBEsent} {SUBSCRIBEsent}} {{SUBACKrcvd} {SUBACKrcvd}} {{UNSUBSCRIBEsent} {UNSUBSCRIBEsent}} {{UNSUBACKrcvd} {UNSUBACKrcvd}} {{DISCONNECTsent} {DISCONNECTsent}} {{DISCONNECTrcvd} {DISCONNECTrcvd}} {{CONNACKdisc} {CONNACKdisc}} {{PUBLISHdisc} {PUBLISHdisc}} {{PUBACKdisc} {PUBACKdisc}} {{PUBRECdisc} {PUBRECdisc}} {{PUBRELdisc} {PUBRELdisc}} {{PUBCOMPdisc} {PUBCOMPdisc}} {{SUBACKdisc} {SUBACKdisc}} {{UNSUBACKdisc} {UNSUBACKdisc}} {{PINGRESPdisc} {PINGRESPdisc}} mimicsh> mimic agent protocol msg MQTT get statistics 1 0 2 11 11 0 0 1 1 0 1 1 2 48 0 2 2 2 2 48 0 2 2 2 1 1 0 0 0 0 0 0 0 0 0 0
This command lets the user query the state of the client. This is particularly useful at agent startup time to wait for MQTT client startup. A sample exchange for this command would be:
mimicsh> mimic agent protocol msg MQTT client get state 0 mimicsh> mimic agent start mimicsh> while { ! [mimic agent protocol msg MQTT client get state] } { sleep 1 }
This command returns the protocol state of the client. The following are the values:
You can change these configurable parameters for the MQTT client after agent startup provided that the agent is HALTED and not connected to a broker (ie. mimic agent protocol msg MQTT client get protstate returns 2). For example:
mimicsh> mimic agent protocol msg MQTT get config {filename=mqtt2.cfg} {broker=192.9.200.71} {port=1883} {clientid=} {username=} {password=} {version=} {is_tls=false} {tls_conf_filename=} mimicsh> mimic agent start # if the agent has a start action script that sets it's state to HALTED, # then the MQTT client does not establish a connection mimicsh> mimic agent get state 3 mimicsh> mimic agent protocol msg MQTT client get protstate 2 mimicsh> mimic agent protocol msg MQTT client set broker iot.eclipse.org
To control the client session at runtime, these commands allow you to abort the session (without sending a DISCONNECT command), disconnect gracefully from the server, and re-connect, respectively. For example:
mimicsh> mimic agent start mimicsh> mimic agent protocol msg MQTT client get state 1 mimicsh> mimic agent protocol msg MQTT client get protstate 6 mimicsh> mimic agent protocol msg MQTT client runtime disconnect mimicsh> mimic agent protocol msg MQTT client get state 1 mimicsh> mimic agent protocol msg MQTT client get protstate 2 mimicsh> mimic agent protocol msg MQTT client runtime connect mimicsh> mimic agent protocol msg MQTT client get state 1 mimicsh> mimic agent protocol msg MQTT client get protstate 6 mimicsh> mimic agent protocol msg MQTT client runtime abort mimicsh> mimic agent protocol msg MQTT client get protstate 2
To control message generation by the client at runtime, these commands allow you to get and set message simulation parameters. The first parameter returns the cardinality of the messages, ie. 0 or more. You can then control each message (starting at 0) with the get and set commands.
ATTR (defined in message configuration below can be one of the following:
Eg. to PUBLISH exactly one of the first configured message on demand (where count was configured to 0):
mimicsh> mimic agent protocol msg MQTT client message card 1 mimicsh> mimic agent protocol msg MQTT client message get 0 count 0 mimicsh> set sent [mimic agent protocol msg MQTT client message get 0 sent] 0 mimicsh> incr sent 1 mimicsh> mimic agent protocol msg MQTT client message set 0 count $sent # message is PUBLISHed at this moment mimicsh> mimic agent protocol msg MQTT client message get 0 count 1 mimicsh> set sent [mimic agent protocol msg MQTT client message get 0 sent] 1
Or, to set and clear the Message Expiry Interval for a message:
mimicsh> mimic agent protocol msg MQTT client message card 1 mimicsh> mimic agent protocol msg MQTT client message get 0 properties User-Property User-Property mimicsh> mimic agent protocol msg MQTT client message get 0 properties.0 prop1 = some value 1 mimicsh> mimic agent protocol msg MQTT client message get 0 properties.1 prop2 = some value 2 mimicsh> mimic agent protocol msg MQTT client message get 0 properties.2 {cannot get property 2} mimicsh> mimic agent protocol msg MQTT client message set 0 properties.Message-Expiry-Interval 100 mimicsh> mimic agent protocol msg MQTT client message get 0 properties.2 100 mimicsh> mimic agent protocol msg MQTT client message set 0 properties.Message-Expiry-Interval {""} mimicsh> mimic agent protocol msg MQTT client message get 0 properties User-Property User-Property mimicsh> mimic agent protocol msg MQTT client message get 0 properties.2 {cannot get property 2}
Or, to change one of the user properties:
mimicsh> mimic agent protocol msg MQTT client message get 0 properties User-Property User-Property mimicsh> mimic agent protocol msg MQTT client message get 0 properties.0 prop3 = some value 3 mimicsh> mimic agent protocol msg MQTT client message get 0 properties.1 prop2 = some value 2 mimicsh> mimic agent protocol msg MQTT client message set 0 properties.1 {{prop3=some value 3}} mimicsh> mimic agent protocol msg MQTT client message get 0 properties.0 prop3 = some value 3 mimicsh> mimic agent protocol msg MQTT client message get 0 properties.1 prop3 = some value 3
To control subscriptions by the client at runtime, these commands allow you to get and set subscription simulation parameters. The first parameter returns the cardinality of the subscriptions, ie. 0 or more. You can then control each subscriptions (starting at 0) with the get and set commands. The unsubscribe command unsubscribes with an UNSUBSCRIBE message, and the resubscribe command resubscribes.
ATTR (defined in subscription configuration below can be one of the following:
For example, to unsubscribe to the current topic, and re-subscribe with a different one
mimicsh> mimic agent protocol msg MQTT client subscribe unsubscribe 0 mimicsh> mimic agent protocol msg MQTT client subscribe get 0 topic some-topic/# mimicsh> mimic agent protocol msg MQTT client subscribe get 0 properties Subscription-Identifier User-Property mimicsh> mimic agent protocol msg MQTT client subscribe get 0 properties.Subscription-Identifier 123456 mimicsh> mimic agent protocol msg MQTT client subscribe set 0 properties.Subscription-Identifier 123457 mimicsh> mimic agent protocol msg MQTT client subscribe get 0 properties.Subscription-Identifier 123457 mimicsh> mimic agent protocol msg MQTT client subscribe set 0 topic some-topic/2 mimicsh> mimic agent protocol msg MQTT client subscribe resubscribe 0
The MIMIC mqttrec and mqttconv utilities work together to transform real-world MQTT client and broker network communications into customizable MIMIC MQTT simulations.
To create a default MQTT simulation, you record MQTT sessions from real-world publisher or subscriber clients (eg. sensors) with the mqttrec utility. This tool works as a proxy between MQTT client and broker, and saves commands and responses in a temporary MQTT session file. Alternatively, you can record from a previously captured packet capture (PCAP) file..
The usage is as below:
mqttrec --help Usage: obj.Linux/mqttrec [--help, -h] print this usage message or --out, -o output file Recording output file --target, -t target Target ip address or host name [--port, -p target port] Target port [--start, -S start] Packet number to start recording from [--stop, -E stop] Ending packet number [--file, -f input file] pcap input file [--directory, -D path] default path combined with filenames [--localport, -l port] Local port [--ipv4, -4] ipv4 (default) [--ipv6, -6] ipv6
The MQTT client to be recorded (ie. the real sensor) should be configured to connect to the MIMIC host system, where mqttrec is running, as its MQTT broker. The mqttrec application will forward mqtt traffic to the broker and broker traffic back to the client.
The MQTT session will be recorded and written to the output file specified. Recorded sessions can be converted by mqttconv into MIMIC MQTT simulations.
mqttconv --help INFO 11/16.15:25:21 - MQTTCONV starting Usage: mqttconv [--help, -h] print this usage message or [--out, -o output file name] mqttconv base output name (no extension) [--in, -i input file] mqttrec input [--path, -p directory path] override default path [--single, -s single output stream] combine MQTT CONNECTs [--multiple, -m multiple output streams] MIMIC cfg file for each CONNECT
mqttconv will generate two types of files, a MIMIC MQTT agent configuration file (.cfg) and script files as detailed below.
mqttconv will process MQTT CONNECT, PUBLISH and SUBSCRIBE commands into a format that can be read by the MIMIC MQTT agent. The MQTT CONNECT information can be further customized using the MIMIC Configure Agents dialog tab.
The stream of PUBLISH commands is evaluated for periodic repetitions that can be consolidated into a single MIMIC MQTT simulation configuration file message statement.
For non-JSON mqtt payloads only identical payloads are consolidated. When a JSON payload is detected the schema is extracted and compared against the other JSON payload schema. If the schema match the timing is used to determine the period of messages.
Currently the MQTT command times are averaged to create the interval value in the MIMIC MQTT configuration message statement.
When a JSON payload is identified several support files are created to facilitate customizing the MIMIC MQTT simulation:
Several general TCL scipts are generated to provide examples of how a MIMIC mqtt simulation can be customized using TCL scripting:
The behavior of the MQTT simulation (data to be published by the simulated sensor, as well as subscriptions) is specified by the MQTT configuration file(s) loaded into the agent instance. This configuration allows to control a simulated MQTT session from establishment until disconnect.
This section documents the global parameters for this client. All of these are optional with specified defaults.
Parameter | Description |
---|---|
init |
There can be zero or more initialization scripts for this simulation. Currently it supports
These can be used to initialize agent store variables, start timer scripts, etc. |
This section documents the connect-time parameters for this client. All of these are optional with specified defaults.
Parameter | Description |
---|---|
will-topic |
The Will Topic as detailed in section 3.1.2.5 and 3.1.3.2 of the MQTT spec . If this option is not specified, no Will Message will be published. |
will-message |
The Will Message payload as detailed in section 3.1.2.5 and 3.1.3.3 of the MQTT spec . If this value is the name of an action script, then the payload will be dynamically generated. |
will-qos |
The Will QOS as detailed in section 3.1.2.5 and 3.1.2.6 of the MQTT spec . The default is 0. |
will-retain |
The Will Retain as detailed in section 3.1.2.5 and 3.1.2.7 of the MQTT spec . The default is 0. |
will-properties |
The will-properties block specifies MQTT 5 Will Properties to be sent with the CONNECT message (optional). This is an example of the properties block:
will-properties = { Will-Delay-Interval = 20 Payload-Format-Indicator = 1 Message-Expiry-Interval = 123 Response-Topic = some-will-topic User-Property = { name = willprop7 value = some value 7 } User-Property = { name = willprop8 value = some value 8 } } |
clean-session |
The Clean Session flag as detailed in section 3.1.2.4 of the MQTT spec . The default is 0. |
keepalive |
The Keep Alive time as detailed in section 3.1.2.10 of the MQTT spec . The default is 60. |
on-disconnect |
This parameter specifies the action script to invoke on receiving a DISCONNECT event on the connection. If absent, then no action is taken on disconnect. |
on-error |
This parameter specifies the action script to invoke on receiving an error event on the connection. If absent, then a default error message is printed in the log. These global variables are passed into the action script:
|
properties |
The properties block specifies MQTT 5 CONNECT Properties to be sent with the CONNECT message (optional). This is an example of the properties block:
properties = { Session-Expiry-Interval = 0xFFFFFFFF Receive-Maximum = 10 Maximum-Packet-Size = 100 Topic-Alias-Maximum = 5 } |
This section documents the PUBLISH message to be sent to the broker as detailed in section 3.3 of the MQTT spec . There can be multiple different messages published from this client. All parameters are mandatory unless specified. Most parameters can be changed at run-time as detailed below.
Parameter | Description | ||||||
---|---|---|---|---|---|---|---|
topic |
The topic that all messages of this type are published to. If this value is the name of an action script, then the action script defines the topic, allowing to make the topic unique for the particular agent instance. | ||||||
interval |
The interval in msec that this message is generated. | ||||||
count |
The count of messages to be generated (optional). If this parameter is absent, then the message is generated indefinitely. | ||||||
qos |
The QOS that this message is sent at, as detailed in this section (optional). By default it is 0. | ||||||
data |
The data section specifies the payload of the message. These are the alternatives:
|
Most of these configurables can be changed at runtime with the mimic agent protocol msg MQTT client message set command above .
This section documents subscriber features of the MQTT client as detailed in section 3.8 of the MQTT spec .
Parameter | Description |
---|---|
topic |
The topic that the subscriber wants to receive. If it is the name of an action script, then that topic is dynamically generated at agent startup time and can be unique for the agent. |
action |
The name of an action script to be invoked upon receipt of PUBLISH messages from the broker on the subscribed topic. These global variables are passed into the action script with information from the PUBLISH message:
An error return, eg. with a "return -code error" statement from this script results in the PUBLISH to be discarded (and not acknowledged if QOS > 1). |
Most of these configurables can be changed at runtime with the mimic agent protocol msg MQTT client subscribe set command above .
This section documents the TLS configuration file.
Parameter | Description |
---|---|
certificate |
The client certificate file. The sample command
openssl req -x509 -newkey rsa:2048 -keyout key.pem -out cert.pem -days 365 -nodescreated the certificate and key file that is shipped in config/mqtt/client.crt . For details see this primer. This command diagnoses the SSL connection to a server at IP address IPADDR and port PORT (usually 8883) for a Certificate Authority specified in CAFILE:
openssl s_client -connect IPADDR:PORT -CAfile CAFILE |
key | The client private key file. |
Click here for the compatibility document. If you get an error opening this URL, you need to download the optional MQTT Compatibility documentationupdate package with the Update Wizard.