Ensuring Success of IoT Projects

Ensuring Success of IoT Projects

MIMIC IoT Simulator is designed to assist in implementation, testing, proof-of-concept, deployment, training of Internet of Things solutions. By employing simulation early in the project, risks associated with failure are reduced at a fraction of the cost.

The Problem

The Internet of Things presents unique challenges that make or break implementation projects. The difficulty and dismal success rate of IoT projects is repeatedly documented in recent news articles and analyst reports such as

• Why IoT Projects Fail Beecham Research, Q1 2020

  "...from a base of 25,000 IoT adopters and buyers. This survey yielded a split between those IoT projects considered to have been successful (42%) and those considered to have been unsuccessful (58%) ..."

  "We therefore consider it highly likely that the real rate of failure in the market is considerably higher and may well be in the region of 70%."

  "Scalability when moving projects from PoC to implementation is often identified as a key challenge - a project catering for a few connected devices represents a completely different challenge when grown to several thousands or tens of thousands of connected devices."

• Microsoft: 30% of IoT projects fail in the proof-of-concept stage VentureBeat, 7/30/2019

  "Microsoft reports that complexity and technical challenges are an IoT dealbreaker for 38% of the decision-makers surveyed...

  "... 30% of respondents say their IoT projects failed in the proof-of-concept stage"

• A Guidance Framework for Testing IoT Solutions Gartner Research, 03/05/2018

  "... An IoT solution may be made up of hundreds or thousands of devices. To test all of the devices in their real environments may be prohibitively expensive or dangerous. However, you also need to ensure that your IoT platform and back-end systems can handle the load of all of those devices and correctly send and receive data as necessary."

• Nearly 3/4th of Internet of Things projects are failing: Cisco The Economic Times, 24/05/2017

  "60 per cent of IoT initiatives stall at the Proof of Concept (PoC) stage and only 26 per cent of companies have had an IoT initiative that they considered a complete success"

  "Sixty per cent of respondents stressed that IoT initiatives often look good on paper but prove much more difficult than anyone expected."

• Real-Time Discovery, Visibility and Control Are Critical for IoT Security Gartner Research, 03/11/2016

The Solution

Using an IoT Simulator early in the project aids in mitigation of risks with failure later on, as documented in these reports:

• IoT Simulation: The Test for Success insight.tech, August 2019

  "... As the number of sensors in the network increases, so does the chance of encountering an unexpected event."

  "... MIMIC supports dynamic rules that can adjust a simulation and introduce new parameters in mid-run."

• Whitepaper: 3 ways to evaluate IoT platforms MachNation, January 2019

  "Performance testing is the only kind of testing that can evaluate an IoT platform’s scalability."

  "Performance testing can simulate very large numbers of devices and data throughput while measuring performance degradation of the full-stack IoT solution as load increases."

• IoT Simulation - Why Simulation is a Critical Component to IoT Systems Leverege, December 2016

  "Simulation is critical across industries like finance, engineering, the military, and more. IoT simulation is just as critical." "Rapidly prototype and iterate before a product goes to market to ensure the highest quality solution and best user experience possible."

Specifically, IoT Simulator solves the following problems:

Security

The Internet of Things opens up many vectors for security vulnerabilities as detailed in RFC 8576.

Vulnerabilities in all stages of a IoT device's life cycle include malware baked in during manufacturing, or patched while operating by exploiting zero-day vulnerabilities, specially after the manufacturer's support of the old device is discontinued (end-of-life). This malware usually causes the IoT device to deviate from its intended function for some nefarious purpose.

Part of any IoT Testing and Proof of Concept (PoC) includes addressing security concerns by adding security monitoring solutions to prevent intrusions, malware, etc in order to prevent failure of the IoT project.

While preventing malware through authentication, authorization and privacy is a first defense, the IoT monitoring solution should detect behavior that is not "normal". A usual test scenario then consists in reproducing cases of IoT devices that deviate from their expected behavior. Unless you have a lab full of hacked devices, this is not easy to do.

MIMIC IoT Simulator is designed to easily recreate scenarios meant to test your IoT monitoring solution for common hacking scenarios, such as misbehaving IoT devices (eg. hacked devices sending unusual Internet traffic, or accessing unauthorized resources), incorrectly configured firewall or load-balancing rules, reported common vulnerabilities and exposures (CVE).

The "normal" behavior of an IoT device can be characterized by the network traffic it emits and the resources it accesses. Monitoring solutions can learn this behavior and alert if it deviates from this pattern. MIMIC can control any simulated device to behave differently at any point in time, and can easily create different behaviors on demand. Thus, the monitoring solution can be exercised to prove that it handles certain scenarios, such as higher traffic rates, network traffic to different destinations and access to restricted resources. Since the simulator creates reproducible scenarios, it can be part of regression tests supporting an agile development cycle.

Scalability

While implementing your MQTT-based application's back-end processing, such as real-time archiving, analytics, edge-processing, graphing, how do you verify that it will store and process all received telemetry messages, with no messages missing, no extra messages, no bytes altered, in the correct order? Furthermore, it is really hard to make sure that it works at required scale and speeds, eg. at millisecond granularity.

You need to come up with a performance test that goes beyond simple load testing and handles each of the test requirements (message integrity, sequencing and frequency).

Contrary to the prevailing testing methodologies, where either dummy MQTT sessions are opened (not doing anything other than maintaining the session with PING responses), or the vast majority of clients subscribe rather than publish, each of the clients should represent a sensor publishing telemetry and possibly an actuator accepting commands. To test the actual performance of the topic switching, you should add your expected number of subscribers on the expected topic hierarchies.

Furthermore, IoT simulators offer a testing methodology such as outlined in our blog post MQTT performance methodology using MIMIC MQTT Simulator to minimize the interference between the test equipment and the system under test.

With this fast turnaround or a simulator you can now do timely investigation (ie. way ahead of deployment) of the real-time performance impact of other variables such as:

1. unencrypted vs. TLS

2. varying message frequency

3. varying payload sizes with arbitrary content (JSON, binary, etc)

4. different QOS levels

5. varying topic hierarchies

6. varying subscription levels

7. disconnect / reconnect rates

8. impact on end-to-end latency

Next Steps

If you would like to share how you are benefiting from MIMIC or if there is an aspect of MIMIC that you would like discussed, please let us know at marketing@gambitcomm.com.