Linux Installation Instructions

  1. Table of Contents

  2. Overview

    MIMIC runs on any version of Linux with a kernel version greater than 2.0, but preferably Red Hat Linux Fedora Core 19 or later (kernel version 3.14 or later) or Red Hat Enterprise Linux 7. MIMIC runs on newer versions of Linux (Fedora 19 or later) in 64-bit mode.

    Although we have reports of MIMIC running on a variety of distributions (such as Debian, SuSE, Caldera, etc), the latest release has been tested at Gambit to run out-of-the-box with

    • Fedora 19 through 30
    • Enterprise Linux 7, 8
    • CENTOS 7, 8

  3. Graphical User Interface

    Even though MIMIC can be run entirely without a graphical user interface (GUI), using a GUI initially will speed your tasks significantly. The native MIMICview GUI is a X Windows client and requires an X server either on the local desktop or remotely.

    Out of the miriad of Linux distributions, your fastest path is to install MIMIC on a Linux desktop variant like

    To display on Microsoft Windows we have had success with the Cygwin/X or MobaXterm X servers, but other implementations like Xming do not work as well. On headless machines you can run a RDP implementation such as Xrdp to display on Windows desktops.

    Additionally, any window manager running on the desktop will impact the behavior of the MIMICview GUI. For example, keyboard focus is entirely dependent on the window manager. If you use keyboard control of MIMICview, the window manager focus policy impacts the behavior after dismissing dialogs.

  4. Swap Space

    MIMIC is a specially memory-intensive application, and it needs plenty of RAM and swap space for the more complex device simulations. We recommend an absolute minimum 1 GB of RAM and 2 GB of swap space to start. Verify that you have enough swap space with the following command from any shell:

    cat /proc/meminfo

    It should display something like

    % cat /proc/meminfo
    MemTotal:      1000448 kB
    MemFree:         37100 kB
    ...
    SwapTotal:     1000448 kB
    SwapFree:      1000448 kB
    ...
    

    This system has 1 GB of RAM (MemTotal), and 1 GB of swap space (SwapTotal).

    You should as a rule of thumb have twice the swap space as your physical memory.

    If you need more swap space, you can easily create it with the mkswap(8) and swapon(8) commands. A 2 GB swap file on the local filesystem can be created with the following commands:

    # dd if=/dev/zero of=SWAP bs=1M count=2048
    2048+0 records in
    2048+0 records out
    2147483648 bytes (2.1 GB) copied, 63.9475 s, 33.6 MB/s
    
    #  mkswap SWAP
    Setting up swapspace version 1, size = 2147483648 bytes
    # swapon [ABSOLUTE-PATH-TO]/SWAP
    

    Different versions of Linux on Intel systems impose different limits on the amount of virtual memory accessible for the MIMIC Simulator. If you run 64-bit Linux the VM limits are of no concern, but on 32-bit systems the absolute limit is below 4 GB due to 32-bit addressing limitations. We have run MIMIC on common Red Hat distributions with upto approximately 3 GB of virtual memory.

  5. 1000/2000/5000/10000 agent support

    You need to perform an extra kernel configuration step on Linux prior to running MIMIC with a large numbers of agents, to increase the global file descriptor table. Eg. to enable 5000 agents, we recommend:

    # cat /proc/sys/fs/file-max
    4096
    # echo "8192" > /proc/sys/fs/file-max
    # cat /proc/sys/fs/file-max
    8192
    

    To apply this parameter permanently, you need to add this command to the boot startup files (eg. in /etc/rc*). Contact your MIS department on this issue.

  6. 20000/50000/100000 agent support

    You can run 20000 and more agents only on 64-bit Linux, and need to apply the appropriate configuration step detailed in the previous section.

    For MIMIC modules (like MQTT) that require ephemeral ports, you need to be aware about system configuration parameters that limit the maximum number of ports .

  7. Yellow Pages with large configurations

    If you are running MIMIC on a system that is using Yellow Pages for name resolution, then you may encounter the following problem: the Yellow Pages client may lock up if you are running a large-scale configuration. The symptoms include delayed command response, error messages in the system log of the form ypbind: YPBINDPROC_DOMAIN: Domain not bound.

    This is likely due to ypbind using broadcast to find the Yellow Pages server, as configured in the file /etc/yp.conf with this entry

    domain SOMETHING broadcast
    

    To solve the problem, comment out the broadcast line and specify an explicit Yellow Pages server, with an entry such as

    ypserver 192.9.200.1
    
  8. Performance impact of certain daemons

    Linux is a general-purpose operating system, and as such runs a variety of daemons to provide services. Certain daemons interfere with the operation of MIMIC. This is a partial list, if you encounter any others, please let us know:

    • avahi-daemon
      The avahi-daemon hogs the CPU for large configurations, because it tries to configure DNS entries for the configured IP aliases.

    • NetworkManager
      The NetworkManager hogs the CPU trying to maintain all those IP aliases that get created and removed when starting and stopping agents.

    • dnsmasq
      You can stop this since you don't need to run a DNS server.

    • ibus-daemon
      We have seen GUI actions slow down considerably with this running. You can stop this if you don't need multilingual input.

    To diagnose the problem, look at CPU consumption of running processes. To solve the problem, stop the daemon.

  9. IPv6 Support

    IPv6 is supported on all Linux platforms that MIMIC runs on.

    Prior to running agents with IPv6 addresses, the loadable IPv6 protocol kernel module needs to be installed. This is a boot-time configurable, eg.

    # ifconfig -a
    eth0      Link encap:Ethernet  HWaddr 00:A0:24:07:C7:8A  
              inet addr:192.9.200.23  Bcast:192.9.200.255  Mask:255.255.255.0
              UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
              RX packets:2740928883 errors:80 dropped:0 overruns:125 frame:80
              TX packets:2196208374 errors:0 dropped:0 overruns:0 carrier:96
              collisions:51506073 txqueuelen:100 
              RX bytes:286144626 (272.8 Mb)  TX bytes:1540102588 (1468.7 Mb)
              Interrupt:10 Base address:0x220 
    
    lo        Link encap:Local Loopback  
              inet addr:127.0.0.1  Mask:255.0.0.0
              UP LOOPBACK RUNNING  MTU:16436  Metric:1
              RX packets:1836705765 errors:0 dropped:0 overruns:0 frame:0
              TX packets:1836705765 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:0 
              RX bytes:3869230693 (3689.9 Mb)  TX bytes:3869230693 (3689.9 Mb)
    
    # modprobe ipv6
    # ifconfig -a
    eth0      Link encap:Ethernet  HWaddr 00:A0:24:07:C7:8A  
              inet addr:192.9.200.23  Bcast:192.9.200.255  Mask:255.255.255.0
              inet6 addr: fe80::2a0:24ff:fe07:c78a/10 Scope:Link
              UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
              RX packets:2740929031 errors:80 dropped:0 overruns:125 frame:80
              TX packets:2196208477 errors:0 dropped:0 overruns:0 carrier:96
              collisions:51506073 txqueuelen:100 
              RX bytes:286160140 (272.9 Mb)  TX bytes:1540116436 (1468.7 Mb)
              Interrupt:10 Base address:0x220 
    
    lo        Link encap:Local Loopback  
              inet addr:127.0.0.1  Mask:255.0.0.0
              inet6 addr: ::1/128 Scope:Host
              UP LOOPBACK RUNNING  MTU:16436  Metric:1
              RX packets:1836705775 errors:0 dropped:0 overruns:0 frame:0
              TX packets:1836705775 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:0 
              RX bytes:3869232221 (3689.9 Mb)  TX bytes:3869232221 (3689.9 Mb)
    
    sit0      Link encap:IPv6-in-IPv4  
              NOARP  MTU:1480  Metric:1
              RX packets:0 errors:0 dropped:0 overruns:0 frame:0
              TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:0 
              RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)
    
    

    When using ping6 to check connectivity to an IPv6 target, for link-local IPv6 destinations you must supply the -I option, eg.

    % ping6 -I eth0 fe80::a00:20ff:feb0:2780
    PING fe80::a00:20ff:feb0:2780(fe80::a00:20ff:feb0:2780) from fe80::d0:b760:a3f7 eth0: 56 data bytes
    Warning: time of day goes back, taking countermeasures.
    64 bytes from fe80::a00:20ff:feb0:2780: icmp_seq=0 hops=255 time=4.618 msec
    64 bytes from fe80::a00:20ff:feb0:2780: icmp_seq=1 hops=255 time=244 usec
    64 bytes from fe80::a00:20ff:feb0:2780: icmp_seq=2 hops=255 time=235 usec
    
    --- fe80::a00:20ff:feb0:2780 ping statistics ---
    3 packets transmitted, 3 packets received, 0% packet loss
    round-trip min/avg/max/mdev = 0.235/1.699/4.618/2.064 ms
    

    When using netstat or route to check or configure routing tables, you need to use the -A inet6 command line option, eg.

    # netstat -A inet6 -r -n
    Kernel IPv6 routing table
    Destination                    Next Hop           Flags Metric Ref    Use Iface
    ::1/128                        ::                 U     0      404      0 lo
    3ffe::1/128                    3ffe::1            UC    0      1        1 eth0
    3ffe:bc0:c56:1::1/128          ::                 U     0      0        0 lo
    3ffe:bc0:c56:1::2/128          ::                 U     0      0        0 lo
    fe80::/10                      ::                 UA    256    0        0 eth0
    ff00::/8                       ::                 UA    256    0        0 eth0
    ::/0                           ::                 UDA   256    0        0 eth0
    
    # route add -A inet6 3ffe::/16 eth0
    

    For more details see Peter Bieringer's Linux IPv6 HOWTO document, eg. here.

    Please check effects of duplicate address detection below.

  10. Core Dumps

    To allow core dumps from MIMICD on new versions of Linux, you need to set the ulimit for core dumps to unlimited.

    On Linux versions that change the kernel core pattern which may prevent core dumps, You can inspect the behavior with something like this:

    $ sysctl kernel.core_pattern
    kernel.core_pattern = |/usr/share/apport/apport %p %s %c %P
    

    Now you have the choice of enabling the apport utility or restoring the core dump behavior:

    $ sudo sysctl kernel.core_pattern=core
    kernel.core_pattern = core
    

    The rest of this section only applies to MIMIC before release 9.00.

    The MIMIC Simulator mimicd runs as a setuid-root daemon on Linux. By default, core files are not produced if it terminates abnormally. If you see the mimicd crashing, you need to run it as root to enable core dumps to help us diagnose the problem:

    su
    export MIMIC_IGNORE_SIGSEGV=1
    ulimit -c unlimited
    ./mimicd
    
  11. Filesystem

    If you are setting up a partition on your hard disk to contain MIMIC data, you will want to consider tuning the block and inode allocation of the filesystem. Since MIMIC data typically consists of many small files, on Unix filesystems it uses up a filesystem resource called "inodes". When these run out, you will get error messages such as "file system full", even though the output of df (1) shows plenty of space available.

    To change the inode allocation from the default for the mkfs (8) utility, use the -b command line option to reduce the block-size to 1024 bytes (the smallest possible, since fragments are not supported), and the -i command line option to set the bytes-per-inode to 1024 (to allocate the maximum number of inodes).

    For details see the mkfs or mke2fs man pages or consult your system administrator.

    In modern Linux systems you can use a loop device to configure a file as a virtual disk to be formatted and mounted with the preferred parameters. This is the recommended procedure to setup a 10GB filesystem, eg. as root user in /usr/local:

    # pwd
    /usr/local
    
    # dd if=/dev/zero of=mimic.img bs=1MB count=10000
    10000+0 records in
    10000+0 records out
    10000000000 bytes (10 GB) copied, 119.391 s, 83.8 MB/s
    
    # ls -la mimic.img
    -rw-r--r-- 1 root root 10000000000 Mar  7 13:47 mimic.img
    
    # mkfs -t ext3 -b 1024 -i 1024 mimic.img
    mke2fs 1.42.7 (21-Jan-2013)
    mimic.img is not a block special device.
    Proceed anyway? (y,n) y
    Discarding device blocks: done
    Filesystem label=
    OS type: Linux
    Block size=1024 (log=0)
    Fragment size=1024 (log=0)
    Stride=0 blocks, Stripe width=0 blocks
    9762144 inodes, 9765624 blocks
    488243 blocks (5.00%) reserved for the super user
    First data block=1
    Maximum filesystem blocks=76995072
    1194 block groups
    8184 blocks per group, 8184 fragments per group
    8176 inodes per group
    Superblock backups stored on blocks:
            8185, 24553, 40921, 57289, 73657, 204601, 220969, 401017, 662905,
            1023001, 1988713, 2807113, 5115001, 5966137
    
    Allocating group tables: done
    Writing inode tables: done
    Creating journal (32768 blocks): done
    Writing superblocks and filesystem accounting information: done
    
    # mount -t auto -o loop /usr/local/mimic.img /usr/local/mimic
    
    # df /usr/local/mimic
    Filesystem     1K-blocks  Used Available Use% Mounted on
    /dev/loop2	 7289346  3983   6797120   1% /usr/local/mimic
    
    

    By setting it up as root, the permissions prevent accidental corruption of the mimic.img file.

    If you need to later increase this virtual disk you can use the resize2fs utility, eg. to add 10GB, first unmount the image, then resize and mount:

    # umount /usr/local/mimic.img
    
    # dd if=/dev/zero bs=1MB of=mimic.img conv=notrunc oflag=append count=10000
    10000+0 records in
    10000+0 records out
    10000000000 bytes (10 GB) copied, 152.072 s, 65.8 MB/s
    
    # e2fsck -f mimic.img
    e2fsck 1.42.9 (28-Dec-2013)
    Pass 1: Checking inodes, blocks, and sizes
    Pass 2: Checking directory structure
    Pass 3: Checking directory connectivity
    Pass 4: Checking reference counts
    Pass 5: Checking group summary information
    mimic.img: 518409/9762144 files (1.3% non-contiguous), 9276883/9765624 blocks
    
    # resize2fs mimic.img
    resize2fs 1.42.9 (28-Dec-2013)
    Resizing the filesystem on mimic.img to 19531248 (1k) blocks.
    The filesystem on mimic.img is now 19531248 blocks long.
    
    # mount -t auto -o loop /usr/local/mimic.img /usr/local/mimic
    
    # df /usr/local/mimic
    Filesystem     1K-blocks  Used Available Use% Mounted on
    /dev/loop2      14613461 6800269   6836706  50% /usr/local/mimic
    
    

  12. NFS

    We recommend not running MIMIC on NFS mounted file systems but on a local disk filesystem as detailed above. When the simulator accesses persistent storage for its configuration or simulation data, NFS could slow things down considerably.

  13. Shared libraries

    If you are getting the error error while loading shared libraries while running MIMIC on a newer version of Linux, you need to install the compatibility libraries as detailed in this Frequently Asked Question .

  14. 64-bit support

    By default, MIMIC runs in 64-bit mode (and requires 64-bit x86_64 Linux on 64-bit systems) on RHEL 7, Fedora 19 or later.

    32-bit support has been discontinued.

  15. Bonded Interfaces

    MIMIC supports NIC bonding in Linux, that is we have tested that agents configured on bonded NICs perform correctly, although we have not seen much performance benefit.

  16. ARP Cache

    The ARP cache on a system resolves IP addresses to MAC addresses. In the case of a large number of MIMIC agents, the Linux ARP cache garbage collection mechanism on the system running the management application can interfere interoperability with MIMIC, limiting performance or even failing. This is the case if the management station and MIMIC system are on the same LAN (without routers in between). In the case of intermediate routers, the issue will apply to the router connected to the MIMIC system, but usually ARP tables are configured adequately on such systems.

    The major clue will be kernel messages of the form

    kernel: Neighbour table overflow.
    

    In this case, you need to configure garbage collection on the other system to allow more ARP table entries, eg. as detailed at this page, eg. to allow 100000 entries

    echo "100000" > /proc/sys/net/ipv4/neigh/default/gc_thresh3
    echo "100000" > /proc/sys/net/ipv4/neigh/default/gc_thresh2

  17. Known Problems

    We are constantly working to remove limitations, but currently we know of the following:

    • Linux kernels upto 2.4.2 have problems dumping core of multi-threaded applications, including MIMIC. There is no such limitation on the 2.6.x kernels in Red Hat Linux 9 and later.

    • Red Hat Linux 6.2 has problems with logging to automounted filesystems. In the default configuration, this will impact only the Trap Wizard if your private area is automounted. Red Hat Linux versions 7.0 and later have no such problems.

    • If you use non-standard network masks with agents in the same network, stopping the agent that was started first will remove all the addresses in the same network. This is a limitation only on Linux kernel version 3.x, our tests show it seems fixed with kernel 4.x after Fedora 20.

    • You can only start upto 200 agents with IPv6 addresses on Red Hat Linux 9.x . After that, the system hangs. There is no such limitation on Fedora Core 3 and later, or Red Hat Enterprise Linux 4 and later.

    • As of kernel 2.6.18 there is a configurable limit of 4096 IPv6 route entries, which limits the number of IPv6 aliases to +/- 4096. This can only be overcome by increasing the number of route entries from a root shell:
      # sysctl net.ipv6.route.max_size
      net.ipv6.route.max_size = 4096
       
      # sysctl -w net.ipv6.route.max_size=8192
      net.ipv6.route.max_size = 8192
      

    • Starting agents with IPv6 addresses on RedHat Linux 9 is slow (approx. 10 agents per second) due to default duplicate address detection. There is no such limitation on later versions of Red Hat Linux (Fedora or Enterprise).

      In order to disable duplicate address detection, you can change the dad_transmits kernel configurable as follows (assuming that eth0 is your network interface):

      # cat /proc/sys/net/ipv6/conf/eth0/dad_transmits
      1
      # echo "0" > /proc/sys/net/ipv6/conf/eth0/dad_transmits
      # cat /proc/sys/net/ipv6/conf/eth0/dad_transmits
      0
      

      Still, while it takes a couple of seconds to start 1000 agents with only IPv4 addresses configured on a common PC, it takes on the order of 140 seconds with just one IPv6 alias per agent with dad_transmits set to 0, and on the order of 400 seconds with dad_transmits set to 1. Optimistic DAD settings don't seem to help performance.

      On newer Linux versions (Fedora 25 and newer), duplicate address detection can interfere with the startup of agents with IPv6 addresses, leading to errors in the MIMIC log containing

            cannot bind receive IP address IPV6-ADDRESS port 161
            bind: Cannot assign requested address
      

      for some IPv6 address IPV6-ADDRESS.

      The workaround is to disable duplicate address detection with

      sysctl -w net.ipv6.conf.INTERFACE.dad_transmits=0
      sysctl -w net.ipv6.conf.INTERFACE.accept_dad=0
      

      for your network interface INTERFACE. Eg. if your network is eth0, you would use

      sysctl -w net.ipv6.conf.eth0.dad_transmits=0
      sysctl -w net.ipv6.conf.eth0.accept_dad=0
      

      On older Linux releases, although DAD does not make MIMIC fail, it does slows down starting many agents. We have sped up agent startup at least a factor of 10 by disabling DAD in our tests.

    • If you want to run MIMIC inside virtual machine software such as VMWare or Xen, please contact Gambit Communications Technical Support (support@gambitcomm.com).

    • Although MIMIC can run inside containers like Docker, there are limitations in the Docker network plugins that don't allow MIMIC to create new IP aliases. So, if all your simulated devices can run on the IP alias for the Docker instance (eg. certain MQTT Simulator scenarios), then you are ok running inside a container.

    • For MIMIC older than version 20.00 on Linux releases older than RHEL 7 or Fedora 19, Linux on AMD64 systems needs 32-bit compatibility libraries installed for the 32-bit MIMIC executables to run. A tell-tale symptom of this problem would be:

      
      % ./mimicd
      -bash: ./mimicd: /lib/ld-linux.so.2: bad ELF interpreter: No such file or directory
      
      % file mimicd
      mimicd: setuid ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV),
      for GNU/Linux 2.6.9, dynamically linked (uses shared libs), stripped
      
      % ldd mimicd
      not a dynamic executable
      

      On Ubuntu, to install these libraries, as root run

      # apt-get install ia32-libs
      

      On Fedora distributions, to install these libraries, as root run

      # yum whatprovides /lib/ld-linux.so.2
      Loaded plugins: refresh-packagekit
      Importing additional filelist information
      glibc-2.10.1-2.i686 : The GNU libc libraries
      Repo        : fedora
      Matched from:
      Filename    : /lib/ld-linux.so.2
      ...
      
      # yum install glibc-2.10.1-2.i686
      ...
      

    • On Fedora Core 6, Java or Perl MIMIC clients will fail to connect to the local MIMIC daemon with an error such as

      ERROR: socket - cannot connect. localhost, 9797
      
      The root cause of this problem is that the /etc/hosts file by default defines localhost with an IPv6 address ::1. Changing this back to an IPv4 address 127.0.0.1 as it has always been, or defining a new entry, will solve this problem. This has been fixed in Fedora 7.

    • If you are running intensive TCP performance tests that open/close many connections in a short amount of time, then you will run into the well-known TIME_WAIT condition: after a while the agents will fail to respond to new connections even though the system seems idle. You can verify this condition with

      % netstat -a -n | grep TIME_WAIT
      

      and it would list many lines. For the details and solutions, search on the Internet for Linux TIME_WAIT, for example this page.

    • Firewalls interfere with the functioning of MIMIC. We have seen all kinds of limitations with different firewalls, among them:

      • no connectivity to simulated devices due to restrictive filters
      • throttling of rates via Linux Netfilter, resulting in spurious connection drops and rejections. This usually results in networking errors all over the system, like

        do_ypcall: clnt_call: RPC: Unable to send; errno = Operation not permitted
        

        or in /var/log/messages:

        Mar 24 15:29:01 dmb kernel: [970079.411385] net_ratelimit: 50 callbacks suppressed
        Mar 24 15:29:01 dmb kernel: [970079.411388] nf_conntrack: table full, dropping packet
        

        If you want to diagnose connectivity problems, turn off your firewall first. This can be done on RHEL systems according to these instructions.

    • Network interfaces with long labels interfere with MIMIC. If the active network interface to be used by MIMIC, eg. as shown with

      % ip link show
      1: lo:  mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default 
          link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
      2: eno16780032:  mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
          link/ether d4:be:d9:cb:58:9f brd ff:ff:ff:ff:ff:ff
      
      % netstat -i
      Kernel Interface table
      Iface      MTU    RX-OK RX-ERR RX-DRP RX-OVR    TX-OK TX-ERR TX-DRP TX-OVR Flg
      lo       65536  1979263      0      0 0       1979263      0      0      0 LRU
      eno16780032 1500 382210306      0  17720 0      542194465      0      0      0 BMRU
      
      

      (in this case eno16780032) is more than 10 characters long, then it can overflow the 16-character limit for network interface names for the IP aliases that MIMIC creates, causing ERROR messages when agents start.

      This is an OS limit, and the only workaround is to rename your network interface.

For any additional support, please contact Gambit Communications Technical Support (support@gambitcomm.com).