Linux Installation Instructions

Table of Contents
Overview
MIMIC runs on any version of Linux with a kernel version greater than 2.0, but preferably Red Hat Linux Fedora Core 19 or later (kernel version 3.14 or later) or Red Hat Enterprise Linux 7. MIMIC runs on newer versions of Linux (Fedora 19 or later) in 64-bit mode.
Although we have reports of MIMIC running on a variety of distributions (such as Debian, SuSE, Caldera, etc), the latest release has been tested at Gambit to run out-of-the-box with
- Fedora 19 through 30
- Enterprise Linux 7, 8
- CENTOS 7, 8
Back to Top
Graphical User Interface
Even though MIMIC can be run entirely without a graphical user interface (GUI), using a GUI initially will speed your tasks significantly. The native MIMICview GUI is a X Windows client and requires an X server either on the local desktop or remotely.
Out of the miriad of Linux distributions, your fastest path is to install MIMIC on a Linux desktop variant like
- RedHat Enterprise Linux Workstation, or
- Fedora Workstation, or
- CentOS (make sure to install Gnome), or
- Ubuntu Desktop, or
- any other variant that includes an X-based desktop.
To display on Microsoft Windows we have had success with the Cygwin/X or MobaXterm X servers, but other implementations like Xming do not work as well. On headless machines you can run a RDP implementation such as Xrdp to display on Windows desktops.
Additionally, any window manager running on the desktop will impact the behavior of the MIMICview GUI. For example, keyboard focus is entirely dependent on the window manager. If you use keyboard control of MIMICview, the window manager focus policy impacts the behavior after dismissing dialogs.
Back to Top
Swap Space
MIMIC is a specially memory-intensive application, and it needs plenty of RAM and swap space for the more complex device simulations. We recommend an absolute minimum 1 GB of RAM and 2 GB of swap space to start. Verify that you have enough swap space with the following command from any shell:
cat /proc/meminfo
It should display something like
```
% cat /proc/meminfo
MemTotal:      1000448 kB
MemFree:         37100 kB
...
SwapTotal:     1000448 kB
SwapFree:      1000448 kB
...
```
This system has 1 GB of RAM (MemTotal), and 1 GB of swap space (SwapTotal).
You should as a rule of thumb have twice the swap space as your physical memory.
If you need more swap space, you can easily create it with the mkswap(8) and swapon(8) commands. A 2 GB swap file on the local filesystem can be created with the following commands:
```
# dd if=/dev/zero of=SWAP bs=1M count=2048
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB) copied, 63.9475 s, 33.6 MB/s

#  mkswap SWAP
Setting up swapspace version 1, size = 2147483648 bytes
# swapon [ABSOLUTE-PATH-TO]/SWAP
```
Different versions of Linux on Intel systems impose different limits on the amount of virtual memory accessible for the MIMIC Simulator. If you run 64-bit Linux the VM limits are of no concern, but on 32-bit systems the absolute limit is below 4 GB due to 32-bit addressing limitations. We have run MIMIC on common Red Hat distributions with upto approximately 3 GB of virtual memory.
Back to Top
1000/2000/5000/10000 agent support
You need to perform an extra kernel configuration step on Linux prior to running MIMIC with a large numbers of agents, to increase the global file descriptor table. Eg. to enable 5000 agents, we recommend:
```
# cat /proc/sys/fs/file-max
4096
# echo "8192" > /proc/sys/fs/file-max
# cat /proc/sys/fs/file-max
8192
```
To apply this parameter permanently, you need to add this command to the boot startup files (eg. in /etc/rc*). Contact your MIS department on this issue.
Back to Top
20000/50000/100000 agent support

You can run 20000 and more agents only on 64-bit Linux, and need to apply the appropriate configuration step detailed in the previous section.
For MIMIC modules (like MQTT) that require ephemeral ports, you need to be aware about system configuration parameters that limit the maximum number of ports .

Back to Top
Yellow Pages with large configurations
If you are running MIMIC on a system that is using Yellow Pages for name resolution, then you may encounter the following problem: the Yellow Pages client may lock up if you are running a large-scale configuration. The symptoms include delayed command response, error messages in the system log of the form ypbind: YPBINDPROC_DOMAIN: Domain not bound.
This is likely due to ypbind using broadcast to find the Yellow Pages server, as configured in the file /etc/yp.conf with this entry
```
domain SOMETHING broadcast
```
To solve the problem, comment out the broadcast line and specify an explicit Yellow Pages server, with an entry such as
```
ypserver 192.9.200.1
```
Back to Top
Performance impact of certain daemons
Linux is a general-purpose operating system, and as such runs a variety of daemons to provide services. Certain daemons interfere with the operation of MIMIC. This is a partial list, if you encounter any others, please let us know:
- avahi-daemon
  The avahi-daemon hogs the CPU for large configurations, because it tries to configure DNS entries for the configured IP aliases.
- NetworkManager
  The NetworkManager hogs the CPU trying to maintain all those IP aliases that get created and removed when starting and stopping agents.
- dnsmasq
  You can stop this since you don't need to run a DNS server.
- ibus-daemon
  We have seen GUI actions slow down considerably with this running. You can stop this if you don't need multilingual input.
To diagnose the problem, look at CPU consumption of running processes. To solve the problem, stop the daemon.
Back to Top

IPv6 Support

IPv6 is supported on all Linux platforms that MIMIC runs on.

Prior to running agents with IPv6 addresses, the loadable IPv6 protocol kernel module needs to be installed. This is a boot-time configurable, eg.

# ifconfig -a
eth0      Link encap:Ethernet  HWaddr 00:A0:24:07:C7:8A  
          inet addr:192.9.200.23  Bcast:192.9.200.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:2740928883 errors:80 dropped:0 overruns:125 frame:80
          TX packets:2196208374 errors:0 dropped:0 overruns:0 carrier:96
          collisions:51506073 txqueuelen:100 
          RX bytes:286144626 (272.8 Mb)  TX bytes:1540102588 (1468.7 Mb)
          Interrupt:10 Base address:0x220 

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:1836705765 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1836705765 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:3869230693 (3689.9 Mb)  TX bytes:3869230693 (3689.9 Mb)

# modprobe ipv6
# ifconfig -a
eth0      Link encap:Ethernet  HWaddr 00:A0:24:07:C7:8A  
          inet addr:192.9.200.23  Bcast:192.9.200.255  Mask:255.255.255.0
          inet6 addr: fe80::2a0:24ff:fe07:c78a/10 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:2740929031 errors:80 dropped:0 overruns:125 frame:80
          TX packets:2196208477 errors:0 dropped:0 overruns:0 carrier:96
          collisions:51506073 txqueuelen:100 
          RX bytes:286160140 (272.9 Mb)  TX bytes:1540116436 (1468.7 Mb)
          Interrupt:10 Base address:0x220 

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:1836705775 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1836705775 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:3869232221 (3689.9 Mb)  TX bytes:3869232221 (3689.9 Mb)

sit0      Link encap:IPv6-in-IPv4  
          NOARP  MTU:1480  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

When using ping6 to check connectivity to an IPv6 target, for link-local IPv6 destinations you must supply the -I option, eg.

% ping6 -I eth0 fe80::a00:20ff:feb0:2780
PING fe80::a00:20ff:feb0:2780(fe80::a00:20ff:feb0:2780) from fe80::d0:b760:a3f7 eth0: 56 data bytes
Warning: time of day goes back, taking countermeasures.
64 bytes from fe80::a00:20ff:feb0:2780: icmp_seq=0 hops=255 time=4.618 msec
64 bytes from fe80::a00:20ff:feb0:2780: icmp_seq=1 hops=255 time=244 usec
64 bytes from fe80::a00:20ff:feb0:2780: icmp_seq=2 hops=255 time=235 usec

--- fe80::a00:20ff:feb0:2780 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max/mdev = 0.235/1.699/4.618/2.064 ms

When using netstat or route to check or configure routing tables, you need to use the -A inet6 command line option, eg.

# netstat -A inet6 -r -n
Kernel IPv6 routing table
Destination                    Next Hop           Flags Metric Ref    Use Iface
::1/128                        ::                 U     0      404      0 lo
3ffe::1/128                    3ffe::1            UC    0      1        1 eth0
3ffe:bc0:c56:1::1/128          ::                 U     0      0        0 lo
3ffe:bc0:c56:1::2/128          ::                 U     0      0        0 lo
fe80::/10                      ::                 UA    256    0        0 eth0
ff00::/8                       ::                 UA    256    0        0 eth0
::/0                           ::                 UDA   256    0        0 eth0

# route add -A inet6 3ffe::/16 eth0

For more details see Peter Bieringer's Linux IPv6 HOWTO document, eg. here.

Please check effects of duplicate address detection below.

Core Dumps
To allow core dumps from MIMICD on new versions of Linux, you need to set the ulimit for core dumps to unlimited.
On Linux versions that change the kernel core pattern which may prevent core dumps, You can inspect the behavior with something like this:
```
$ sysctl kernel.core_pattern
kernel.core_pattern = |/usr/share/apport/apport %p %s %c %P
```
Now you have the choice of enabling the apport utility or restoring the core dump behavior:
```
$ sudo sysctl kernel.core_pattern=core
kernel.core_pattern = core
```
The rest of this section only applies to MIMIC before release 9.00.
The MIMIC Simulator mimicd runs as a setuid-root daemon on Linux. By default, core files are not produced if it terminates abnormally. If you see the mimicd crashing, you need to run it as root to enable core dumps to help us diagnose the problem:
```
su
export MIMIC_IGNORE_SIGSEGV=1
ulimit -c unlimited
./mimicd
```
Back to Top

Filesystem

If you are setting up a partition on your hard disk to contain MIMIC data, you will want to consider tuning the block and inode allocation of the filesystem. Since MIMIC data typically consists of many small files, on Unix filesystems it uses up a filesystem resource called "inodes". When these run out, you will get error messages such as "file system full", even though the output of df (1) shows plenty of space available.

To change the inode allocation from the default for the mkfs (8) utility, use the -b command line option to reduce the block-size to 1024 bytes (the smallest possible, since fragments are not supported), and the -i command line option to set the bytes-per-inode to 1024 (to allocate the maximum number of inodes).

For details see the mkfs or mke2fs man pages or consult your system administrator.

In modern Linux systems you can use a loop device to configure a file as a virtual disk to be formatted and mounted with the preferred parameters. This is the recommended procedure to setup a 10GB filesystem, eg. as root user in /usr/local:

# pwd
/usr/local

# dd if=/dev/zero of=mimic.img bs=1MB count=10000
10000+0 records in
10000+0 records out
10000000000 bytes (10 GB) copied, 119.391 s, 83.8 MB/s

# ls -la mimic.img
-rw-r--r-- 1 root root 10000000000 Mar  7 13:47 mimic.img

# mkfs -t ext3 -b 1024 -i 1024 mimic.img
mke2fs 1.42.7 (21-Jan-2013)
mimic.img is not a block special device.
Proceed anyway? (y,n) y
Discarding device blocks: done
Filesystem label=
OS type: Linux
Block size=1024 (log=0)
Fragment size=1024 (log=0)
Stride=0 blocks, Stripe width=0 blocks
9762144 inodes, 9765624 blocks
488243 blocks (5.00%) reserved for the super user
First data block=1
Maximum filesystem blocks=76995072
1194 block groups
8184 blocks per group, 8184 fragments per group
8176 inodes per group
Superblock backups stored on blocks:
        8185, 24553, 40921, 57289, 73657, 204601, 220969, 401017, 662905,
        1023001, 1988713, 2807113, 5115001, 5966137

Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

# mount -t auto -o loop /usr/local/mimic.img /usr/local/mimic

# df /usr/local/mimic
Filesystem     1K-blocks  Used Available Use% Mounted on
/dev/loop2	 7289346  3983   6797120   1% /usr/local/mimic

By setting it up as root, the permissions prevent accidental corruption of the mimic.img file.

If you need to later increase this virtual disk you can use the resize2fs utility, eg. to add 10GB, first unmount the image, then resize and mount:

# umount /usr/local/mimic.img

# dd if=/dev/zero bs=1MB of=mimic.img conv=notrunc oflag=append count=10000
10000+0 records in
10000+0 records out
10000000000 bytes (10 GB) copied, 152.072 s, 65.8 MB/s

# e2fsck -f mimic.img
e2fsck 1.42.9 (28-Dec-2013)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
mimic.img: 518409/9762144 files (1.3% non-contiguous), 9276883/9765624 blocks

# resize2fs mimic.img
resize2fs 1.42.9 (28-Dec-2013)
Resizing the filesystem on mimic.img to 19531248 (1k) blocks.
The filesystem on mimic.img is now 19531248 blocks long.

# mount -t auto -o loop /usr/local/mimic.img /usr/local/mimic

# df /usr/local/mimic
Filesystem     1K-blocks  Used Available Use% Mounted on
/dev/loop2      14613461 6800269   6836706  50% /usr/local/mimic

NFS

We recommend not running MIMIC on NFS mounted file systems but on a local disk filesystem as detailed above. When the simulator accesses persistent storage for its configuration or simulation data, NFS could slow things down considerably.

Back to Top
Shared libraries

If you are getting the error error while loading shared libraries while running MIMIC on a newer version of Linux, you need to install the compatibility libraries as detailed in this Frequently Asked Question .

Back to Top
64-bit support

By default, MIMIC runs in 64-bit mode (and requires 64-bit x86_64 Linux on 64-bit systems) on RHEL 7, Fedora 19 or later.
32-bit support has been discontinued.

Back to Top
Bonded Interfaces

MIMIC supports NIC bonding in Linux, that is we have tested that agents configured on bonded NICs perform correctly, although we have not seen much performance benefit.

Back to Top
ARP Cache
The ARP cache on a system resolves IP addresses to MAC addresses. In the case of a large number of MIMIC agents, the Linux ARP cache garbage collection mechanism on the system running the management application can interfere interoperability with MIMIC, limiting performance or even failing. This is the case if the management station and MIMIC system are on the same LAN (without routers in between). In the case of intermediate routers, the issue will apply to the router connected to the MIMIC system, but usually ARP tables are configured adequately on such systems.
The major clue will be kernel messages of the form
```
kernel: Neighbour table overflow.
```
In this case, you need to configure garbage collection on the other system to allow more ARP table entries, eg. as detailed at this page, eg. to allow 100000 entries
echo "100000" > /proc/sys/net/ipv4/neigh/default/gc_thresh3 echo "100000" > /proc/sys/net/ipv4/neigh/default/gc_thresh2
Back to Top
Known Problems
We are constantly working to remove limitations, but currently we know of the following:
- Linux kernels upto 2.4.2 have problems dumping core of multi-threaded applications, including MIMIC. There is no such limitation on the 2.6.x kernels in Red Hat Linux 9 and later.
- Red Hat Linux 6.2 has problems with logging to automounted filesystems. In the default configuration, this will impact only the Trap Wizard if your private area is automounted. Red Hat Linux versions 7.0 and later have no such problems.
- If you use non-standard network masks with agents in the same network, stopping the agent that was started first will remove all the addresses in the same network. This is a limitation only on Linux kernel version 3.x, our tests show it seems fixed with kernel 4.x after Fedora 20.
- You can only start upto 200 agents with IPv6 addresses on Red Hat Linux 9.x . After that, the system hangs. There is no such limitation on Fedora Core 3 and later, or Red Hat Enterprise Linux 4 and later.
- As of kernel 2.6.18 there is a configurable limit of 4096 IPv6 route entries, which limits the number of IPv6 aliases to +/- 4096. This can only be overcome by increasing the number of route entries from a root shell:
  # sysctl net.ipv6.route.max_size net.ipv6.route.max_size = 4096 # sysctl -w net.ipv6.route.max_size=8192 net.ipv6.route.max_size = 8192
- Starting agents with IPv6 addresses on RedHat Linux 9 is slow (approx. 10 agents per second) due to default duplicate address detection. There is no such limitation on later versions of Red Hat Linux (Fedora or Enterprise).
  In order to disable duplicate address detection, you can change the dad_transmits kernel configurable as follows (assuming that eth0 is your network interface):
  # cat /proc/sys/net/ipv6/conf/eth0/dad_transmits 1 # echo "0" > /proc/sys/net/ipv6/conf/eth0/dad_transmits # cat /proc/sys/net/ipv6/conf/eth0/dad_transmits 0
  
  Still, while it takes a couple of seconds to start 1000 agents with only IPv4 addresses configured on a common PC, it takes on the order of 140 seconds with just one IPv6 alias per agent with dad_transmits set to 0, and on the order of 400 seconds with dad_transmits set to 1. Optimistic DAD settings don't seem to help performance.
  On newer Linux versions (Fedora 25 and newer), duplicate address detection can interfere with the startup of agents with IPv6 addresses, leading to errors in the MIMIC log containing
  
  cannot bind receive IP address IPV6-ADDRESS port 161 bind: Cannot assign requested address
  
  for some IPv6 address IPV6-ADDRESS.
  The workaround is to disable duplicate address detection with
  
  sysctl -w net.ipv6.conf.INTERFACE.dad_transmits=0 sysctl -w net.ipv6.conf.INTERFACE.accept_dad=0
  
  for your network interface INTERFACE. Eg. if your network is eth0, you would use
  
  sysctl -w net.ipv6.conf.eth0.dad_transmits=0 sysctl -w net.ipv6.conf.eth0.accept_dad=0
  
  On older Linux releases, although DAD does not make MIMIC fail, it does slows down starting many agents. We have sped up agent startup at least a factor of 10 by disabling DAD in our tests.
- If you want to run MIMIC inside virtual machine software such as VMWare or Xen, please contact Gambit Communications Technical Support (support@gambitcomm.com).
- Although MIMIC can run inside containers like Docker, there are limitations in the Docker network plugins that don't allow MIMIC to create new IP aliases. So, if all your simulated devices can run on the IP alias for the Docker instance (eg. certain MQTT Simulator scenarios), then you are ok running inside a container.
- For MIMIC older than version 20.00 on Linux releases older than RHEL 7 or Fedora 19, Linux on AMD64 systems needs 32-bit compatibility libraries installed for the 32-bit MIMIC executables to run. A tell-tale symptom of this problem would be:
  
  % ./mimicd -bash: ./mimicd: /lib/ld-linux.so.2: bad ELF interpreter: No such file or directory % file mimicd mimicd: setuid ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), for GNU/Linux 2.6.9, dynamically linked (uses shared libs), stripped % ldd mimicd not a dynamic executable
  
  On Ubuntu, to install these libraries, as root run
  
  # apt-get install ia32-libs
  
  On Fedora distributions, to install these libraries, as root run
  
  # yum whatprovides /lib/ld-linux.so.2 Loaded plugins: refresh-packagekit Importing additional filelist information glibc-2.10.1-2.i686 : The GNU libc libraries Repo : fedora Matched from: Filename : /lib/ld-linux.so.2 ... # yum install glibc-2.10.1-2.i686 ...
- On Fedora Core 6, Java or Perl MIMIC clients will fail to connect to the local MIMIC daemon with an error such as
  
  ERROR: socket - cannot connect. localhost, 9797
  The root cause of this problem is that the /etc/hosts file by default defines localhost with an IPv6 address ::1. Changing this back to an IPv4 address 127.0.0.1 as it has always been, or defining a new entry, will solve this problem. This has been fixed in Fedora 7.
- If you are running intensive TCP performance tests that open/close many connections in a short amount of time, then you will run into the well-known TIME_WAIT condition: after a while the agents will fail to respond to new connections even though the system seems idle. You can verify this condition with
  
  % netstat -a -n | grep TIME_WAIT
  
  and it would list many lines. For the details and solutions, search on the Internet for Linux TIME_WAIT, for example this page.
- Firewalls interfere with the functioning of MIMIC. We have seen all kinds of limitations with different firewalls, among them:
  
  no connectivity to simulated devices due to restrictive filters
  throttling of rates via Linux Netfilter, resulting in spurious connection drops and rejections. This usually results in networking errors all over the system, like
  
  do_ypcall: clnt_call: RPC: Unable to send; errno = Operation not permitted
  
  or in /var/log/messages:
  
  Mar 24 15:29:01 dmb kernel: [970079.411385] net_ratelimit: 50 callbacks suppressed Mar 24 15:29:01 dmb kernel: [970079.411388] nf_conntrack: table full, dropping packet
  
  If you want to diagnose connectivity problems, turn off your firewall first. This can be done on RHEL systems according to these instructions.
- Network interfaces with long labels interfere with MIMIC. If the active network interface to be used by MIMIC, eg. as shown with
  
  % ip link show 1: lo: mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2: eno16780032: mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000 link/ether d4:be:d9:cb:58:9f brd ff:ff:ff:ff:ff:ff % netstat -i Kernel Interface table Iface MTU RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg lo 65536 1979263 0 0 0 1979263 0 0 0 LRU eno16780032 1500 382210306 0 17720 0 542194465 0 0 0 BMRU
  
  (in this case eno16780032) is more than 10 characters long, then it can overflow the 16-character limit for network interface names for the IP aliases that MIMIC creates, causing ERROR messages when agents start.
  
  This is an OS limit, and the only workaround is to rename your network interface.
Back to Top

For any additional support, please contact Gambit Communications Technical Support (support@gambitcomm.com).

MIMIC® Online Documentation