{"id":13616,"date":"2022-08-03T23:27:03","date_gmt":"2022-08-03T20:27:03","guid":{"rendered":"https:\/\/kifarunix.com\/?p=13616"},"modified":"2024-03-09T21:03:29","modified_gmt":"2024-03-09T18:03:29","slug":"install-malcolm-network-traffic-analysis-tool-on-ubuntu","status":"publish","type":"post","link":"https:\/\/kifarunix.com\/install-malcolm-network-traffic-analysis-tool-on-ubuntu\/","title":{"rendered":"Install Malcolm Network Traffic Analysis Tool on Ubuntu 22.04"},"content":{"rendered":"\n

Follow through this tutorial to learn how to install Malcolm network traffic analysis tool on Ubuntu 22.04. Malcolm<\/a> is a network traffic analysis tool suite for full packet capture artifacts (PCAP files) and Zeek logs. The PCAP files or Zeek logs can be uploaded to Malcolm via browser, forwarded via the forwarders or can capture live traffic, parses and normalize the traffic for visualization via OpenSearch dashboards or Arkime<\/a>.<\/p>\n\n\n\n

Read more about Malcolm network traffic analysis tool and its features on their page.<\/p>\n\n\n\n

Install Malcolm on Ubuntu 22.04<\/h2>\n\n\n\n

Malcolm provides scripts that can be used to easy deploy them on a Linux system.<\/p>\n\n\n\n

There are two ways in which you can obtain the Malcolm installation script.<\/p>\n\n\n\n

    \n
  1. Either by downloading the install.py<\/code> and the malcolm_YYYYMMDD_HHNNSS_xxxxxxx.tar.gz<\/code> from the Malcolm Github releases page.<\/a><\/li>\n\n\n\n
  2. Or by simply cloning Malcolm Github repository.<\/li>\n<\/ol>\n\n\n\n

    In this tutorial, we will use the second method to install Malcolm.<\/p>\n\n\n\n

    System Requirements<\/h3>\n\n\n\n

    Check Malcolm deployment system requirements.<\/a><\/p>\n\n\n\n

    In our environment, these are the system resources we are using;<\/p>\n\n\n\n

    CPU cores;<\/p>\n\n\n\n

    sudo dmidecode -t 4 | grep -i 'core count'<\/code><\/pre>\n\n\n\n
    \tCore Count: 4<\/code><\/pre>\n\n\n\n
    RAM;<\/p>\n\n\n\n
    free -h<\/code><\/pre>\n\n\n\n
                   total        used        free      shared  buff\/cache   available\nMem:            15Gi        14Gi       153Mi       7.0Mi       723Mi       372Mi\nSwap:             0B          0B          0B\n<\/code><\/pre>\n\n\n\n
    Disk Space;<\/p>\n\n\n\n
    df -hT -P \/<\/code><\/pre>\n\n\n\n
    Filesystem     Type  Size  Used Avail Use% Mounted on\n\/dev\/sda1      ext4  150G   14G  131G  10% \/<\/code><\/pre>\n\n\n\n
    Create Malcolm System Account<\/h3>\n\n\n\n
    Create an account to run Malcolm with;<\/p>\n\n\n\n
    useradd -m -d \/opt\/malcolm -s \/bin\/bash -G sudo malcolm<\/code><\/pre>\n\n\n\n
    passwd malcolm<\/code><\/pre>\n\n\n\n
    Get the ID of the user as this is required while configuring Malcolm below.<\/p>\n\n\n\n
    id malcolm<\/code><\/pre>\n\n\n\n
    Sample output;<\/p>\n\n\n\n
    uid=1001(malcolm) gid=1001(malcolm) groups=1001(malcolm),27(sudo)<\/code><\/pre>\n\n\n\n
    Run system update;<\/h3>\n\n\n\n
    Ensure system package cache is up-to-date.<\/p>\n\n\n\n
    sudo apt update<\/code><\/pre>\n\n\n\n
    Install Git package on Ubuntu 22.04;<\/h3>\n\n\n\n
    You will need git to clone the Malcolm Github repository<\/p>\n\n\n\n
    sudo apt install git<\/code><\/pre>\n\n\n\n
    Clone Malcolm Github Repository<\/h3>\n\n\n\n
    Run the command below to clone Malcolm Github repository;<\/p>\n\n\n\n
    su - malcolm<\/code><\/pre>\n\n\n\n
    git clone https:\/\/github.com\/idaholab\/Malcolm<\/code><\/pre>\n\n\n\n
    Install Required Malcolm Packages and Setup System Tunables<\/h3>\n\n\n\n
    Next, change into cloned Malcolm Github repository directory;<\/p>\n\n\n\n
    cd Malcolm<\/code><\/pre>\n\n\n\n
    Malcolm installation scripts are located under scripts directory. Thus, to launch the installation, execute the command below.<\/p>\n\n\n\n
    sudo .\/scripts\/install.py<\/code><\/pre>\n\n\n\n
    Malcolm runs atop Docker. Thus, when the script runs;<\/p>\n\n\n\n
      \n
    • You will be prompted to install required Docker packages. When prompted, accept to proceed with installation.<\/li>\n\n\n\n
    • Once the installation of Docker packages is done, you are prompted on whether to add a non-root user account to Docker group. And by all means, please add your standard user account into the group!<\/li>\n\n\n\n
    • Next, install Docker compose.<\/li>\n<\/ul>\n\n\n\n
      Installing required packages: ['apache2-utils', 'make', 'openssl', 'python3-dialog']\n\n\"docker info\" failed, attempt to install Docker? (Y\/n): y\n\nAttempt to install Docker using official repositories? (Y\/n): y\nInstalling required packages: ['apt-transport-https', 'ca-certificates', 'curl', 'gnupg-agent', 'software-properties-common']\nInstalling docker packages: ['docker-ce', 'docker-ce-cli', 'containerd.io']\nInstallation of docker packages apparently succeeded\n\nAdd a non-root user to the \"docker\" group?: y\n\nEnter user account: kifarunix \n\nAdd another non-root user to the \"docker\" group?: no\n\n\"docker-compose version\" failed, attempt to install docker-compose? (Y\/n): y\n\nInstall docker-compose directly from docker github? (Y\/n): y\nDownload and installation of docker-compose apparently succeeded\n...\n<\/code><\/pre>\n\n\n\n
        \n
      • Next, you will be prompted to update some kernel tunables. Accept for all prompts.<\/li>\n<\/ul>\n\n\n\n
        fs.file-max increases allowed maximum for file handles\nfs.file-max= appears to be missing from \/etc\/sysctl.conf, append it? (Y\/n): y\n\n\nfs.inotify.max_user_watches increases allowed maximum for monitored files\nfs.inotify.max_user_watches= appears to be missing from \/etc\/sysctl.conf, append it? (Y\/n): y\n\n\nfs.inotify.max_queued_events increases queue size for monitored files\nfs.inotify.max_queued_events= appears to be missing from \/etc\/sysctl.conf, append it? (Y\/n): y\n\n\nfs.inotify.max_user_instances increases allowed maximum monitor file watchers\nfs.inotify.max_user_instances= appears to be missing from \/etc\/sysctl.conf, append it? (Y\/n): y\n\n\nvm.max_map_count increases allowed maximum for memory segments\nvm.max_map_count= appears to be missing from \/etc\/sysctl.conf, append it? (Y\/n): y\n\n\nnet.core.somaxconn increases allowed maximum for socket connections\nnet.core.somaxconn= appears to be missing from \/etc\/sysctl.conf, append it? (Y\/n): y\n\n\nvm.swappiness adjusts the preference of the system to swap vs. drop runtime memory pages\nvm.swappiness= appears to be missing from \/etc\/sysctl.conf, append it? (Y\/n): y\n\n\nvm.dirty_background_ratio defines the percentage of system memory fillable with \"dirty\" pages before flushing\nvm.dirty_background_ratio= appears to be missing from \/etc\/sysctl.conf, append it? (Y\/n): y\n\n\nvm.dirty_ratio defines the maximum percentage of dirty system memory before committing everything\nvm.dirty_ratio= appears to be missing from \/etc\/sysctl.conf, append it? (Y\/n): y\n\n\n\/etc\/security\/limits.d\/limits.conf increases the allowed maximums for file handles and memlocked segments\n\/etc\/security\/limits.d\/limits.conf does not exist, create it? (Y\/n): y\n<\/code><\/pre>\n\n\n\n
          \n
        • Once that is done, the script will exit.<\/li>\n<\/ul>\n\n\n\n
          Run Malcolm System Configurations<\/h3>\n\n\n\n
          Once the install script above completes, proceed to configure Malcolm;<\/p>\n\n\n\n
          sudo .\/scripts\/install.py --configure<\/code><\/pre>\n\n\n\n
          When configuration starts;<\/p>\n\n\n\n
            \n
          • Choose a user with which to run Malcolm as. We created malcolm<\/strong> user account above, whose UID and GID is 1001<\/strong>. Hence, say no to user with UID\/GID 1000.<\/li>\n<\/ul>\n\n\n\n
            Malcolm processes will run as UID 1000 and GID 1000. Is this OK? (Y\/n): n<\/strong><\/code><\/pre>\n\n\n\n
              \n
            • Enter the user ID and group ID of malcolm system account.<\/li>\n<\/ul>\n\n\n\n
              Enter user ID (UID) for running non-root Malcolm processes; 1001<\/code><\/pre>\n\n\n\n
              Enter group ID (GID) for running non-root Malcolm processes; 1001<\/code><\/pre>\n\n\n\n
              Confirm;<\/p>\n\n\n\n
              Malcolm processes will run as UID 998 and GID 998. Is this OK? yes<\/strong><\/code><\/pre>\n\n\n\n
                \n
              • OpenSearch and logstash settings;<\/li>\n<\/ul>\n\n\n\n
                Setting 10g for OpenSearch and 3g for Logstash. Is this OK? yes<\/strong><\/code><\/pre>\n\n\n\n
                Setting 3 workers for Logstash pipelines. Is this OK? (Y\/n): yes<\/strong><\/code><\/pre>\n\n\n\n
                  \n
                • Restart Malcolm when system or Docker restarts;<\/li>\n<\/ul>\n\n\n\n
                  Restart Malcolm upon system or Docker daemon restart? Yes<\/code><\/pre>\n\n\n\n
                  Choose when to restart Malcolm. You have four options here; no, on-failure, always, unless-stopped<\/strong>. Choose the default option, unless-stopped<\/strong>.<\/p>\n\n\n\n
                    \n
                  • Choose whether to setup Malcolm with HTTPS. We select yes<\/strong> (SSL certificates will be generated at a later step)<\/li>\n\n\n\n
                  • Choose whether Malcolm will run behind any proxy. You are given some options like Caddy, Traefik..We choose No<\/strong>.<\/li>\n\n\n\n
                  • On Default networking, press enter to accept the defaults.<\/li>\n\n\n\n
                  • If prompted whether to use LDAP, select no<\/strong> for now.<\/li>\n\n\n\n
                  • Store OpenSearch index snapshosts locally in \/opt\/malcolm\/Malcom\/opensearch-backup? You can choose to accept the default or just enter the new path, \/opt\/malcolm\/<\/strong><\/code>.<\/li>\n\n\n\n
                  • Choose to Compress OpenSearch index snapshots (yes<\/strong>)<\/li>\n\n\n\n
                  • When prompted to delete the oldest indices when the database exceeds a certain size choose no to avoid loosing data.<\/li>\n\n\n\n
                  • Automatically analyze all PCAP files with Suricata: yes<\/li>\n\n\n\n
                  • Automatically analyze all PCAP files with Zeek: yes<\/strong><\/li>\n\n\n\n
                  • Disable <\/strong>reverse DNS lookup locally for source and destination IP addresses in logs.<\/li>\n\n\n\n
                  • Enable<\/strong> hardware vendor OUI lookups for MAC addresses.<\/li>\n\n\n\n
                  • Perform string randomness scoring on some fields: yes<\/strong><\/li>\n\n\n\n
                  • Expose OpenSearch port to external hosts: no<\/strong><\/li>\n\n\n\n
                  • Expose Logstash port to external hosts: no<\/strong><\/li>\n\n\n\n
                  • Forward Logstash logs to external OpenSearch instance: no<\/strong><\/li>\n\n\n\n
                  • Expose Filebeat TCP port to external hosts: no<\/strong><\/li>\n\n\n\n
                  • Enable file extraction with Zeek: yes<\/strong><\/li>\n\n\n\n
                  • choose interesting<\/strong> as the extraction behavior.<\/li>\n\n\n\n
                  • Select quarantine<\/strong> as the file preservation method.<\/li>\n\n\n\n
                  • Scan extracted files\/PE files with ClamAV: yes<\/strong><\/li>\n\n\n\n
                  • Scan extracted files\/PE files with Yara: yes<\/strong><\/li>\n\n\n\n
                  • Scan extracted files\/PE files with Capa: yes<\/strong><\/li>\n\n\n\n
                  • Lookup extracted file hashes with VirusTotal: no<\/strong><\/li>\n\n\n\n
                  • Download updated scanner signatures periodically: yes<\/strong><\/li>\n\n\n\n
                  • Should Malcolm capture network traffic to PCAP files: yes<\/strong><\/li>\n\n\n\n
                  • Specify capture interface(s) (comma-separated) on which Malcolm will use to network traffic: enp0s8<\/strong><\/li>\n\n\n\n
                  • Capture packets using netsniff-ng? (Y\/n): yes<\/strong><\/li>\n\n\n\n
                  • Capture packets using tcpdump? (y\/N): no<\/strong><\/li>\n\n\n\n
                  • Capture filter (tcpdump-like filter expression; leave blank to capture all traffic) (): You can disable traffic related to Elasticsearch (port 9200), Logstash (5044), Arkime(8005): not port 9200 and not port 5044 and not port 8005<\/code><\/strong>.<\/li>\n\n\n\n
                  • Disable capture interface hardware offloading and adjust ring buffer sizes? (y\/N): n<\/strong><\/li>\n<\/ul>\n\n\n\n
                    Malcolm has now been installed to \/home\/$USER\/Malcolm.<\/p>\n\n\n\n
                    Scripts for starting and stopping Malcolm and changing authentication-related settings can be found in \/home\/$USER\/Malcolm\/scripts.<\/p>\n\n\n\n
                    Run system reboot to apply some of the system changes;<\/p>\n\n\n\n
                    sudo systemctl reboot -i<\/code><\/pre>\n\n\n\n
                    Create Malcolm User Account<\/h3>\n\n\n\n
                    Navigate back to the Malcolm Github repository directory;<\/p>\n\n\n\n
                    su - malcolm<\/code><\/pre>\n\n\n\n
                    cd ~\/Malcolm<\/code><\/pre>\n\n\n\n
                    Run the authentication setup script;<\/p>\n\n\n\n
                    .\/scripts\/auth_setup<\/code><\/pre>\n\n\n\n