Discovering a ShellBot

Today, I found some malware running on one of my research group’s linux servers. It seems to be a variant of a Shellbot. Here’s how it went down:

  1. I wanted to train a VGGNet neural network on this server since it had a GPU. Because the training was very slow, I checked htop to see if my implementation was using the CPU or GPU.

  2. There, I noticed that a user named testuser had many processes named kswapd0 running. I googled kswapd0 and read that it is a common linux process that manages memory and swap, so it would make sense that it could take lots of CPU. The most common google search suggested was “kswapd0 high cpu”. So I decided to mind my own business and just let the model train.

  3. About 30 minutes later I went back to check htop, and realized that the machine wasn’t using much memory, and there was no swap used. So then why would kswapd0 be running? What is it trying to swap?

  4. What is kswapd0 anyway? Why are there 18 of them, with each of them using 100% of a CPU core? I checked the origin of one of the processes with sudo ls -l /proc/1393441/exe where 1393441 was the process ID from htop. This revealed that these processes were launched from /home/testuser/.configrc5/a/kswapd0. This is when I began to feel some trickery was afoot.

  5. A user’s directory? That’s not where system programs usually live 🤔

  6. Let’s look inside. That kswapd0 file was a binary, so I looked in its parent directory and found a file named run:

    echo "QWxhZGRpbjpvcGVu=IHNlc2FtZQ...." | base64 --decode > kswapd0

  7. I took that base64 string, decoded it, and got a Perl script out of it. I put that script into ChatGPT and was told:

    This Perl script appears to be a sophisticated and multifunctional IRC bot, commonly referred to as a “Shellbot”. These types of bots are often used for automated tasks on IRC networks, but they can also be employed for malicious activities such as DDoS attacks, unauthorized access, and spreading malware. Here’s a breakdown of its key components and functionalities: …

  8. I then immediately killed all processes running by testuser , with sudo pkill -U testuser , and I monitored top to see that none were restarted. Thankfully none of them restarted themselves immediately.

  9. I then removed executable privileges for all users for all files in the folder containing malware by running sudo chmod -R a-x .configrc5/ so that nobody can run these files by accident

  10. I then checked cron jobs: sudo crontab -u testuser -l and found five cron jobs that were set up to start these malicious scripts at 6:05 and 8:05am every Sunday, at every system reboot, and at midnight every three days. I commented all these out.

  11. I then checked systemd just in case. Nothing here looked suspicious, and I doubt the attackers would go to the effort of putting duplicate jobs in both cron and systemd.

My guess is that the malware got installed by someone who brute force guessed a password to the testuser account. I assume that whoever set up the account originally chose a simple password just for testing, and this was easily guessed since the machine was set to allow SSH connections from the entire internet. Unfortunately, logrotate was set to discard logs after 4 weeks, and it seems that the initial login was more than 4 weeks ago, so I wasn’t able to figure out how the original attack happened.

This attack seems to be a classic example of a ShellBot: https://asec.ahnlab.com/en/49769/