收到一封腾讯云的邮件，内容如下
一开始没在意，以为是腾讯云误判(还是我太年轻)，毕竟我也没调用任何别人6379端口 的业务，俺自己的redis端口也改了不是这个，所以有点莫名其妙，就没理。
谁知道当天就出现业务异常卡顿，网络通信存在异常延迟等问题，顺带查看了一下腾讯云自带的免费云监控(没有广告费)，7天网络如下图
最高的时候到了15M，感觉有点小DDOS的意思了…难怪腾讯云以为我攻击别人，现在才引起我注意，连上服务器敲一个top，就看到了挖矿程序：kdevtmpfsi
含有守护进程并带有定时任务，所以往往清理后，隔几天又会出现
1.查询kdevtmpfsi进程id
[root@VM-0-2-centos ~]# ps -ef | grep kdevtmpfsi root 11451 1286 99 09:58 ? 01:50:07 /tmp/kdevtmpfsi root 14267 10669 0 11:02 pts/2 00:00:00 grep --color=auto kdevtmpfsi
2.根据进程id查询守护进程
[root@VM-0-2-centos ~]# systemctl status 11451 # 阿哦~ 系统指令也都被黑了，所有关键系统文件的权限均被修改 -bash: /usr/bin/systemctl: Permission denied # 赋予文件755权限 [root@VM-0-2-centos ~]# chmod 755 /usr/bin/systemctl [root@VM-0-2-centos ~]# systemctl status 11451 Failed to get unit for PID 11451: PID 11451 does not belong to any loaded unit.
这里进程11451已经被我刚刚kill了，我想试试kill指令能不能用来着，所以这里显示id不存在
3.删除kdevtmpfsi进程
# 挖矿程序 kill kdevtmpfsi # 挖矿程序的守护进程 kill kinsing # 验证是否已杀死 ps -aux | grep kdevtmpfsi ps -aux | grep kinsing
4.删除kdevtmpfsi相关文件夹
解决步骤较多，都在下面了，具体解释看注释
[root@VM-0-2-centos tmp]# cd /tmp # 试图删除kdevtmpfsi文件夹，结果被锁了 [root@VM-0-2-centos tmp]# rm -rf kdevtmpfsi rm: cannot remove 'kdevtmpfsi’: Operation not permitted # 发现kdevtmpfsi文件夹被加了i属性，具体含义参考：https://www.runoob.com/linux/linux-comm-chattr.html [root@VM-0-2-centos tmp]# lsattr -------------e-- ./cpuidle_support.log -------------e-- ./net_affinity.log ----ia-------e-- ./svcupdates -------------e-- ./yum_save_tx.2021-06-10.10-47.RIGnYJ.yumtx -------------e-- ./yum_save_tx.2021-06-09.14-42.9567MU.yumtx ----ia-------e-- ./svcguard -------------e-- ./setRps.log -------------e-- ./nv_gpu_conf.log -------------e-- ./yum_save_tx.2021-06-09.14-45.ewXMhx.yumtx -------------e-- ./yum_save_tx.2021-06-08.20-18.o5LMma.yumtx -------------e-- ./yum_save_tx.2021-06-08.22-39.fMM654.yumtx -------------e-- ./yum_save_tx.2021-06-08.18-50.nzlZMH.yumtx ----i--------e-- ./redis2 -------------e-- ./crontab.PqH9Tb -------------e-- ./yum_save_tx.2021-06-08.18-50.C9KsvO.yumtx -------------e-- ./yum_save_tx.2021-06-08.21-53.5ay7xI.yumtx -------------e-- ./virtio_blk_affinity.log -------------e-- ./virtio_blk_affinity_udev.log ----i--------e-- ./kdevtmpfsi -------------e-- ./yum_save_tx.2021-06-09.15-03.7KNsei.yumtx -------------e-- ./crontab.7lC9aC -------------e-- ./systemd-private-85826f165bb14fc9b36ced7d99e7989b-ntpd.service-aDaYPL ----ia-------e-- ./svcupdate ----ia-------e-- ./newsvc.sh -------------e-- ./yum_save_tx.2021-06-09.15-33.pDiw4o.yumtx -------------e-- ./yum_save_tx.2021-06-08.18-50.uPB19d.yumtx ----ia-------e-- ./svcworkmanager # 用chattr命令来删除i属性，果然不让我删 [root@VM-0-2-centos tmp]# chattr -i kdevtmpfsi -bash: /usr/bin/chattr: Permission denied [root@VM-0-2-centos tmp]# cd /usr/bin/ # 复制chattr来替换使用 [root@VM-0-2-centos bin]# ls -lh chattr;lsattr chattr -rw-r--r-- 1 root root 12K Apr 1 2020 chattr ----i--------e-- chattr [root@VM-0-2-centos bin]# cp chattr chattr.new [root@VM-0-2-centos bin]# chmod a+x chattr.new [root@VM-0-2-centos bin]# chattr.new -i chattr [root@VM-0-2-centos bin]# rm -f chattr.new [root@VM-0-2-centos bin]# chmod a+x chattr [root@VM-0-2-centos bin]# ls -lh chattr;lsattr chattr -rwxr-xr-x 1 root root 12K Apr 1 2020 chattr -------------e-- chattr [root@VM-0-2-centos bin]# cd /tmp/ # 删除kdevtmpfsi文件夹的i属性 [root@VM-0-2-centos tmp]# chattr -i kdevtmpfsi [root@VM-0-2-centos bin]# [root@VM-0-2-centos bin]# [root@VM-0-2-centos bin]# [root@VM-0-2-centos bin]# [root@VM-0-2-centos bin]# [root@VM-0-2-centos tmp]# cd /var/spool [root@VM-0-2-centos spool]# lsattr -------------e-- ./at -------------e-- ./mail -------------e-- ./plymouth -------------e-- ./postfix ----ia-------e-- ./cron -------------e-- ./anacron -------------e-- ./lpd -------------e-- ./abrt -------------e-- ./abrt-upload # 删除定时任务文件的i属性 [root@VM-0-2-centos spool]# chattr -ai /var/spool/cron [root@VM-0-2-centos spool]# lsattr -------------e-- ./at -------------e-- ./mail -------------e-- ./plymouth -------------e-- ./postfix -------------e-- ./cron -------------e-- ./anacron -------------e-- ./lpd -------------e-- ./abrt -------------e-- ./abrt-upload [root@VM-0-2-centos spool]#
5.删除kdevtmpfsi相关文件夹
rm -rf /tmp/kdevtmpfsi rm -rf /var/tmp/kinsing # 查看是否有残留，查到就rm -rf find / -name "*kdevtmpfsi*" find / -name "*kinsing*"
6.清理定时任务
# 查看当前已有的定时任务 crontab -l # 清理所有定时任务 crontab -r
7.删除免密登录ssh
[root@VM-0-2-centos .ssh]# cd /root/.ssh # 发现有个authorized_keys2，以及当前目录全部锁了i [root@VM-0-2-centos .ssh]# lsattr -a -------------e-- ./.. ----ia-------e-- ./authorized_keys2 ----ia---------- ./authorized_keys ----i----------- ./. [root@VM-0-2-centos .ssh]# chattr -i ./ [root@VM-0-2-centos .ssh]# chattr -ia authorized_keys [root@VM-0-2-centos .ssh]# chattr -ia authorized_keys2 # 删除所有密钥文件 [root@VM-0-2-centos .ssh]# rm -rf ./* # 创建空的私钥文件 [root@VM-0-2-centos .ssh]# touch authorized_keys # 修改ssh配置文件。确保PubkeyAuthentication为no或者已被注释掉即可 [root@VM-0-2-centos .ssh]# vim /etc/ssh/sshd_config
1.防火墙该开开
2.安全组该开开
3.端口该关关
过几天再看看会不会卷土重来
流量监控等也均恢复正常啦，满血复活！
2021-06-13再次收到腾讯云的警告邮件，内容一致，还是说我在攻击其他人6379端口，看来之前清理的kdevtmpfsi还是有残留，这里就接着对剩余残留处理。
[root@VM-0-2-centos ~]# netstat -an Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address Foreign Address State tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN tcp 0 1 172.17.0.2:47660 65.5.49.179:6379 SYN_SENT tcp 0 1 172.17.0.2:50264 65.5.50.177:6379 SYN_SENT tcp 0 0 172.19.0.1:44890 172.19.0.3:80 TIME_WAIT tcp 0 0 172.19.0.1:43788 172.19.0.3:80 TIME_WAIT tcp 0 0 172.19.0.1:55582 172.19.0.3:80 TIME_WAIT tcp 0 1 172.17.0.2:44704 65.5.48.59:6379 SYN_SENT tcp 0 1 172.17.0.2:41186 65.5.51.15:6379 SYN_SENT tcp 0 0 172.19.0.1:47904 172.19.0.3:80 TIME_WAIT tcp 0 0 172.19.0.1:56860 172.19.0.3:80 TIME_WAIT tcp 0 1 172.17.0.2:44702 65.5.51.173:6379 SYN_SENT tcp 0 1 172.17.0.2:57440 65.5.50.50:6379 SYN_SENT tcp 0 1 172.17.0.2:40746 65.5.50.104:6379 SYN_SENT tcp 0 1 172.17.0.2:58656 65.5.51.254:6379 SYN_SENT tcp 0 1 172.17.0.2:60436 65.5.51.171:6379 SYN_SENT
# 发现病毒进程：1584/pnscan [root@VM-0-2-centos ~]# netstat -antp |grep 1 tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 6971/sshd tcp 0 1 172.17.0.2:33416 65.141.50.158:6379 SYN_SENT 1584/pnscan tcp 0 0 172.19.0.1:36736 172.19.0.3:80 TIME_WAIT - tcp 0 0 172.19.0.1:58506 172.19.0.3:80 TIME_WAIT - tcp 0 0 172.19.0.1:41298 172.19.0.3:80 TIME_WAIT - tcp 0 0 172.19.0.1:45006 172.19.0.3:80 TIME_WAIT - tcp 0 0 172.19.0.1:51746 172.19.0.3:80 TIME_WAIT - tcp 0 0 172.19.0.1:34208 172.19.0.3:80 TIME_WAIT - tcp 0 1 172.17.0.2:45738 65.141.50.77:6379 SYN_SENT 1584/pnscan tcp 0 0 172.19.0.1:45406 172.19.0.3:80 TIME_WAIT - tcp 0 1 172.17.0.2:59096 65.141.49.180:6379 SYN_SENT 1584/pnscan tcp 0 0 172.19.0.1:47682 172.19.0.3:80 TIME_WAIT - tcp 0 0 172.19.0.1:55756 172.19.0.3:80 TIME_WAIT - tcp 0 0 172.19.0.1:57948 172.19.0.3:80 ESTABLISHED 853/docker-proxy tcp 0 0 172.19.0.1:46024 172.19.0.3:80 ESTABLISHED 853/docker-proxy tcp 0 0 172.19.0.1:51486 172.19.0.3:80 TIME_WAIT - tcp 0 0 172.19.0.1:47870 172.19.0.3:80 TIME_WAIT - tcp 0 0 172.19.0.1:36458 172.19.0.3:80 TIME_WAIT - tcp 0 1 172.17.0.2:46668 65.141.49.81:6379 SYN_SENT 1584/pnscan tcp 0 1 172.17.0.2:55490 65.141.48.147:6379 SYN_SENT 1584/pnscan # 查找病毒pnscan所在路径 [root@VM-0-2-centos ~]# find / -name "*pnscan*" /usr/local/bin/pnscan /usr/local/share/man/man1/pnscan.1.gz # 杀它！！！！ [root@VM-0-2-centos ~]# rm -rf /usr/local/bin/pnscan /usr/local/share/man/man1/pnscan.1.gz # 再次验证是否有发起6379的网络请求，可以看到没有6379的请求端口了，但是！ [root@VM-0-2-centos ~]# netstat -anptl Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 6971/sshd tcp 0 0 172.19.0.1:49710 172.19.0.3:80 TIME_WAIT - tcp 0 0 172.19.0.1:51082 172.19.0.3:80 TIME_WAIT - tcp 0 0 172.19.0.1:53556 172.19.0.3:80 ESTABLISHED 853/docker-proxy tcp 0 0 172.19.0.1:49298 172.19.0.3:80 TIME_WAIT - tcp 0 0 172.19.0.1:47786 172.19.0.3:80 TIME_WAIT - tcp 0 0 172.19.0.1:50184 172.19.0.3:80 TIME_WAIT - tcp 0 0 172.19.0.1:53496 172.19.0.3:80 ESTABLISHED 853/docker-proxy tcp 0 0 172.19.0.1:53818 172.19.0.3:80 ESTABLISHED 853/docker-proxy tcp 0 0 172.19.0.1:52173 172.19.0.3:80 TIME_WAIT - tcp 0 0 172.19.0.1:53024 172.19.0.3:80 TIME_WAIT - tcp 0 0 172.19.0.1:59698 172.19.0.3:80 CLOSE_WAIT 853/docker-proxy tcp 0 0 172.19.0.1:50394 172.19.0.3:80 TIME_WAIT - tcp 0 0 172.19.0.1:52694 172.19.0.3:80 TIME_WAIT - tcp 0 0 172.19.0.1:50382 172.19.0.3:80 TIME_WAIT - tcp 0 0 172.19.0.1:48692 172.19.0.3:80 TIME_WAIT - tcp 0 0 172.19.0.1:56954 172.19.0.3:80 CLOSE_WAIT 853/docker-proxy tcp 0 0 172.19.0.1:50710 172.19.0.3:80 TIME_WAIT - tcp 0 0 172.19.0.1:49884 172.19.0.3:80 TIME_WAIT - tcp 0 0 172.19.0.1:51434 172.19.0.3:80 TIME_WAIT - tcp 0 0 172.19.0.1:52858 172.19.0.3:80 TIME_WAIT - tcp 0 0 172.19.0.1:33880 172.19.0.3:80 CLOSE_WAIT 853/docker-proxy tcp 0 0 172.19.0.1:52416 172.19.0.3:80 TIME_WAIT - tcp 0 0 172.19.0.1:49504 172.19.0.3:80 TIME_WAIT - tcp 0 0 172.19.0.1:51296 172.19.0.3:80 TIME_WAIT - tcp 0 0 172.19.0.1:50628 172.19.0.3:80 TIME_WAIT - tcp 0 0 172.19.0.1:48666 172.19.0.3:80 TIME_WAIT - tcp 0 0 172.19.0.1:42912 172.19.0.3:80 CLOSE_WAIT 853/docker-proxy tcp 0 0 172.19.0.1:53812 172.19.0.3:80 ESTABLISHED 853/docker-proxy tcp 0 0 172.19.0.1:50402 172.19.0.3:80 TIME_WAIT - tcp 0 0 172.19.0.1:52370 172.19.0.3:80 TIME_WAIT - tcp 0 0 172.19.0.1:52538 172.19.0.3:80 TIME_WAIT - tcp 0 0 172.17.0.2:60168 139.99.102.70:14444 ESTABLISHED 683/[crypto] tcp 0 280 172.17.0.2:22 45.55.134.210:33818 ESTABLISHED 27443/sshd: [accept # 着重看末尾两个；ESTABLISHED 27443/sshd: [accept应该是黑客在尝试登录，输入一下who看到登录用户只有我自己 # ESTABLISHED 683/[crypto]：又发现一个病毒，在请求139.99.102.70的ip [root@VM-0-2-centos ~]# lsof -i COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME [crypto] 683 root 13u IPv4 1200665664 0t0 TCP VM-0-2-centos:60168->70.ip-139-99-102.eu:14444 (ESTABLISHED) # 搞它！ [root@VM-0-2-centos ~]# kill -9 683 [root@VM-0-2-centos ~]# find / -name "*\\[crypto\\]*" /usr/share/[crypto].log /usr/share/[crypto] /usr/share/[crypto].sh /usr/share/[crypto].pid # 又是执行脚本、log文件七七八八的，直接删它！！！！ [root@VM-0-2-centos ~]# rm -rf /usr/share/[crypto].log /usr/share/[crypto] /usr/share/[crypto].sh /usr/share/[crypto].pid
最后再查看一下top以及网络请求等等，目前为止都正常了…如果又冒出来此文再接着更新。— 2021-06-13
联系客服
