ban_ip_with_memcached_flood这篇文章根据 互联网网站的反爬虫策略浅析 使用php实现
假设现有一phpwind搭建论坛,在global.php文件内require_once(‘ban_ip_with_memcached_flood.php’)
ban_ip_with_memcached_flood.php文件
05 | $memcache_obj = new Memcache; |
06 | $memcache_host = "127.0.0.1" ; |
07 | $memcache_obj ->connect( $memcache_host , 11211) or die ( "Could not connect memcached at localserver" ); |
13 | if ( $HTTP_SERVER_VARS [ "HTTP_X_FORWARDED_FOR" ]) |
15 | $ip = $HTTP_SERVER_VARS [ "HTTP_X_FORWARDED_FOR" ]; |
17 | elseif ( $HTTP_SERVER_VARS [ "HTTP_CLIENT_IP" ]) |
19 | $ip = $HTTP_SERVER_VARS [ "HTTP_CLIENT_IP" ]; |
21 | elseif ( $HTTP_SERVER_VARS [ "REMOTE_ADDR" ]) |
23 | $ip = $HTTP_SERVER_VARS [ "REMOTE_ADDR" ]; |
25 | elseif ( getenv ( "HTTP_X_FORWARDED_FOR" )) |
27 | $ip = getenv ( "HTTP_X_FORWARDED_FOR" ); |
29 | elseif ( getenv ( "HTTP_CLIENT_IP" )) |
31 | $ip = getenv ( "HTTP_CLIENT_IP" ); |
33 | elseif ( getenv ( "REMOTE_ADDR" )) |
35 | $ip = getenv ( "REMOTE_ADDR" ); |
47 | $iptable_statement = "iptables -A INPUT -i eth0 -j DROP -p tcp --dport 80 -s $client_ip" ; |
48 | # echo ( $iptable_statement ); |
49 | $ip_counter = $memcache_obj ->increment( $client_ip ); |
50 | #var_dump( $ip_counter ); |
52 | $memcache_obj ->set( $client_ip ,1, 0, 60); |
53 | } elseif ( $ip_counter >300) { |
54 | $crawler_counter = $memcache_obj ->increment( "crawler/$client_ip" ); |
55 | if (! $crawler_counter ) { |
56 | $memcache_obj ->set( "crawler/$client_ip" ,1, 0, 60); |
58 | elseif ( $crawler_counter >50){ |
59 | #BlackList.add(ip_sec) |
61 | # echo exec ( $iptable_statement ); |
62 | echo exec ( "echo 'deny $client_ip;\n'>>/d/nginx/conf/ip_ban.conf" ); |
64 | # echo exec ( "nginx -s reload" ); |
65 | header( 'HTTP/1.1 403 Forbidden' ); |
66 | die ( "Unauthorized access forbidden! 爬虫再见" ); |
68 | #var_dump( $crawler_counter ); |
70 | header( 'HTTP/1.0 401 Unauthorized' ); |
71 | die ( "Unauthorized access forbidden! 未认证授权" ); |
72 | #render :template => 'test' , :status => 401 and return false |
这里的 ip_ban.conf 是nginx的 httpaccess module 加载配置文件,利用它和crontab来实现每2秒重新reload
nginx配置文件禁止ip访问,从0.8版本开始nginx的reload方法 /d/env/nginx/sbin/nginx -s reload
01 | mkdir -p /d/env/crontab |
03 | nano /d/env/crontab/nginx_ban_ip.sh |
08 | /d/nginx/sbin/nginx -s reload |
11 | chmod +x /d/env/crontab/nginx_ban_ip.sh |
13 | #给ip_ban加上www执行权限,由于我的nginx和php都是www用户执行 |
14 | chmod 777 /d/nginx/conf/ip_ban.conf |
15 | chown www:www /d/nginx/conf/ip_ban.conf |
16 | ll /d/nginx/conf/ip_ban.conf |
18 | #crontab -e 添加2分钟执行shell的语句 |
19 | */2 * * * * /d/env/crontab/nginx_ban_ip.sh |
找到一款不错的网站压力测试工具webbench[原创]
使用该压力测试对nginx服务器进行虐待,测试以上反爬虫效果,注意linux系统专用
webbench -c 200 -t 30 http://www.xxxxxxxxxxxxxxxxxxxxxxxx.com/
30秒内开启浏览器访问网址吧 ,应该会显示“爬虫再见”,2分钟之后nginx就会重读配置文件,之后返回403,ip地址被禁止访问了。
(403页面可以自定义)