Before kernel 2.6.30 [2]: 702 rp_filter - BOOLEAN 703 1 - do source validation by reversed path, as specified in RFC1812 704 Recommended option for single homed hosts and stub network 705 rout问题现象： tunnel模式的lvs在系统从fedora8升级成fc17之后，同样的lvs配置，同样的架构下，lvs工作不正常。
具体在現象（fc17内核3.3.7）
1.从客户端ping vip地址，OK；
2.Telnet访问vip的应该用端口80现象为连接timeout;
3.完全同样的架构和配置在fc8内核2.6.26下则工作正常；
系统构成说明图：
1 系统采用的是lvs-tunnel模式，关于lvs的三种模式请参看：
http://zh.linuxvirtualserver.org/handbooks2上图中整个数据流的过程应该是：
2.1客户端访问lvs的vip地址,此时数据包源地址为客户端地址192.168.91.196，目的地址为lvsvip地址192.168.91.204；
2.2lvs通过其配置的算法，决定请求发给底下的某台realserver，采用的是tunnel模式，此后往下转发的数据包应该是被封装的，源地址为LVS 转发器director的内部地址192.168.91.209，目的地址为realserver的th0地址192.168.91.78（此处假设请求被分配各左边的这台relaserver）
2.3 realserver的eth0上接收到目的地址为自己ip数据包后，解封装，并且交给上层处理，协议栈对payload分析后是ipip封装包，则进一步把该解封装后的数据包交给tunl0处理，因此tunel0上收到的数据包应该是解封装后的数据包，源地址为真实客户端地址192.168.91.196目的地址为vip 192.168.91.204，tunl0接受该数据包后交由上层tcp处理。
2.4针对该请求的返回数据包则按照以下的动作流程：应用层生成返回的信息之后交给tcp/ip层，最后在ip层生产数据包，此时的数据包源地址为VIP192.168.91.204，目的地址为真实客户端地址192.168.91.196
2.5 该数据包在参考适当的路由，上图中实际情况则是参考192.168.0.0/16via 192.168.91.65 dev eth0这条路由，通过eth0发送该数据包到网关C-Router192.168.91.65（如果内核允许），直接发送到外部世界,最终被真实客户端接收。
分析过程
1从客户端ping VIP192.168.91.204通，说明vip地址已经起来；
2从客户端不断的telnet 192.168.91.204的80端口，模拟访问应用程序，并且同时在上图左边的realserver上抓数据包，得到结果如下：
.在th0端口上，能够抓到从lvs-director发给realersver的封装后的ip数据包，源地址和目的地址分别为192.168.91.209，192.168.91.78；
.而在tunlo上，能够抓到解封装后的ip数据包源和目的分别是192.168.91.196，192.168.91.204。
由此可见，来自客户端来的数据包，已经被lvsdirector正确的封装后转发到后端真实的服务器上，而后端真实服务器上的ip栈也已经正确的接收了封装后的数据包，并且正确解封装交由ipip栈处理，但是返回的数据包没有在eth0上抓到，可见问题出在返回数据包上。
3确认真实服务器上的tcp端口正常启动之后，问题怀疑点就落在了返回数据包，没有被正确路由或者干脆丢弃了的假设上。 于是很快想到了内核参数rp_filter（关于什么是rp_filter请参看http://en.wikipedia.org/wiki/Reverse_path_filtering）总结起来就是说如果rp_filter被启用，则服务器在某个接口上接收到了某个数据包，则目的地址为该数据包源地址的返回包必须通过同样的接口发送出去，也就是用于返回数据包的路由项的出口接口如果和该接口不一致，则返回的数据包就直接被内核丢弃
于是查看realserver的内核配置
$ sudo/sbin/sysctl -a|fgrep .rp_filternet.ipv4.conf.all.rp_filter = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.lo.rp_filter = 1
net.ipv4.conf.eth0.rp_filter = 1
net.ipv4.conf.tunl0.rp_filter = 1 <---果然开启了rp_filter,需要关闭。
####关闭的命令如下
echo 0>/proc/sys/net/ipv4/conf/tunl0/rp_filter
4修改rp_filter的配置之后，再次从客户端连接vip的80端口，显示正常，于是问题解决。可还有个问题没有解释清楚的是，为什么在fc8下同样的配置没有出问题呢？
5 很快在网上查到了以下的知识：
Linux kernel 2.6.30 和 kernel2.6.31，内核参数rp_filter的定义和计算其值的算法发生了变化：
这种变化包含，
1，内核参数rp_filter类型从BOOLEAN变成了INTEGER；源代码
http://patchwork.ozlabs.org/patch/23513/2，增即了一个值2，用于宽松模式的反向路径转发
3，并且kernel 2.6.31时，为了修正kernel2.6.32中反向路径转发使用带标记的策略路由的相关bug，计算内核参数rp_filter值的算法，发生了变化；参看
http://git.freesmartphone.org/?p=linux-2.6.git;a=blobdiff;f=include/linux/inetdevice.h;h=ad27c7da87986da346da3d62f29e88bec957280a;hp=acef2a770b6bce589b2cf6e84680e892b235a351;hb=27fed4175acf81ddd91d9a4ee2fd298981f60295;hpb=6d7760a88c25057c2c2243e5dfe2d731064bd31dBefore kernel 2.6.30 [2]: 702 rp_filter - BOOLEAN 703 1 - do source validation by reversed path, as specified in RFC1812 704 Recommended option for single homed hosts and stub network 705 routers. Could cause troubles for complicated (not loop free) 706 networks running a slow unreliable protocol (sort of RIP), 707 or using static routes. 708 709 0 - No source validation. 710 711 conf/all/rp_filter must also be set to TRUE to do source validation 712 on the interface 713 714 Default value is 0. Note that some distributions enable it 715 in startup scripts.Since kernel 2.6.31 [3]: 702 rp_filter - INTEGER 703 0 - No source validation. 704 1 - Strict mode as defined in RFC3704 Strict Reverse Path 705 Each incoming packet is tested against the FIB and if the interface 706 is not the best reverse path the packet check will fail. 707 By default failed packets are discarded. 708 2 - Loose mode as defined in RFC3704 Loose Reverse Path 709 Each incoming packet's source address is also tested against the FIB 710 and if the source address is not reachable via any interface 711 the packet check will fail. 712 713 Current recommended practice in RFC3704 is to enable strict mode 714 to prevent IP spoofing from DDos attacks. If using asymmetric routing 715 or other complicated routing, then loose mode is recommended. 716 717 conf/all/rp_filter must also be set to non-zero to do source validation 718 on the interface 719 720 Default value is 0. Note that some distributions enable it 721 in startup scripts.Before kernel 2.6.31 :Actual rp_filter for <interface> = net.ipv4.conf.<interface>.rp_filter AND net.ipv4.conf.all.rp_filterI.e. reverse path filtering is enabled in strict mode if rp_filter=1 for both "all" and the interface.Since kernel 2.6.31 :Actual rp_filter for <interface> = MAX(net.ipv4.conf.<interface>.rp_filter, net.ipv4.conf.all.rp_filter)I.e. reverse path filtering is enabled in strict mode if rp_filter=1 for either "all" or the interface. 由上面的内容可以看出，在 2.6.31之前的版本中，判断一个端口的rp_filter是不是有效，是对该端口的rp_filter和net.ipv4.conf.all.rp_filter的求与运算，而在之后的版本里面，是对该端口的rp_filter值和net.ipv4.conf.all.rp_filter的求最大值。
6再次来分析系统默认的参数配置
FC8和fc17都是一下的默认内核参数配置
$ sudo/sbin/sysctl -a|fgrep .rp_filternet.ipv4.conf.all.rp_filter = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.lo.rp_filter = 1
net.ipv4.conf.eth0.rp_filter = 1
net.ipv4.conf.tunl0.rp_filter = 1
因此 关于net.ipv4.conf.tunl0.rp_filter最终值的计算过程如下
内核2.6.26；对net.ipv4.conf.all.rp_filter = 0 和 net.ipv4.conf.tunl0.rp_filter =1 求余运算后结果为0,所以在tunl0上rp_filter并没有被启用，因此返回的数据包能够通过eth0端口发送出去，LVS工作正常；
内核3.3.7; 对net.ipv4.conf.all.rp_filter = 0 和 net.ipv4.conf.tunl0.rp_filter = 1 求最大值得到1，表示在该tunl0上 rp_filter是被启用的，也就是说在tunl0上收到的数据包，针对该数据包的返回数据包（目的地址为接受到的数据包的源地址，而源地址为接收到数据包的目的地址），必须通过tunl0发送出去，否则丢弃。在本case中，由于路由选择为eth0而非tunl0所以，数据包丢弃。 从而lvs工作不正常。
总结：
为了确保不出错，在lvs-tunnel配置的时候统一配置内核如下
net.ipv4.conf.all.rp_filter= 0 net.ipv4.conf.default.rp_filter = 0
net.ipv4.conf.lo.rp_filter = 0
net.ipv4.conf.eth0.rp_filter = 0
net.ipv4.conf.tunl0.rp_filter = 0
参考资料：
[1]. http://en.wikipedia.org/wiki/Reverse_path_filtering
[2].http://lxr.free-electrons.com/source/Documentation/networking/ip-sysctl.txt?v=2.6.29#L702
[3].http://lxr.free-electrons.com/source/Documentation/networking/ip-sysctl.txt?v=2.6.30#L702
[4]. http://www.spinics.net/lists/linux-net/msg17162.html
[5]. http://www.spinics.net/lists/netfilter/msg47124.html
[6].http://patchwork.ozlabs.org/patch/23513/
－－－edit by andy.chouers. Could cause troubles for complicated (not loop free) 706 networks running a slow unreliable protocol (sort of RIP), 707 or using static routes. 708 709 0 - No source validation. 710 711 conf/all/rp_filter must also be set to TRUE to do source validation 712 on the interface 713 714 Default value is 0. Note that some distributions enable it 715 in startup scripts.Since kernel 2.6.31 [3]: 702 rp_filter - INTEGER 703 0 - No source validation. 704 1 - Strict mode as defined in RFC3704 Strict Reverse Path 705 Each incoming packet is tested against the FIB and if the interface 706 is not the best reverse path the packet check will fail. 707 By default failed packets are discarded. 708 2 - Loose mode as defined in RFC3704 Loose Reverse Path 709 Each incoming packet's source address is also tested against the FIB 710 and if the source address is not reachable via any interface 711 the packet check will fail. 712 713 Current recommended practice in RFC3704 is to enable strict mode 714 to prevent IP spoofing from DDos attacks. If using asymmetric routing 715 or other complicated routing, then loose mode is recommended. 716 717 conf/all/rp_filter must also be set to non-zero to do source validation 718 on the interface 719 720 Default value is 0. Note that some distributions enable it 721 in startup scripts.
