Multi gateway balancing with iptables

I have been testing load balancing via both iptables and ip route nexthop for a couple of days now.  They both work pretty well too.
This only balances outgoing traffic as incoming traffic balanced via DNS RR and the firewall just returns the traffic on the interface it arrived on as per the previous post.
On the whole, I prefer the iptables solution. It seems to balance the traffic better. ip route balances outgoing connections based on nexthop of the route to that host is not already in it’s routing cache. While iptables balances traffic by alternate outgoing connections. The only downside I have seen is occasional connection drops to the BlackBerry servers.
After 24 hours of iptables balancing:
ppp0      Link encap:Point-to-Point Protocol
          RX bytes:1186783900 (1.1 GB)  TX bytes:1290603327 (1.2 GB)
ppp1      Link encap:Point-to-Point Protocol
          RX bytes:1109227490 (1.1 GB)  TX bytes:1140565429 (1.1 GB)

This is using inclusion rules for determining balanced traffic. These are the rules that ended up on the production server:

# Load balancing rules (Split 50/50 between fwmark 1/2)
iptables -t mangle -A balance1 -d      -j RETURN
iptables -t mangle -A balance1 -d          -j RETURN
iptables -t mangle -A balance1 -m connmark ! --mark 0 -j RETURN
iptables -t mangle -A balance1 -m state --state ESTABLISHED,RELATED -j RETURN
iptables -t mangle -A balance1 -m statistic --mode nth --every 2 --packet 0 -j CONNMARK --set-mark 1
iptables -t mangle -A balance1 -m statistic --mode nth --every 2 --packet 1 -j CONNMARK --set-mark 2

# Check to see if we have already marked a packet
iptables -t mangle -A PREROUTING  -m state --state ESTABLISHED,RELATED -j CONNMARK --restore-mark
iptables -t mangle -A OUTPUT      -m state --state ESTABLISHED,RELATED -j CONNMARK --restore-mark

# Mark incoming connections to return on the interface they came in on
iptables -t mangle -A PREROUTING          -i ppp0                     -m state --state NEW  -j CONNMARK --set-mark 1
iptables -t mangle -A PREROUTING          -i ppp1                     -m state --state NEW  -j CONNMARK --set-mark 2

# New outgoing packets
iptables -t mangle -A PREROUTING  -i eth0          -p tcp --dport  22 -m state --state NEW  -j balance1
iptables -t mangle -A PREROUTING  -i eth0          -p tcp --dport  25 -m state --state NEW  -j balance1
iptables -t mangle -A PREROUTING  -i eth0          -p tcp --dport  80 -m state --state NEW  -j balance1
iptables -t mangle -A PREROUTING  -i eth0          -p tcp --dport 443 -m state --state NEW  -j balance1
iptables -t mangle -A OUTPUT                       -p tcp --dport  80 -m state --state NEW  -j balance1

# Choose our route and save the mark
iptables -t mangle -A PREROUTING  -m connmark --mark 1 -j MARK --set-mark 1
iptables -t mangle -A PREROUTING  -m connmark --mark 2 -j MARK --set-mark 2
iptables -t mangle -A PREROUTING  -m state --state NEW -m connmark ! --mark 0  -j CONNMARK --save-mark

That’s all of the CONNMARK and MARK related rules I use.
The new outgoing packets section is where I choose what packets should be balanced and accounts for about 95% of our outgoing traffic.
The balance1 chain just has some checks at the beginning to catch further traffic that should not be balanced in case some rule gets messed up.
Of the new outgoing packets rules, the PREROUTE lines are for forwarded traffic and the OUTPUT rule is for traffic generated on that host by a transparent squid proxy.
Hope somebody finds that useful one day.

This entry was posted in Server, Ubuntu and tagged , , , , , . Bookmark the permalink. Both comments and trackbacks are currently closed. 
Source: http://blog.khax.net/2009/12/01/multi-gateway-balancing-with-iptables/


No comments:

Post a Comment