Two isp routing
From Finninday
Contents
- 1 background
- 2 No response from comcast dhcp or mac spoofing FTW
- 3 iptables
- 4 what I thought worked
- 5 it didn't really work
- 6 another view of the routing table
- 7 Trying again, with routing
- 8 Success
- 9 Making it permanent, or Where does Ubuntu put static routes?
- 10 broken by a reboot and manual repair
- 11 Making it permanent, ubuntu static routes
- 12 After another reboot, still broken
- 13 got a new IP address assigned by comcast
- 14 clean out routes for old ip address
- 15 success again
- 16 practicing a reboot
- 16.1 external services fail
- 16.2 remove a route from the spiritone table
- 16.3 try switching the default route
- 16.4 examine a packet capture to see what is happening
- 16.4.1 first a normal transaction from a remote host to my web server
- 16.4.2 now switch the default route and retry
- 16.4.3 recover from test by setting the default route back
- 16.4.4 missing rules to pin responses to the originating network
- 16.4.5 Changes to /etc/network/interfaces
- 16.4.6 Disabled network manager
- 16.4.7 Full /etc/network/interfaces
- 17 when comcast borks, how do I manually switch my route to spiritone?
- 18 spiritone changes authentication scheme
- 19 working ubuntu config for two ISP providers
background
I'm using Ubuntu.
I have two ISPs. I didn't plan on getting two ISPs. I've had one for ages that I'm reasonably happy with that I've shopped long and hard for. It gives me a static IP and doesn't charge too much. It has a 1Mbps down and 370Kbps up.
Then my son wanted to vastly upgrade our connectivity and our existing link just couldn't fill the bill. So we got another. His link is 30Mbps down and 3Mbps up. I want some of that. But I don't want to disrupt the existing services on my static IP and the original link.
Skip to the end of this document to see what actually worked and not be bored with all my mistakes.
My routing table looks like this now.
root@weasel:/# route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 216.99.216.1 0.0.0.0 UG 0 0 0 eth0 10.0.0.0 0.0.0.0 255.0.0.0 U 0 0 0 eth1 169.254.0.0 0.0.0.0 255.255.0.0 U 1000 0 0 eth1 216.99.216.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
I want both links to be active. It looks like a lot of my questions will be answered here: http://www.linuxhorizon.ro/iproute2.html
This turns out to be a better reference: http://www.lartc.org/howto/lartc.rpdb.multiple-links.html
Another good reference here: http://www.tipsternet.com/articles/advance%20routing.htm
Eventually, I'll be configuring the routes. Starting with the default here:
root@weasel:/# cat /etc/iproute2/rt_tables # # reserved values # 255 local 254 main 253 default 0 unspec # # local # #1 inr.ruhep
No response from comcast dhcp or mac spoofing FTW
The service guy set up the connection to a windows machine that works just fine. When I try to connect with my laptop or server, I get no response from the dhcp server. Actually, that's not quite true. I get an IPv6 address, but not an IPv4. The windows machine gets an IPv4 just fine.
Maybe I have to spoof my mac address to proceed. Here is my old laptop mac address, so I can change it back:
f0:1f:af:16:f3:39
Here is the mac address on the windows machine:
bc:5f:f4:54:c9:d1
Switch laptop mac address to match what comcast expects.
ifconfig em1 hw ether bc:5f:f4:54:c9:d1
ifconfig em1 reports that it was successfully changed.
yay, it worked.
And better yet, it was easy to clean up.
ifdown em1 ifconfig em1
reports that I've got my old mac address again.
Now test on server.
# original mac address here
eth2 Link encap:Ethernet HWaddr 68:05:ca:19:af:e0
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
Interrupt:19 Memory:fd9c0000-fd9e0000
root@weasel:~# ifconfig eth2 hw ether bc:5f:f4:54:c9:d1
root@weasel:~# ifconfig eth2
eth2 Link encap:Ethernet HWaddr bc:5f:f4:54:c9:d1
inet6 addr: fe80::6a05:caff:fe19:afe0/64 Scope:Link
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:11837 errors:0 dropped:0 overruns:0 frame:0
TX packets:870 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:863363 (863.3 KB) TX bytes:565985 (565.9 KB)
Interrupt:19 Memory:fd9c0000-fd9e0000
# plug cable and watch dhcp log
Sep 28 15:36:00 weasel dhclient: DHCPDISCOVER on eth2 to 255.255.255.255 port 67 interval 3 (xid=0x2f2cf17d)
Sep 28 15:36:00 weasel dhclient: DHCPREQUEST of 24.20.234.228 on eth2 to 255.255.255.255 port 67 (xid=0x2f2cf17d)
Sep 28 15:36:00 weasel dhclient: DHCPOFFER of 24.20.234.228 from 73.94.124.1
Sep 28 15:36:00 weasel dhclient: DHCPACK of 24.20.234.228 from 73.94.124.1
Sep 28 15:36:00 weasel dhclient: bound to 24.20.234.228 -- renewal in 1167 seconds.
Awesome.
Make it permanent by defining it in /etc/network/interfaces like this:
auto eth2 iface eth2 inet dhcp hwaddress bc:5f:f4:54:c9:d1
And here is the original (slower) uplink, also defined in /etc/network/interfaces:
auto eth0 iface eth0 inet static pre-up iptables-restore < /etc/default/iptables address 216.99.216.99 netmask 255.255.255.0 gateway 216.99.216.1 dns-nameservers 8.8.8.8 216.99.193.19 dns-search finninday.net
iptables
Useful iptables reference: https://help.ubuntu.com/community/IptablesHowTo
I've already made changes to my iptables config to make eth2 and eth0 (the two ISPs) be treated generally the same. But the rules are too restrictive for eth2.
root@weasel:/etc/default# ping -I eth2 google.com PING google.com (74.125.239.104) from 24.20.234.228 eth2: 56(84) bytes of data. ping: sendmsg: Operation not permitted ping: sendmsg: Operation not permitted # open up iptables for eth2 root@weasel:/etc/default# ping -I eth2 google.com PING google.com (173.194.33.9) from 24.20.234.228 eth2: 56(84) bytes of data. 64 bytes from sea09s01-in-f9.1e100.net (173.194.33.9): icmp_req=1 ttl=56 time=12.0 ms 64 bytes from sea09s01-in-f9.1e100.net (173.194.33.9): icmp_req=2 ttl=56 time=11.0 ms
So I have to think harder about what iptables rules I need. Let's look at the basics. If I leave out the rules for security and network hygiene, it looks like this:
root@weasel:/etc/default# grep eth2 iptables # eth2 = untrusted link to comcast [0:0] -A POSTROUTING -o eth2 -j MASQUERADE [0:0] -A FORWARD -m state --state NEW -o eth2 -j ACCEPT [0:0] -A FORWARD -i eth2 -o eth0 -m state --state RELATED,ESTABLISHED -j ACCEPT
iptables troubleshooting
I wanted a way to know which rules were being triggered so I can know how to tweak my iptables rules. My best tool so far is this script:
root@weasel:~/bin# cat iptables-diff-nat rm /tmp/iptables-diff* iptables -t nat -L -v -n > /tmp/iptables-diff1 sleep 10 iptables -t nat -L -v -n > /tmp/iptables-diff2 diff /tmp/iptables-diff1 /tmp/iptables-diff2 | less
which generates output like this:
pkts bytes target prot opt in out source destination 3c3 < 2665 553K ACCEPT all -- eth1 * 10.0.0.0/8 0.0.0.0/0 --- > 2706 565K ACCEPT all -- eth1 * 10.0.0.0/8 0.0.0.0/0 67c67 < 563 129K ACCEPT all -- eth1 * 10.0.0.0/8 0.0.0.0/0 --- > 601 133K ACCEPT all -- eth1 * 10.0.0.0/8 0.0.0.0/0 70c70 < 468 126K ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED --- > 503 138K ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED 98c98 < 834 298K ACCEPT all -- * eth1 216.99.216.99 10.0.0.0/24 --- > 856 319K ACCEPT all -- * eth1 216.99.216.99 10.0.0.0/24 100c100 < 1250 116K ACCEPT all -- * eth1 10.0.0.0/8 10.0.0.0/8 --- > 1262 118K ACCEPT all -- * eth1 10.0.0.0/8 10.0.0.0/8
Based on the output of that script run against the nat and filter tables, I now think that my changes to iptables should look like this:
root@weasel:/etc/default# egrep "24.20|eth2" iptables # eth2 = untrusted link to comcast [0:0] -A POSTROUTING -o eth2 -j MASQUERADE [0:0] -A INPUT -s 10.0.0.0/8 -i eth2 -j DROP [0:0] -A INPUT -s 172.16.0.0/12 -i eth2 -j DROP [0:0] -A INPUT -s 192.168.0.0/12 -i eth2 -j DROP [0:0] -A INPUT -p tcp --destination-port 6667 -i eth2 -j DROP [0:0] -A INPUT -p udp --destination-port 6667 -i eth2 -j DROP [0:0] -A FORWARD -m state --state NEW -o eth2 -j ACCEPT [0:0] -A FORWARD -i eth2 -o eth1 -m state --state RELATED,ESTABLISHED -j ACCEPT [2:88] -A FORWARD -i eth1 -o eth2 -j ACCEPT [0:0] -A OUTPUT -s 24.20.234.228 -d 10.0.0.0/24 -o eth1 -j ACCEPT [0:0] -A OUTPUT -d 10.0.0.0/8 -o eth2 -j DROP
That grep doesn't show you that the MASQUERADE rule is applied to the nat tables and the rest are applied to the filter table. Most of those rules (the DROPs) are just for good hygiene, eliminating clearly bogus packets.
what I thought worked
root@weasel:/etc/default# egrep "24.20|eth2" iptables # eth2 = untrusted link to comcast # nat table rule -A POSTROUTING -o eth2 -j MASQUERADE # filter table rules -A FORWARD -m state --state NEW -o eth2 -j ACCEPT -A INPUT -s 75.75.75.75 -p tcp --destination-port 53 -i eth2 -j ACCEPT -A INPUT -d 24.20.234.228 -i eth2 -m state --state RELATED,ESTABLISHED -j ACCEPT -A FORWARD -i eth2 -o eth1 -m state --state RELATED,ESTABLISHED -j ACCEPT -A FORWARD -i eth1 -o eth2 -j ACCEPT -A OUTPUT -s 24.20.234.228 -d 10.0.0.0/24 -o eth1 -j ACCEPT -A OUTPUT -s 24.20.234.228 -o eth2 -j ACCEPT # hygiene -A INPUT -s 10.0.0.0/8 -i eth2 -j DROP -A INPUT -s 172.16.0.0/12 -i eth2 -j DROP -A INPUT -s 192.168.0.0/12 -i eth2 -j DROP -A INPUT -p tcp --destination-port 6667 -i eth2 -j DROP -A INPUT -p udp --destination-port 6667 -i eth2 -j DROP -A OUTPUT -d 10.0.0.0/8 -o eth2 -j DROP
Also had to fiddle with DHCP server which was giving out the old DNS servers to clients. And Xymon needed to be updated to know about the new routes. Not sure if this config will survive a reboot.
root@weasel:/etc/network# route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 24.20.234.1 0.0.0.0 UG 0 0 0 eth2 10.0.0.0 0.0.0.0 255.0.0.0 U 0 0 0 eth1 24.20.234.0 0.0.0.0 255.255.254.0 U 0 0 0 eth2 169.254.0.0 0.0.0.0 255.255.0.0 U 1000 0 0 eth2 216.99.216.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
it didn't really work
External traffic (mail, web) came in over the slow link and then apparently it went out over the fast link and then was dropped. The failure wasn't seen by my internal monitors which were routed properly and showed all services functioning. I need a way to see when external traffic is not being routed properly. Perhaps count the number of emails arriving by hour and alert when it drops. That way I can get some use out of the spammers hitting my mail server.
another view of the routing table
I'm not sure what is different about routel and route -n. Just that routel is much more interesting:
root@weasel:/var/log# routel
target gateway source proto scope dev tbl
default 216.99.216.1 eth0
10.0.0.0 8 10.0.0.3 kernel link eth1
169.254.0.0 16 169.254.11.125 kernel link eth2
216.99.216.0 24 216.99.216.99 kernel link eth0
10.0.0.0 broadcast 10.0.0.3 kernel link eth1 local
10.0.0.3 local 10.0.0.3 kernel host eth1 local
10.255.255.255 broadcast 10.0.0.3 kernel link eth1 local
127.0.0.0 broadcast 127.0.0.1 kernel link lo local
127.0.0.0 8 local 127.0.0.1 kernel host lo local
127.0.0.1 local 127.0.0.1 kernel host lo local
127.255.255.255 broadcast 127.0.0.1 kernel link lo local
169.254.0.0 broadcast 169.254.11.125 kernel link eth2 local
169.254.11.125 local 169.254.11.125 kernel host eth2 local
169.254.255.255 broadcast 169.254.11.125 kernel link eth2 local
216.99.216.0 broadcast 216.99.216.99 kernel link eth0 local
216.99.216.99 local 216.99.216.99 kernel host eth0 local
216.99.216.255 broadcast 216.99.216.99 kernel link eth0 local
:: 96 :: sit0
2001:470:1f08:52c:: 64 :: kernel sit1
fe80:: 64 kernel eth0
fe80:: 64 kernel eth1
default sit1
default unreachable kernel lo unspec
::1 :: none lo local
::10.0.0.3 :: none lo local
::127.0.0.1 :: none lo local
::216.99.216.99 :: none lo local
2001:470:1f08:52c::2 :: none lo local
fe80::a00:3 :: none lo local
fe80::d863:d863 :: none lo local
fe80::213:d3ff:fea6:5cfc :: none lo local
fe80::260:8ff:fe11:b59e :: none lo local
ff00:: 8 eth0 local
ff00:: 8 eth1 local
ff00:: 8 sit1 local
default unreachable kernel lo unspec
root@weasel:/var/log# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 216.99.216.1 0.0.0.0 UG 0 0 0 eth0
10.0.0.0 0.0.0.0 255.0.0.0 U 0 0 0 eth1
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth2
216.99.216.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
If I turn off the IPv6 stuff, it looks simpler:
root@weasel:/var/log# routel
target gateway source proto scope dev tbl
default 216.99.216.1 eth0
10.0.0.0 8 10.0.0.3 kernel link eth1
169.254.0.0 16 169.254.11.125 kernel link eth2
216.99.216.0 24 216.99.216.99 kernel link eth0
10.0.0.0 broadcast 10.0.0.3 kernel link eth1 local
10.0.0.3 local 10.0.0.3 kernel host eth1 local
10.255.255.255 broadcast 10.0.0.3 kernel link eth1 local
127.0.0.0 broadcast 127.0.0.1 kernel link lo local
127.0.0.0 8 local 127.0.0.1 kernel host lo local
127.0.0.1 local 127.0.0.1 kernel host lo local
127.255.255.255 broadcast 127.0.0.1 kernel link lo local
169.254.0.0 broadcast 169.254.11.125 kernel link eth2 local
169.254.11.125 local 169.254.11.125 kernel host eth2 local
169.254.255.255 broadcast 169.254.11.125 kernel link eth2 local
216.99.216.0 broadcast 216.99.216.99 kernel link eth0 local
216.99.216.99 local 216.99.216.99 kernel host eth0 local
216.99.216.255 broadcast 216.99.216.99 kernel link eth0 local
fe80:: 64 kernel eth0
fe80:: 64 kernel eth1
default unreachable kernel lo unspec
::1 :: none lo local
fe80::213:d3ff:fea6:5cfc :: none lo local
fe80::260:8ff:fe11:b59e :: none lo local
ff00:: 8 eth0 local
ff00:: 8 eth1 local
default unreachable kernel lo unspec
Trying again, with routing
- first plug in the interface and bring it up
Add this to /etc/network/interfaces
auto eth2 iface eth2 inet dhcp hwaddress bc:5f:f4:54:c9:d1 gateway 24.20.234.1
- then turn it on
root@weasel:/etc/iproute2# ifup eth2 Internet Systems Consortium DHCP Client 4.2.4 Copyright 2004-2012 Internet Systems Consortium. All rights reserved. For info, please visit https://www.isc.org/software/dhcp/ run-parts: failed to stat component /etc/dhcp/dhclient-enter-hooks.d/samba: No such file or directory Listening on LPF/eth2/bc:5f:f4:54:c9:d1 Sending on LPF/eth2/bc:5f:f4:54:c9:d1 Sending on Socket/fallback DHCPDISCOVER on eth2 to 255.255.255.255 port 67 interval 3 (xid=0x50d19edd) DHCPREQUEST of 24.20.234.228 on eth2 to 255.255.255.255 port 67 (xid=0x50d19edd) DHCPOFFER of 24.20.234.228 from 73.94.124.1 DHCPACK of 24.20.234.228 from 73.94.124.1 run-parts: failed to stat component /etc/dhcp/dhclient-enter-hooks.d/samba: No such file or directory status: Unknown job: squid bound to 24.20.234.228 -- renewal in 917 seconds. ssh stop/waiting ssh start/running, process 26659
And now my routes look like:
root@weasel:/etc/iproute2# ip route show default via 216.99.216.1 dev eth0 10.0.0.0/8 dev eth1 proto kernel scope link src 10.0.0.3 24.20.234.0/23 dev eth2 proto kernel scope link src 24.20.234.228 169.254.0.0/16 dev eth2 scope link metric 1000 216.99.216.0/24 dev eth0 proto kernel scope link src 216.99.216.99
I have also made changes to /etc/iproute2/rt_tables like so:
root@weasel:/etc/iproute2# cat rt_tables # # reserved values # 255 local 254 main 253 default 0 unspec # # local # #1 inr.ruhep 1 spiritone 2 comcast
Now I should be able to just add the split access routes according to the lartc howto. The variables that lartc says is required for split access are as follows:
T1 = spiritone T2 = comcast IF1 = eth0 IF2 = eth2 IP1 = 216.99.216.99 IP2 = 24.20.234.228 P1 = 216.99.216.1 P2 = 24.20.234.1 P1_net = 216.99.216.0/24 P2_net = 24.20.234.0/23
Which means that I need to apply these routes. But the first one is a duplicate?
root@weasel:/etc/iproute2# ip route add 216.99.216.0/24 eth0 src 216.99.216.99 table spiritone Error: either "to" is duplicate, or "eth0" is a garbage.
The others worked just fine.
root@weasel:/etc/iproute2# ip route add default via 216.99.216.1 table spiritone root@weasel:/etc/iproute2# ip route add 24.20.234.0/23 dev eth2 src 24.20.234.228 table comcast root@weasel:/etc/iproute2# ip route add default via 24.20.234.1 table comcast
That didn't break anything too bad, except turning on eth2 overwrote my resolv.conf with comcast servers. After I put 8.8.8.8 back at the top of resolv.conf, I could ping again:
root@weasel:/etc/iproute2# ping google.com PING google.com (67.50.19.20) 56(84) bytes of data. 64 bytes from 67.50.19.20: icmp_req=1 ttl=60 time=45.4 ms 64 bytes from 67.50.19.20: icmp_req=2 ttl=60 time=44.5 ms 64 bytes from 67.50.19.20: icmp_req=3 ttl=60 time=44.7 ms ^C --- google.com ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2002ms rtt min/avg/max/mdev = 44.525/44.916/45.442/0.423 ms
Also, I'm more careful to check my external services. Using my cell phone to come in from the outside (not via wifi) I can access my home web service. Another important test, can the local natted machines still see the internet, and the answer is yes.
At this point, spiritone is still the default route:
root@weasel:~# traceroute google.com traceroute to google.com (67.50.19.20), 30 hops max, 60 byte packets 1 spiritone-router.finninday.net (216.99.216.1) 41.409 ms 43.249 ms 45.165 ms 2 b1.colo.spiritone.com (216.99.192.28) 49.114 ms 53.067 ms 57.050 ms ^C
Moving on with ip rules to pin traffic to the interface that it arrived on.
root@weasel:~# ip rule add from 216.99.216.99 table spiritone root@weasel:~# ip rule add from 24.20.234.228 table comcast
And more routes for tidying up:
root@weasel:~# ip route add 10.0.0.0/8 dev eth1 table spiritone root@weasel:~# ip route add 24.20.234.0/23 dev eth2 table spiritone root@weasel:~# ip route add 127.0.0.0/8 dev lo table spiritone root@weasel:~# ip route add 10.0.0.0/8 dev eth1 table comcast root@weasel:~# ip route add 216.99.216.0/24 dev eth0 table comcast root@weasel:~# ip route add 127.0.0.0/8 dev lo table comcast
Cool. And now for load balancing over the two routes. Read here about multiple routes to the same destination: http://www.policyrouting.org/PolicyRoutingBook/ONLINE/CH05.web.html
It looks like I have two options: a) alternate packets over the two routes b) alternate flows over the two routes. I'll try alternating flows so I don't break sessions.
root@weasel:~# ip route add default scope global nexthop via 216.99.216.1 dev eth0 weight 1 nexthop via 24.20.234.1 dev eth2 weight 1 root@weasel:~# ip route default nexthop via 216.99.216.1 dev eth0 weight 1 nexthop via 24.20.234.1 dev eth2 weight 1 10.0.0.0/8 dev eth1 proto kernel scope link src 10.0.0.3 24.20.234.0/23 dev eth2 proto kernel scope link src 24.20.234.228 169.254.0.0/16 dev eth2 scope link metric 1000 216.99.216.0/24 dev eth0 proto kernel scope link src 216.99.216.99
Load balancing was acting a little sketchy, causing stalls, so I backed it out in favor of simply using a default route to comcast like this:
root@weasel:/etc/default# ip route default via 24.20.234.1 dev eth2 10.0.0.0/8 dev eth1 proto kernel scope link src 10.0.0.3 24.20.234.0/23 dev eth2 proto kernel scope link src 24.20.234.228 169.254.0.0/16 dev eth2 scope link metric 1000 216.99.216.0/24 dev eth0 proto kernel scope link src 216.99.216.99
I'll try it again after I'm sure I've got iptables straight.
Now to allow traffic through iptables.
root@weasel:/etc/default# diff -u iptables-before-comcast iptables --- iptables-before-comcast 2013-09-28 14:07:05.000000000 -0700 +++ iptables 2013-10-08 00:22:20.000000000 -0700 @@ -1,5 +1,6 @@ # eth1 = trusted link to LAN -# eth0 = untrusted link to internet +# eth0 = untrusted link to spiritone +# eth2 = untrusted link to comcast # Generated by iptables-save v1.2.7a on Fri Jul 4 22:39:57 2003 *nat @@ -23,6 +24,7 @@ # could use MASQUERADE instead like this: # -t nat -A POSTROUTING -o eth0 -j MASQUERADE # masquerade is prefered to SNAT and doesn't require me to know my public IP addr +[0:0] -A POSTROUTING -o eth2 -j MASQUERADE COMMIT # Completed on Fri Jul 4 22:39:57 2003 @@ -121,12 +123,10 @@ # accept established connections to eth0 # hard-coded local ip address [0:0] -A OUTPUT -m state --state NEW -o eth0 -j ACCEPT -#[0:0] -A INPUT -d 24.21.143.6 -i eth0 -m state --state RELATED,ESTABLISHED -j ACCEPT -#[0:0] -A INPUT -d 24.21.185.50 -i eth0 -m state --state RELATED,ESTABLISHED -j ACCEPT -#[0:0] -A INPUT -d 69.30.77.115 -i eth0 -m state --state RELATED,ESTABLISHED -j ACCEPT -#[0:0] -A INPUT -d 69.11.182.134 -i eth0 -m state --state RELATED,ESTABLISHED -j ACCEPT [0:0] -A INPUT -d 216.99.216.99 -i eth0 -m state --state RELATED,ESTABLISHED -j ACCEPT +[0:0] -A INPUT -d 24.20.234.228 -i eth2 -m state --state RELATED,ESTABLISHED -j ACCEPT [0:0] -A FORWARD -m state --state NEW -o eth0 -j ACCEPT +[0:0] -A FORWARD -m state --state NEW -o eth2 -j ACCEPT [0:0] -A FORWARD -m state --state ESTABLISHED,RELATED -j ACCEPT # accept imaps connections @@ -176,6 +176,7 @@ [0:0] -A INPUT -s 192.168.100.1 -p tcp --destination-port 53 -i eth0 -j ACCEPT [0:0] -A INPUT -s 8.8.8.8 -p tcp --destination-port 53 -i eth0 -j ACCEPT [0:0] -A INPUT -s 216.99.193.19 -p tcp --destination-port 53 -i eth0 -j ACCEPT +[0:0] -A INPUT -s 75.75.75.75 -p tcp --destination-port 53 -i eth2 -j ACCEPT # accept squid traffic [0:0] -A INPUT -p tcp --destination-port 3128 -i eth0 -j ACCEPT # accept vnc traffic @@ -191,8 +192,10 @@ #### FORWARD RULES # accept inbound established packets [0:0] -A FORWARD -i eth0 -o eth1 -m state --state RELATED,ESTABLISHED -j ACCEPT +[0:0] -A FORWARD -i eth2 -o eth1 -m state --state RELATED,ESTABLISHED -j ACCEPT # accept outbound packets [2:88] -A FORWARD -i eth1 -o eth0 -j ACCEPT +[2:88] -A FORWARD -i eth1 -o eth2 -j ACCEPT # deny everything else [0:0] -A FORWARD -p tcp -d 10.0.0.85 --dport 6112:6119 -j ACCEPT [0:0] -A FORWARD -p udp -d 10.0.0.85 --dport 6112:6119 -j ACCEPT @@ -207,6 +210,8 @@ #[0:0] -A OUTPUT -s 69.30.77.115 -d 10.0.0.0/24 -o eth1 -j ACCEPT #[0:0] -A OUTPUT -s 69.11.182.134 -d 10.0.0.0/24 -o eth1 -j ACCEPT [0:0] -A OUTPUT -s 216.99.216.99 -d 10.0.0.0/24 -o eth1 -j ACCEPT +[0:0] -A OUTPUT -s 24.20.234.228 -d 10.0.0.0/24 -o eth1 -j ACCEPT +[0:0] -A OUTPUT -s 24.20.234.228 -o eth2 -j ACCEPT # accept packets on LAN [0:0] -A OUTPUT -s 10.0.0.0/8 -d 10.0.0.0/8 -o eth1 -j ACCEPT # deny packets to eth0 that have eth1 addresses
Success
Here is what the routes now look like:
root@weasel:/var/log# routel
target gateway source proto scope dev tbl
default 216.99.216.1 eth0 spiritone
10.0.0.0 8 link eth1 spiritone
24.20.234.0 23 link eth2 spiritone
127.0.0.0 8 link lo spiritone
default 24.20.234.1 eth2 comcast
10.0.0.0 8 link eth1 comcast
24.20.234.0 23 24.20.234.228 link eth2 comcast
127.0.0.0 8 link lo comcast
216.99.216.0 24 link eth0 comcast
default 24.20.234.1 eth2
10.0.0.0 8 10.0.0.3 kernel link eth1
24.20.234.0 23 24.20.234.228 kernel link eth2
169.254.0.0 16 link eth2
216.99.216.0 24 216.99.216.99 kernel link eth0
10.0.0.0 broadcast 10.0.0.3 kernel link eth1 local
10.0.0.3 local 10.0.0.3 kernel host eth1 local
10.255.255.255 broadcast 10.0.0.3 kernel link eth1 local
24.20.234.0 broadcast 24.20.234.228 kernel link eth2 local
24.20.234.228 local 24.20.234.228 kernel host eth2 local
24.20.235.255 broadcast 24.20.234.228 kernel link eth2 local
127.0.0.0 broadcast 127.0.0.1 kernel link lo local
127.0.0.0 8 local 127.0.0.1 kernel host lo local
127.0.0.1 local 127.0.0.1 kernel host lo local
127.255.255.255 broadcast 127.0.0.1 kernel link lo local
216.99.216.0 broadcast 216.99.216.99 kernel link eth0 local
216.99.216.99 local 216.99.216.99 kernel host eth0 local
216.99.216.255 broadcast 216.99.216.99 kernel link eth0 local
fe80:: 64 kernel eth0
fe80:: 64 kernel eth1
fe80:: 64 kernel eth2
default fe80::21d:70ff:feaf:d7e2 ra eth2
default unreachable kernel lo unspec
::1 :: none lo local
fe80::213:d3ff:fea6:5cfc :: none lo local
fe80::260:8ff:fe11:b59e :: none lo local
fe80::be5f:f4ff:fe54:c9d1 :: none lo local
ff00:: 8 eth0 local
ff00:: 8 eth1 local
ff00:: 8 eth2 local
default unreachable kernel lo unspec
And a speed test:
root@weasel:/etc/default# curl -o /dev/null http://speedtest.wdc01.softlayer.com/downloads/test500.zip
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
11 500M 11 55.9M 0 0 1457k 0 0:05:51 0:00:39 0:05:12 1489k^C
So a report of 1.457MB/sec means that I'm getting about 12Mb/sec which seems about right.
Making it permanent, or Where does Ubuntu put static routes?
It appears that the reasonable thing to do is put static routes into /etc/network/interfaces along with everything else. But there is a trick to allow the ifup and ifdown scripts to work: http://askubuntu.com/questions/168033/how-to-set-routes
So I'd like /etc/network/interfaces to look like this:
# The loopback network interface auto lo iface lo inet loopback # up ip route add 127.0.0.0/8 dev lo table spiritone || true # up ip route add 127.0.0.0/8 dev lo table comcast || true # This is a list of hotpluggable network interfaces. # They will be activated automatically by the hotplug subsystem. auto eth0 iface eth0 inet static pre-up iptables-restore < /etc/default/iptables address 216.99.216.99 netmask 255.255.255.0 gateway 216.99.216.1 dns-nameservers 8.8.8.8 216.99.193.19 dns-search finninday.net # up ip route add 216.99.216.0/24 eth0 src 216.99.216.99 table spiritone || true # up ip route add default via 216.99.216.1 table spiritone || true # The primary network interface # local network auto eth1 iface eth1 inet static address 10.0.0.3 netmask 255.0.0.0 # up ip route add 10.0.0.0/8 dev eth1 table spiritone || true # up ip route add 10.0.0.0/8 dev eth1 table comcast || true auto eth2 #pre-up iptables-restore < /etc/default/iptables iface eth2 inet dhcp hwaddress bc:5f:f4:54:c9:d1 gateway 24.20.234.1 # up ip route add 24.20.234.0/23 eth2 src 24.20.234.228 table comcast || true # up ip route add default via 24.20.234.1 table comcast || true # up ip route add 24.20.234.0/23 dev eth2 table spiritone || true
broken by a reboot and manual repair
I noticed my bandwidth usage dropped significantly a few days ago. The abrupt change in the graph coincides with this event:
reboot system boot 3.8.0-32-generic Sat Oct 26 09:04 - 10:19 (3+01:15)
where I had to reboot to get a new kernel.
Now, my default route is through pokey old spiritone:
root@weasel:~# route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 216.99.216.1 0.0.0.0 UG 0 0 0 eth0 10.0.0.0 0.0.0.0 255.0.0.0 U 0 0 0 eth1 169.254.0.0 0.0.0.0 255.255.0.0 U 1000 0 0 eth1 216.99.216.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0 root@weasel:~# ip route default via 216.99.216.1 dev eth0 10.0.0.0/8 dev eth1 proto kernel scope link src 10.0.0.3 169.254.0.0/16 dev eth1 scope link metric 1000 216.99.216.0/24 dev eth0 proto kernel scope link src 216.99.216.99
And eth2 is conspicuously absent from the routing table.
Speed test reports 260k download speeds. bleck.
root@weasel:~# curl -o /dev/null http://speedtest.wdc01.softlayer.com/downloads/test500.zip
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
2 500M 2 10.6M 0 0 262k 0 0:32:29 0:00:41 0:31:48 266k^C
The cool thing is that the failure was graceful in that my family hasn't noticed yet.
Did my iptables rules get applied at boot?
Yes.
But eth2 did not get an ipv4 address at boot. Just ipv6.
root@weasel:~# ifdown eth2 Internet Systems Consortium DHCP Client 4.2.4 Copyright 2004-2012 Internet Systems Consortium. All rights reserved. For info, please visit https://www.isc.org/software/dhcp/ Listening on LPF/eth2/bc:5f:f4:54:c9:d1 Sending on LPF/eth2/bc:5f:f4:54:c9:d1 Sending on Socket/fallback DHCPRELEASE on eth2 to 76.96.95.4 port 67 (xid=0x37b15006) status: Unknown job: squid root@weasel:~# ifup eth2 Internet Systems Consortium DHCP Client 4.2.4 Copyright 2004-2012 Internet Systems Consortium. All rights reserved. For info, please visit https://www.isc.org/software/dhcp/ Listening on LPF/eth2/bc:5f:f4:54:c9:d1 Sending on LPF/eth2/bc:5f:f4:54:c9:d1 Sending on Socket/fallback DHCPDISCOVER on eth2 to 255.255.255.255 port 67 interval 3 (xid=0x314176f4) DHCPDISCOVER on eth2 to 255.255.255.255 port 67 interval 6 (xid=0x314176f4) DHCPDISCOVER on eth2 to 255.255.255.255 port 67 interval 8 (xid=0x314176f4) DHCPREQUEST of 24.20.234.228 on eth2 to 255.255.255.255 port 67 (xid=0x314176f4) DHCPOFFER of 24.20.234.228 from 73.94.124.1 DHCPACK of 24.20.234.228 from 73.94.124.1 status: Unknown job: squid bound to 24.20.234.228 -- renewal in 1362 seconds. ssh stop/waiting ssh start/running, process 19325
Not sure why squid is getting notified about eth2 bouncing.
Now eth2 has an ipv4 address and it shows up in the routing table.
root@weasel:~# route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 216.99.216.1 0.0.0.0 UG 0 0 0 eth0 10.0.0.0 0.0.0.0 255.0.0.0 U 0 0 0 eth1 24.20.234.0 0.0.0.0 255.255.254.0 U 0 0 0 eth2 169.254.0.0 0.0.0.0 255.255.0.0 U 1000 0 0 eth1 216.99.216.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
I manually applied the routes that I still had commented out in /etc/network/interfaces and traffic is still flowing. But I need to add the rules to mark packets according to the interface they arrive on and change the master default route. Those directives are not captured in /etc/network/interfaces yet. I suspect I'll need to do a few reboots before I get the syntax right and capture everything.
Pings on spiritone:
root@weasel:~# route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 216.99.216.1 0.0.0.0 UG 0 0 0 eth0 10.0.0.0 0.0.0.0 255.0.0.0 U 0 0 0 eth1 24.20.234.0 0.0.0.0 255.255.254.0 U 0 0 0 eth2 169.254.0.0 0.0.0.0 255.255.0.0 U 1000 0 0 eth1 216.99.216.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0 root@weasel:~# ping google.com PING google.com (67.50.19.16) 56(84) bytes of data. 64 bytes from cache.google.com (67.50.19.16): icmp_req=1 ttl=60 time=73.4 ms 64 bytes from cache.google.com (67.50.19.16): icmp_req=2 ttl=60 time=65.7 ms 64 bytes from cache.google.com (67.50.19.16): icmp_req=3 ttl=60 time=69.8 ms 64 bytes from cache.google.com (67.50.19.16): icmp_req=4 ttl=60 time=50.8 ms 64 bytes from cache.google.com (67.50.19.16): icmp_req=5 ttl=60 time=47.8 ms ^C --- google.com ping statistics --- 5 packets transmitted, 5 received, 0% packet loss, time 4005ms rtt min/avg/max/mdev = 47.857/61.581/73.489/10.308 ms
Switch default routes
root@weasel:~# route del default; route add default gw 24.20.234.228
Pings on comcast:
root@weasel:~# route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 24.20.234.228 0.0.0.0 UG 0 0 0 eth2 10.0.0.0 0.0.0.0 255.0.0.0 U 0 0 0 eth1 24.20.234.0 0.0.0.0 255.255.254.0 U 0 0 0 eth2 169.254.0.0 0.0.0.0 255.255.0.0 U 1000 0 0 eth1 216.99.216.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0 root@weasel:~# ping google.com PING google.com (173.194.33.73) 56(84) bytes of data. 64 bytes from sea09s15-in-f9.1e100.net (173.194.33.73): icmp_req=1 ttl=56 time=18.6 ms 64 bytes from sea09s15-in-f9.1e100.net (173.194.33.73): icmp_req=2 ttl=56 time=11.1 ms 64 bytes from sea09s15-in-f9.1e100.net (173.194.33.73): icmp_req=3 ttl=56 time=11.5 ms 64 bytes from sea09s15-in-f9.1e100.net (173.194.33.73): icmp_req=4 ttl=56 time=11.1 ms 64 bytes from sea09s15-in-f9.1e100.net (173.194.33.73): icmp_req=5 ttl=56 time=11.8 ms ^C --- google.com ping statistics --- 5 packets transmitted, 5 received, 0% packet loss, time 4005ms rtt min/avg/max/mdev = 11.124/12.848/18.642/2.911 ms
Making it permanent, ubuntu static routes
Here is my first shot at putting my routes in /etc/network/interfaces:
auto eth0 iface eth0 inet static pre-up iptables-restore < /etc/default/iptables address 216.99.216.99 netmask 255.255.255.0 # gateway 216.99.216.1 dns-nameservers 8.8.8.8 216.99.193.19 dns-search finninday.net up ip route add 216.99.216.0/24 eth0 src 216.99.216.99 table spiritone || true up ip route add default via 216.99.216.1 table spiritone || true auto eth2 #pre-up iptables-restore < /etc/default/iptables iface eth2 inet dhcp hwaddress bc:5f:f4:54:c9:d1 gateway 24.20.234.1 up ip route add 24.20.234.0/23 eth2 src 24.20.234.228 table comcast || true up ip route add default via 24.20.234.1 table comcast || true up ip route add 24.20.234.0/23 dev eth2 table spiritone || true up ip route add default via 24.20.234.1 || true
Things to notice:
- I'm not specifying a gateway for the skinny pipe (eth0) anymore.
- I got an error when I tried to have eth2 load my iptables config in pre-up, so that is commented out and I rely on eth0 pre-up to do it.
- I am setting my mac address on eth2 to match what comcast expects
- I'm trying to set a default route for all interfaces in the last command in eth2 up
After another reboot, still broken
- first problem, eth2 was not active after reboot
I manually did "ifup eth2" and it looked like things were working, but routing did not allow for services on the spiritone interface
- I tried manually adding all my static routes. Some of them were already present and threw errors:
root@weasel:/etc/network# ip route add 216.99.216.0/24 eth0 src 216.99.216.99 table spiritone Error: either "to" is duplicate, or "eth0" is a garbage. root@weasel:/etc/network# ip route add default via 216.99.216.1 table spiritone RTNETLINK answers: File exists root@weasel:/etc/network# ip route add 24.20.234.0/23 dev eth2 src 24.20.234.228 table comcast root@weasel:/etc/network# ip route add default via 24.20.234.1 table comcast RTNETLINK answers: File exists root@weasel:/etc/network# ip rule add from 216.99.216.99 table spiritone root@weasel:/etc/network# ip rule add from 24.20.234.228 table comcast root@weasel:/etc/network# ip route add 10.0.0.0/8 dev eth1 table spiritone RTNETLINK answers: File exists root@weasel:/etc/network# ip route add 24.20.234.0/23 dev eth2 table spiritone RTNETLINK answers: File exists root@weasel:/etc/network# ip route add 127.0.0.0/8 dev lo table spiritone RTNETLINK answers: File exists root@weasel:/etc/network# ip route add 10.0.0.0/8 dev eth1 table comcast RTNETLINK answers: File exists root@weasel:/etc/network# ip route add 216.99.216.0/24 dev eth0 table comcast root@weasel:/etc/network# ip route add 127.0.0.0/8 dev lo table comcast
But even after adding those routes, comcast had lots of packet loss and the spiritone interface was not serving mail and web.
got a new IP address assigned by comcast
Now I need to recalculate this:
T1 = spiritone T2 = comcast IF1 = eth0 IF2 = eth2 IP1 = 216.99.216.99 IP2 = 67.189.73.245 P1 = 216.99.216.1 P2 = 67.189.72.1 (not sure about how to find this without actually sending packets) P1_net = 216.99.216.0/24 P2_net = 67.189.72.0/22
clean out routes for old ip address
# ip route show table comcast 10.0.0.0/8 dev eth1 scope link 127.0.0.0/8 dev lo scope link 216.99.216.0/24 dev eth0 scope link # ip route del 216.99.216.0/24 table comcast # ip route show table comcast 10.0.0.0/8 dev eth1 scope link 127.0.0.0/8 dev lo scope link
- edit /etc/hosts for comcast-router
- edit /etc/network/interfaces
- edit /etc/default/iptables
Switch default route
root@weasel:~# route del default; route add default gw 67.189.72.1
success again
mtr
root@weasel:/etc/network# mtr -r -c 1 google.com HOST: weasel.finninday.net Loss% Snt Last Avg Best Wrst StDev 1.|-- ??? 100.0 1 0.0 0.0 0.0 0.0 0.0 2.|-- te-0-0-0-1-ur03.troutdale 0.0% 1 8.5 8.5 8.5 8.5 0.0 3.|-- ae-4-0-ar03.beaverton.or. 0.0% 1 10.4 10.4 10.4 10.4 0.0 4.|-- ae-1-0-ar03.troutdale.or. 0.0% 1 14.9 14.9 14.9 14.9 0.0 5.|-- he-2-2-0-0-10-cr01.seattl 0.0% 1 12.4 12.4 12.4 12.4 0.0 6.|-- be-12-pe03.seattle.wa.ibo 0.0% 1 13.7 13.7 13.7 13.7 0.0 7.|-- 75.149.231.90 0.0% 1 12.4 12.4 12.4 12.4 0.0 8.|-- 209.85.249.32 0.0% 1 11.9 11.9 11.9 11.9 0.0 9.|-- 209.85.244.61 0.0% 1 12.3 12.3 12.3 12.3 0.0 10.|-- sea09s17-in-f9.1e100.net 0.0% 1 11.7 11.7 11.7 11.7 0.0
interfaces
auto eth0 iface eth0 inet static pre-up iptables-restore < /etc/default/iptables address 216.99.216.99 netmask 255.255.255.0 # gateway 216.99.216.1 dns-nameservers 8.8.8.8 216.99.193.19 dns-search finninday.net up ip route add 216.99.216.0/24 eth0 src 216.99.216.99 table spiritone || true up ip route add default via 216.99.216.1 table spiritone || true # local unrouted network auto eth1 iface eth1 inet static address 10.0.0.3 netmask 255.0.0.0 up ip route add 10.0.0.0/8 dev eth1 table spiritone || true up ip route add 10.0.0.0/8 dev eth1 table comcast || true auto eth2 iface eth2 inet dhcp hwaddress bc:5f:f4:54:c9:d1 # gateway 24.20.234.1 up ip route add 67.189.72.0/22 eth2 src 67.189.73.245 table comcast || true up ip route add default via 67.189.72.1 table comcast || true up ip route add 67.189.72.0/22 dev eth2 table spiritone || true up ip route add default via 67.189.72.1 || true
iptables
root@weasel:/etc/network# iptables -L -nv | grep eth2
11028 4672K ACCEPT all -- eth2 * 0.0.0.0/0 67.189.73.245 state RELATED,ESTABLISHED
0 0 ACCEPT tcp -- eth2 * 75.75.75.75 0.0.0.0/0 tcp dpt:53
1149 271K DROP all -- eth2 * 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT all -- * eth2 0.0.0.0/0 0.0.0.0/0 state NEW
0 0 ACCEPT all -- eth2 eth1 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
0 0 ACCEPT all -- eth1 eth2 0.0.0.0/0 0.0.0.0/0
12236 937K ACCEPT all -- * eth2 67.189.73.245 0.0.0.0/0
ping times
root@weasel:/etc/network# ping comcast-router PING comcast-router.finninday.net (67.189.72.1) 56(84) bytes of data. 64 bytes from comcast-router.finninday.net (67.189.72.1): icmp_req=1 ttl=255 time=8.17 ms 64 bytes from comcast-router.finninday.net (67.189.72.1): icmp_req=2 ttl=255 time=6.54 ms ^C --- comcast-router.finninday.net ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1001ms rtt min/avg/max/mdev = 6.542/7.357/8.173/0.819 ms root@weasel:/etc/network# ping spiritone-router PING spiritone-router.finninday.net (216.99.216.1) 56(84) bytes of data. 64 bytes from spiritone-router.finninday.net (216.99.216.1): icmp_req=1 ttl=64 time=42.4 ms 64 bytes from spiritone-router.finninday.net (216.99.216.1): icmp_req=2 ttl=64 time=42.6 ms ^C --- spiritone-router.finninday.net ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1001ms rtt min/avg/max/mdev = 42.484/42.561/42.638/0.077 ms
ip route show
# ip route show default via 67.189.72.1 dev eth2 10.0.0.0/8 dev eth1 proto kernel scope link src 10.0.0.3 67.189.72.0/22 dev eth2 proto kernel scope link src 67.189.73.245 169.254.0.0/16 dev eth1 scope link metric 1000 216.99.216.0/24 dev eth0 proto kernel scope link src 216.99.216.99 # ip route show table comcast default via 67.189.72.1 dev eth2 10.0.0.0/8 dev eth1 scope link 127.0.0.0/8 dev lo scope link # ip route show table spiritone default via 216.99.216.1 dev eth0 10.0.0.0/8 dev eth1 scope link 67.189.72.0/22 dev eth2 scope link 127.0.0.0/8 dev lo scope link
ip neigh
root@weasel:/etc/network# ip neigh fe80::21d:70ff:feaf:d7e2 dev eth2 lladdr 00:1d:70:af:d7:e2 router STALE 216.99.216.1 dev eth0 lladdr 00:10:67:00:71:08 REACHABLE 67.189.72.1 dev eth2 lladdr 00:1d:70:af:d7:e2 REACHABLE
routel main
root@weasel:/etc/network# routel main
target gateway source proto scope dev tbl
default 67.189.72.1 eth2
10.0.0.0 8 10.0.0.3 kernel link eth1
67.189.72.0 22 67.189.73.245 kernel link eth2
169.254.0.0 16 link eth1
216.99.216.0 24 216.99.216.99 kernel link eth0
routel comcast
root@weasel:/etc/network# routel comcast
target gateway source proto scope dev tbl
default 67.189.72.1 eth2
10.0.0.0 8 link eth1
127.0.0.0 8 link lo
routel spiritone
root@weasel:/etc/network# routel spiritone
target gateway source proto scope dev tbl
default 216.99.216.1 eth0
10.0.0.0 8 link eth1
67.189.72.0 22 link eth2
127.0.0.0 8 link lo
practicing a reboot
/etc/network/interfaces is still not correct because after a reboot, eth2 did not have an ipv4 address.
root@weasel:~# ifconfig
eth2 Link encap:Ethernet HWaddr bc:5f:f4:54:c9:d1
inet6 addr: fe80::be5f:f4ff:fe54:c9d1/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:1532 errors:0 dropped:0 overruns:0 frame:0
TX packets:305 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:122299 (122.2 KB) TX bytes:68801 (68.8 KB)
Interrupt:19 Memory:fd9c0000-fd9e0000
I issued "dhclient eth2" to fix that
root@weasel:~# dhclient eth2
As soon as I issued the "dhclient eth2", the ping from a NAT'ed machine in the network to 8.8.8.8 started to succeed.
But then routes for eth2 in the comcast and spiritone tables were not in place:
root@weasel:~# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 67.189.72.1 0.0.0.0 UG 0 0 0 eth2
10.0.0.0 0.0.0.0 255.0.0.0 U 0 0 0 eth1
67.189.72.0 0.0.0.0 255.255.252.0 U 0 0 0 eth2
169.254.0.0 0.0.0.0 255.255.0.0 U 1000 0 0 eth1
216.99.216.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
root@weasel:~# routel main
target gateway source proto scope dev tbl
default 67.189.72.1 eth2
10.0.0.0 8 10.0.0.3 kernel link eth1
67.189.72.0 22 67.189.73.245 kernel link eth2
169.254.0.0 16 link eth1
216.99.216.0 24 216.99.216.99 kernel link eth0
root@weasel:~# routel comcast
target gateway source proto scope dev tbl
10.0.0.0 8 link eth1
127.0.0.0 8 link lo
root@weasel:~# routel spiritone
target gateway source proto scope dev tbl
default 216.99.216.1 eth0
10.0.0.0 8 link eth1
127.0.0.0 8 link lo
I issued "ifdown/ifup eth2" to fix that.
root@weasel:~# ifdown eth2 Internet Systems Consortium DHCP Client 4.2.4 Copyright 2004-2012 Internet Systems Consortium. All rights reserved. For info, please visit https://www.isc.org/software/dhcp/ Listening on LPF/eth2/bc:5f:f4:54:c9:d1 Sending on LPF/eth2/bc:5f:f4:54:c9:d1 Sending on Socket/fallback DHCPRELEASE on eth2 to 76.96.95.4 port 67 (xid=0x44d97468) status: Unknown job: squid root@weasel:~# ifup eth2 Internet Systems Consortium DHCP Client 4.2.4 Copyright 2004-2012 Internet Systems Consortium. All rights reserved. For info, please visit https://www.isc.org/software/dhcp/ Listening on LPF/eth2/bc:5f:f4:54:c9:d1 Sending on LPF/eth2/bc:5f:f4:54:c9:d1 Sending on Socket/fallback DHCPDISCOVER on eth2 to 255.255.255.255 port 67 interval 3 (xid=0x4108a47c) DHCPDISCOVER on eth2 to 255.255.255.255 port 67 interval 4 (xid=0x4108a47c) DHCPREQUEST of 67.189.73.245 on eth2 to 255.255.255.255 port 67 (xid=0x4108a47c) DHCPOFFER of 67.189.73.245 from 73.94.124.1 DHCPACK of 67.189.73.245 from 73.94.124.1 status: Unknown job: squid bound to 67.189.73.245 -- renewal in 1601 seconds. Error: either "to" is duplicate, or "eth2" is a garbage. RTNETLINK answers: File exists ssh stop/waiting ssh start/running, process 4756
At this point the routing tables look right:
root@weasel:~# routel spiritone
target gateway source proto scope dev tbl
default 216.99.216.1 eth0
10.0.0.0 8 link eth1
67.189.72.0 22 link eth2
127.0.0.0 8 link lo
root@weasel:~# routel comcast
target gateway source proto scope dev tbl
default 67.189.72.1 eth2
10.0.0.0 8 link eth1
127.0.0.0 8 link lo
root@weasel:~# routel main
target gateway source proto scope dev tbl
default 67.189.72.1 eth2
10.0.0.0 8 10.0.0.3 kernel link eth1
67.189.72.0 22 67.189.73.245 kernel link eth2
169.254.0.0 16 link eth1
216.99.216.0 24 216.99.216.99 kernel link eth0
external services fail
Also, my test of external services fails to deliver a web page to my cell phone. That's troubling because it looks like the iptables and routing tables are all correct. What did I miss?
To make it work, I did "ifdown eth2" and set the default gw to spiritone and now external services work.
ifdown eth2 root@weasel:~# route del default; route add default gw 216.99.216.1
What do the routing tables look like now?
root@weasel:~# routel main
target gateway source proto scope dev tbl
default 216.99.216.1 eth0
10.0.0.0 8 10.0.0.3 kernel link eth1
169.254.0.0 16 link eth1
216.99.216.0 24 216.99.216.99 kernel link eth0
root@weasel:~# routel spiritone
target gateway source proto scope dev tbl
default 216.99.216.1 eth0
10.0.0.0 8 link eth1
127.0.0.0 8 link lo
root@weasel:~# routel comcast
target gateway source proto scope dev tbl
10.0.0.0 8 link eth1
127.0.0.0 8 link lo
Now to turn comcast back on...
root@weasel:~# ifup eth2 Internet Systems Consortium DHCP Client 4.2.4 Copyright 2004-2012 Internet Systems Consortium. All rights reserved. For info, please visit https://www.isc.org/software/dhcp/ Listening on LPF/eth2/bc:5f:f4:54:c9:d1 Sending on LPF/eth2/bc:5f:f4:54:c9:d1 Sending on Socket/fallback DHCPDISCOVER on eth2 to 255.255.255.255 port 67 interval 3 (xid=0x15c19324) DHCPDISCOVER on eth2 to 255.255.255.255 port 67 interval 4 (xid=0x15c19324) DHCPDISCOVER on eth2 to 255.255.255.255 port 67 interval 8 (xid=0x15c19324) DHCPREQUEST of 67.189.73.245 on eth2 to 255.255.255.255 port 67 (xid=0x15c19324) DHCPOFFER of 67.189.73.245 from 73.94.124.1 DHCPREQUEST of 67.189.73.245 on eth2 to 255.255.255.255 port 67 (xid=0x15c19324) DHCPACK of 67.189.73.245 from 73.94.124.1 status: Unknown job: squid bound to 67.189.73.245 -- renewal in 1621 seconds. Error: either "to" is duplicate, or "eth2" is a garbage. RTNETLINK answers: File exists ssh stop/waiting ssh start/running, process 7256
Routing tables?
root@weasel:~# routel main
target gateway source proto scope dev tbl
default 216.99.216.1 eth0
10.0.0.0 8 10.0.0.3 kernel link eth1
67.189.72.0 22 67.189.73.245 kernel link eth2
169.254.0.0 16 link eth1
216.99.216.0 24 216.99.216.99 kernel link eth0
root@weasel:~# routel comcast
target gateway source proto scope dev tbl
default 67.189.72.1 eth2
10.0.0.0 8 link eth1
127.0.0.0 8 link lo
root@weasel:~# routel spiritone
target gateway source proto scope dev tbl
default 216.99.216.1 eth0
10.0.0.0 8 link eth1
67.189.72.0 22 link eth2
127.0.0.0 8 link lo
External services still work at this point.
Change default route...
route del default; route add default gw 67.189.72.1
External services fail.
Change default route back
route del default; route add default gw 216.99.216.1
External services work again.
Maybe there are extraneous and harmful routes being applied when eth2 comes up.
remove a route from the spiritone table
root@weasel:/etc/network# routel comcast
target gateway source proto scope dev tbl
default 67.189.72.1 eth2
10.0.0.0 8 link eth1
127.0.0.0 8 link lo
root@weasel:/etc/network# routel spiritone
target gateway source proto scope dev tbl
default 216.99.216.1 eth0
10.0.0.0 8 link eth1
67.189.72.0 22 link eth2
127.0.0.0 8 link lo
root@weasel:/etc/network# ip route del 67.189.72.0/22 table spiritone
root@weasel:/etc/network# routel spiritone
target gateway source proto scope dev tbl
default 216.99.216.1 eth0
10.0.0.0 8 link eth1
127.0.0.0 8 link lo
try switching the default route
External services work. Switch default to comcast.
# route del default; route add default gw 67.189.72.1
external services fail.
root@weasel:/etc/network# routel main
target gateway source proto scope dev tbl
default 67.189.72.1 eth2
10.0.0.0 8 10.0.0.3 kernel link eth1
67.189.72.0 22 67.189.73.245 kernel link eth2
169.254.0.0 16 link eth1
216.99.216.0 24 216.99.216.99 kernel link eth0
root@weasel:/etc/network# routel comcast
target gateway source proto scope dev tbl
default 67.189.72.1 eth2
10.0.0.0 8 link eth1
127.0.0.0 8 link lo
root@weasel:/etc/network# routel spiritone
target gateway source proto scope dev tbl
default 216.99.216.1 eth0
10.0.0.0 8 link eth1
127.0.0.0 8 link lo
Switch default route back.
root@weasel:~# route del default; route add default gw 216.99.216.1
external services work.
root@weasel:~# routel main
target gateway source proto scope dev tbl
default 216.99.216.1 eth0
10.0.0.0 8 10.0.0.3 kernel link eth1
67.189.72.0 22 67.189.73.245 kernel link eth2
169.254.0.0 16 link eth1
216.99.216.0 24 216.99.216.99 kernel link eth0
root@weasel:~# routel comcast
target gateway source proto scope dev tbl
10.0.0.0 8 link eth1
127.0.0.0 8 link lo
root@weasel:~# routel spiritone
target gateway source proto scope dev tbl
default 216.99.216.1 eth0
10.0.0.0 8 link eth1
127.0.0.0 8 link lo
examine a packet capture to see what is happening
first a normal transaction from a remote host to my web server
- on the client
[ryan.day@bash01 ~]$ curl finninday.net
- on the server
root@weasel:~# tcpdump -X port 80 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
- three-way handshake
09:41:57.800722 IP 70.103.8.101.52417 > 216-99-216-99.static.dsl.spiritone.com.http: Flags [S], seq 4242878906, win 14600, options [mss 1380,sackOK,TS val 1291124496 ecr 0,nop,wscale 7], length 0 09:41:57.800830 IP 216-99-216-99.static.dsl.spiritone.com.http > 70.103.8.101.52417: Flags [S.], seq 3606047033, ack 4242878907, win 14480, options [mss 1460,sackOK,TS val 360929 ecr 1291124496,nop,wscale 7], length 0 09:41:57.846749 IP 70.103.8.101.52417 > 216-99-216-99.static.dsl.spiritone.com.http: Flags [.], ack 1, win 115, options [nop,nop,TS val 1291124544 ecr 360929], length 0
- http get
09:41:57.848736 IP 70.103.8.101.52417 > 216-99-216-99.static.dsl.spiritone.com.http: Flags [P.], seq 1:169, ack 1, win 115, options [nop,nop,TS val 1291124544 ecr 360929], length 168 0x0000: 4500 00dc bc04 4000 3a06 8484 4667 0865 E.....@.:...Fg.e 0x0010: d863 d863 ccc1 0050 fce5 31bb d6ef e93a .c.c...P..1....: 0x0020: 8018 0073 fe42 0000 0101 080a 4cf4 ff40 ...s.B......L..@ 0x0030: 0005 81e1 4745 5420 2f20 4854 5450 2f31 ....GET./.HTTP/1 0x0040: 2e31 0d0a 5573 6572 2d41 6765 6e74 3a20 .1..User-Agent:. 0x0050: 6375 726c 2f37 2e31 392e 3720 2878 3836 curl/7.19.7.(x86 0x0060: 5f36 342d 7265 6468 6174 2d6c 696e 7578 _64-redhat-linux 0x0070: 2d67 6e75 2920 6c69 6263 7572 6c2f 372e -gnu).libcurl/7. 0x0080: 3139 2e37 204e 5353 2f33 2e31 332e 312e 19.7.NSS/3.13.1. 0x0090: 3020 7a6c 6962 2f31 2e32 2e33 206c 6962 0.zlib/1.2.3.lib 0x00a0: 6964 6e2f 312e 3138 206c 6962 7373 6832 idn/1.18.libssh2 0x00b0: 2f31 2e32 2e32 0d0a 486f 7374 3a20 6669 /1.2.2..Host:.fi 0x00c0: 6e6e 696e 6461 792e 6e65 740d 0a41 6363 nninday.net..Acc 0x00d0: 6570 743a 202a 2f2a 0d0a 0d0a ept:.*/*....
- ack
09:41:57.848824 IP 216-99-216-99.static.dsl.spiritone.com.http > 70.103.8.101.52417: Flags [.], ack 169, win 122, options [nop,nop,TS val 360941 ecr 1291124544], length 0
- http 200
09:41:57.849627 IP 216-99-216-99.static.dsl.spiritone.com.http > 70.103.8.101.52417: Flags [P.], seq 1:339, ack 169, win 122, options [nop,nop,TS val 360941 ecr 1291124544], length 338 0x0000: 4500 0186 68d8 4000 4006 d106 d863 d863 E...h.@.@....c.c 0x0010: 4667 0865 0050 ccc1 d6ef e93a fce5 3263 Fg.e.P.....:..2c 0x0020: 8018 007a 9f4d 0000 0101 080a 0005 81ed ...z.M.......... 0x0030: 4cf4 ff40 4854 5450 2f31 2e31 2032 3030 L..@HTTP/1.1.200 0x0040: 204f 4b0d 0a44 6174 653a 2053 6174 2c20 .OK..Date:.Sat,.
- ack
09:41:57.906766 IP 70.103.8.101.52417 > 216-99-216-99.static.dsl.spiritone.com.http: Flags [.], ack 339, win 123, options [nop,nop,TS val 1291124598 ecr 360941], length 0
- fin
09:41:57.906769 IP 70.103.8.101.52417 > 216-99-216-99.static.dsl.spiritone.com.http: Flags [F.], seq 169, ack 339, win 123, options [nop,nop,TS val 1291124598 ecr 360941], length 0
- fin
09:41:57.907017 IP 216-99-216-99.static.dsl.spiritone.com.http > 70.103.8.101.52417: Flags [F.], seq 339, ack 170, win 122, options [nop,nop,TS val 360956 ecr 1291124598], length 0 09:41:57.952728 IP 70.103.8.101.52417 > 216-99-216-99.static.dsl.spiritone.com.http: Flags [.], ack 340, win 123, options [nop,nop,TS val 1291124650 ecr 360956], length 0 ^C 10 packets captured 10 packets received by filter 0 packets dropped by kernel
now switch the default route and retry
- on the server
root@weasel:~# route del default; route add default gw 67.189.72.1 root@weasel:~# ping 8.8.8.8 PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data. 64 bytes from 8.8.8.8: icmp_req=1 ttl=48 time=20.6 ms 64 bytes from 8.8.8.8: icmp_req=2 ttl=48 time=19.5 ms 64 bytes from 8.8.8.8: icmp_req=3 ttl=48 time=19.2 ms 64 bytes from 8.8.8.8: icmp_req=4 ttl=48 time=18.8 ms
root@weasel:~# tcpdump -X port 80 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
- syn
10:02:24.539742 IP 70.103.8.101.53638 > 216-99-216-99.static.dsl.spiritone.com.http: Flags [S], seq 2119244259, win 14600, options [mss 1380,sackOK,TS val 1292351134 ecr 0,nop,wscale 7], length 0
- syn
10:02:25.577731 IP 70.103.8.101.53638 > 216-99-216-99.static.dsl.spiritone.com.http: Flags [S], seq 2119244259, win 14600, options [mss 1380,sackOK,TS val 1292352134 ecr 0,nop,wscale 7], length 0
- syn
10:02:27.559711 IP 70.103.8.101.53638 > 216-99-216-99.static.dsl.spiritone.com.http: Flags [S], seq 2119244259, win 14600, options [mss 1380,sackOK,TS val 1292354134 ecr 0,nop,wscale 7], length 0
- syn
10:02:31.561660 IP 70.103.8.101.53638 > 216-99-216-99.static.dsl.spiritone.com.http: Flags [S], seq 2119244259, win 14600, options [mss 1380,sackOK,TS val 1292358134 ecr 0,nop,wscale 7], length 0
Where are the acks going? Not on eth0...
recover from test by setting the default route back
# route del default; route add default gw 216.99.216.1
and verify from remote client
[ryan.day@bash01 ~]$ curl finninday.net <html> <head> <meta http-equiv="refresh" content="0; URL=http://finninday.net/wiki"> </head> </html>
missing rules to pin responses to the originating network
root@weasel:~# ip rule 0: from all lookup local 32766: from all lookup main 32767: from all lookup default root@weasel:~# ip rule add from 216.99.216.99 table spiritone root@weasel:~# ip rule add from 24.20.234.228 table comcast root@weasel:~# ip rule 0: from all lookup local 32764: from 24.20.234.228 lookup comcast 32765: from 216.99.216.99 lookup spiritone 32766: from all lookup main 32767: from all lookup default
Losing this rule with reboot would cause the symptom that I'm seeing of responses going out over different networks than the corresponding requests.
Now that we have the rules in place, switch test switching default route.
Changes to /etc/network/interfaces
root@weasel:/etc/network# diff -u interfaces /tmp/interfaces --- interfaces 2013-11-16 11:07:43.000000000 -0800 +++ /tmp/interfaces 2013-11-16 11:06:53.000000000 -0800 @@ -20,6 +20,7 @@ dns-search finninday.net up ip route add 216.99.216.0/24 eth0 src 216.99.216.99 table spiritone || true up ip route add default via 216.99.216.1 table spiritone || true + up ip rule add from 216.99.216.99 table spiritone || true # The primary network interface @@ -38,7 +39,9 @@ up ip route add 67.189.72.0/22 eth2 src 67.189.73.245 table comcast || true up ip route add default via 67.189.72.1 table comcast || true up ip route add 67.189.72.0/22 dev eth2 table spiritone || true + up route del default || true up ip route add default via 67.189.72.1 || true + up ip rule add from 24.20.234.228 table comcast || true #Tunnel to work #iface tun0 inet static
Disabled network manager
On reboot, dhclient on eth2 would succeed, but then the ip would be discarded and the interface left without an address. I tried various goofy hacks in /etc/network/interfaces, like having an extra "post up dhclient eth2", but to no avail. The right thing to do turned out to be to disable network manager so it didn't throw out the address. Funny that I still had network manager active on this server for so long and never noticed until now. Shame on me.
Disable like this:
# echo "manual" > /etc/init/network-manager.override
This proves to me that network manager is disabled:
# nm-tool ** (process:5234): WARNING **: Could not initialize NMClient /org/freedesktop/NetworkManager: The name org.freedesktop.NetworkManager was not provided by any .service files NetworkManager Tool State: unknown
Now after a reboot, everything comes up clean and there is no odd repair work to do.
On Fedora, disable network manager with
systemctl stop NetworkManager systemctl disable NetworkManager nmcli n
On Fedora22, if you disable network manager, you must enable legacy network with this:
chkconfig network on
It took me a long time to notice that wasn't on...
Full /etc/network/interfaces
After a few successful reboots, I'm convinced this is a solid configuration.
# This file describes the network interfaces available on your system # and how to activate them. For more information, see interfaces(5). # The loopback network interface auto lo iface lo inet loopback up ip route add 127.0.0.0/8 dev lo table spiritone || true up ip route add 127.0.0.0/8 dev lo table comcast || true # This is a list of hotpluggable network interfaces. # They will be activated automatically by the hotplug subsystem. # spiritone auto eth0 iface eth0 inet static pre-up iptables-restore < /etc/default/iptables address 216.99.216.99 netmask 255.255.255.0 # gateway 216.99.216.1 dns-nameservers 8.8.8.8 216.99.193.19 dns-search finninday.net up ip route add 216.99.216.0/24 eth0 src 216.99.216.99 table spiritone || true up ip route add default via 216.99.216.1 table spiritone || true up ip rule add from 216.99.216.99 table spiritone || true # The primary network interface # local unrouted network auto eth1 iface eth1 inet static address 10.0.0.3 netmask 255.0.0.0 up ip route add 10.0.0.0/8 dev eth1 table spiritone || true up ip route add 10.0.0.0/8 dev eth1 table comcast || true # comcast auto eth2 iface eth2 inet dhcp hwaddress bc:5f:f4:54:c9:d1 up ip route add 67.189.72.0/22 eth2 src 67.189.73.245 table comcast || true up ip route add default via 67.189.72.1 table comcast || true up ip route add 67.189.72.0/22 dev eth2 table spiritone || true up route del default || true up ip route add default via 67.189.72.1 || true up ip rule add from 24.20.234.228 table comcast || true
when comcast borks, how do I manually switch my route to spiritone?
On Nov 10, 2014, Comcast packet loss went over 40% for a short time. My music streaming died. As I was reading this document to figure out how to switch my traffic manually over to spiritone, the packet loss issue went away. So I haven't tested the procedure to flip networks yet.
For future reference, it should be these commands:
- set comcast as default route
route del default ip route add default via 67.189.72.1
- set spiritone as default route
route del default ip route add default via 216.99.216.1
This command shows the current settings:
root@weasel:~# routel | grep default
default 216.99.216.1 eth0 spiritone
default 67.189.72.1 eth2 comcast
default 67.189.72.1 eth2
default fe80::1edf:fff:fe02:2ee2 ra eth2
default unreachable kernel lo unspec
default unreachable kernel lo unspec
So currently all IPv4 in the main table is set to go out through comcast on eth2.
Likewise, this shows comcast as the default route (ignoring internal interfaces):
root@weasel:~# ip route | grep -v eth1 default via 67.189.72.1 dev eth2 67.189.72.0/22 dev eth2 proto kernel scope link src 67.189.73.245 216.99.216.0/24 dev eth0 proto kernel scope link src 216.99.216.99
spiritone changes authentication scheme
Spiritone switches to pppoe for my adsl modem. I think the previous protocol was 1483 Bridged IP. After the changes, my first outbound hop becomes 96.120.60.169.
For instance, with comcast as my default route,
root@weasel:~# mtr -r -c 1 -n google.com Start: Tue Jun 16 14:06:51 2015 HOST: weasel.finninday.net Loss% Snt Last Avg Best Wrst StDev 1.|-- 96.120.60.169 0.0% 1 7.9 7.9 7.9 7.9 0.0 2.|-- 68.87.218.153 0.0% 1 8.5 8.5 8.5 8.5 0.0 3.|-- 68.87.222.194 0.0% 1 9.0 9.0 9.0 9.0 0.0 4.|-- 68.87.216.253 0.0% 1 9.1 9.1 9.1 9.1 0.0 5.|-- 68.86.93.25 0.0% 1 14.4 14.4 14.4 14.4 0.0 6.|-- 68.86.84.198 0.0% 1 14.2 14.2 14.2 14.2 0.0 7.|-- 66.208.233.242 0.0% 1 12.9 12.9 12.9 12.9 0.0 8.|-- 72.14.234.208 0.0% 1 13.0 13.0 13.0 13.0 0.0 9.|-- 209.85.244.61 0.0% 1 12.6 12.6 12.6 12.6 0.0 10.|-- 173.194.33.135 0.0% 1 12.3 12.3 12.3 12.3 0.0
root@weasel:~# tracepath -n google.com 1?: [LOCALHOST] pmtu 1500 1: 96.120.60.169 15.777ms 1: 96.120.60.169 8.907ms 2: 68.87.218.153 8.201ms 3: 68.87.222.194 10.512ms asymm 2 4: 68.85.243.137 12.052ms 5: 68.86.93.25 17.854ms asymm 4 6: 68.86.84.198 15.510ms asymm 5 7: no reply 8: no reply ^C
And one more mtr, but with names enabled
root@weasel:/etc# mtr -r google.com Start: Tue Jun 16 14:28:58 2015 HOST: weasel.finninday.net Loss% Snt Last Avg Best Wrst StDev 1.|-- 96.120.60.169 0.0% 10 8.3 8.6 7.9 9.4 0.0 2.|-- GE-2-44-ur01.troutdale.or 0.0% 10 8.3 8.6 7.3 10.2 0.7 3.|-- et-5-0-0-1-0-ar03.troutda 0.0% 10 8.2 10.3 6.1 23.4 4.8 4.|-- he-0-4-0-0-11-cr02.seattl 0.0% 10 14.3 14.0 12.3 15.2 0.7 5.|-- he-0-10-0-1-pe05.seattle. 0.0% 10 14.0 13.2 11.8 15.5 0.9 6.|-- as35176-3.11greatoaks.ca. 0.0% 10 12.2 11.8 8.1 15.6 1.8 7.|-- 72.14.234.208 0.0% 10 11.7 12.4 8.7 14.9 1.5 8.|-- 216.239.51.159 0.0% 10 13.1 13.2 9.8 19.0 2.4 9.|-- sea15s01-in-f142.1e100.ne 0.0% 10 13.8 12.3 11.4 13.8 0.6
working ubuntu config for two ISP providers
I got things working, but didn't really feel confident about it until I tried to migrate everything to fedora. I still haven't proven that I can load balance across the two links, but I think it is possible.
- make sure forwarding is enabled
echo 1 > /proc/sys/net/ipv4/ip_forward
or use /etc/sysctl.d
# cat net.ipv4.ip_forward.conf net.ipv4.ip_forward = 1
- define custom routing tables in /etc/iproute2/rt_tables:
# # reserved values # 255 local 254 main 253 default 0 unspec # # local # #1 inr.ruhep 1 spiritone 2 comcast
- create those routing tables for each interface in /etc/network/interfaces
# The loopback network interface
auto lo
iface lo inet loopback
up ip route add 127.0.0.0/8 dev lo table spiritone || true
up ip route add 127.0.0.0/8 dev lo table comcast || true
# spiritone
auto eth0
iface eth0 inet static
pre-up iptables-restore < /etc/default/iptables
pre-up ip6tables-restore < /etc/default/ip6tables
address {device_ip}
netmask 255.255.255.240
dns-nameservers 8.8.8.8 8.8.4.4
dns-search finninday.net
up ip route add {cidr_network_range} eth0 src {device_ip} table spiritone || true
up ip route add default via {gateway_ip} table spiritone || true
up ip rule add from {device_ip} table spiritone || true
# local unrouted network
auto eth1
iface eth1 inet static
address 10.0.0.3
netmask 255.0.0.0
up ip route add 10.0.0.0/8 dev eth1 table spiritone || true
up ip route add 10.0.0.0/8 dev eth1 table comcast || true
# comcast
auto eth2
iface eth2 inet dhcp
hwaddress bc:5f:f4:54:c9:d1
up ip route add {cidr_network_range} eth2 src {device_ip} table comcast || true
up ip route add default via {gateway_ip} table comcast || true
up ip route add {cidr_network_range} dev eth2 table spiritone || true
up route del default || true
up ip route add default via {gateway_ip} || true
up ip rule add from {device_ip} table comcast || true
- set up iptables to do natting and stuff
# eth1 = trusted link to LAN # eth0 = untrusted link to spiritone # eth2 = untrusted link to comcast *nat :PREROUTING ACCEPT [0:0] :POSTROUTING ACCEPT [0:0] :OUTPUT ACCEPT [0:0] [0:0] -A POSTROUTING -o eth0 -j MASQUERADE [0:0] -A POSTROUTING -o eth2 -j MASQUERADE COMMIT # insert your rules to block and accept as appropriate for you
