Two isp routing

From Finninday
Revision as of 19:07, 7 October 2013 by Rday (Talk | contribs)

Jump to: navigation, search

I'm using Ubuntu.

I have two ISPs. I didn't plan on getting two ISPs. I've had one for ages that I'm reasonably happy with that I've shopped long and hard for. It gives me a static IP and doesn't charge too much. It has a 1Mbps down and 370Kbps up.

Then my son wanted to vastly upgrade our connectivity and our existing link just couldn't fill the bill. So we got another. His link is 30Mbps down and 3Mbps up. I want some of that. But I don't want to disrupt the existing services on my static IP and the original link.

My routing table looks like this now. 

root@weasel:/# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         216.99.216.1    0.0.0.0         UG    0      0        0 eth0
10.0.0.0        0.0.0.0         255.0.0.0       U     0      0        0 eth1
169.254.0.0     0.0.0.0         255.255.0.0     U     1000   0        0 eth1
216.99.216.0    0.0.0.0         255.255.255.0   U     0      0        0 eth0

I want both links to be active. It looks like a lot of my questions will be answered here: http://www.linuxhorizon.ro/iproute2.html

Not sure if I'll have to manually configure the routes. Starting with the default here: 

root@weasel:/# cat /etc/iproute2/rt_tables 
#
# reserved values
#
255	local
254	main
253	default
0	unspec
#
# local
#
#1	inr.ruhep

No response from comcast dhcp

The service guy set up the connection to a windows machine that works just fine. When I try to connect with my laptop or server, I get no response from the dhcp server. Actually, that's not quite true. I get an IPv6 address, but not an IPv4. The windows machine gets an IPv4 just fine.

Maybe I have to spoof my mac address to proceed. Here is my old laptop mac address, so I can change it back:

f0:1f:af:16:f3:39

Here is the mac address on the windows machine:

bc:5f:f4:54:c9:d1

Switch laptop mac address to match what comcast expects. 

ifconfig em1 hw ether bc:5f:f4:54:c9:d1

ifconfig em1 reports that it was successfully changed.

yay, it worked.

And better yet, it was easy to clean up. 

ifdown em1
ifconfig em1

reports that I've got my old mac address again.

Now test on server. 

# original mac address here

eth2      Link encap:Ethernet  HWaddr 68:05:ca:19:af:e0  
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
          Interrupt:19 Memory:fd9c0000-fd9e0000 

root@weasel:~# ifconfig eth2 hw ether bc:5f:f4:54:c9:d1
root@weasel:~# ifconfig eth2
eth2      Link encap:Ethernet  HWaddr bc:5f:f4:54:c9:d1  
          inet6 addr: fe80::6a05:caff:fe19:afe0/64 Scope:Link
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:11837 errors:0 dropped:0 overruns:0 frame:0
          TX packets:870 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:863363 (863.3 KB)  TX bytes:565985 (565.9 KB)
          Interrupt:19 Memory:fd9c0000-fd9e0000 
# plug cable and watch dhcp log

Sep 28 15:36:00 weasel dhclient: DHCPDISCOVER on eth2 to 255.255.255.255 port 67 interval 3 (xid=0x2f2cf17d)
Sep 28 15:36:00 weasel dhclient: DHCPREQUEST of 24.20.234.228 on eth2 to 255.255.255.255 port 67 (xid=0x2f2cf17d)
Sep 28 15:36:00 weasel dhclient: DHCPOFFER of 24.20.234.228 from 73.94.124.1
Sep 28 15:36:00 weasel dhclient: DHCPACK of 24.20.234.228 from 73.94.124.1
Sep 28 15:36:00 weasel dhclient: bound to 24.20.234.228 -- renewal in 1167 seconds.

Awesome.

Make it permanent by defining it in /etc/network/interfaces like this: 

auto eth2
pre-up iptables-restore < /etc/default/iptables
iface eth2 inet dhcp
hwaddress bc:5f:f4:54:c9:d1


And here is the original (slower) uplink, also defined in /etc/network/interfaces: 

auto eth0
iface eth0 inet static
 pre-up iptables-restore < /etc/default/iptables
 address 216.99.216.99
 netmask 255.255.255.0
 gateway 216.99.216.1
 dns-nameservers 8.8.8.8 216.99.193.19
 dns-search finninday.net


iptables

Useful iptables reference: https://help.ubuntu.com/community/IptablesHowTo

I've already made changes to my iptables config to make eth2 and eth0 (the two ISPs) be treated generally the same. But the rules are too restrictive for eth2. 

root@weasel:/etc/default# ping -I eth2 google.com
PING google.com (74.125.239.104) from 24.20.234.228 eth2: 56(84) bytes of data.
ping: sendmsg: Operation not permitted
ping: sendmsg: Operation not permitted

# open up iptables for eth2

root@weasel:/etc/default# ping -I eth2 google.com
PING google.com (173.194.33.9) from 24.20.234.228 eth2: 56(84) bytes of data.
64 bytes from sea09s01-in-f9.1e100.net (173.194.33.9): icmp_req=1 ttl=56 time=12.0 ms
64 bytes from sea09s01-in-f9.1e100.net (173.194.33.9): icmp_req=2 ttl=56 time=11.0 ms

So I have to think harder about what iptables rules I need. Let's look at the basics. If I leave out the rules for security and network hygiene, it looks like this: 

root@weasel:/etc/default# grep eth2 iptables

# eth2 = untrusted link to comcast
[0:0] -A POSTROUTING -o eth2 -j MASQUERADE
[0:0] -A FORWARD -m state --state NEW -o eth2 -j ACCEPT
[0:0] -A FORWARD -i eth2 -o eth0 -m state --state RELATED,ESTABLISHED -j ACCEPT

iptables troubleshooting

I wanted a way to know which rules were being triggered so I can know how to tweak my iptables rules. My best tool so far is this script: 

root@weasel:~/bin# cat iptables-diff-nat
rm /tmp/iptables-diff*
iptables -t nat -L -v -n > /tmp/iptables-diff1
sleep 10
iptables -t nat -L -v -n > /tmp/iptables-diff2
diff /tmp/iptables-diff1 /tmp/iptables-diff2 | less

which generates output like this: 

   pkts bytes target     prot opt in     out     source               destination     
3c3
<  2665  553K ACCEPT     all  --  eth1   *       10.0.0.0/8           0.0.0.0/0           
---
>  2706  565K ACCEPT     all  --  eth1   *       10.0.0.0/8           0.0.0.0/0           
67c67
<   563  129K ACCEPT     all  --  eth1   *       10.0.0.0/8           0.0.0.0/0           
---
>   601  133K ACCEPT     all  --  eth1   *       10.0.0.0/8           0.0.0.0/0           
70c70
<   468  126K ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED
---
>   503  138K ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED
98c98
<   834  298K ACCEPT     all  --  *      eth1    216.99.216.99        10.0.0.0/24         
---
>   856  319K ACCEPT     all  --  *      eth1    216.99.216.99        10.0.0.0/24         
100c100
<  1250  116K ACCEPT     all  --  *      eth1    10.0.0.0/8           10.0.0.0/8          
---
>  1262  118K ACCEPT     all  --  *      eth1    10.0.0.0/8           10.0.0.0/8

Based on the output of that script run against the nat and filter tables, I now think that my changes to iptables should look like this: 

root@weasel:/etc/default# egrep "24.20|eth2" iptables
# eth2 = untrusted link to comcast
[0:0]   -A POSTROUTING -o eth2 -j MASQUERADE
[0:0] -A INPUT -s 10.0.0.0/8     -i eth2 -j DROP
[0:0] -A INPUT -s 172.16.0.0/12  -i eth2 -j DROP
[0:0] -A INPUT -s 192.168.0.0/12  -i eth2 -j DROP
[0:0] -A INPUT -p tcp --destination-port 6667 -i eth2 -j DROP
[0:0] -A INPUT -p udp --destination-port 6667 -i eth2 -j DROP
[0:0] -A FORWARD -m state --state NEW -o eth2 -j ACCEPT
[0:0] -A FORWARD -i eth2 -o eth1 -m state --state RELATED,ESTABLISHED -j ACCEPT 
[2:88] -A FORWARD -i eth1 -o eth2 -j ACCEPT 
[0:0] -A OUTPUT -s 24.20.234.228  -d 10.0.0.0/24 -o eth1 -j ACCEPT 
[0:0] -A OUTPUT -d 10.0.0.0/8 -o eth2 -j DROP

That grep doesn't show you that the MASQUERADE rule is applied to the nat tables and the rest are applied to the filter table. Most of those rules (the DROPs) are just for good hygiene, eliminating clearly bogus packets.

what I thought worked

root@weasel:/etc/default# egrep "24.20|eth2" iptables
# eth2 = untrusted link to comcast
# nat table rule
 -A POSTROUTING -o eth2 -j MASQUERADE
# filter table rules
 -A FORWARD -m state --state NEW -o eth2 -j ACCEPT
 -A INPUT -s 75.75.75.75   -p tcp --destination-port 53 -i eth2 -j ACCEPT
 -A INPUT -d 24.20.234.228 -i eth2 -m state --state RELATED,ESTABLISHED -j ACCEPT
 -A FORWARD -i eth2 -o eth1 -m state --state RELATED,ESTABLISHED -j ACCEPT 
 -A FORWARD -i eth1 -o eth2 -j ACCEPT 
 -A OUTPUT -s 24.20.234.228  -d 10.0.0.0/24 -o eth1 -j ACCEPT 
 -A OUTPUT -s 24.20.234.228  -o eth2 -j ACCEPT 
# hygiene
 -A INPUT -s 10.0.0.0/8     -i eth2 -j DROP
 -A INPUT -s 172.16.0.0/12  -i eth2 -j DROP
 -A INPUT -s 192.168.0.0/12  -i eth2 -j DROP
 -A INPUT -p tcp --destination-port 6667 -i eth2 -j DROP
 -A INPUT -p udp --destination-port 6667 -i eth2 -j DROP
 -A OUTPUT -d 10.0.0.0/8 -o eth2 -j DROP

Also had to fiddle with DHCP server which was giving out the old DNS servers to clients. And Xymon needed to be updated to know about the new routes. Not sure if this config will survive a reboot. 

root@weasel:/etc/network# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         24.20.234.1     0.0.0.0         UG    0      0        0 eth2
10.0.0.0        0.0.0.0         255.0.0.0       U     0      0        0 eth1
24.20.234.0     0.0.0.0         255.255.254.0   U     0      0        0 eth2
169.254.0.0     0.0.0.0         255.255.0.0     U     1000   0        0 eth2
216.99.216.0    0.0.0.0         255.255.255.0   U     0      0        0 eth0

It would be nice to have traffic fail over to eth0, but that will be a future project.

it didn't really work

External traffic (mail, web) came in over the slow link and then apparently it went out over the fast link and then was dropped.

Retrieved from "https://finninday.net/wiki/index.php?title=Two_isp_routing&oldid=4025"