Two isp routing

From finninday
Revision as of 05:57, 8 October 2013 by Rday (talk | contribs)
Jump to navigation Jump to search

background

I'm using Ubuntu.

I have two ISPs. I didn't plan on getting two ISPs. I've had one for ages that I'm reasonably happy with that I've shopped long and hard for. It gives me a static IP and doesn't charge too much. It has a 1Mbps down and 370Kbps up.

Then my son wanted to vastly upgrade our connectivity and our existing link just couldn't fill the bill. So we got another. His link is 30Mbps down and 3Mbps up. I want some of that. But I don't want to disrupt the existing services on my static IP and the original link.

My routing table looks like this now. 

root@weasel:/# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         216.99.216.1    0.0.0.0         UG    0      0        0 eth0
10.0.0.0        0.0.0.0         255.0.0.0       U     0      0        0 eth1
169.254.0.0     0.0.0.0         255.255.0.0     U     1000   0        0 eth1
216.99.216.0    0.0.0.0         255.255.255.0   U     0      0        0 eth0

I want both links to be active. It looks like a lot of my questions will be answered here: http://www.linuxhorizon.ro/iproute2.html

This turns out to be a better reference: http://www.lartc.org/howto/lartc.rpdb.multiple-links.html

Not sure if I'll have to manually configure the routes. Starting with the default here: 

root@weasel:/# cat /etc/iproute2/rt_tables 
#
# reserved values
#
255	local
254	main
253	default
0	unspec
#
# local
#
#1	inr.ruhep

No response from comcast dhcp

The service guy set up the connection to a windows machine that works just fine. When I try to connect with my laptop or server, I get no response from the dhcp server. Actually, that's not quite true. I get an IPv6 address, but not an IPv4. The windows machine gets an IPv4 just fine.

Maybe I have to spoof my mac address to proceed. Here is my old laptop mac address, so I can change it back:

f0:1f:af:16:f3:39

Here is the mac address on the windows machine:

bc:5f:f4:54:c9:d1

Switch laptop mac address to match what comcast expects. 

ifconfig em1 hw ether bc:5f:f4:54:c9:d1

ifconfig em1 reports that it was successfully changed.

yay, it worked.

And better yet, it was easy to clean up. 

ifdown em1
ifconfig em1

reports that I've got my old mac address again.

Now test on server. 

# original mac address here

eth2      Link encap:Ethernet  HWaddr 68:05:ca:19:af:e0  
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
          Interrupt:19 Memory:fd9c0000-fd9e0000 

root@weasel:~# ifconfig eth2 hw ether bc:5f:f4:54:c9:d1
root@weasel:~# ifconfig eth2
eth2      Link encap:Ethernet  HWaddr bc:5f:f4:54:c9:d1  
          inet6 addr: fe80::6a05:caff:fe19:afe0/64 Scope:Link
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:11837 errors:0 dropped:0 overruns:0 frame:0
          TX packets:870 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:863363 (863.3 KB)  TX bytes:565985 (565.9 KB)
          Interrupt:19 Memory:fd9c0000-fd9e0000 
# plug cable and watch dhcp log

Sep 28 15:36:00 weasel dhclient: DHCPDISCOVER on eth2 to 255.255.255.255 port 67 interval 3 (xid=0x2f2cf17d)
Sep 28 15:36:00 weasel dhclient: DHCPREQUEST of 24.20.234.228 on eth2 to 255.255.255.255 port 67 (xid=0x2f2cf17d)
Sep 28 15:36:00 weasel dhclient: DHCPOFFER of 24.20.234.228 from 73.94.124.1
Sep 28 15:36:00 weasel dhclient: DHCPACK of 24.20.234.228 from 73.94.124.1
Sep 28 15:36:00 weasel dhclient: bound to 24.20.234.228 -- renewal in 1167 seconds.

Awesome.

Make it permanent by defining it in /etc/network/interfaces like this: 

auto eth2
iface eth2 inet dhcp
hwaddress bc:5f:f4:54:c9:d1


And here is the original (slower) uplink, also defined in /etc/network/interfaces: 

auto eth0
iface eth0 inet static
 pre-up iptables-restore < /etc/default/iptables
 address 216.99.216.99
 netmask 255.255.255.0
 gateway 216.99.216.1
 dns-nameservers 8.8.8.8 216.99.193.19
 dns-search finninday.net


iptables

Useful iptables reference: https://help.ubuntu.com/community/IptablesHowTo

I've already made changes to my iptables config to make eth2 and eth0 (the two ISPs) be treated generally the same. But the rules are too restrictive for eth2. 

root@weasel:/etc/default# ping -I eth2 google.com
PING google.com (74.125.239.104) from 24.20.234.228 eth2: 56(84) bytes of data.
ping: sendmsg: Operation not permitted
ping: sendmsg: Operation not permitted

# open up iptables for eth2

root@weasel:/etc/default# ping -I eth2 google.com
PING google.com (173.194.33.9) from 24.20.234.228 eth2: 56(84) bytes of data.
64 bytes from sea09s01-in-f9.1e100.net (173.194.33.9): icmp_req=1 ttl=56 time=12.0 ms
64 bytes from sea09s01-in-f9.1e100.net (173.194.33.9): icmp_req=2 ttl=56 time=11.0 ms

So I have to think harder about what iptables rules I need. Let's look at the basics. If I leave out the rules for security and network hygiene, it looks like this: 

root@weasel:/etc/default# grep eth2 iptables

# eth2 = untrusted link to comcast
[0:0] -A POSTROUTING -o eth2 -j MASQUERADE
[0:0] -A FORWARD -m state --state NEW -o eth2 -j ACCEPT
[0:0] -A FORWARD -i eth2 -o eth0 -m state --state RELATED,ESTABLISHED -j ACCEPT

iptables troubleshooting

I wanted a way to know which rules were being triggered so I can know how to tweak my iptables rules. My best tool so far is this script: 

root@weasel:~/bin# cat iptables-diff-nat
rm /tmp/iptables-diff*
iptables -t nat -L -v -n > /tmp/iptables-diff1
sleep 10
iptables -t nat -L -v -n > /tmp/iptables-diff2
diff /tmp/iptables-diff1 /tmp/iptables-diff2 | less

which generates output like this: 

   pkts bytes target     prot opt in     out     source               destination     
3c3
<  2665  553K ACCEPT     all  --  eth1   *       10.0.0.0/8           0.0.0.0/0           
---
>  2706  565K ACCEPT     all  --  eth1   *       10.0.0.0/8           0.0.0.0/0           
67c67
<   563  129K ACCEPT     all  --  eth1   *       10.0.0.0/8           0.0.0.0/0           
---
>   601  133K ACCEPT     all  --  eth1   *       10.0.0.0/8           0.0.0.0/0           
70c70
<   468  126K ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED
---
>   503  138K ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED
98c98
<   834  298K ACCEPT     all  --  *      eth1    216.99.216.99        10.0.0.0/24         
---
>   856  319K ACCEPT     all  --  *      eth1    216.99.216.99        10.0.0.0/24         
100c100
<  1250  116K ACCEPT     all  --  *      eth1    10.0.0.0/8           10.0.0.0/8          
---
>  1262  118K ACCEPT     all  --  *      eth1    10.0.0.0/8           10.0.0.0/8

Based on the output of that script run against the nat and filter tables, I now think that my changes to iptables should look like this: 

root@weasel:/etc/default# egrep "24.20|eth2" iptables
# eth2 = untrusted link to comcast
[0:0]   -A POSTROUTING -o eth2 -j MASQUERADE
[0:0] -A INPUT -s 10.0.0.0/8     -i eth2 -j DROP
[0:0] -A INPUT -s 172.16.0.0/12  -i eth2 -j DROP
[0:0] -A INPUT -s 192.168.0.0/12  -i eth2 -j DROP
[0:0] -A INPUT -p tcp --destination-port 6667 -i eth2 -j DROP
[0:0] -A INPUT -p udp --destination-port 6667 -i eth2 -j DROP
[0:0] -A FORWARD -m state --state NEW -o eth2 -j ACCEPT
[0:0] -A FORWARD -i eth2 -o eth1 -m state --state RELATED,ESTABLISHED -j ACCEPT 
[2:88] -A FORWARD -i eth1 -o eth2 -j ACCEPT 
[0:0] -A OUTPUT -s 24.20.234.228  -d 10.0.0.0/24 -o eth1 -j ACCEPT 
[0:0] -A OUTPUT -d 10.0.0.0/8 -o eth2 -j DROP

That grep doesn't show you that the MASQUERADE rule is applied to the nat tables and the rest are applied to the filter table. Most of those rules (the DROPs) are just for good hygiene, eliminating clearly bogus packets.

what I thought worked

root@weasel:/etc/default# egrep "24.20|eth2" iptables
# eth2 = untrusted link to comcast
# nat table rule
 -A POSTROUTING -o eth2 -j MASQUERADE
# filter table rules
 -A FORWARD -m state --state NEW -o eth2 -j ACCEPT
 -A INPUT -s 75.75.75.75   -p tcp --destination-port 53 -i eth2 -j ACCEPT
 -A INPUT -d 24.20.234.228 -i eth2 -m state --state RELATED,ESTABLISHED -j ACCEPT
 -A FORWARD -i eth2 -o eth1 -m state --state RELATED,ESTABLISHED -j ACCEPT 
 -A FORWARD -i eth1 -o eth2 -j ACCEPT 
 -A OUTPUT -s 24.20.234.228  -d 10.0.0.0/24 -o eth1 -j ACCEPT 
 -A OUTPUT -s 24.20.234.228  -o eth2 -j ACCEPT 
# hygiene
 -A INPUT -s 10.0.0.0/8     -i eth2 -j DROP
 -A INPUT -s 172.16.0.0/12  -i eth2 -j DROP
 -A INPUT -s 192.168.0.0/12  -i eth2 -j DROP
 -A INPUT -p tcp --destination-port 6667 -i eth2 -j DROP
 -A INPUT -p udp --destination-port 6667 -i eth2 -j DROP
 -A OUTPUT -d 10.0.0.0/8 -o eth2 -j DROP

Also had to fiddle with DHCP server which was giving out the old DNS servers to clients. And Xymon needed to be updated to know about the new routes. Not sure if this config will survive a reboot. 

root@weasel:/etc/network# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         24.20.234.1     0.0.0.0         UG    0      0        0 eth2
10.0.0.0        0.0.0.0         255.0.0.0       U     0      0        0 eth1
24.20.234.0     0.0.0.0         255.255.254.0   U     0      0        0 eth2
169.254.0.0     0.0.0.0         255.255.0.0     U     1000   0        0 eth2
216.99.216.0    0.0.0.0         255.255.255.0   U     0      0        0 eth0

It would be nice to have traffic fail over to eth0, but that will be a future project.

it didn't really work

External traffic (mail, web) came in over the slow link and then apparently it went out over the fast link and then was dropped. The failure wasn't seen by my internal monitors which were routed properly and showed all services functioning. I need a way to see when external traffic is not being routed properly. Perhaps count the number of emails arriving by hour and alert when it drops. That way I can get some use out of the spammers hitting my mail server.

another view of the routing table

I'm not sure what is different about routel and route -n. Just that routel is much more interesting: 

root@weasel:/var/log# routel
         target            gateway          source    proto    scope    dev tbl
        default       216.99.216.1                                     eth0 
       10.0.0.0 8                         10.0.0.3   kernel     link   eth1 
    169.254.0.0 16                  169.254.11.125   kernel     link   eth2 
   216.99.216.0 24                   216.99.216.99   kernel     link   eth0 
       10.0.0.0          broadcast        10.0.0.3   kernel     link   eth1 local
       10.0.0.3              local        10.0.0.3   kernel     host   eth1 local
 10.255.255.255          broadcast        10.0.0.3   kernel     link   eth1 local
      127.0.0.0          broadcast       127.0.0.1   kernel     link     lo local
      127.0.0.0 8            local       127.0.0.1   kernel     host     lo local
      127.0.0.1              local       127.0.0.1   kernel     host     lo local
127.255.255.255          broadcast       127.0.0.1   kernel     link     lo local
    169.254.0.0          broadcast  169.254.11.125   kernel     link   eth2 local
 169.254.11.125              local  169.254.11.125   kernel     host   eth2 local
169.254.255.255          broadcast  169.254.11.125   kernel     link   eth2 local
   216.99.216.0          broadcast   216.99.216.99   kernel     link   eth0 local
  216.99.216.99              local   216.99.216.99   kernel     host   eth0 local
 216.99.216.255          broadcast   216.99.216.99   kernel     link   eth0 local
             :: 96              ::                                     sit0 
2001:470:1f08:52c:: 64              ::                   kernel            sit1 
         fe80:: 64                                   kernel            eth0 
         fe80:: 64                                   kernel            eth1 
        default                                                        sit1 
        default        unreachable                   kernel              lo unspec
            ::1                 ::                     none              lo local
     ::10.0.0.3                 ::                     none              lo local
    ::127.0.0.1                 ::                     none              lo local
::216.99.216.99                 ::                     none              lo local
2001:470:1f08:52c::2                 ::                     none              lo local
    fe80::a00:3                 ::                     none              lo local
fe80::d863:d863                 ::                     none              lo local
fe80::213:d3ff:fea6:5cfc                 ::                     none              lo local
fe80::260:8ff:fe11:b59e                 ::                     none              lo local
         ff00:: 8                                                      eth0 local
         ff00:: 8                                                      eth1 local
         ff00:: 8                                                      sit1 local
        default        unreachable                   kernel              lo unspec
root@weasel:/var/log# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         216.99.216.1    0.0.0.0         UG    0      0        0 eth0
10.0.0.0        0.0.0.0         255.0.0.0       U     0      0        0 eth1
169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0 eth2
216.99.216.0    0.0.0.0         255.255.255.0   U     0      0        0 eth0

If I turn off the IPv6 stuff, it looks simpler: 

root@weasel:/var/log# routel
         target            gateway          source    proto    scope    dev tbl
        default       216.99.216.1                                     eth0 
       10.0.0.0 8                         10.0.0.3   kernel     link   eth1 
    169.254.0.0 16                  169.254.11.125   kernel     link   eth2 
   216.99.216.0 24                   216.99.216.99   kernel     link   eth0 
       10.0.0.0          broadcast        10.0.0.3   kernel     link   eth1 local
       10.0.0.3              local        10.0.0.3   kernel     host   eth1 local
 10.255.255.255          broadcast        10.0.0.3   kernel     link   eth1 local
      127.0.0.0          broadcast       127.0.0.1   kernel     link     lo local
      127.0.0.0 8            local       127.0.0.1   kernel     host     lo local
      127.0.0.1              local       127.0.0.1   kernel     host     lo local
127.255.255.255          broadcast       127.0.0.1   kernel     link     lo local
    169.254.0.0          broadcast  169.254.11.125   kernel     link   eth2 local
 169.254.11.125              local  169.254.11.125   kernel     host   eth2 local
169.254.255.255          broadcast  169.254.11.125   kernel     link   eth2 local
   216.99.216.0          broadcast   216.99.216.99   kernel     link   eth0 local
  216.99.216.99              local   216.99.216.99   kernel     host   eth0 local
 216.99.216.255          broadcast   216.99.216.99   kernel     link   eth0 local
         fe80:: 64                                   kernel            eth0 
         fe80:: 64                                   kernel            eth1 
        default        unreachable                   kernel              lo unspec
            ::1                 ::                     none              lo local
fe80::213:d3ff:fea6:5cfc                 ::                     none              lo local
fe80::260:8ff:fe11:b59e                 ::                     none              lo local
         ff00:: 8                                                      eth0 local
         ff00:: 8                                                      eth1 local
        default        unreachable                   kernel              lo unspec

Trying again, with routing

  • first plug in the interface and bring it up

Add this to /etc/network/interfaces 

auto eth2
iface eth2 inet dhcp
 hwaddress bc:5f:f4:54:c9:d1
 gateway 24.20.234.1
  • then turn it on
root@weasel:/etc/iproute2# ifup eth2
Internet Systems Consortium DHCP Client 4.2.4
Copyright 2004-2012 Internet Systems Consortium.
All rights reserved.
For info, please visit https://www.isc.org/software/dhcp/

run-parts: failed to stat component /etc/dhcp/dhclient-enter-hooks.d/samba: No such file or directory
Listening on LPF/eth2/bc:5f:f4:54:c9:d1
Sending on   LPF/eth2/bc:5f:f4:54:c9:d1
Sending on   Socket/fallback
DHCPDISCOVER on eth2 to 255.255.255.255 port 67 interval 3 (xid=0x50d19edd)
DHCPREQUEST of 24.20.234.228 on eth2 to 255.255.255.255 port 67 (xid=0x50d19edd)
DHCPOFFER of 24.20.234.228 from 73.94.124.1
DHCPACK of 24.20.234.228 from 73.94.124.1
run-parts: failed to stat component /etc/dhcp/dhclient-enter-hooks.d/samba: No such file or directory
status: Unknown job: squid
bound to 24.20.234.228 -- renewal in 917 seconds.
ssh stop/waiting
ssh start/running, process 26659

And now my routes look like: 

root@weasel:/etc/iproute2# ip route show
default via 216.99.216.1 dev eth0 
10.0.0.0/8 dev eth1  proto kernel  scope link  src 10.0.0.3 
24.20.234.0/23 dev eth2  proto kernel  scope link  src 24.20.234.228 
169.254.0.0/16 dev eth2  scope link  metric 1000 
216.99.216.0/24 dev eth0  proto kernel  scope link  src 216.99.216.99

I have also made changes to /etc/iproute2/rt_tables like so: 

root@weasel:/etc/iproute2# cat rt_tables
#
# reserved values
#
255	local
254	main
253	default
0	unspec
#
# local
#
#1	inr.ruhep
1 spiritone
2 comcast

Now I should be able to just add the split access routes according to the lartc howto. The variables that lartc says is required for split access are as follows: 

T1 = spiritone
T2 = comcast
IF1 = eth0
IF2 = eth2
IP1 = 216.99.216.99
IP2 = 24.20.234.228
P1 = 216.99.216.1
P2 = 24.20.234.1
P1_net = 216.99.216.0/24
P2_net = 24.20.234.0/23


Which means that I need to apply these routes. But the first one is a duplicate? 

root@weasel:/etc/iproute2# ip route add 216.99.216.0/24 eth0 src 216.99.216.99 table spiritone
Error: either "to" is duplicate, or "eth0" is a garbage.

The others worked just fine. 

root@weasel:/etc/iproute2# ip route add default via 216.99.216.1 table spiritone
root@weasel:/etc/iproute2# ip route add 24.20.234.0/23 dev eth2 src 24.20.234.228 table comcast
root@weasel:/etc/iproute2# ip route add default via 24.20.234.1 table comcast
Retrieved from "https://finninday.net/wiki/index.php?title=Two_isp_routing&oldid=4034"