Linux QoS and Router Howto

In this howto I would like to demonstrate how to build a powerful router that supports QoS, stateful firewalling and many other features that linux is capable of. All using a spare computer or a box that you could purchase used for less than $100. It combines a little bit of everything from basic networking to building a custom linux kernel.

This is a complete howto, so you might want to skip ahead if you already know this part...

The Distro

Choosing a Linux distribution you're comfortable with is important. I'm most familiar with Red Hat Linux so the examples and steps in this howto will assume that you're working with RedHat. If you're comfortable with Linux and you don't like RedHat, change it to which ever distro makes you the most comfortable...

The Machine

You can build a router out of just about any machine, even an old i386 will work, but bear in mind that it's damned near impossible to find ISA NICs anymore unless you look on eBAY or something. The machine must have 2 NICs installed, one is for the WAN connection and the other for the LAN connection. These can be 10Base-T NICs but again, they're harder to find and it's cheap enough to just buy an SMC or LinkSys card... The machine should also have at least 12MB of RAM installed.

Installation

You can do this a couple of ways. If you have a hard drive laying around and it's at least 1GB that will be perfect. If you don't have one and you don't want to spend the money that's ok too, if you have a server or another linux box that has free hard disk space you can create a small filesystem on that machine and export it via NFS. Install a floppy drive or a CD-ROM in your router box and you can create a diskless system that boots off of NFS. The rest of this howto assumes that you DO have a hard drive in your machine. Creating a diskless system is a whole other topic and beyond the scope of this howto but it CAN be done and it's not that hard...

Boot your system using the Linux install CD (or if your machine is too old to boot from CD, you can download a bootable floppy image from ftp.redhat.com). The idea here is to create a very small distro so I would advise against installing XFree-86 and graphics apps and toys and stick to things like FTP, WWW, and DNS servers, you'll also want SSH and all of the developer libraries and includes. You should keep whittling the packages down until your distro is no bigger than 250MB.

The Kernel (www.kernel.org)

If you've never created your own kernel before I suggest you go here The Linux Kernel HOWTO. It's an important skill for any Linux user to have, most distro kernels are bloated with unnecessary modules and options and are not optimized for your specific machine.

If you have a fast linux machine and you're comfortable with transferring the modules and the kernel image, I suggest not building the kernel on your router box unless it's a fast machine or you have a LOT of patience. A kernel will take anywhere from 15 minutes to an hour on a P-200 depending on it's size and the modules you use. I can't even imagine doing it on an i486 or i386.

Download the latest 2.4 or 2.6 kernel, unpack it into /usr/src and do a 'make menuconfig'. Try and compile as many things as you can as modules to save memory. Compiling the kernel for your specific processor may help a little with speed and efficiency. Make sure that you DO include all of the NETFILTER modules, especially the connection tracking, NAT and packet marking ones, you'll need them! I suggest compiling your hard disk controller drivers and your filesystem directly into the kernel. It's just easier than trying to create an initrd image and it doesn't take any more memory than loading the drivers as modules. The only reason to use an initrd really is to make a kernel fit on a floppy or to create your root filesystem on a device that normally can't be loaded at boot time like a loop on a windows partition or something wacky like that... It can be a pain in the arse and if you don't do it correctly and you're not careful you can end up with an unbootable system...

Once you're finished customizing the box, do a 'make dep && make bzImage && make modules' go have a pizza, watch some TV, play a video game, whatever... it's going to be compiling for a while...

The Dreaded NetFilter (www.netfilter.org)

Oh come on, you knew this part was coming! It's not that bad! Actually netfilter is really quite flexible and powerful. First make sure that you download the very latest version, it will have bugfixes and all of the newest doodads and goodies for if you're feeling really brave. Make a copy of the iptables init script under /etc/rc.d/init.d/iptables and then do an rpm -e iptables to remove the current version. Untar your new downloaded version and extract it to /usr/src. The default setup of netfilter places it in /usr/local, I would change the prefix in the Makefile to /usr so that it installs in the standard directories but it will work in /usr/local as well...

Patch-O-Matic - You can use this if you wish, it is really unnecessary and contains some pretty experimental stuff, most of which can lock up your kernel.

IPROUTE2 - IP Utilities no unix user can live without... (www.lartc.org)

In order to have HTB support as well as the newest CBQ schedulers, you will need the latest iproute2 utilities Download Now

Compile them and install...

Putting it all together

Obviously a lesson on how to use netfilter and the IPROUTE2 utilities are beyond the scope of this HOWTO and I won't presume to redo work that others have already done. This will however show you all the necessary steps in making a fully functional NAT router with firewalling capabilities and QoS using a low cost linux box. After all, that's why you clicked here isn't it?

IPTABLES, The Basics

Firewalling is done in the FILTER table of iptables. The INPUT chain is for packets that enter and are destined for a process on the router machine. The FORWARD chain is for packets that enter on one of the interfaces of the router and are destined to exit on another interface. The OUTPUT chain, as the name implies, is for packets that leave one of the interfaces on the router and is destined for another machine.

NAT is done in the NAT table... DUH! In the NAT table, we have PREROUTING, POSTROUTING and OUTPUT. PREROUTING is for packets which arrive on an interface in the router, we know they're not destined for a local process but we don't know where they're going yet. In PREROUTING you can do things like DNAT (Destination NAT) which used to be called PORT FORWARDING. You can also create transparent proxies by using the REDIRECT target which is basically like doing DNAT to the local machine's IP address... In POSTROUTING, this is where we specify which interface that NAT will be going out on. If you've plugged your Cable, DSL, Router, etc into eth0 then you would do something like this 'iptables -t nat -A POSTROUTING -o eth0 -j SNAT --to-source IP.ADDRESS.OF.ETH0'. Likewise if you plugged it into eth1 then you would substitute eth1. Notice I've used the SNAT target and that it specifies an IP address... If you use DHCP like on a cable or DSL connection, you'll want to use the MASQUERADE target (like in the ipchains days). This target DOES NOT require an IP address to be specified.

To build a simple router, all you really would have to do is this....

iptables -P INPUT DROP
iptables -A INPUT -i (interface of LOCAL, private net goes here) -j ACCEPT
iptables -P FORWARD DROP
iptables -A FORWARD -i (interface of LOCAL, private net goes here) -o (interface of WAN, public net goes here) -j ACCEPT
iptables -A FORWARD -i (interface of PUBLIC net goes here) -o (interface of LOCAL, private net goes here)
           -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -t nat -A POSTROUTING -o (interface of PUBLIC net goes here) -j SNAT --to-source IP.OF.PUBLIC.NET

Set all of your machines in the local LAN to use the ip of the PRIVATE address on the router as their default gateway. Be sure to set up DNS as well or it will appear that things aren't working when they really are... This will give you a basic functioning NAT router, it has good security because it denies ALL connections that are not from the local net... Obviously you'll want to play with this a little bit but this should get you started!

Packet Mangling

Ok, now that you have a functioning albeit basic router, we need to have a way to prioritize traffic so that we can do all of that neat QoS stuff that this HOWTO is ultimately about. I think packet mangling is a horrible word for it, it doesn't invoke pleasant images, but it's essentially what you're doing.

Packet mangling is done... you guessed it... in the MANGLE table of iptables. PREROUTING has the same meaning here as it does in NAT, there are special circumstances in which you would want to use this chain but we won't go into that here. FORWARD has the same meaning as it does in the FILTER chain. We'll be doing our NAT mangling here.

Using my setup at home as an example, I've placed traffic leaving my local network into 3 categories, they are VoIP traffic which gets the highest priority and can preempt ALL other traffic, Interactive traffic which requires little bandwidth but needs to have low latency as well, and bulk traffic like P2P and downloading... To keep things simple, I kept the packet marks to 1, 2, and 3 respectively.

We'll set up chains for ASTERISK and for COMMONly used tcp and udp ports, all other traffic is classified bulk and will go in lowest priority

iptables -t mangle -N common
iptables -t mangle -N asterisk
iptables -t mangle -A common -p tcp --dport 80 -j MARK --set-mark 2
iptables -t mangle -A common -p tcp --dport 8080 -j MARK --set-mark 2
iptables -t mangle -A common -p tcp --dport 443 -j MARK --set-mark 2
iptables -t mangle -A common -p tcp --dport 110 -j MARK --set-mark 2
iptables -t mangle -A common -p tcp --dport 119 -j MARK --set-mark 2
iptables -t mangle -A common -p tcp --dport 25 -j MARK --set-mark 2
iptables -t mangle -A common -p udp --dport 53 -j MARK --set-mark 2
iptables -t mangle -A common -p udp --dport 68 -j MARK --set-mark 2
iptables -t mangle -A asterisk -p udp --sport 5060 -j MARK --set-mark 1
iptables -t mangle -A asterisk -p udp --dport 5060 -j MARK --set-mark 1
iptables -t mangle -A asterisk -p tcp --dport 5036 -j MARK --set-mark 1
iptables -t mangle -A asterisk -p udp --dport 5036 -j MARK --set-mark 1
iptables -t mangle -A asterisk -p udp --dport 4569 -j MARK --set-mark 1
iptables -t mangle -A asterisk -p udp --sport RTP PORT LOW RANGE:RTP PORT HIGH RANGE -j MARK --set-mark 1
iptables -t mangle -A FORWARD -i <LOCAL LAN INTERFACE> -o <PUBLIC WAN INTERFACE> -j MARK --set-mark 3
iptables -t mangle -A FORWARD -i <LOCAL LAN INTERFACE> -o <PUBLIC WAN INTERFACE> -j common
iptables -t mangle -A FORWARD -i <LOCAL LAN INTERFACE> -o <PUBLIC WAN INTERFACE> -j asterisk
iptables -t mangle -A FORWARD -i <LOCAL LAN INTERFACE> -o <PUBLIC WAN INTERFACE> -p icmp -j MARK --set-mark 1

Ok, so what was all that? Well it's pretty simple, I just classified some well known and commonly used TCP and UDP ports into a chain called 'common', then I classified all of the Asterisk control and RTP ports and placed them into a chain called 'asterisk' The marks are like I stated above, 1 for high priority, 2 for interactive, and 3 for bulk. Then I placed 4 rules in the forward chain. The order of these rules is important because IPTABLES reads from top to bottom unless you use the "JUMP" target. I placed the bulk traffic at the top of the chain, anything that's not asterisk-related or interactive is bulk. Then common ports next and finally asterisk. The reason I did ICMP high priority is because usually a network doesn't generate a whole lot of ICMP traffic, this also allows you to do PING timing without too much skewing..

"************************************TO BE CONTINUED!**************************************"


You can also try the free Linux LiveCD Router http://www.wifi.com.ar
Is perfect to share and firewall your dedicated Internet connection. It supports DSL, Cablemodem, T1, WiFi 802.11a/b/g and more. It is a LiveCD, so it does not require installation! Includes firewall Shorewall, traffic control and QoS

The company also makes a LiveCD VoIP Server
http://www.wifi.com.ar/english/voip.html