QoS with Linux using PRIO and HTB
Created by: Carceri,Last modification on Thu 03 of Aug, 2006 [12:37 UTC]
QoS with Linux using PRIO and HTB
I have used the Wondershaper and other QoS scripts for Linux, and they were surely better than nothing, but they were never perfect. The problem is that they all use fair queing schemes where one connection cannot steal all the bandwidth, but this is exactly what I want. When I use VoIP I don't want any other traffic sent out on my link when there is VoIP data that wants to go out. I solved it using the following script:
#!/bin/bash
TC=/sbin/tc
DEV=eth1
if [ "$1" = "prio" ]
then
CEIL=170
DOWNLINK=768
else
CEIL=370
DOWNLINK=2048
fi
# low priority OUTGOING traffic - you can leave this blank if you want
# low priority source netmasks
NOPRIOHOSTSRC=
# low priority destination netmasks
NOPRIOHOSTDST=
# low priority source ports
NOPRIOPORTSRC=
# low priority destination ports
NOPRIOPORTDST=
if [ "$1" = "status" ]
then
$TC -s qdisc ls dev $DEV
$TC -s class ls dev $DEV
exit
fi
# clean existing down- and uplink qdiscs, hide errors
$TC qdisc del dev $DEV root 2> /dev/null > /dev/null
$TC qdisc del dev $DEV ingress 2> /dev/null > /dev/null
if [ "$1" = "stop" ]
then
exit
fi
###### uplink
$TC qdisc add dev ${DEV} root handle 1: prio bands 2 priomap 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
$TC qdisc add dev ${DEV} parent 1:1 handle 11: pfifo
$TC qdisc add dev ${DEV} parent 1:2 handle 12: htb r2q 3
$TC class add dev ${DEV} parent 12: classid 12:1 htb rate ${CEIL}kbit burst 2k
$TC class add dev ${DEV} parent 12:1 classid 12:10 htb \
rate $[50*$CEIL/100]kbit ceil ${CEIL}kbit burst 2k prio 1
$TC class add dev ${DEV} parent 12:1 classid 12:11 htb \
rate $[30*$CEIL/100]kbit ceil ${CEIL}kbit burst 2k prio 2
$TC class add dev ${DEV} parent 12:1 classid 12:12 htb \
rate $[20*$CEIL/100]kbit ceil ${CEIL}kbit burst 2k prio 3
$TC qdisc add dev ${DEV} parent 12:10 handle 1210: sfq perturb 10
$TC qdisc add dev ${DEV} parent 12:11 handle 1211: sfq perturb 10
$TC qdisc add dev ${DEV} parent 12:12 handle 1212: sfq perturb 10
# VoIP traffic always get first in line
$TC filter add dev ${DEV} parent 1: prio 1 protocol ip u32 \
match ip tos 0x68 0xff \
match ip protocol 0x11 0xff \
flowid 1:1
$TC filter add dev ${DEV} parent 1: prio 1 protocol ip u32 \
match ip tos 0xb8 0xff \
match ip protocol 0x11 0xff \
flowid 1:1
# All non-VoIP traffic on the second band
$TC filter add dev ${DEV} parent 1: protocol ip prio 3 u32 \
match ip src 0.0.0.0/0 \
flowid 1:2
# TOS Minimum Delay
$TC filter add dev ${DEV} parent 12: protocol ip prio 10 u32 \
match ip tos 0x10 0xff \
flowid 12:10
# ICMP (ip protocol 1) in the interactive class
$TC filter add dev ${DEV} parent 12: protocol ip prio 11 u32 \
match ip protocol 1 0xff flowid 12:10
# To speed up downloads while an upload is going on, put ACK packets in
# the interactive class:
$TC filter add dev ${DEV} parent 12: protocol ip prio 12 u32 \
match ip protocol 6 0xff \
match u8 0x05 0x0f at 0 \
match u16 0x0000 0xffc0 at 2 \
match u8 0x10 0xff at 33 \
flowid 12:10
# some traffic however suffers a worse fate
for a in $NOPRIOPORTDST
do
$TC filter add dev $DEV parent 12: protocol ip prio 30 u32 \
match ip protocol 0x6 0xff \
match ip dport $a 0xffff \
flowid 12:12
done
for a in $NOPRIOPORTSRC
do
$TC filter add dev $DEV parent 12: protocol ip prio 31 u32 \
match ip protocol 0x6 0xff \
match ip sport $a 0xffff \
flowid 12:12
done
for a in $NOPRIOHOSTSRC
do
$TC filter add dev $DEV parent 12: protocol ip prio 32 u32 \
match ip protocol 0x6 0xff \
match ip src $a \
flowid 12:12
done
for a in $NOPRIOHOSTDST
do
$TC filter add dev $DEV parent 12: protocol ip prio 33 u32 \
match ip protocol 0x6 0xff \
match ip dst $a \
flowid 12:12
done
# rest is 'non-interactive' ie 'bulk' and ends up in the default queue
$TC filter add dev ${DEV} parent 12: protocol ip prio 20 u32 \
match ip src 0.0.0.0/0 \
flowid 12:11
########## downlink #############
# slow downloads down to somewhat less than the real speed to prevent
# queuing at our ISP. Tune to see how high you can set it.
# ISPs tend to have *huge* queues to make sure big downloads are fast
#
# attach ingress policer:
if [ "$1" = "prio" ]
then
$TC qdisc add dev $DEV handle ffff: ingress
# filter *everything* to it (0.0.0.0/0), drop everything that's
# coming in too fast:
$TC filter add dev $DEV parent ffff: protocol ip prio 50 u32 \
match ip protocol 0x6 0xff police rate ${DOWNLINK}kbit burst 10k drop \
flowid :1
fi
TC=/sbin/tc
DEV=eth1
if [ "$1" = "prio" ]
then
CEIL=170
DOWNLINK=768
else
CEIL=370
DOWNLINK=2048
fi
# low priority OUTGOING traffic - you can leave this blank if you want
# low priority source netmasks
NOPRIOHOSTSRC=
# low priority destination netmasks
NOPRIOHOSTDST=
# low priority source ports
NOPRIOPORTSRC=
# low priority destination ports
NOPRIOPORTDST=
if [ "$1" = "status" ]
then
$TC -s qdisc ls dev $DEV
$TC -s class ls dev $DEV
exit
fi
# clean existing down- and uplink qdiscs, hide errors
$TC qdisc del dev $DEV root 2> /dev/null > /dev/null
$TC qdisc del dev $DEV ingress 2> /dev/null > /dev/null
if [ "$1" = "stop" ]
then
exit
fi
###### uplink
$TC qdisc add dev ${DEV} root handle 1: prio bands 2 priomap 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
$TC qdisc add dev ${DEV} parent 1:1 handle 11: pfifo
$TC qdisc add dev ${DEV} parent 1:2 handle 12: htb r2q 3
$TC class add dev ${DEV} parent 12: classid 12:1 htb rate ${CEIL}kbit burst 2k
$TC class add dev ${DEV} parent 12:1 classid 12:10 htb \
rate $[50*$CEIL/100]kbit ceil ${CEIL}kbit burst 2k prio 1
$TC class add dev ${DEV} parent 12:1 classid 12:11 htb \
rate $[30*$CEIL/100]kbit ceil ${CEIL}kbit burst 2k prio 2
$TC class add dev ${DEV} parent 12:1 classid 12:12 htb \
rate $[20*$CEIL/100]kbit ceil ${CEIL}kbit burst 2k prio 3
$TC qdisc add dev ${DEV} parent 12:10 handle 1210: sfq perturb 10
$TC qdisc add dev ${DEV} parent 12:11 handle 1211: sfq perturb 10
$TC qdisc add dev ${DEV} parent 12:12 handle 1212: sfq perturb 10
# VoIP traffic always get first in line
$TC filter add dev ${DEV} parent 1: prio 1 protocol ip u32 \
match ip tos 0x68 0xff \
match ip protocol 0x11 0xff \
flowid 1:1
$TC filter add dev ${DEV} parent 1: prio 1 protocol ip u32 \
match ip tos 0xb8 0xff \
match ip protocol 0x11 0xff \
flowid 1:1
# All non-VoIP traffic on the second band
$TC filter add dev ${DEV} parent 1: protocol ip prio 3 u32 \
match ip src 0.0.0.0/0 \
flowid 1:2
# TOS Minimum Delay
$TC filter add dev ${DEV} parent 12: protocol ip prio 10 u32 \
match ip tos 0x10 0xff \
flowid 12:10
# ICMP (ip protocol 1) in the interactive class
$TC filter add dev ${DEV} parent 12: protocol ip prio 11 u32 \
match ip protocol 1 0xff flowid 12:10
# To speed up downloads while an upload is going on, put ACK packets in
# the interactive class:
$TC filter add dev ${DEV} parent 12: protocol ip prio 12 u32 \
match ip protocol 6 0xff \
match u8 0x05 0x0f at 0 \
match u16 0x0000 0xffc0 at 2 \
match u8 0x10 0xff at 33 \
flowid 12:10
# some traffic however suffers a worse fate
for a in $NOPRIOPORTDST
do
$TC filter add dev $DEV parent 12: protocol ip prio 30 u32 \
match ip protocol 0x6 0xff \
match ip dport $a 0xffff \
flowid 12:12
done
for a in $NOPRIOPORTSRC
do
$TC filter add dev $DEV parent 12: protocol ip prio 31 u32 \
match ip protocol 0x6 0xff \
match ip sport $a 0xffff \
flowid 12:12
done
for a in $NOPRIOHOSTSRC
do
$TC filter add dev $DEV parent 12: protocol ip prio 32 u32 \
match ip protocol 0x6 0xff \
match ip src $a \
flowid 12:12
done
for a in $NOPRIOHOSTDST
do
$TC filter add dev $DEV parent 12: protocol ip prio 33 u32 \
match ip protocol 0x6 0xff \
match ip dst $a \
flowid 12:12
done
# rest is 'non-interactive' ie 'bulk' and ends up in the default queue
$TC filter add dev ${DEV} parent 12: protocol ip prio 20 u32 \
match ip src 0.0.0.0/0 \
flowid 12:11
########## downlink #############
# slow downloads down to somewhat less than the real speed to prevent
# queuing at our ISP. Tune to see how high you can set it.
# ISPs tend to have *huge* queues to make sure big downloads are fast
#
# attach ingress policer:
if [ "$1" = "prio" ]
then
$TC qdisc add dev $DEV handle ffff: ingress
# filter *everything* to it (0.0.0.0/0), drop everything that's
# coming in too fast:
$TC filter add dev $DEV parent ffff: protocol ip prio 50 u32 \
match ip protocol 0x6 0xff police rate ${DOWNLINK}kbit burst 10k drop \
flowid :1
fi
On a drawing the qdiscs and classes look like this:
Some explaination is probably needed.
I want to use the PRIO qdisc to create two classes where no traffic will be sent from class 2 as long as there is still traffic in class 1. VoIP will then be placed in class 1. Still, I want to prioritise the rest of the traffic. At the root I assign a PRIO qdisc with two subclasses. To subclass 1 (to be used for VoIP) I assign a PFIFO qdisc and to subclass 2 (all other traffic) I assign a HTB qdisc to shape that traffic. Below the HTB qdisc I add a HTB class which sets the rate for all other traffic. To the HTB class I add three subclasses. The first one for interactive traffic, the second for all other traffic and the third for certain low priority traffic. These three subclasses share all available bandwitdh, but with a high priority to interactive traffic when available.
The problem with this configuration is that the HTB qdisc will send at full speed, and the PRIO qdisc will in addition to this traffic also allow VoIP traffic out, which will cause queues in the ADSL modem or whatever you have. There are two possible solutions to this problem. One is to add a TBF qdisc above the PRIO qdisc to limit the overall bandwidth, but then you just risk having the TBF qdisc dropping or delaying VoIP packets to satisfy the rate. The second solution is to limit the bandwidth of the HTB qdisc so that it, combined with VoIP traffic, will not cause queues in the ADSL modem. This will of cause limit your upstream significantly for normal use, since a part of it will always be dedicated to VoIP traffic.
This is what I use the "prio" parameter for. With this parameter, the script will limit the upstream of all non-VoIP traffic significantly, and when it is off there will not be reserved any traffic for VoIP. Note that still with this configuration, it's still a fairly decent script for VoIP traffic. Furthermore, when "prio" is specified, the downstream traffic is severely reduced for TCP connections. This is done by dropping packets, causing the TCP protocol to reduce it's transmission rate. This will ensure that UDP packets (VoIP) also have good bandwidth to get from the recipient to my phone.
So how does this all fit together? I am using a Sipura 2000 adapter for VoIP and I have written a small program that will monitor the adapter, and when a call is started it will run this script with the "prio" parameter. When the call is ended, the normal script is run. That way upstream and downstream is only significantly lowered while I am using the phone, and my phone conversations are always clear.
Hope this is useful. If you have any questions, feel free to ask.
NOTE: The values are for a 2048/512 kbps ADSL connection
Comments
333
333box at home with web server and *
The HTB manual gave me the answer: mtu. I set the mtu to the maximum packet size, while keeping burst and cburst low, and it finally obeyed ceil.
If you are running a box with any sort of file sharing (http, scp, ftp), I imagine this will help. I have a box at home with a broadband router and comcast cable (6Mbps/340kbps). The box has asterisk, apache, ssh, vsftpd, and more.
The relevant command for the script on this page:
tc class add dev eth0 parent 12:1 classid 12:10 htb rate 1kbit ceil 250kbit prio 1 burst 1b cburst 1b mtu 47000
Do the same with all the htb classes. I haven't tested other burst sizes, but I'm thinking burst is unwanted in this scenario. 250kbit ceil works well for me, providing decent call quality with decent uplink bandwidth. I used ethereal to capture scp and http traffic, and then looked through the packet lengths to find the maximum, 46500, then I chose 47000 for mtu.
In case it's not apparent already, I'm using part of the script given on this page, where voip goes through the pfifo 11: and other things go through htb 12:10-12:12. My box serves local clients and Internet clients, so my tc filters only select Internet-bound traffic, leaving the local traffic to flow at maximum speed. I don't control the downlink, but that may become an issue for me.
333