15.8. The Ultimate Traffic Conditioner: Low Latency, Fast Up & Downloads
Note: This script has recently been upgraded and previously only worked for
Linux clients in your network! So you might want to update if you have
Windows machines or Macs in your network and noticed that they were not able
to download faster while others were uploading.
I attempted to create the holy grail:
- Maintain low latency for interactive traffic at all times
This means that downloading or uploading files should not disturb SSH or
even telnet. These are the most important things, even 200ms latency is
sluggish to work over.
- Allow 'surfing' at reasonable speeds while up or downloading
Even though http is 'bulk' traffic, other traffic should not drown it out
too much.
- Make sure uploads don't harm downloads, and the other way around
This is a much observed phenomenon where upstream traffic simply destroys
download speed.
It turns out that all this is possible, at the cost of a tiny bit of
bandwidth. The reason that uploads, downloads and ssh hurt each other is the
presence of large queues in many domestic access devices like cable or DSL
modems.
The next section explains in depth what causes the delays, and how we can
fix them. You can safely skip it and head straight for the script if you
don't care how the magic is performed.
15.8.1. Why it doesn't work well by default
ISPs know that they are benchmarked solely on how fast people can download.
Besides available bandwidth, download speed is influenced heavily by packet
loss, which seriously hampers TCP/IP performance. Large queues can help
prevent packet loss, and speed up downloads. So ISPs configure large queues.
These large queues however damage interactivity. A keystroke must first
travel the upstream queue, which may be seconds (!) long and go to your
remote host. It is then displayed, which leads to a packet coming back, which
must then traverse the downstream queue, located at your ISP, before it
appears on your screen.
This HOWTO teaches you how to mangle and process the queue in many ways, but
sadly, not all queues are accessible to us. The queue over at the ISP is
completely off-limits, whereas the upstream queue probably lives inside your
cable modem or DSL device. You may or may not be able to configure it. Most
probably not.
So, what next? As we can't control either of those queues, they must be
eliminated, and moved to your Linux router. Luckily this is possible.
- Limit upload speed
By limiting our upload speed to slightly less than the truly available rate,
no queues are built up in our modem. The queue is now moved to Linux.
- Limit download speed
This is slightly trickier as we can't really influence how fast the internet
ships us data. We can however drop packets that are coming in too fast,
which causes TCP/IP to slow down to just the rate we want. Because we don't
want to drop traffic unnecessarily, we configure a 'burst' size we allow at
higher speed.
Now, once we have done this, we have eliminated the downstream queue totally
(except for short bursts), and gain the ability to manage the upstream queue
with all the power Linux offers.
What remains to be done is to make sure interactive traffic jumps to the
front of the upstream queue. To make sure that uploads don't hurt downloads,
we also move ACK packets to the front of the queue. This is what normally
causes the huge slowdown observed when generating bulk traffic both ways.
The ACKnowledgements for downstream traffic must compete with upstream
traffic, and get delayed in the process.
If we do all this we get the following measurements using an excellent ADSL
connection from xs4all in the Netherlands:
Baseline latency:
round-trip min/avg/max = 14.4/17.1/21.7 ms
Without traffic conditioner, while downloading:
round-trip min/avg/max = 560.9/573.6/586.4 ms
Without traffic conditioner, while uploading:
round-trip min/avg/max = 2041.4/2332.1/2427.6 ms
With conditioner, during 220kbit/s upload:
round-trip min/avg/max = 15.7/51.8/79.9 ms
With conditioner, during 850kbit/s download:
round-trip min/avg/max = 20.4/46.9/74.0 ms
When uploading, downloads proceed at ~80% of the available speed. Uploads
at around 90%. Latency then jumps to 850 ms, still figuring out why. |
What you can expect from this script depends a lot on your actual uplink
speed. When uploading at full speed, there will always be a single packet
ahead of your keystroke. That is the lower limit to the latency you can
achieve - divide your MTU by your upstream speed to calculate. Typical
values will be somewhat higher than that. Lower your MTU for better effects!
Next, two versions of this script, one with Devik's excellent HTB, the other
with CBQ which is in each Linux kernel, unlike HTB. Both are tested and work
well.
15.8.2. The actual script (CBQ)
Works on all kernels. Within the CBQ
qdisc we place two Stochastic Fairness Queues that make sure that multiple
bulk streams don't drown each other out.
Downstream traffic is policed using a tc filter containing a Token Bucket
Filter.
You might improve on this script by adding 'bounded' to the line that starts
with 'tc class add .. classid 1:20'. If you lowered your MTU, also lower the
allot & avpkt numbers!
#!/bin/bash
# The Ultimate Setup For Your Internet Connection At Home
#
#
# Set the following values to somewhat less than your actual download
# and uplink speed. In kilobits
DOWNLINK=800
UPLINK=220
DEV=ppp0
# clean existing down- and uplink qdiscs, hide errors
tc qdisc del dev $DEV root 2> /dev/null > /dev/null
tc qdisc del dev $DEV ingress 2> /dev/null > /dev/null
###### uplink
# install root CBQ
tc qdisc add dev $DEV root handle 1: cbq avpkt 1000 bandwidth 10mbit
# shape everything at $UPLINK speed - this prevents huge queues in your
# DSL modem which destroy latency:
# main class
tc class add dev $DEV parent 1: classid 1:1 cbq rate ${UPLINK}kbit \
allot 1500 prio 5 bounded isolated
# high prio class 1:10:
tc class add dev $DEV parent 1:1 classid 1:10 cbq rate ${UPLINK}kbit \
allot 1600 prio 1 avpkt 1000
# bulk and default class 1:20 - gets slightly less traffic,
# and a lower priority:
tc class add dev $DEV parent 1:1 classid 1:20 cbq rate $[9*$UPLINK/10]kbit \
allot 1600 prio 2 avpkt 1000
# both get Stochastic Fairness:
tc qdisc add dev $DEV parent 1:10 handle 10: sfq perturb 10
tc qdisc add dev $DEV parent 1:20 handle 20: sfq perturb 10
# start filters
# TOS Minimum Delay (ssh, NOT scp) in 1:10:
tc filter add dev $DEV parent 1:0 protocol ip prio 10 u32 \
match ip tos 0x10 0xff flowid 1:10
# ICMP (ip protocol 1) in the interactive class 1:10 so we
# can do measurements & impress our friends:
tc filter add dev $DEV parent 1:0 protocol ip prio 11 u32 \
match ip protocol 1 0xff flowid 1:10
# To speed up downloads while an upload is going on, put ACK packets in
# the interactive class:
tc filter add dev $DEV parent 1: protocol ip prio 12 u32 \
match ip protocol 6 0xff \
match u8 0x05 0x0f at 0 \
match u16 0x0000 0xffc0 at 2 \
match u8 0x10 0xff at 33 \
flowid 1:10
# rest is 'non-interactive' ie 'bulk' and ends up in 1:20
tc filter add dev $DEV parent 1: protocol ip prio 13 u32 \
match ip dst 0.0.0.0/0 flowid 1:20
########## downlink #############
# slow downloads down to somewhat less than the real speed to prevent
# queuing at our ISP. Tune to see how high you can set it.
# ISPs tend to have *huge* queues to make sure big downloads are fast
#
# attach ingress policer:
tc qdisc add dev $DEV handle ffff: ingress
# filter *everything* to it (0.0.0.0/0), drop everything that's
# coming in too fast:
tc filter add dev $DEV parent ffff: protocol ip prio 50 u32 match ip src \
0.0.0.0/0 police rate ${DOWNLINK}kbit burst 10k drop flowid :1 |
If you want this script to be run by ppp on connect, copy it to
/etc/ppp/ip-up.d.
If the last two lines give an error, update your tc tool to a newer version!
15.8.3. The actual script (HTB)
The following script achieves all goals using the wonderful HTB queue, see
the relevant chapter. Well worth patching your kernel for!
#!/bin/bash
# The Ultimate Setup For Your Internet Connection At Home
#
#
# Set the following values to somewhat less than your actual download
# and uplink speed. In kilobits
DOWNLINK=800
UPLINK=220
DEV=ppp0
# clean existing down- and uplink qdiscs, hide errors
tc qdisc del dev $DEV root 2> /dev/null > /dev/null
tc qdisc del dev $DEV ingress 2> /dev/null > /dev/null
###### uplink
# install root HTB, point default traffic to 1:20:
tc qdisc add dev $DEV root handle 1: htb default 20
# shape everything at $UPLINK speed - this prevents huge queues in your
# DSL modem which destroy latency:
tc class add dev $DEV parent 1: classid 1:1 htb rate ${UPLINK}kbit burst 6k
# high prio class 1:10:
tc class add dev $DEV parent 1:1 classid 1:10 htb rate ${UPLINK}kbit \
burst 6k prio 1
# bulk & default class 1:20 - gets slightly less traffic,
# and a lower priority:
tc class add dev $DEV parent 1:1 classid 1:20 htb rate $[9*$UPLINK/10]kbit \
burst 6k prio 2
# both get Stochastic Fairness:
tc qdisc add dev $DEV parent 1:10 handle 10: sfq perturb 10
tc qdisc add dev $DEV parent 1:20 handle 20: sfq perturb 10
# TOS Minimum Delay (ssh, NOT scp) in 1:10:
tc filter add dev $DEV parent 1:0 protocol ip prio 10 u32 \
match ip tos 0x10 0xff flowid 1:10
# ICMP (ip protocol 1) in the interactive class 1:10 so we
# can do measurements & impress our friends:
tc filter add dev $DEV parent 1:0 protocol ip prio 10 u32 \
match ip protocol 1 0xff flowid 1:10
# To speed up downloads while an upload is going on, put ACK packets in
# the interactive class:
tc filter add dev $DEV parent 1: protocol ip prio 10 u32 \
match ip protocol 6 0xff \
match u8 0x05 0x0f at 0 \
match u16 0x0000 0xffc0 at 2 \
match u8 0x10 0xff at 33 \
flowid 1:10
# rest is 'non-interactive' ie 'bulk' and ends up in 1:20
########## downlink #############
# slow downloads down to somewhat less than the real speed to prevent
# queuing at our ISP. Tune to see how high you can set it.
# ISPs tend to have *huge* queues to make sure big downloads are fast
#
# attach ingress policer:
tc qdisc add dev $DEV handle ffff: ingress
# filter *everything* to it (0.0.0.0/0), drop everything that's
# coming in too fast:
tc filter add dev $DEV parent ffff: protocol ip prio 50 u32 match ip src \
0.0.0.0/0 police rate ${DOWNLINK}kbit burst 10k drop flowid :1 |
If you want this script to be run by ppp on connect, copy it to
/etc/ppp/ip-up.d.
If the last two lines give an error, update your tc tool to a newer version!
