3. How it Works
There are two basic steps to optimize upstream bandwidth. First we have to find a way to
prevent the ADSL modem from queuing packets since we have no control over how it handles
the queue. In order to do this we will throttle the amount of data the router sends out eth0
to be slightly less than the total upstream bandwidth of the ADSL modem. This will result
in the router having to queue packets that arrive from the Local Network faster than it is allowed
to send them.
The second step is to set up priority queuing discipline on the router.
We'll investigate a queue that can be configured to give priority to interactive
traffic such as telnet and multi-player games.
The final step is to configure the firewall to prioritize packets by using fwmark.
3.1. Throttling Outbound Traffic with Linux HTB
Although the connection between the router and the modem is at 10Mbit/s, the modem
is only able to send data at 128kbit/s. Any data sent in excess of that rate will be queued
at the modem. Thus, a ping packet sent from the router may go to the modem immediately, but
may take a few seconds to actually get sent out to the Internet if the queue in the modem
has any packets in it. Unfortunately most ADSL modems provide no mechanism to specify how
packets are dequeued or how large the queue is, so our first objective is to move the place
where the outbound packets are queued to somewhere where we have more control over the queue.
We'll do this by using the HTB queue to limit
the rate at which we send packets to the ADSL modem. Even though our upstream bandwidth may be
128kbit/s we'll have to limit the rate at which we send packets to be slightly below that. If
we want to lower the latency we have to be SURE that not a single packet is ever queued at the
modem. Through experimentation I have found that limiting the outbound traffic to about 90kbit/s
gives me almost 95% of the bandwidth I could achieve without HTB rate control. With HTB enabled at this
rate, we've prevented the ADSL modem from queuing packets.
3.2. Priority Queuing with HTB
At this point we still haven't realized any change in the performance. We've merely moved the
FIFO queue from the ADSL modem to the router. In fact, with Linux configured to a default queue
size of 100 packets we've probably made our problem worse at this point! But not for long...
Each neighbor class in an HTB queue can be assigned a priority. By placing different types
of traffic in different classes and then assigning these classes different priorities, we can
control the order in which packets are dequeued and sent. HTB makes it possible to do this while still
avoiding starvation of any one class, since we're able to specify a minimum guaranteed rate for each
class. In addition to this, HTB allows for us to tell a class that it may use any unused bandwidth
from other classes up to a certain ceiling.
Once we have our classes set up, we set up filters to place traffic in classes. There are several
ways to do this, but the method described in this document uses the familiar iptables/ipchains to
mark packets with an fwmark. The filters place traffic into the classes of the HTB queue based on
their fwmark. This way, we're able to set up matching rules in iptables to send certain types of
traffic to certain classes.
3.3. Classifying Outbound Packets with iptables
The final step in configuring your router to give priority to interactive traffic is
to set up the firewall to define how traffic should be classified. This is done by setting the
packet's fwmark field.
Without getting into too much detail, here is a simplified description of how outbound packets
might be classified into 4 classes with the highest priority class being 0x00:
Mark ALL packets as 0x03. This places all packets, by default, into the lowest priority queue.
Mark ICMP packets as 0x00. We want ping to show the latency for the highest priority packets.
Mark all packets that have a destination port 1024 or less as 0x01. This gives priority to system
services such as Telnet and SSH. FTP's control port will also fall into this range however FTP data transfer
takes place on high ports and will remain in the 0x03 band.
Mark all packets that have a destination port of 25 (SMTP) as 0x03. If someone sends an email with
a large attachment we don't want it to swamp interactive traffic.
Mark all packets that are going to a multi-player game server as 0x02. This will give gamers low latency but
will keep them from swamping out the the system applications that require low latency.
Mark any "small" packets as 0x02. Outbound ACK packets from inbound downloads should be sent promptly
to assure efficient downloads. This is possible using the iptables length module.
Obviously, this can be customized to fit your needs.
3.4. A few more tweaks...
There are two more things that you can do to improve your latency. First, you can set the Maximum Transmittable
Unit (mtu) to be lower than the default of 1500 bytes. Lowering this number will lower the average time you have
to wait to send a priority packet if there is already a full-sized low-priority packet being sent. Lowering this
number will also slightly decrease your throughput because each packet contains at least 40 bytes worth of IP and
TCP header information.
The other thing you can do to improve latency even on your low-priority traffic is to lower your queue length
from the default of 100, which on an ADSL line could take as much as 10 seconds to empty with a 1500 byte mtu.
3.5. Attempting to Throttle Inbound Traffic
By using the Intermediate Queuing Device (IMQ), we can run all incoming packets through a queue in the same
way that we queue outbound packets. Packet priority is much simpler in this case. Since we can only (attempt to)
control inbound TCP traffic, we'll put all non-TCP traffic in the 0x00 class, and all TCP traffic in the 0x01 class.
We'll also place "small" TCP packets in the 0x00 class since these are most likely ACK packets for outbound data that
has already been sent. We'll set up a standard FIFO queue on the 0x00 class, and we'll set up a Random Early Drop (RED)
queue on the 0x01 class. RED is better than a FIFO (tail-drop) queue at controlling TCP because it will drop packets
before the queue overflows in an attempt to slow down transfers that look like they're about to get out of control.
We'll also rate-limit both classes to some maximum inbound rate which is less than your true inbound speed over the
ADSL modem.
3.5.1. Why Inbound Traffic Limiting isn't all That Good
We want to limit our inbound traffic to avoid filling up the queue at the ISP, which can sometimes buffer
as much as 5 seconds worth of data. The problem is that currently the only way to limit inbound TCP traffic
is to drop perfectly good packets. These packets have already taking up some share of bandwidth on the ADSL modem
only to be dropped by the Linux box in an effort to slow down future packets. These dropped packets will eventually
be retransmitted consuming more bandwidth. When we limit traffic, we are limiting the rate of packets which we
will accept into our network. Since the actual inbound data rate is somewhere above this because
of the packets we drop, we'll actually have to limit our downstream to much lower than the
actual rate of the ADSL modem in order to assure low latency. In practice I had to limit my 1.5mbit/s downstream
ADSL to 700kbit/sec in order to keep the latency acceptable with 5 concurrent downloads. The more TCP sessions
you have, the more bandwidth you'll waste with dropped packets, and the lower you'll have to set your limit rate.
A much better way to control inbound TCP traffic would be TCP window manipulation, but as of this writing
there exists no (free) implementation of it for Linux (that I know of...).
