ADSL Bandwidth Management HOWTO by Dan Singletary (inspirational books for students TXT) 📕

|

| |

+---------------------------------------------------------------------------+

At this point we still haven't realized any change in the performance. We've

merely moved the FIFO queue from the ADSL modem to the router. In fact, with

Linux configured to a default queue size of 100 packets we've probably made

our problem worse at this point! But not for long...

Each neighbor class in an HTB queue can be assigned a priority. By placing

different types of traffic in different classes and then assigning these

classes different priorities, we can control the order in which packets are

dequeued and sent. HTB makes it possible to do this while still avoiding

starvation of any one class, since we're able to specify a minimum guaranteed

rate for each class. In addition to this, HTB allows for us to tell a class

that it may use any unused bandwidth from other classes up to a certain

ceiling.

Once we have our classes set up, we set up filters to place traffic in

classes. There are several ways to do this, but the method described in this

document uses the familiar iptables/ipchains to mark packets with an fwmark.

The filters place traffic into the classes of the HTB queue based on their

fwmark. This way, we're able to set up matching rules in iptables to send

certain types of traffic to certain classes.

3.3. Classifying Outbound Packets with iptables

+---------------------------------------------------------------------------+

| |

| Note: originally this document used ipchains to classify packets. The |

| newer iptables is now used. |

| |

+---------------------------------------------------------------------------+

The final step in configuring your router to give priority to interactive

traffic is to set up the firewall to define how traffic should be classified.

This is done by setting the packet's fwmark field.

Without getting into too much detail, here is a simplified description of how

outbound packets might be classified into 4 classes with the highest priority

class being 0x00:

Mark ALL packets as 0x03. This places all packets, by default, into the

lowest priority queue.

Mark ICMP packets as 0x00. We want ping to show the latency for the

highest priority packets.

Mark all packets that have a destination port 1024 or less as 0x01. This

gives priority to system services such as Telnet and SSH. FTP's control

port will also fall into this range however FTP data transfer takes place

on high ports and will remain in the 0x03 band.

Mark all packets that have a destination port of 25 (SMTP) as 0x03. If

someone sends an email with a large attachment we don't want it to swamp

interactive traffic.

Mark all packets that are going to a multi-player game server as 0x02.

This will give gamers low latency but will keep them from swamping out

the the system applications that require low latency.

Mark any "small" packets as 0x02. Outbound ACK packets from inbound

downloads should be sent promptly to assure efficient downloads. This is

possible using the iptables length module.

Obviously, this can be customized to fit your needs.

3.4. A few more tweaks...

There are two more things that you can do to improve your latency. First, you

can set the Maximum Transmittable Unit (mtu) to be lower than the default of

1500 bytes. Lowering this number will lower the average time you have to wait

to send a priority packet if there is already a full-sized low-priority

packet being sent. Lowering this number will also slightly decrease your

throughput because each packet contains at least 40 bytes worth of IP and TCP

header information.

The other thing you can do to improve latency even on your low-priority

traffic is to lower your queue length from the default of 100, which on an

ADSL line could take as much as 10 seconds to empty with a 1500 byte mtu.

3.5. Attempting to Throttle Inbound Traffic

By using the Intermediate Queuing Device (IMQ), we can run all incoming

packets through a queue in the same way that we queue outbound packets.

Packet priority is much simpler in this case. Since we can only (attempt to)

control inbound TCP traffic, we'll put all non-TCP traffic in the 0x00 class,

and all TCP traffic in the 0x01 class. We'll also place "small" TCP packets

in the 0x00 class since these are most likely ACK packets for outbound data

that has already been sent. We'll set up a standard FIFO queue on the 0x00

class, and we'll set up a Random Early Drop (RED) queue on the 0x01 class.

RED is better than a FIFO (tail-drop) queue at controlling TCP because it

will drop packets before the queue overflows in an attempt to slow down

transfers that look like they're about to get out of control. We'll also

rate-limit both classes to some maximum inbound rate which is less than your

true inbound speed over the ADSL modem.

3.5.1. Why Inbound Traffic Limiting isn't all That Good

We want to limit our inbound traffic to avoid filling up the queue at the

ISP, which can sometimes buffer as much as 5 seconds worth of data. The

problem is that currently the only way to limit inbound TCP traffic is to

drop perfectly good packets. These packets have already taking up some share

of bandwidth on the ADSL modem only to be dropped by the Linux box in an

effort to slow down future packets. These dropped packets will eventually be

retransmitted consuming more bandwidth. When we limit traffic, we are

limiting the rate of packets which we will accept into our network. Since the

actual inbound data rate is somewhere above this because of the packets we

drop, we'll actually have to limit our downstream to much lower than the

actual rate of the ADSL modem in order to assure low latency. In practice I

had to limit my 1.5mbit/s downstream ADSL to 700kbit/sec in order to keep the

latency acceptable with 5 concurrent downloads. The more TCP sessions you

have, the more bandwidth you'll waste with dropped packets, and the lower

you'll have to set your limit rate.

A much better way to control inbound TCP traffic would be TCP window

manipulation, but as of this writing there exists no (free) implementation of

it for Linux (that I know of...).

Implementation

Now with all of the explanation out of the way it's time to implement

bandwidth management with Linux.

4.1. Caveats

Limiting the actual rate of data sent to the DSL modem is not as simple as it

may seem. Most DSL modems are really just ethernet bridges that bridge data

back and forth between your linux box and the gateway at your ISP. Most DSL

modems use ATM as a link layer to send data. ATM sends data in cells that are

always 53 bytes long. 5 of these bytes are header information, leaving 48

bytes available for data. Even if you are sending 1 byte of data, an entire

53 bytes of bandwidth are consumed sent since ATM cells are always 53 bytes

long. This means that if you are sending a typical TCP ACK packet which

consists of 0 bytes data + 20 bytes TCP header + 20 bytes IP header + 18

bytes Ethernet header. In actuality, even though the ethernet packet you are

sending has only 40 bytes of payload (TCP and IP header), the minimum payload

for an Ethernet packet is 46 bytes of data, so the remaining 6 bytes are

padded with nulls. This means that the actual length of the Ethernet packet

plus header is 18 + 46 = 64 bytes. In order to send 64 bytes over ATM, you

have to send two ATM cells which consume 106 bytes of bandwidth. This means

for every TCP ACK packet, you're wasting 42 bytes of bandwidth. This would be

okay if Linux accounted for the encapsulation that the DSL modem uses, but

instead, Linux only accounts the TCP header, IP header, and 14 bytes of the

MAC address (Linux doesn't count the 4 bytes CRC since this is handled at the

hardware level). Linux doesn't count the minimum Ethernet packet size of 46

bytes, nor does it take into account the fixed ATM cell size.

What all of this means is that you'll have to limit your outbound bandwidth

to somewhat less than your true capacity (until we can figure out a packet

scheduler that can account for the various types of encapsulation being

used). You may find that you've figured out a good number to limit your

bandwidth to, but then you download a big file and the latency starts to

shoot up over 3 seconds. This is most likely because the bandwidth those

small ACK packets consume is being miscalculated by Linux.

I have been working on a solution to this problem for a few months and have

almost settled on a solution that I will soon release to the public for

further testing. The solution involves using a user-space queue instead of

linux's QoS to rate-limit packets. I've basically implemented a simple HTB

queue using linux user-space queues. This solution (so far) has been able to

regulate outbound traffic SO WELL that even during a massive bulk download

(several streams) and bulk upload (gnutella, several streams) the latency

PEAKS at 400ms over my nominal no-traffic latency of about 15ms. For more

information on this QoS method, subscribe to the email list for updates or

check back on updates to this HOWTO.

4.2. Script: myshaper

The following is a listing of the script which I use to control bandwidth on

my Linux router. It uses several of the concepts covered in the document.

Outbound traffic is placed into one of 7 queues depending on type. Inbound

traffic is placed into two queues with TCP packets being dropped first

(lowest priority) if the inbound data is over-rate. The rates given in this

script seem to work OK for my setup but your results may vary.

+---------------------------------------------------------------------------+

|