Archive for January, 2011
What QoS Looks Like
How to use Wireshark and Pathview
to verify QoS Performance
Quality of Service, aka QoS, is one of those industry specific buzz words that everyone in the IT and telecom world is at least conceptually aware of. When pressed for the details of the nuts and bolts, though, there is often an indignant “well, don’t you know?” response followed by immediate loss of eye contact and shrug of the shoulders. In the Precambrian VoIP era (roughly 2001 and earlier), many of us were migrating from traditional “digital” voice technology and were sometimes asked how we could prioritize voice traffic over any other network traffic to which we would respond “Why, QoS it of course!” .
“Great! Then bring your QoS thingy out and let’s get these IP phones working …” was usually the customer’s response. But as soon as the door was closed or the phone hung up, there would be that soul searching, gut wrenching moment of “why did I commit to that?” followed by “Who do I know that can do this?”. In the beginning, for many of us, QoS, like the earth in Genesis “was without form and void, and darkness was over the face of the deep.”
IP Packet Cliff Notes
Many IT guys know – and many more voice guys should know – that when you are capturing packets – usually with Wireshark – what you are seeing is a combination of traffic that is LAN specific and traffic that is on its way to another network. Most of it is “IP” and some of it is not. Non-IP packets, for example, would be ARPs that you see in the capture saying “Who has 192.168.1.1? Tell 192.168.1.104” – commonly known as a ‘request’. These sort of requests are all over the place in a busy network and you will see associated with them the replies like “192.168.1.1 is at 00:23:ae:63:21:aa” (a MAC address) . You will also see the occasional “Gratuitous ARP” which is basically some host making sure that there is not another host on the network that has the same IP address it was just assigned (either manually or by a DHCP server). What makes ARP a non-IP packet is that it does not have an IP field in its header and is therefore un-routable off the local network.
Typical ARP Packet
Comparing that with a typical TCP packet, you will see an Internet Protocol field along with all the goodies that come with it – including the holy grail (at least for this article) where you can actually tell whether the packet is given preferential treatment or not – the ToS (for Type of Service) field aka the Differentiated Services Field.
There are, or better, there were two main flavors of ToS – IP Precedence and DSCP. IP Precedence is the older of the two and consists of the first 3 “most significant” bits in the ToS field (i.e. reading the bits from left to right). From that, the router could determine what ‘precedence’ this particular packet had (if any) in terms of how it got routed e.g. route it using a particular interface or use the path with the best known metric, etc. and not necessarily whether it got to go in front of the line or not. Hence the term “IP Precedence” and not “IP Preference” . When you see this, think DSCP as in just about every instance nowadays this term and method has superseded it.
A common QoS misconception that is shared by most non-packet oriented office members (think sales people who have just heard of the term and want to show off at the next sales meeting) is that you want to prioritize everything from a particular IP address. On the surface that might seem okay until you realize the shear number of protocols, applications and chatter that just one semi-busy machine generates on the network. There are times when you do this but not many. Don’t be THAT GUY in the meeting with an owner/CEO and his IT staff that refers to QoS configuration in this manner – you will lose the job the minute they are able to talk about you behind your back. And don’t pronounce it as “Kwoss” either, it’s Q-O-S. Saying it any other way makes you sound like a hick telephone guy or a salesman trying to sound like he knows what he’s talking about. It must be said smoothly, confidently and with a certain air of “I know what this means, do you?”
Most QoS is a combination of “marking” packets of a particular protocol at the source level in combination with a matching QoS outbound map/policy/rule at the router level and/or the managed path level in the case of MPLS networks. You can prioritize packets solely based on ports, IP addresses even entire subnets but that is done at the router level only – usually one with a lot of horse power too – and frankly, it is less common.
DSCP (aka “DS”, “DiffServ”) stands for Differentiated Service Code Point – not that anyone actually says that unless they are trying to impress you – and it comes in several decimal, binary or hexadecimal values. In Wireshark, the DS field itself is the second byte in the IPv4 header packet (which you will still see as the ToS field in some text books) and consists of two sections –
- The actual DS binary value which occupies the first six most significant bits .
- The ECN value (aka Explicit Congestion Notification) which is used to notify end points that a router along the path is reaching its saturation point and to “lighten up” until it has a chance to work through its issue. This is a little used feature and most of the time you will see it set to zero.
Technically, you can have up to 64 (0 through 63, that’s 2^6) values or “classes” but I’ve never seen a small or medium size business that needed more than 5. For most scenarios, you will be using one of the following 3 classes;
1) Best Effort – which is like sending a message in a bottle hoping that it gets to the other side … eventually. Most sites we have walked into that are not using VoIP usually are using this. The numeric value is always zero.
2) Assured Forwarding – This has decimal values between 10 and 38 and is a step or two above just throwing your packets into the Internet ocean. It’s not FedEx but it’s not carrier pigeon either if programmed right. In most cases you will use this to prioritize different services such as video, a Point of Sale application, security , utilities, email etc. I will sometimes put just the signaling packets of VoIP in this class with a decimal value of 26 or 28. In other cases you will use it to control access to a shared circuit – e.g. you may have 2 or more tenants using the same Internet T-1 and you want to control their percentage of usage equally.
3) Expedited Forwarding - When it comes to the RTP voice packets, this is the only class you want because it is the one that gets your information there ahead of everyone else. On top of that, you will usually reserve some bandwidth, to the determent of all other packets, when it absolutely positively has to get there …. now. Its decimal value is 46. Special note – if your application requires either hex or binary and you don’t do conversions well in your head, get an app for your iPhone or Droid that does this – their cheap and often free.
QoS from Wiresharks' point of view
The actual decimal value in the Differentiated Services field that you see in Wireshark will not seem to match what you thought you programmed until you look at it in the packet details pane. The reason for this is because there is an “off set” which, in this case, allows you to address two semi-separate needs (DSCP and ECN) in one byte – that happens sometimes in packets btw.
Getting from Here to There in Style …. And Knowing It.
Keep in mind that the voice packets are usually going to be “marked” at the end points by what ever application is creating the data traffic – in this case the “VoIP” app. These points include the actual IP-PBX or Voice Server, the physical IP set or even the “softphone”. Routers within an MPLS cloud will sometimes alter the marking by either changing them to “Best Effort” – which defeats our purpose – or some other value either intentionally or inadvertently. Also, QoS of any sort only kicks in when the outgoing bandwidth from your WAN is saturated.
Customer "The Internet is slow sometimes but that should not affect anything you're doing, right?" IT Dept - "We have everything locked down pretty tight. I doubt we're using more than 1 meg."
From a VoIP standpoint, there are effectively 2 protocol groups you care about;
1) SIP, SDP, H.323, RTSP and H225 (there are more but these are the most used ones) which controls the non-voice part of a conversation such as the set up and teardown of calls (as a voice guy, think ‘signaling’) and almost anything that is not actually carrying packets with audio in it. This does not HAVE to have the top priority as it is not nearly as time sensitive as the second group but in cases when network traffic is really congested, you should give it some type of priority.
2) RTP and RTCP which are referred to as the “Transport”. RTP is the protocol that actually has voice audio in it and you will want this to get the highest priority possible. RTCP is the control link for a particular RTP session. It has a few flavors and is another form of signaling but you want to keep this with the same QoS treatment as RTP.
A typical question you will hear when the MPLS carrier or generic router guy starts setting up QoS for you is “How are you marking your packets?”. To which you would inform him that the packets will be marked such and such to which they will usually say “No problem” and then disappear with nary a call or email. So the question becomes a) Did they do it? and b) How do I verify that they did it? After all, it’s you and “that phone system your company sold us” that will be held accountable. This is where two important rules come into play with regard to doing anything on someone’s network the first time around.
First Rule – Get a baseline of the health and use of their circuit(s) BEFORE you start tinkering. This does not necessarily mean capturing packets with Wireshark (at least, not at first) as you are not so interested in the details of WHAT is going through (web traffic, emails, hosted services etc.) as much as you are interested in HOW it is going through. Are the circuits you are working with clean and by that I mean is there any continuity loss? High utilization? Jitter? High latency? Packet loss of either voice or data packets – i.e. anything that will affect a real time application like voice.. Fyi, normally data packets have larger payloads than voice packets and are sometimes treated differently as they travel across the network. Programming QoS on a circuit that is not fit for the doghouse (even if you’re not in it at the time) will be a big waste of unbillable time.
Second Rule – Do not rely on the customer, their IT department or the carrier to give you anything close to accuracy or the truth as to the condition of their circuit(s). They are usually clueless, misinformed or indifferent as to its importance. Invest in a piece of software that can reliably measure all of the above. The key word here is “reliably” because there is plenty of software (free and not free) that will claim to do it but with some caveats. If your reputation is being the “go to” guy, don’t morph into the “go too cheap” guy. If the app you’re using misdiagnoses, comes up with lots of false positives or simply misses the problem altogether, get something else … quick! I am partial to Pathview (developed by Apparent Networks, btw we’re partners with them and can hook you up!) for that reason. With this type of software, you can keep everyone honest – you, the customer and most importantly, the carrier/ISP.
Nothing gets you off the hook faster than being able to prove that a customer’s network or (more likely) Internet connection was screwed up BEFORE you walked into the building than a nice colorful graph or set of tests taken ahead of time. I have run into situations where the IT department knew that the network was messed up but wanted to keep it that way just enough to allow them to increase their budget for more “ IT things” (not Pathview or anything like it by the way). It was one of those “silent battles” that occasionally come up – just go in knowing that sometimes not everyone wants you to fix the problem.
Once both Rules have been followed, you will then have something to go to work with – and eventually pick up your check. What you should see is things like jitter, MOS, packet loss and latency drop for those packets that are prioritized (see the graphs showing QoS enabled and not enabled). QoS is not perfect but in critical situations – when done right – it’s pretty good.
Some Final Pointers
Implementing QoS is like any other network project. Get the big picture first with baselines because it will definitely help color in the details later. Links to remote sites – especially end users working at their homes as more and more are doing nowadays – are almost always the trouble points so use an app that is good for that and does not take all your hard drive.
Going in with a packet sniffer like Wireshark first – unless you are really familiar with the site or the problem – is arduous and may even distract you from the primary objective. Besides, packet analysis is intense and time consuming – if you have that skill you are entitled to get top dollar but do it efficiently. If you don’t have that skill and want it, check out Wireshark Network Analysis by Laura Chappell – you will be a much sought after, well informed (and well compensated) network geek when you are through.
If you are consulting or selling managed services, it is important to straighten out your sales staff that contrary to what they read in the “IT Sales Action Hero” comic book, QoS is not magic and can not traverse into a remote network to prioritize THEIR data traffic.
Finally, be sure to let your customer and sales staff know that in the event that the entire circuit is down, nothing will get through – not even QoS’ed packets. As obvious as this sounds, I actually had someone ask me that. It’s hard to make this stuff up …..
Copyright Eric Knaus, WCNA Jan 2011
Incoming search terms for the article:
