This document is intended for anyone looking to analyze network path information, especially Traceroute reports, and make a determination about whether or not the network path is having a material effect on the user experience.
Network diagnosis based on Traceroute leads to a high rate of false positives. This is primarily due to misinterpretation of metrics in Traceroute reports. This document is to help interpret Traceroute reports accurately and reduce false positive diagnosis.
What is Traceroute?
Traceroute is the primary tool for troubleshooting problems on the Internet. It is a command-line utility that, when given an IP address or hostname, shows a list of router hops between the source and destination, associated metrics such as latency, lost packets and router details.
Brief overview of Traceroute:
- The source device (SRC) launches a probe packet towards the destination device (DST), with a TTL of 1.
- Every router hop decreases the TTL of the packet by 1.
- When the TTL hits 0, the packet is dropped, and the router sends an ‘ICMP TTL Exceeded’ packet to SRC with the original probe packet as payload
- SRC receives this ICMP message, displays a Traceroute “hop”
- This is repeated from step 1, with the TTL value increased by 1 each time until the DST host receives a probe packet.
- When the DST host receives the probe packet, it returns “ICMP Dest Unreachable”.
- SRC stops the Traceroute upon receipt of “ICMP Dest Unreachable”.
- If the DST never receives a probe packet, or never responds to one, then traceroute will usually time out after a number of failed attempts.
Example Output (Linux):
Traceroute to www.ntt.net (130.94.58.116), 64 hops max, 52 byte packets 1 ge0-34.aggrFZ155-2.ord6.us.scnet.net (204.93.176.73) 4.558 ms 2.030 ms 2.730 ms 2 ge9-47.ar1.ord6.us.scnet.net (75.102.0.65) 0.405 ms 0.297 ms 0.265 ms 3 61.po4.ar1.ord1.us.scnet.net (75.102.3.225) 1.305 ms 1.249 ms 1.232 ms 4 ae0-81.cr1.ord1.us.nlayer.net (69.31.111.1) 1.135 ms 59.441 ms 1.144 ms 5 ae1.ar2.ord1.us.nlayer.net (69.31.111.146) 1.419 ms 2.249 ms 1.452 ms 6 as2914.xe-6-0-3.ar2.ord1.us.nlayer.net (69.31.111.233) 1.450 ms as2914.xe-6-0-2.ar1.ord1.us.nlayer.net (69.31.111.209) 1.608 ms as2914.xe-6-0-3.ar2.ord1.us.nlayer.net (69.31.111.233) 1.497 ms 7 ae-7.r21.chcgil09.us.bb.gin.ntt.net (129.250.4.201) 9.476 ms ae-6.r21.chcgil09.us.bb.gin.ntt.net (129.250.2.26) 1.389 ms 9.325 ms 8 ae-5.r20.snjsca04.us.bb.gin.ntt.net (129.250.3.107) 52.695 ms 54.304 ms 57.892 ms 9 ae-1.r06.snjsca04.us.bb.gin.ntt.net (129.250.5.13) 54.316 ms 54.275 ms 52.426 ms 10 130.94.58.116 (130.94.58.116) 52.211 ms 58.061 ms 54.065 ms
Common Tools
- Classic UNIX Traceroute, which uses UDP packets with destination ports starting at 33434, and incrementing by 1 with each probe. Typical defaults are 3 probes per hop (or TTL increment), but this is usually configurable. When the probe packet reaches the final destination, the host will return an ICMP Destination Unreachable packet (assuming no application is listening on those UDP ports, which is not common), denoting the end of the Traceroute.
- Windows Traceroute (or more specifically, tracert.exe) is notable for its use of ICMP Echo Request probes, rather than the UDP probes of classic UNIX Traceroute implementations. When the probe packet reaches the final destination, an ICMP Echo Reply packet is returned, indicating the end of the Traceroute.
- “My Traceroute” (MTR) is another popular program which can also use ICMP probe packets. The most notable difference between MTR and other programs is that it performs its Traceroute probes in an infinite loop. MTR is extremely good at finding multiple parallel paths, since it sends many probes over a long period of time.
Modern Traceroute implementations allow the user to specify UDP, ICMP, or TCP probe packets.
Common Issues with Traceroute Analysis
The following highlight why a misinterpretation of Traceroute metrics often leads to misdiagnosis, particularly when analyzing the three primary use cases: path identification, latency, and packet loss.
Path Identification: Asymmetric Forwarding Paths
Traceroute only shows the forward path. It is a common misconception that the path reported by traceroute is for both directions. The reverse path is invisible to Traceroute and can be completely different at every hop, and often differs from different networks along the path. Traceroute helps determine if the forward path to a destination is reasonable. To verify the reverse path, a separate Traceroute from the destination back to the origin is required.
Latency Analysis: Delays from ICMP Prioritization/Rate Limiting
Latency spikes on individual hops are often caused by the router de-prioritizing and rate-limiting the generation of the ‘ICMP TTL Exceed’ packets. By contrast, data packets are forwarded much faster by dedicated circuitry that can switch packets at line rate and doesn’t have to involve the router’s CPU when it might be busy doing something more important. High latency detected at a specific hop doesn’t automatically indicate an impact on traffic to the final destination. It is crucial to verify whether the increased latency persists in the subsequent hops.
Example (Windows):
>tracert www.amazon.com Tracing route to cf.47cf2c8c9-frontier.amazon.com [3.164.99.110] over a maximum of 30 hops: 1 7 ms 5 ms 6 ms mynetwork.home [192.168.2.1] 2 13 ms 39 ms 19 ms 142.124.37.243 3 * * * Request timed out. 4 * * * Request timed out. 5 * * * Request timed out. 6 12 ms 10 ms 11 ms 142.124.123.156 7 14 ms 24 ms 9 ms 142.124.125.80 8 15 ms 36 ms 7 ms 64.230.97.145 9 * * * Request timed out. 10 * * * Request timed out. 11 * * * Request timed out. 12 * * * Request timed out. 13 * * * Request timed out. 14 * * * Request timed out. 15 9 ms 6 ms 12 ms server-3-164-99-110.yto53.r.cloudfront.net [3.164.99.110]
Latency to destination is lower than latency reported in earlier hops. So the higher latency at middle hops can be ignored.
Hop/Packet Loss Analysis: Transient Drops That Do Not Persist
Traceroute appears to stop or drop packets at the intermediate hops could be misleading. Packet drops at an intermediate hop does not necessarily mean that the traffic for the final destination will also be dropped. It is important to check packet drops at subsequent hops.
Latency or packet loss seen on a Traceroute should be ignored if subsequent hops show lower latency or less/no packetloss, as every packet that reaches a later hop has to be forwarded by the earlier one. If latency or packet loss appears on a Traceroute and continues all the way through to the end, then it may indicate a real issue. That issue may be on the forward path right before the first affected hop, or it might be on the return path from that and subsequent hops.
Example (Windows):
>tracert www.google.com Tracing route to www.google.com [142.250.137.147] over a maximum of 30 hops: 1 9 ms 6 ms 4 ms mynetwork.home [192.168.2.1] 2 8 ms 9 ms 5 ms 142.124.37.243 3 * * * Request timed out. <------ Hidden hops 4 * * * Request timed out. 5 * * * Request timed out. 6 14 ms 10 ms 8 ms 142.124.123.154 7 * 9 ms 27 ms 142.124.125.82 <------ Busy routers 8 13 ms 6 ms * 64.230.97.147 9 * * * Request timed out. 10 * 17 ms 17 ms 192.178.99.39 11 9 ms 15 ms 11 ms 192.178.99.24 12 14 ms 24 ms 12 ms 108.170.229.1 13 10 ms 11 ms 13 ms 142.250.208.188 14 * * * Request timed out. <------ Hidden hops 15 * * * Request timed out. 16 * * * Request timed out. 17 * * * Request timed out. 18 * * * Request timed out. 19 * * * Request timed out. 20 15 ms 9 ms 12 ms pnyyzb-in-f147.1e100.net [142.250.137.147] Trace complete.
Despite the many “Request timed out” errors, the trace confirms connection to Google is great.
- The Destination Was Reached: The final line (Hop 20) shows it successfully connected to Google.
- Latency is Excellent: Round-trip time to the destination is roughly 12 ms (milliseconds), which is excellent.
- The “Timeouts” (*) Are Normal:
- Hops 3–5: These are ISP’s internal routers. They are configured to be “invisible”, but they passed subsequent packets through successfully.
- Hops 14–19: This is Google’s security firewall. It blocks the traceroute tracking tools but allows actual data to reach the final server.
Traceroute Metrics
Key metrics reported by Traceroute that help diagnose network issues are described below:
Router Hops (Path Identification)
Traceroute shows a list of router hops between a source and a destination. There are multiple factors that affect the path reported by Traceroute.
- Asymmetric Routing
When a traceroute is performed from a source device to a destination device, the path represented by the results is referred to as “the forward path”. From the perspective of that same source device, the path from the destination device back to the source device is referred to as “the reverse path”. In networking, knowing the forward path does not mean that you automatically know the reverse path. In some cases the paths can be different, which is referred to as “asymmetric routing”. Ideally, these paths should be identical or at least very similar. When they differ greatly, performance issues can arise. For this reason, it is very important to confirm both the forward path and reverse path, which means that traceroutes need to be run from both the source and destination devices.
- Load Balancing and Multiple Paths
There could be Network layer 3 (L3) load balancers in the network path which distribute packets to multiple routers. In such cases, probe packets could reach different routers that are actually at the same “hop” in the network path. While assessing the hops, it is essential to consider the possibility of multiple, sometimes unequal, forward paths being represented within a single traceroute result.
Example Output:
4 p16-1-0-0.r21.asbnva01.us.bb.gin.ntt.net (129.250.5.21) 0.571 ms 0.604 ms 0.594 ms 5 p16-4-0-0.r00.chcgil06.us.bb.gin.ntt.net (129.250.5.102) 25.981 ms p16-1-2-2.r21.nycmny01.us.bb.gin.ntt.net (129.250.4.26) 7.279 ms 7.260 ms 6 p16-2-0-0.r21.sttlwa01.us.bb.gin.ntt.net (129.250.2.180) 71.027 ms p16-1-1-3.r20.sttlwa01.us.bb.gin.ntt.net (129.250.2.6) 66.730 ms 66.535 ms
In this example, traffic is load balanced across the following two paths:
- Ashburn VA – Chicago IL – Seattle WA
- Ashburn VA – New York NY – Seattle WA
MPLS/Tunnels and Traceroute
When networks use MPLS or tunnels to carry IP traffic across their backbone, the packet encapsulation employed by these technologies may result in multiple network hops being hidden in the traceroute results, or even the same latency being reported across multiple hops.
Example Output:
1 te2-4.ar5.PAO2.gblx.net (69.22.153.209) 1.160 ms 1.060 ms 1.029 ms 2 192.205.34.245 (192.205.34.245) 3.984 ms 3.810 ms 3.786 ms 3 tbr1.sffca.ip.att.net (12.123.12.25) 74.848 ms 74.859 ms 74.936 ms 4 cr1.sffca.ip.att.net (12.122.19.1) 74.344 ms 74.612 ms 74.072 ms 5 cr1.cgcil.ip.att.net (12.122.4.122) 74.827 ms 75.061 ms 74.640 ms 6 cr2.cgcil.ip.att.net (12.122.2.54) 75.279 ms 74.839 ms 75.238 ms 7 cr1.n54ny.ip.att.net (12.122.1.1) 74.667 ms 74.501 ms 77.266 ms 8 gbr7.n54ny.ip.att.net (12.122.4.133) 74.443 ms 74.357 ms 75.397 ms 9 ar3.n54ny.ip.att.net (12.123.0.77) 74.648 ms 74.369 ms 74.415 ms 10 12.126.0.29 (12.126.0.29) 76.104 ms 76.283 ms 76.174 ms 11 route-server.cbbtier3.att.net (12.0.1.28) 74.360 ms 74.303 ms 74.272 ms
Latency (RTT)
Latency reported by Traceroute indicates ‘round trip’ time for a packet to get to the hop and back to the source. During transit, there are multiple factors that contribute to latency.
- Propagation Delay
Time for the signal to reach the destination over the physical medium. For optical networks, 100 kilometers of fiber will result in approximately 1ms of round-trip propagation delay.
- Serialization Delay
It is generally not possible to begin transmitting the packet to the egress interface until the entire packet has been received. To calculate the serialization delay, simply divide the size of the packet by the speed of the link. Higher the linkspeed, lower the serialization delay.
- Queueing Delay
When an interface is busy, the packets to be transmitted must be queued. As an interface approaches the point of saturation, the percentage of packets which must be queued increases exponentially and can add significant latency.
The latency values reported by Traceroute are has the following components,
- The time taken for the probe packet to reach a specific router, plus
- The time taken for that router to generate an ICMP TTL Exceed packet, plus
- The time taken for the ICMP packet to return to the Traceroute source
Routers often give low priority to ICMP packets (slow path), processed by the CPU which could be delayed by other high priority tasks handled by the CPU. Data packets passing through the router take a fast path which does not incur these delays. If there is an actual issue, the latency will continue to increase in subsequent hops.
Packet Drops
Traceroute reports where it “stops” or “drops packets”. This does not necessarily mean there is a connection problem to the destination. Packet drops at a hop are often cited as network problems in the path, which usually is not the case. Routers may drop the Traceroute probe packets for various reasons (rate limiting, CPU contention, interface congestion etc).
Packet drops at a hop followed by packet drops at subsequent hops at a similar frequency is indicative of a problem. However, if packets are dropped at hop but no drops in subsequent hops, this is often not an issue.
Reverse DNS
Traceroute tools can do reverse DNS lookup for router IPs and report the names. These values might not follow a standard format, but greatly helps understand various aspects of the network path which can help with troubleshooting.
- Location Identifiers – IATA airport codes, CLLI Codes, arbitrary values.
- Router Types / Roles – core router, edgerouter etc.
- Network Boundaries – network boundaries are handoff points where administrative policies change and capacity is most constrained, often leading to altered return paths or congestion.
- Interface Types – 10 Gigabit Ethernet, Gigabit Ethernet, ATM etc.
For example, consider xe-11-1-0.edge1.NewYork1.Level3.net. The naming scheme xe-#-#-# is a Juniper 10GE interface: FPC 11, PIC 1, port 0. The presence of FPC 11 indicates at least a 12-slot chassis.
Sometimes, it is easy to detect network boundaries from DNS names:
4 te1-2-10g.ar3.DCA3.gblx.net (67.17.108.146) 5 sl-st21-ash-8-0-0.sprintlink.net (144.232.18.65)
Moving Beyond Traceroute: Digital Experience Management
While Traceroute provides foundational information about the network path, most users often require deeper insights, less risk for misinterpretation, and a proactive approach to managing user experience. Netskope’s Digital Experience Management (DEM) is designed to address the inherent limitations of Traceroute by integrating deep performance metrics into the platform. DEM provides you with context-aware visibility into every step of the traffic flow. This provides a more accurate and actionable view of network performance.
1. Path Identification: Asymmetric Forwarding Paths
The primary limitation for path identification is that Traceroute only shows the forward path: the reverse path and end-to-end flow are invisible. This lack of visibility introduces significant ambiguity regarding the actual route and the source of any delays.
To remove this ambiguity and dive deeper into path identification, DEM provides complete visibility from device to application. DEM monitors the real user experience and the performance of every step from the device to the application. This multi-layer visibility accurately measures the entire path taken by user traffic, ensuring that the entire round-trip flow is understood. To learn more, please see User Overview.
2. Latency Analysis: Delays from ICMP Prioritization/Rate Limiting
A common misdiagnosis occurs because latency spikes are often false positives caused by the router de-prioritizing and rate-limiting the ICMP TTL Exceed response. This means the data reflects the router’s CPU load, not the network’s forwarding capacity.
To gain clarity on true network and application latency, Netskope DEM uses real user activity monitoring. DEM measures performance based on monitoring real user activity and traffic analysis, not on the control plane response. By tracking performance metrics of real traffic, DEM reports true application and network performance.
3. Hop/Packet Loss Analysis: Spikes and Drops That Do Not Persist
Diagnosing packet loss or high latency with Traceroute is difficult because false reports occur when temporary spikes or drops don’t persist, making it hard to differentiate real issues from network noise.
To provide definitive clarity on the impact and scope of impairments, DEM uses alerts to diagnose, predict, and prioritize events, effectively reducing false positives. Metrics provide valuable data about user experience, including device health, network, and application performance, which immediately clarifies the scope and impact of any impairment. This contextual data offers a comprehensive diagnosis of any potential issues with packet loss or latency.