An alert fires, users report a sluggish application, and someone asks the question every SRE knows: “Is it the network?” A single dropped ping rarely settles that argument. You need a repeatable packet-loss test, a second vantage point, and logs that let you compare the network symptom with application failures at the same time.
Packet loss testing turns that vague complaint into a measurable rate. The basic calculation is lost packets divided by transmitted packets, usually expressed as a percentage. The practical challenge is choosing the right probe, sample, path, and context so that the number reflects what users experience rather than what one manual command happened to observe.
Table of Contents
- Why Packet Loss Testing Matters in Real Operations
- Active Probing Tools and How to Read Their Output
- Active Probes vs Passive Interface Measurement
- Designing Test Scenarios for LAN, WAN, Cloud, and Wi-Fi
- Lab vs Production Trade-offs and Reproducing Loss Safely
- Automating Tests, Setting Thresholds, and Alerting with Fluxtail
- Troubleshooting Flow and a Practical Closing Checklist
Why Packet Loss Testing Matters in Real Operations
Packet loss is a network-quality problem that becomes useful during an incident only after you express it numerically. Send a known number of probes, count the replies, and record the resulting rate. That gives you a value you can compare across time, sites, interfaces, and routes. Packet-loss rate fundamentals describe both the percentage calculation and the use of active probes such as ping, while also noting that operators can measure loss passively from router and interface statistics.
Operational thresholds make that measurement actionable. Industry guidance commonly treats below 1% as acceptable, 1% to 2.5% as degraded, and above 5% as critical. These thresholds aren't a substitute for service-specific baselines, but they provide a practical first filter during triage. A 100-packet test makes the arithmetic easy: 99 successful replies represent about 1% packet loss.
Practical rule: Never treat one dropped packet as a root cause. Treat it as a reason to run a repeatable test.
Why one ping check misleads
A short, manual test can miss bursty loss. Congestion changes with time of day, traffic volume, and route selection, so a clean result taken during a quiet interval doesn't disprove an intermittent problem. Field guidance on packet-loss testing recommends repeated or continuous sampling, preferably from more than one location, because a single snapshot can't characterize a changing path reliably.
Loss also needs company. Record latency and jitter with the loss rate. A path can deliver every probe while producing severe latency variation, or it can show modest loss that causes TCP retransmissions and visible application delays. The combination tells you more than any individual column.
What a useful test delivers
A good test produces a timestamped result with the source, destination, protocol, packet count, loss percentage, round-trip latency, and jitter. Repeat the same measurement over time, then compare it with a known-good path or control location. That turns “the app feels slow” into a question such as whether loss increased on one route while the control path remained stable.
For standards-based work, RFC 2680 defines one-way packet-loss measurement around a known Type-P test packet, synchronized clocks, and a declared loss-detection threshold. The standard matters when you need defensible results across teams or providers, not just a quick indication that something is wrong.
Active Probing Tools and How to Read Their Output
Start with ping when you need a fast baseline. A controlled test sends multiple packets and calculates loss as (sent - received) / sent. For example:
ping -c 100 -i 1 target.example
Read the summary for transmitted packets, received packets, loss percentage, and round-trip statistics. The -i 1 interval spaces probes instead of firing them as quickly as the host allows. That reduces unnecessary test traffic and gives you a time series that is easier to correlate with logs. The mistake is stopping after a handful of packets. Short samples are especially weak against bursty loss.
Use mtr to locate a path pattern
mtr combines repeated probing with a route view:
mtr -r -n -c 100 target.example
The report's important fields are Loss%, Snt, Last, Avg, Best, Wrst, and StDev. A high loss value at one intermediate hop isn't automatically a bad link. If later hops and the destination remain healthy, that router may just rate-limit or deprioritize probe responses. Loss becomes more convincing when it begins at a hop and continues at a similar rate through the destination.
Use a probe that resembles the affected traffic when ICMP isn't representative:
sudo mtr --tcp --port 443 -r -n -c 100 target.example
A TCP-mode result can expose filtering or prioritization that an ICMP test won't show. For automation, save report output with timestamps, or use a machine-readable format supported by the installed mtr version.
Add load with iperf, carefully
iperf3 tests behavior under traffic rather than idle conditions:
iperf3 -s
iperf3 -c server.example -t 60
iperf3 -c server.example -u -b 10M -t 60
TCP output helps reveal retransmission and throughput behavior. UDP output exposes reported loss, jitter, and datagram throughput, but don't read the jitter column as packet loss. UDP tests also need a deliberately chosen rate. An aggressive stream can create the congestion you're trying to diagnose, while a timid stream may never reach the failure mode.
Capture proof with Wireshark
When an application reports stalls, capture traffic at the relevant endpoint or observation point:
sudo tcpdump -i eth0 -w incident.pcap host target.example
Open the capture in Wireshark and inspect TCP retransmissions, duplicate acknowledgments, out-of-order segments, and reset behavior. Retransmissions support the conclusion that delivery failed or arrived too late, but they don't identify which hop dropped the packet. Pair the capture with path tests and interface counters.
| Tool | Best for | Key output | Watch out for |
|---|---|---|---|
ping |
Fast reachability baseline | Replies, loss percentage, RTT | Small samples miss bursts |
mtr |
Path and hop patterns | Per-hop loss, latency, variation | Intermediate ICMP loss may be rate limiting |
iperf3 |
Loss under controlled load | TCP retransmissions, UDP loss and jitter | Test traffic can cause congestion |
| Wireshark | Application-level evidence | Retransmissions, duplicate ACKs, resets | A capture shows symptoms, not the failing hop |
Choose the probe based on the symptom. For protocol context and how traffic moves through network layers, the network-layer protocol guide is a useful reference. Don't use a large UDP stream to answer a simple reachability question, and don't use idle ICMP alone to claim that an application path is healthy.
Active Probes vs Passive Interface Measurement
Active and passive measurement answer different questions. An active probe asks, “What happens to traffic sent from this source to that destination right now?” A passive counter asks, “Is this interface or device reporting drops in the traffic it handles?” Serious incident response uses both because either view can be incomplete.
What active probes reveal
Ping, mtr, and synthetic UDP streams create controlled traffic and measure the end-to-end experience from a selected vantage point. They're valuable for testing a user-facing route, comparing paths, and finding failures that don't appear in a local interface counter. Their weakness is equally important: a probe can miss a short microburst, follow a different route than production traffic, or add enough traffic to influence a congested link.
Use more than one source when the incident affects only some users. A probe from the application host, a separate monitoring host, and an external location can separate a local access problem from a provider or destination problem.
What passive telemetry reveals
SNMP and similar telemetry expose router and interface statistics without generating synthetic traffic. These counters can show whether a physical link, queue, or device is dropping traffic even when a ping from your monitoring host succeeds. They also help distinguish a transient event from a persistent fault when collected continuously.
RFC 8321 describes alternate marking for production traffic. Packets receive a color or marking, and operators compare source and destination counters for the same marked interval. Loss is calculated by subtracting the destination count from the source count, allowing measurement of loss, delay, and jitter on live traffic. RFC 8321 documents the method and its production-traffic focus.
A clean ping and a rising interface-drop counter aren't contradictory. They may be observing different traffic, paths, or time windows.
Reconcile the two views
Compare timestamps before drawing conclusions. A passive counter spike with no active failure can indicate drops affecting traffic that your probe doesn't traverse. An active failure without a corresponding local counter change can point downstream, toward a carrier, remote network, or destination policy. Neither result alone proves the fault location.
The strongest evidence is convergent: active loss on the affected path, passive drops on the relevant interface, and application errors in the same interval. If the views disagree, vary the vantage point and protocol before escalating.
Designing Test Scenarios for LAN, WAN, Cloud, and Wi-Fi
Test design should follow the failure domain. A local switch, a carrier path, a cloud route, and a wireless cell can all produce “packet loss,” but they need different controls. Keep the destination, source, interval, and recording format consistent within each comparison.
LAN
Run parallel tests from more than one host to the local gateway or switch-facing service:
ping -c 100 -i 1 gateway.example
A single host can have a bad cable, NIC, driver, or access port, so parallel sources matter. Compare loss percentage, mean RTT, and jitter over the same window. Check interface error and drop counters as well. If only one host loses packets, investigate that endpoint first. If several hosts show the same pattern, examine the switch, uplink, queue, and physical path.
WAN
Use mtr toward the actual service path, not only a convenient public target:
mtr -r -n -c 100 target.example
Repeat from the remote side when possible because forward and return routes may differ. Preserve the complete report, timestamp, source location, destination, loss percentage, mean RTT, and jitter. A loss value that appears only at an intermediate hop but disappears at the destination may reflect ICMP treatment rather than forwarding loss.
For a broader distinction between local and wide-area paths, consult LAN versus WAN characteristics.
Cloud
Run iperf3 between controlled virtual machines when you need to see how a path behaves under load. Test the suspected region or availability-zone path, then repeat from a control location. Record loss, mean RTT, jitter, throughput, and the selected protocol. Keep the control test separate from the suspected path so that an edge firewall, host policy, or local egress issue doesn't look like a provider-wide problem.
Wi-Fi
Compare a wired baseline with a wireless test over the same period. Wi-Fi retransmissions are normal, so a wireless result needs context from signal quality, client location, access-point association, roaming events, and channel conditions. Record loss percentage, mean RTT, jitter, and the client's radio observations, then compare them with the wired host's results.
The video below provides a visual introduction to planning tests across common network environments.
Lab vs Production Trade-offs and Reproducing Loss Safely
During an incident, production diagnosis and resilience testing require separate controls. Production work observes the existing path without changing its behavior. Lab work introduces a measured impairment, then checks how the service, client, retry policy, and alerts respond.
Inject loss in a lab
Linux tc with netem can add loss, delay, and corruption to a virtual or test interface:
sudo tc qdisc add dev eth0 root netem loss 5%
sudo tc qdisc change dev eth0 root netem delay 50ms loss 5%
sudo tc qdisc del dev eth0 root netem
Run this only on an isolated interface or disposable test host, and remove the rule explicitly when the experiment ends. Validate the condition with the same probes and application checks used in production. The objective is not a dramatic outage. It is evidence that the service degrades predictably and that monitoring records the change.
The RFC 2680's measurement methodology supports recording the packet type, clock precision, and loss-detection threshold. Apply that discipline in the lab, or the test result will be difficult to compare with production evidence.
Diagnose production without polluting it
Do not apply netem to a live path to prove that the path is impaired. Use a second probe vantage point, passive interface telemetry, and an application-level check. Keep probe traffic low overhead, and mark the measurement interval separately from the incident interval.
A continuous sweep can hide burst patterns when its spacing is fixed. Short repeated runs can show whether loss clusters around particular moments. The Modern packet-loss measurement discussion examines probe spacing, sampling bias, and the risk that testing traffic contributes to congestion.
During incident response, correlate each probe window with Fluxtail log streams. Match the source, destination, timestamp, and application symptoms, then check whether retransmits, timeouts, or interface errors rise in the same interval. This connects a one-off ping result to the production behavior that matters.
Make the result reproducible
For every run, store the source, destination, protocol, packet count, interval, route, timestamp, loss percentage, latency, and jitter. Use identical configuration for baseline and comparison runs. If the destination rate-limits probes, choose a protocol or endpoint that represents the traffic under investigation instead of treating probe suppression as forwarding loss.
RFC 6703 recommends waiting long enough to distinguish true loss from finite delay. About 51 seconds is sufficient in general measurement circumstances, while approximately 50 seconds is considered manageable in practice. Report long-term loss only for intervals where connectivity can be demonstrated, and summarize other periods as time without apparent connectivity.
Automating Tests, Setting Thresholds, and Alerting with Fluxtail
Manual commands help during the first minutes of an incident, but they don't establish a durable baseline or wake the on-call engineer. Schedule repeatable probes with cron or a systemd timer, write timestamped results, and send the records into the same operational log stream as service errors.
A simple scheduled command might run mtr in report mode and emit machine-readable output. A wrapper should add source, destination, protocol, timestamp, loss percentage, mean RTT, and jitter before forwarding the event. Keep the payload small enough that the monitoring path doesn't become a second network problem.
Alert on operational bands
Use the thresholds as routing rules:
- Warning: loss below the degraded boundary but above the acceptable baseline.
- Error: loss at or above 2.5%, based on the cited operational guidance.
- Critical: loss above 5%, which the same guidance treats as critical.
The exact policy should reflect your service. A voice or transaction path may need a tighter response than a batch transfer. Pair loss alerts with latency and jitter, include the probe location, and require repeated evidence before paging when transient probe failures are common. The alerting best-practices guide covers the broader principles of routing actionable alerts without creating avoidable noise.
Correlate the network event with logs
Send probe results and application logs into separate named streams, then compare them in the live tail during the incident. Fluxtail ingests logs over HTTP, Syslog, OTLP, GELF, and collector traffic, routes them into named streams, and exposes timestamps, severity, stream, host, and message for triage. It doesn't replace the packet-loss probe. It gives the probe result a useful operational home beside timeout errors, retries, gateway failures, and deployment events.
The MCP server adds a practical investigation path for teams using compatible AI clients. An on-call engineer can query recent loss records across regions and compare them with error logs without copying dashboard screenshots into chat. Build that query into the runbook before the next incident, including the expected stream name and time window.
Troubleshooting Flow and a Practical Closing Checklist
Start with a 100-ping baseline from a known-good host, then vary the source and destination. Check whether loss reaches the destination, inspect the mtr pattern, compare passive counters, and only then investigate the application. Record loss percentage, latency, jitter, route, and timestamps, then correlate the result with logs in a central stream.
Packet-loss testing turns a fuzzy network complaint into one comparable number, then gives that number the context needed to find the fault.
Use Fluxtail to keep probe events, interface observations, and application errors in readable streams during incident response. Set up a loss-testing stream, correlate its timestamps with service failures, and give your on-call team a direct path from a packet-loss alert to the logs that explain it.