The day my ping took countermeasures
Recorded: May 24, 2026, 2:58 a.m.
| Original | Summarized |
The day my ping took countermeasures Once my holidays had passed, I found myself reluctantly reemerging into the world of the living. I powered on a corporate laptop, scared to check on my email inbox. However, before turning on the browser, obviously, I had to run a ping. Debugging the network is a mandatory first step after a boot, right? As expected, the network was perfectly healthy but what caught me off guard was this message: I was not expecting ping to take countermeasures that early on in a day. Gosh, I wasn't expecting any countermeasures that Monday!Once I got over the initial confusion, I took a deep breath and collected my thoughts. You don't have to be Sherlock Holmes to figure out what has happened. I'm really fast - I started ping before the system NTP daemon synchronized the time. In my case, the computer clock was rolled backward, confusing ping.While this doesn't happen too often, a computer clock can be freely adjusted either forward or backward. However, it's pretty rare for a regular network utility, like ping, to try to manage a situation like this. It's even less common to call it "taking countermeasures". I would totally expect ping to just print a nonsensical time value and move on without hesitation.Ping developers clearly put some thought into that. I wondered how far they went. Did they handle clock changes in both directions? Are the bad measurements excluded from the final statistics? How do they test the software?I can't just walk past ping "taking countermeasures" on me. Now I have to understand what ping did and why. Understanding ping An investigation like this starts with a quick glance at the source code: The code is straightforward: the message in question is printed when the measured RTT is negative. In this case ping resets the latency measurement to zero. Here you are: "taking countermeasures" is nothing more than just marking an erroneous measurement as if it was 0ms.But what precisely does ping measure? Is it the wall clock? The man page comes to the rescue. Ping has two modes.The "old", -U mode, in which it uses the wall clock. This mode is less accurate (has more jitter). It calls gettimeofday before sending and after receiving the packet.The "new", default, mode in which it uses "network time". It calls gettimeofday before sending, and gets the receive timestamp from a more accurate SO_TIMESTAMP CMSG. More on this later. Tracing gettimeofday is hard Let's start with a good old strace: int gettimeofday(struct timeval *restrict tv, void *restrict tz) { time_t time(time_t *tloc) { int clock_gettime(clockid_t clockid, struct timespec *tp) { $ LD_PRELOAD=./vdso_override.so \ clock_gettime(CLOCK_REALTIME, {tv_sec=1688656245 ...}) = 0 To suid or not to suid I forgot that ping might need special permissions to read and write raw packets. Historically it had a suid bit set, which granted the program elevated user identity. However LD_PRELOAD doesn't work with suid. When a program is being loaded a dynamic linker checks if it has suid bit, and if so, it ignores LD_PRELOAD and LD_LIBRARY_PATH settings.However, does ping need suid? Nowadays it's totally possible to send and receive ICMP Echo messages without any extra privileges, like this: sd = socket(AF_INET, SOCK_DGRAM, IPPROTO_ICMP) sd.send(struct.pack("!BBHHH10s", 8, 0, 0, 0, 1234, b'payload')) Rootless is perfectly fine Let's remove the CAP_NET_RAW and try out LD_PRELOAD hack again: $ LD_PRELOAD=./vdso_override.so strace -f ./ping -n -c1 1.1 Fault injection - fooling ping With strace up and running we can finally do something interesting. You see, strace has a little known fault injection feature, named "tampering" in the manual: With a couple of command line parameters we can overwrite the result of the gettimeofday call. I want to set it forward to confuse ping into thinking the SO_TIMESTAMP time is in the past: PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data. --- 1.1.1.1 ping statistics --- Fool me once, fool me twice At this point we could conclude our investigation. We're now able to reliably trigger the "taking countermeasures" message using strace fault injection.There is one more thing though. When sending ICMP Echo Request messages, does ping remember the send timestamp in some kind of hash table? That might be wasteful considering a long-running ping sending thousands of packets.Ping is smart, and instead puts the timestamp in the ICMP Echo Request packet payload!Here's how the full algorithm works:Ping sets the SO_TIMESTAMP_OLD socket option to receive timestamps.It looks at the wall clock with gettimeofday.It puts the current timestamp in the first bytes of the ICMP payload.After receiving the ICMP Echo Reply packet, it inspects the two timestamps: the send timestamp from the payload and the receive timestamp from CMSG.It calculates the RTT delta.This is pretty neat! With this algorithm, ping doesn't need to remember much, and can have an unlimited number of packets in flight! (For completeness, ping maintains a small fixed-size bitmap to account for the DUP! packets).What if we set a packet length to be less than 16 bytes? Let's see: --- 139.162.188.91 ping statistics --- --- 139.162.188.91 ping statistics --- def custom_action(echo_req): echo_reply = scapy.IP( scapy.sniff(filter="icmp and icmp[0] = 8", iface=iface, prn=custom_action) Leap second In practice, how often does time change on a computer? The NTP daemon adjusts the clock all the time to account for any drift. However, these are very small changes. Apart from initial clock synchronization after boot or sleep wakeup, big clock shifts shouldn't really happen.There are exceptions as usual. Systems that operate in virtual environments or have unreliable Internet connections often experience their clocks getting out of sync.One notable case that affects all computers is a coordinated clock adjustment called a leap second. It causes the clock to move backwards, which is particularly troublesome. An issue with handling leap second caused our engineers a headache in late 2016. Leap seconds often cause issues, so the current consensus is to deprecate them by 2035. However, according to Wikipedia the solution seem to be to just kick the can down the road:A suggested possible future measure would be to let the discrepancy increase to a full minute, which would take 50 to 100 years, and then have the last minute of the day taking two minutes in a "kind of smear" with no discontinuity.In any case, there hasn't been a leap second since 2016, there might be some in the future, but there likely won't be any after 2035. Many environments already use a leap second smear to avoid the problem of clock jumping back.In most cases, it might be completely fine to ignore the clock changes. When possible, to count time durations use CLOCK_MONOTONIC, which is bulletproof.We haven't mentioned daylight savings clock adjustments here because, from a computer perspective they are not real clock changes! Most often programmers deal with the operating system clock, which is typically set to the UTC timezone. DST timezone is taken into account only when pretty printing the date on screen. The underlying software operates on integer values. Let's consider an example of two timestamps, which in my Warsaw timezone, appear as two different DST timezones. While it may like the clock rolled back, this is just a user interface illusion. The integer timestamps are sequential: $ date --date=@$[1698541199+1] Lessons Arguably, the clock jumping backwards is a rare occurrence. It's very hard to test for such cases, and I was surprised to find that ping made such an attempt. To avoid the problem, to measure the latency ping might use CLOCK_MONOTONIC, its developers already use this time source in another place.Unfortunately this won't quite work here. Ping needs to compare send timestamp to receive timestamp from SO_TIMESTAMP CMSG, which uses the non-monotonic system clock. Linux API's are sometimes limited, and dealing with time is hard. For time being, clock adjustments will continue to confuse ping.In any case, now we know what to do when ping is "taking countermeasures"! Pull down your periscope and check the NTP daemon status! NetworkFollow on XMarek Majkowski|@majek04Cloudflare|@cloudflareRelated postsApril 08, 2026From bytecode to bytes: automated magic packet generationBy applying symbolic execution and the Z3 theorem prover to BPF bytecode, we’ve automated the generation of malware trigger packets, cutting analysis time from hours to seconds....By Axel BoesenachMalware, Network, Z3, BPF, Reverse Engineering November 05, 2025How Workers VPC Services connects to your regional private networks from anywhere in the worldWorkers VPC Services enter open beta today. We look under the hood to see how Workers VPC connects your globally-deployed Workers to your regional private networks by using Cloudflare's global network, while abstracting cross-cloud networking complexity....By Thomas Gauvin, Matt Alonso, Eric FalcãoCloudflare Workers, Workers VPC, Cloudflare Tunnel, Network, Hybrid Cloud, Security, VPC, Private Network October 31, 2025BGP zombies and excessive path huntingA BGP “zombie” is essentially a route that has become stuck in the Default-Free Zone (DFZ) of the Internet, potentially due to a missed or lost prefix withdrawal. We’ll walk through some situations where BGP zombies are more likely to rise from the dead and wreak havoc. |
The investigation began when the author observed the network utility ping taking countermeasures after a system clock rollback, prompting an inquiry into the utility's handling of time. This led to examining the origins of ping, which was originally developed by Mike Muuss in 1983 while working at the U.S. Army Ballistic Research Laboratory. The analysis delved into the source code, specifically the gather_statistics function, revealing that "taking countermeasures" occurs when the measured Round Trip Time (RTT) is negative, causing ping to reset the latency measurement to zero. The utility operates in two modes: an older mode that uses the wall clock and a newer, default mode that utilizes network time information obtained via the SO_TIMESTAMP CMSG. A significant technical hurdle in tracing these operations on modern Linux systems involved understanding how system calls were executed. The author explored the use of strace and discovered that modern Linux utilizes virtual system call entry points known as vdso, which execute faster in userspace rather than making traditional system calls visible to tools like strace. Attempts to bypass this mechanism through manipulating the Auxiliary Vector or using LD_PRELOAD failed to capture the necessary system call traces. The author experimented with overriding the behavior of functions like gettimeofday by redefining them to use slower, explicit syscalls, achieving temporary access to system call tracing data, although this method proved fragile when applied to ping. Further investigation clarified ping's internal timing mechanism. Ping uses the wall clock via gettimeofday, and in its modern mode, it relies on network time obtained through the CMSG to calculate latency. The core algorithm involves setting the SO_TIMESTAMP_OLD socket option to receive timestamps, fetching time using gettimeofday, encoding the send time in the ICMP payload, and comparing both the send time and the received timestamp from the CMSG to calculate the RTT delta. This structure allows ping to function without needing to maintain extensive historical timestamp data. The analysis then explored advanced manipulations of the ping process. By truncating the ICMP payload to less than sixteen bytes, ping correctly skips the RTT calculation, demonstrating efficiency. Furthermore, by proactively manipulating the timestamps sent in the ICMP Echo Request packet, the author successfully induced ping to report the "taking countermeasures" warning, showing that the message is triggered when time moves in one direction. Experiments also showed that forwarding a packet response could confuse ping, and setting the send timestamp backward resulted in nonsensical RTT values. This established that countermeasures are successfully triggered only when time moves in a specific direction. The discussion extended to the broader context of timekeeping, addressing leap seconds and the use of system clocks. While the NTP daemon constantly adjusts the clock, large shifts are rare, with exceptions occurring in virtual environments or when dealing with leap seconds. The text argues for using monotonic clocks, such as CLOCK_MONOTONIC, for measuring durations, as these are immune to practical clock adjustments. The conclusion suggests that while current system time adjustments continue to confuse ping, the method of reliably triggering the countermeasures message has been established, directing attention toward monitoring the NTP daemon status. |