ESP32 firmware update recovery time | Groundrun

groundrun.io

Platform
Blog
About
Docs
Book a demo

20 September 2026
We broke an Over-The-Air update on the ESP32 on purpose

Over-The-Air (OTA) updates are one of the most fundamental mechanisms in any connected product. Thanks to OTAs, we can fix problems even after the product has shipped simply by downloading a new firmware over the air. In fact, most connected products perform an OTA update the first thing they do when they are installed: this ensures they always have all the latest fixes as quickly as possible.
OTAs entirely change the entire software in our system, so they are delicate: there are plenty of chances where the mechanism could go wrong. (This is why you sometimes see a message saying "do not switch off until the update is applied".)
But because the OTA mechanism is so important, it has to be completely reliable. There should be no way to break the OTA mechanism, no matter how hard we try. Even if we download only a partial OTA, or if power is cut at the worst possible moment. Nothing should break it. And we should recover as quickly as possible.
OTAs are such an important mechanism that it must be automatically tested on every change to the system. We should have our devices go through the worst possible ways we could wreck havoc to the update mechanism, and see that they still work. The update needs to either fully apply or not apply at all: if the update fails, we should be back at a known-good state.
We need to:

Survive the interruption. The device should still be alive after an interrupted OTA update, running the original version of the code.
Recover from the interruption. The device should redo the OTA update, thus ending up on the new version of the code.

So this we need to test: an update either finishes and applies cleanly, or it gets interrupted. The power goes out mid-download, the device reboots on its own, someone hits reset at the wrong moment. When that happens, the device has to recover on its own and finish the job anyway.
In this post we use the Groundrun system to try to crush OTAs on an ESP32-C6 system and to see that we are able to successfully recover every time. We kill the OTA in three different ways: (1) we do a software reset in the middle of the update, (2) we do a hardware reset in the middle of the update, and (3) we pull the power plug on the entire system in the middle of the update. We then ensure that the system is able to recover, restart the update, and complete it. We measure the time it is able to do so.
This is something that any connected product should do as part of its development workflow to ensure that the system is as stable as possible, after every change to the system.
How Over the air updates work
An over-the-air update pushes new firmware to a device over its own network connection, then installes the new firmware to itself. The new image comes from a cloud backend, which knows both which version each device is running and which version the device shold be updated to. Whether the device fetches it right away depends on the product: some updates apply quietly in the background, and others show up in the app first, so the person using the product decides when to accept one. Either way, the device downloads the new image, applies it, and reboots into it.

New image
ready in the cloud

Sent in the
background

Shown in the
app first

Device
downloads it

Applies and
reboots into it

A new image starts in the cloud, reaches the device either quietly or through the app, and the device applies it.

The tricky part is if the OTA process gets interrupted. In the worst case, a firmware image applied halfway through a write can leave a device unable to boot at all, which is the absolutely worst state that a connected product can end up in. This is typically known as a bricked device: a device that is about as useful as a brick. Completely dead to the world.
Any OTA mechanism must be fully able to avoid bricking the device. The common defense is to never touch the firmware a device is currently running until the new image has confirmed itself. The new image goes into its own separate storage first, gets verified before anything switches over, and the device falls back to what it was running if the new image never completes. Today, every chip vendor provides an OTA mechanism that has this level of production. In our case, the Espressif's ESP-IDF, which the ESP32 in this experiment runs, is one of them. With the default mechanism, an interrupted update cannot leave a device unable to boot.
Since the OTA mechanim protects us against bricking the device, the worst state we can end up in is a device stuck on the old version, with nothing making sure a fresh attempt ever happens. Our system must therefore be able to detect this and recover. We should automatically start a new attempt with the end goal that our device ends up on the target version.
The ESP32
The ESP32 is Espressif's family of Wi-Fi and Bluetooth microcontrollers. The ESP32 is a very common microcontroller, used across smart plugs, sensors, appliances, and plenty of other connected products. The board in this experiment is an ESP32-C6, one of the newer chips in the line, adding Thread alongside Wi-Fi and Bluetooth Low Energy on the same chip. Its firmware runs on ESP-IDF, Espressif's own development framework.
How the ESP32 handles OTA recovery
ESP-IDF implements the general pattern above with two flash partitions, ota_0 and ota_1. The device boots from whichever one its bootloader currently points at, and an update downloads to whichever one is inactive.
The board in this experiment joins Wi-Fi the way a shipped device would join a home network, and the update itself comes from an ordinary HTTPS server, the same path esp_https_ota() uses in a real deployment.
The update downloads to the inactive slot, gets verified, and only then does the bootloader switch the boot pointer and reboot into it. If the new image boots and confirms itself, the bootloader marks it valid and moves on. If it doesn't, CONFIG_BOOTLOADER_APP_ROLLBACK_ENABLE sends the device back to the slot it just came from, automatically, with nothing running on top of the firmware involved.

Before

ota_0
running v1

ota_1
unused
Boot pointer

After a successful update

ota_0
holds v1

ota_1
running v2
Boot pointer

The two flash partitions the ESP32 uses for an update: the boot pointer marks one active, and a new image lands in the other.

We interrupt the update in three ways
A clean run of this update, undisturbed, takes about ninety seconds on the rig's own network. At some point point during the download we cut the update three different ways.

Software reset. Sends the board's own console a force_reboot command mid-download, closer to a crash somewhere else in the firmware.
Reset-pin toggle. Pulses its hardware reset line directly, the same debug hardware used to flash it in the first place.
Hard power cut. The rig's own power switch cuts current to the board, the way a brownout or someone unplugging the product would. This one is a real hardware fault, not a simulated one, and it's the rig doing it automatically as part of the scenario, not a person standing at the outlet.

Each of these methods has the same effect: they stop the OTA partway through writing the new image to the inactive slot. They interrupt before we reach the point where the bootloader would switch the boot pointer, so the device that reboots is still running the old firmware, with the same rollback mechanism ready to walk it through a fresh attempt.

They interrupt before we reach the point where the bootloader would switch the boot pointer, so the device that reboots is still running the old firmware, with the same rollback mechanism ready to walk it through a fresh attempt.

While the three methods has a similar effect, in that they all interrupt the firmware download, they could potentially break the system in different ways. We want to make sure that our system survives and recovers in every case.
How to recover from a failed update
Recovery here means one thing: the device reboots, confirms it is still on the old version, completes a fresh update to the new one, and holds that version through a second, deliberate reset. That second reset rules out a lingering PENDING_VERIFY state, where the bootloader still thinks the new image might need to roll back.

Update
downloading

Software reset

Reset-pin toggle

Hard power cut

Reboots on
old version

Second attempt
succeeds

Three ways an update gets interrupted, all converging on the same recovery path.

In every trial where the interruption landed, that is exactly what happened. The board came back on the old version, took a second update attempt cleanly, and stayed there. ESP-IDF's own rollback mechanism did the whole job, with nothing on Groundrun's side stepping in, in any of the eighteen trials.
But we need to do more than just confirm that the second firmware works. We need to be through: a thorough test goes one step further than confirming the device runs v2: it drives one more update, from v2 to v3, to check that v2 did not lose its own ability to take a future update along the way.
Comparison
Three interruption methods, two rollback settings, three trials each: eighteen runs total, on real hardware. The numbers are mean recovery time, from the moment the update starts to the confirmed second reset.

Rollback enabledRollback disabled

Hard power cut99.6s93.5s
Software reset165.7s155.3s
Reset-pin toggle157.5s158.5s

0s

20s

40s

60s

80s

100s

120s

140s

160s

180s

99.6s

93.5s
Hard power cut

165.7s

155.3s
Software reset

157.5s

158.5s
Reset-pin toggle

Rollback enabled

Rollback disabled

Mean recovery time by interruption method and rollback setting, across three trials each.

Recovery time clustered by how the update was interrupted more than by whether rollback was enabled. A hard power cut recovered fastest, under two minutes either way, because that recovery run is a clean second attempt with no re-provisioning needed. Software reset and the reset-pin toggle took longer, around two and a half minutes, because their recovery run also rejoins Wi-Fi from scratch.
Varying when interruption occurs
To be even more thorough, we also vary at which point the interruption happens. This does not change the correctness of our experiemnt: we can still recover every time. But it changes how long recovery takes. The comparison above held that fixed, cutting every mechanism at the same instant, right as the download starts.
We used the reset-pin toggle for this one, since it the fastest of the mechamisms to apply, so we can control the timing on a millisecond level. First we measured three clean, uninterrupted downloads to get a real baseline: 89 seconds, in tight agreement across all three runs. Then we timed the reset-pin toggle to land at 25%, 50%, and 75% of that baseline, rollback enabled throughout, three trials at each point.

140s

160s

180s

200s

220s

240s

0%
25%
50%
75%
Cut point, as a percentage of the baseline download

157.5s

183.2s

200.8s

223.2s
Fixed-point reference
(reset-pin, rollback enabled)

Mean recovery time against how far into the download the reset-pin toggle fired, from a fixed-point reference (effectively a 0% cut) through 75% of the baseline download.

Every one of the nine cut-point trials recovered automatically, the same as the fixed-point comparison above. What changed was how long it took. Recovery at the earliest cut point, 25% into the download, took just over three minutes. By the time the cut landed at 75%, recovery had grown to close to three and three-quarters. The extra recovery time tracks the added delay closely: it's the download's own progress, redone twice, once before the interruption and once again after.

The extra recovery time tracks the added delay closely: it's the download's own progress, redone twice, once before the interruption and once again after.

Conclusions
Our experiment show that we are able to both survive and recover our interrupted OTA attempts. This is what we expected: we were expecting the raw OTA mechanism in the Espressif IDF to be solid. Across every configuration we interrupted, the device found its own way back through ESP-IDF's own rollback mechanism. Exercised by three different interruption paths, it recovers reliably regardless of a legitimate configuration choice like whether that mechanism is even turned on.
The value of our experiment is in the confidence the mechanism itself brings. We can now run these scenarios repeatedly and be certain that whatever we do, our most important mechanism still always continues to wokr.

Groundrun platform overview →

More from the blog

Automating Wi-Fi setup testing on the ESP32
A second product on the same app and cloud: that's when you want Groundrun
Docs, for people and agents

Platform
Blog
About
Docs

groundrun.io
© groundrun.io

Over-The-Air (OTA) updates represent a fundamental mechanism for connected products, allowing fixes to be deployed remotely. Because OTA updates involve significant changes to the software, the mechanism must possess absolute reliability, ensuring that even under adverse conditions, it can recover gracefully. This reliability necessitates that the system must survive interruptions—such as power loss or unexpected resets—and automatically recover to a known-good state. Therefore, the core requirement is that an update must either complete successfully or revert cleanly, preventing the device from entering an unusable, "bricked" state. This requirement demands that the system survive the interruption and possess the capability to automatically reattempt and complete the update.

The experiment conducted by the authors utilized the Groundrun system to rigorously test the resilience of the OTA mechanism on an ESP32-C6 system. The objective was to subject the system to various forms of interruption during the firmware download and verify successful recovery. The researchers simulated three distinct failure modes: performing a software reset mid-download, executing a hardware reset during the update, and cutting the power supply to the entire system during the process. These methods were designed to interrupt the writing of the new image at various stages.

The mechanism governing the ESP32's OTA process, implemented via the ESP-IDF, relies on two flash partitions, ota_0 and ota_1, where the boot pointer determines which partition is active and which is inactive for the update. The update downloads to the inactive slot, is verified, and only then the bootloader switches the pointer. A critical safeguard is the CONFIG_BOOTLOADER_APP_ROLLBACK_ENABLE feature, which manages the rollback process. If a newly booted image fails self-confirmation, this mechanism automatically reverts the device to the previously running firmware partition, ensuring that the system avoids being left in an indeterminate state.

When the update was interrupted using the three failure methods, the system was designed to recover by rebooting into the original firmware, initiating a fresh update attempt, and maintaining that stable version through a second deliberate reset, thereby eliminating any lingering pending verification states. The study confirmed that the ESP-IDF's inherent rollback mechanism successfully handled all interruption scenarios reliably, regardless of whether rollback was explicitly enabled or disabled in testing.

The measured recovery times varied based on the interruption method. The hard power cut demonstrated the fastest recovery time, achieving completion in under two minutes across all tested configurations. In contrast, interruptions involving a software reset or a reset-pin toggle resulted in longer recovery times, approximately two and a half minutes, as these methods required the recovered device to re-establish its network connection and re-provision itself. Furthermore, the time taken for recovery was directly correlated with where the interruption occurred during the download; interrupts near the beginning of the download tested faster than those occurring later in the process, demonstrating that the recovery delay reflects the amount of download progress that had to be redone. Overall, the experiment strongly supports the confidence that the OTA mechanism reliably ensures the survival and recovery of firmware updates by autonomously utilizing internal rollback procedures.