Converting from Arduino to ESP-IDF

...for real this time.

A few months ago I wrote about converting TiltBridge to use Arduino as an ESP-IDF component – swapping the build system from framework = arduino to framework = espidf while keeping Arduino available as a fallback. That post ended with a working project that still used Arduino for WiFi, BLE, HTTP, Serial, and basically everything else. The framework had changed but the code hadn't.

This post is about the other side of that coin: actually removing Arduino. Replacing every Arduino API call, every Arduino library, and every implicit dependency on Arduino.h with native ESP-IDF equivalents –and then deleting the Arduino component entirely.

I recently did this conversion for TiltBridge - and am doing the conversion for BrewPi-ESP - and having spent a fair bit of time to complete the conversion I learned a lot of things the hard way that I couldn't find documented anywhere else. This is my attempt to write the guide I wish I'd had when I started.

But first, why bother converting?

The obvious question. Arduino-as-component was working. Why not stop there?

A few reasons. First, the Arduino component is *large*. It pulls in WiFi libraries, Bluetooth stacks, HTTP servers, filesystem wrappers, and a mountain of compatibility shims – most of which I was no longer using. That's dead weight in a firmware binary where every kilobyte matters.

Second, having two of everything creates confusion. When your project has both esp_http_client.h and HTTPClient.h available, it's only a matter of time before someone (probably future me) includes the wrong one. The Arduino component also brings its own versions of system headers that can conflict with the ESP-IDF originals – I spent an embarrassing amount of time debugging a build failure that turned out to be Arduino's IPAddress.h redefining a macro that lwIP also defines.

Most importantly, however, remains the reasons I covered in my earlier post: The tooling and level of support for Arduino has gone downhill since Espressif and Platformio had their falling out. Eventually I anticipate that leading to my build pipeline ceasing to function, and there is nothing more disheartening than getting to what you feel is the end of a project, and finding that you can't actually build or release it.

A Key Insight: Arduino is Already ESP-IDF

The thing that makes this conversion manageable – and I cannot stress this enough – is that Arduino-ESP32 is built on top of ESP-IDF. Every delay() is a vTaskDelay(). Every digitalRead() is a gpio_get_level(). Every WiFi.begin() is calling esp_wifi_start() under the hood. The ESP-IDF APIs are right there, accessible even while you're still building with framework = arduino.

This means you don't have to do everything at once. You can replace Arduino APIs with their ESP-IDF equivalents one file at a time, testing as you go, while the project continues to build and run on Arduino. By the time you actually flip the switch and remove the Arduino component, most of your code is already using native APIs and the final cut is anticlimactic.

I broke TiltBridge's conversion into four phases, and I'd recommend anyone attempting this do the same.

Phase 1: Replace Arduino Built-ins (while still on Arduino)

This is the tedious-but-safe phase. You're replacing Arduino's convenience functions with the ESP-IDF calls they wrap, one at a time, while keeping framework = arduino in your platformio.ini. Nothing changes about how the project builds or runs. You're just peeling away the abstraction layer.

The Easy Stuff

Some replacements are pure find-and-replace. Arduino's F() macro, for example, stores string literals in flash on AVR microcontrollers. On ESP32, strings are already in flash. F() has always been a no-op on this platform. Delete it everywhere:

// Before
Log.notice(F("Starting up.\r\n"));

// After
Log.notice("Starting up.\r\n");

delay() is similarly straightforward:

// Before
delay(1000);

// After
#include <freertos/FreeRTOS.h>
#include <freertos/task.h>
vTaskDelay(pdMS_TO_TICKS(1000));

ESP.restart() becomes esp_restart(). ESP.getFreeHeap() becomes esp_get_free_heap_size(). These are all one-line swaps.

The Harder Stuff: Killing Arduino String

This is the single most time-consuming change in the entire conversion, and if you use a lot of Strings, it can touch almost everything. Every String object becomes a char[] or char*. Every concatenation becomes an snprintf. Every .length() becomes a strlen().

// Before
String url = "http://" + String(host) + ":" + String(port) + path;

// After
char url[256];
snprintf(url, sizeof(url), "http://%s:%d%s", host, port, path);

It's not glamorous work, but it's important. Arduino's String class does heap allocation behind your back on every concatenation, which fragments memory over time on a device with limited RAM. The C-string approach is more verbose but completely predictable.

My advice: start with leaf files -- utilities, data structures, anything that doesn't depend on many other modules -- and work inward. You can do this one file at a time over days or weeks.

If you're using ArduinoJson (and if you're building an IoT project, you probably are), watch out for two things. First, doc["field"].is<String>() becomes doc["field"].is<const char*>() -- ArduinoJson's String type check is Arduino-specific. Second, when serializing to a char* buffer, you need to pass the buffer size: serializeJson(doc, buf, sizeof(buf)). Omitting the size parameter silently truncates your output with no error. Ask me how I know.

(Oh, and one more thing – Despite the name, ArduinoJSON is ESP-IDF compatible. You don't need to convert to a new library!)

Ticker to FreeRTOS Timers

Arduino's Ticker library is a thin wrapper around ESP-IDF's timer system. The replacement is FreeRTOS software timers, which are available in both frameworks:

// Before (Arduino Ticker)
Ticker myTicker;
myTicker.once(30, myCallback);

// After (FreeRTOS)
TimerHandle_t myTimer = xTimerCreate(
    "MyTimer", pdMS_TO_TICKS(30000), pdFALSE, nullptr, myTimerCallback);
xTimerStart(myTimer, 0);

Phase 2: Replace Arduino Libraries

With the built-in Arduino functions gone, the next step is swapping third-party Arduino libraries for alternatives that work with both frameworks.

The biggest one for TiltBridge was the logging library. ArduinoLog depends on Arduino's Print class, which doesn't exist in ESP-IDF. I ended up writing thorlog -- a fork that provides the same Log.notice(), Log.warning(), Log.error() API but can output through either Arduino's Serial or ESP-IDF's printf() but I could have just as easily used ESP-IDF's native ESP_LOGX() functions. If you're in a similar situation, the key insight is that you only need to abstract the output layer, not the entire logging API.

Other library swaps will depend on your project. TiltBridge replaced LCBUrl (an Arduino URL parser) with a handful of C functions, and replaced an Arduino I2C power management library with direct ESP-IDF i2c_master driver calls. The pattern is the same each time: find the Arduino dependency, figure out which ESP-IDF API it wraps, write the replacement, and remove the old library from lib_deps.

Phase 3: Flip the Build System

This is the phase I covered in my earlier post, but I'll summarize the key bits for completeness.

Change platformio.ini:

framework = espidf
platform = espressif32
lib_compat_mode = off

Create src/idf_component.yml to declare your ESP-IDF component dependencies:

dependencies:
  idf:
    version: '>=4.1.0'
  espressif/arduino-esp32: ^3.1    # Keep Arduino for now
  h2zero/esp-nimble-cpp: ^2.3.4
  espressif/mdns: "^1.2"
  joltwallet/littlefs: "^1.14"

Create CMakeLists.txt files, generate sdkconfig files via menuconfig, and set CONFIG_AUTOSTART_ARDUINO=y so that Arduino's setup()/loop() entry point still works.

Critical setting: CONFIG_FREERTOS_HZ=1000. Arduino defaults to a 1ms FreeRTOS tick. ESP-IDF defaults to 100Hz. If you miss this, every pdMS_TO_TICKS() call gets 10x less granularity and your timers behave strangely. Put it in sdkconfig.defaults so it applies everywhere.

With Arduino still available as a component, you can now incrementally replace the remaining Arduino-specific libraries with native ESP-IDF equivalents. I did these one subsystem at a time, testing after each:

BLE: NimBLE-Arduino to esp-nimble-cpp

This was the easiest swap. Both libraries wrap the same NimBLE stack, so the API is nearly identical. Remove NimBLE-Arduino from lib_deps, add h2zero/esp-nimble-cpp to idf_component.yml, and your BLE scan callbacks, advertisement parsing, and device discovery code all work without changes.

MQTT: Arduino MQTT to esp_mqtt

Arduino MQTT libraries depend on WiFiClient for their TCP connection. ESP-IDF's esp_mqtt manages its own sockets, so the dependency vanishes:

esp_mqtt_client_config_t mqtt_cfg = {};
mqtt_cfg.broker.address.hostname = broker_host;
mqtt_cfg.broker.address.port = broker_port;
mqtt_cfg.broker.address.transport = MQTT_TRANSPORT_OVER_TCP;
mqtt_cfg.credentials.client_id = device_id;

esp_mqtt_client_handle_t client = esp_mqtt_client_init(&mqtt_cfg);
esp_mqtt_client_register_event(client, ESP_EVENT_ANY_ID, mqtt_event_handler, this);
esp_mqtt_client_start(client);

The biggest conceptual shift is that ESP-IDF MQTT is event-driven rather than polled. You register a callback that fires on connect, disconnect, and error events. If your MQTT code lives inside a class, pass this as the handler context and cast it back in the static callback -- a pattern that shows up repeatedly in ESP-IDF.

HTTP Client: HTTPClient to esp_http_client

Same pattern as MQTT. The ESP-IDF HTTP client manages its own TCP/TLS connections and uses an event handler for response body capture:

esp_http_client_config_t config = {};
config.url = url;
config.method = HTTP_METHOD_POST;
config.timeout_ms = 6000;
config.event_handler = http_event_handler;
config.crt_bundle_attach = esp_crt_bundle_attach;  // Built-in TLS cert bundle

esp_http_client_handle_t client = esp_http_client_init(&config);
esp_http_client_set_header(client, "Content-Type", "application/json");
esp_http_client_set_post_field(client, payload, strlen(payload));
esp_http_client_perform(client);
esp_http_client_cleanup(client);

One gotcha: esp_http_client doesn't resolve .local mDNS hostnames. If your targets include mDNS names (mine did – TiltBridge can send data to Fermentrack instances identified via mDNS), you need to resolve the hostname first using mdns_query_a() and rewrite the URL with the resolved IP before handing it to the HTTP client.

HTTP Server: ESPAsyncWebServer to esp_http_server

This was the most involved swap (outside of the WiFi Manager). ESPAsyncWebServer is async and callback-based; ESP-IDF's HTTP server is not. The route registration model changes completely:

// ESPAsyncWebServer
server.on("/api/data", HTTP_GET, [](AsyncWebServerRequest *request) {
    request->send(200, "application/json", jsonString);
});

// esp_http_server
static esp_err_t api_data_handler(httpd_req_t *req) {
    httpd_resp_set_type(req, "application/json");
    httpd_resp_send(req, json_string, strlen(json_string));
    return ESP_OK;
}

httpd_uri_t uri = {
    .uri = "/api/data",
    .method = HTTP_GET,
    .handler = api_data_handler,
};
httpd_register_uri_handler(server, &uri);

The biggest thing I had to keep in mind here was that the HTTPD server tracks a limited number of URI handlers which must be configured at setup. This became especially important when setting up the WiFi Manager later.

Filesystem: Arduino LittleFS to VFS + POSIX I/O

Arduino's FILESYSTEM.open() becomes standard fopen(). The only catch is that file paths need the VFS mount-point prefix:

// Before
File file = FILESYSTEM.open("/config.json", "r");

// After
FILE *file = fopen("/littlefs/config.json", "r");

If you're using ArduinoJson, note that deserializeJson() doesn't accept a POSIX FILE* -- you need to read the file into a char* buffer first and pass that instead.

Phase 4: Remove Arduino Entirely

If you did Phases 1-3 thoroughly, this phase is surprisingly small - outside of the swap to the WiFi Manager. The Arduino component at this point is only providing three things: the setup()/loop() entry point, Serial, and WiFi management.

Entry Point

Remove CONFIG_AUTOSTART_ARDUINO=y and provide your own app_main():

extern "C" void app_main(void) {
    setup();

    xTaskCreatePinnedToCore(
        [](void*) { for (;;) { loop(); vTaskDelay(pdMS_TO_TICKS(10)); } },
        "loopTask", 8192, nullptr, 1, nullptr, 1
    );
}

The extern "C" linkage is critical -- ESP-IDF expects a C entry point.

Serial

Arduino's Serial.begin() disappears. ESP-IDF auto-configures UART0 for console output via menuconfig. If you're using thorlog (or a similar logging library), switch the output adapter:

// Was: Log.begin(LOG_LEVEL, &Serial, true);
static EspIdfPrint espIdfAdapter;
Log.begin(LOG_LEVEL, &espIdfAdapter, true);

WiFi

This is the last big piece - and unfortunately, it's a big one. Arduino's WiFi class and WiFiManager get replaced by esp_wifi APIs and an ESP-IDF-native WiFi manager. The setup code gains some initialization boilerplate that Arduino was handling silently:

// NVS (required for WiFi credential storage)
esp_err_t ret = nvs_flash_init();
if (ret == ESP_ERR_NVS_NO_FREE_PAGES || ret == ESP_ERR_NVS_NEW_VERSION_FOUND) {
    ESP_ERROR_CHECK(nvs_flash_erase());
    ret = nvs_flash_init();
}
ESP_ERROR_CHECK(ret);

// TCP/IP stack and event loop
ESP_ERROR_CHECK(esp_netif_init());
ESP_ERROR_CHECK(esp_event_loop_create_default());

For WiFiManager I ended up adopting tuanpmt's esp_wifi_manager (after making a bunch of changes – use the version in my org) to handle captive portal AP mode, credential storage, and auto-reconnection through ESP-IDF's event system. It's not a drop-in replacement for the Arduino WiFiManager, but it covers the same functionality – and adds a bunch of additional features as well. I won't cover the full conversion, but recommend checking out TiltBridge's wifi_setup.cpp for an explanation of how it all works.

Remove the Component

Finally, delete espressif/arduino-esp32 from idf_component.yml, remove any remaining Arduino libraries from lib_deps, and remove all #include <Arduino.h> from your source files. Replace with the specific standard headers each file actually needs: <cstdint>, <cstring>, <cstdio>, etc.

Here's TiltBridge's final idf_component.yml:

dependencies:
  idf:
    version: '>=4.1.0'
  h2zero/esp-nimble-cpp: ^2.3.4
  espressif/mdns: "^1.2"
  joltwallet/littlefs: "^1.14"

No Arduino in sight.

Things That Bit Me

ESP-IDF's compiler is stricter than Arduino's defaults. Even if your code compiled cleanly on Arduino, expect a wave of new errors after switching frameworks. Here are are some of what I encountered:

printf format specifiers. Arduino's GCC is lenient about %d for uint32_t. ESP-IDF treats mismatched format specifiers as errors. You might need <cinttypes> and its PRIx32, PRIu32 macros.

const on return types. const int getCount() is technically ill-formed in C++ but Arduino doesn't flag it. ESP-IDF does. Remove the const.

Unused code. ESP-IDF builds commonly use -Werror=unused-function. Dead code that compiled silently on Arduino will now fail the build. Delete it or #ifdef it.

Struct initialization. ESP-IDF config structs often have more fields than you expect. If you use designated initializers and miss a field that was added in a recent ESP-IDF version, the compiler will tell you about it. I got in the habit of zero-initializing with = {} and then setting fields individually.

mDNS service names need underscores. Arduino's MDNS.addService("http", "tcp", 80) becomes mdns_service_add(NULL, "_http", "_tcp", 80, NULL, 0). Note the underscore prefix. This one is easy to miss and produces no error -- your service just silently doesn't advertise correctly.

Was It Worth It?

Yep.

The binary is smaller, the build is faster, there are no more mysterious header conflicts, and I have direct access to every ESP-IDF feature without going through an abstraction layer that may or may not expose what I need. The WiFi stack is more stable. The BLE stack is more stable. And when something does go wrong, I'm debugging one framework instead of two. The conversion took several weeks of focused work, but in the end I'm not worrying about waking up one morning and having framework support completely break.

If you're maintaining an ESP32 Arduino project and have been eyeing ESP-IDF, the incremental approach is the way to go. Don't try to convert everything at once. Replace the Arduino built-ins first, one file at a time, while keeping the project buildable on Arduino the entire time. By the time you flip the framework switch, you'll find there's not much left to change.

The full TiltBridge source is available on GitHub if you want to see the end result.

Final Thoughts & Detailed Notes

If you're planning to convert a project yourself, I've attached detailed notes below - with a specific emphasis on the HTTP Server and WiFi Manager conversions. These can also be handed to AI agents (e.g. Claude Code) which can do a significant amount of the work for you.