Converting to Arduino as an ESP-IDF Component

This post is intended as a response to my recent frustration finding guidance on how to convert Arduino-framework projects such as BrewPi-ESP and TiltBridge to use Arduino as an ESP-IDF component instead. If you're interested in why this change needed to be made for my projects, keep reading. If you are looking to make the same change in your own project, feel free to jump ahead.

History and Background

To date, I’ve built all of my embedded projects for the ESP32 and ESP8266 using the Arduino framework. Arduino was my introduction to embedded programming, offering cross-microcontroller support and a wealth of libraries that simplified hardware abstraction. Because the Arduino ecosystem is beginner-friendly and widely adopted, it’s ideal for open-source projects—it lowers barriers to entry and encourages community contributions, even from a less-technical user base. Arduino’s official development environment, however, is incredibly limited, even lacking the ability to split a project across multiple files.

To overcome those limitations, I needed to move away from Arduino's IDE while retaining the framework – and ultimately settled on Platformio while developing BrewPi-ESP. Platformio made it much easier to keep frameworks updated, switch between controller architectures, and replicate build environments across different PCs - all within a professional development environment. This allowed me to focus on creating and refining my projects, rather than debugging issues related to environment setup and configuration.

Espressif and Platformio Fall Out

Unfortunately, over the past year, the relationship between Espressif (makers of the ESP32/ESP8266 chips) and Platformio has changed in a way that makes using Arduino with Platformio increasingly difficult. Although not officially acknowledged as such, it appears that while Espressif was preparing their Arduino 3.0 release Platformio was attempting to negotiate for official, financial support. When Espressif changed their documentation to clarify that Platformio support was "third party", Platformio responded by announcing they would not support versions beyond Arduino 2.0, encouraging users to pressure Espressif if they wanted future updates. Absent official support from Espressif for Arduino, Platformio pointed towards Espressif's ESP-IDF framework as where their efforts would be focused going forward.

At first, the community worked to maintain Arduino 3.x support in Platformio, but recently Platformio has begun to refuse to merge these changes making it nearly impossible to officially use the latest Arduino libraries or incorporate recent bugfixes on the ESP32/ESP8266. Although some dedicated forks of Platformio aim to restore Arduino 3.x support, these projects are still in their infancy, and have yet to reach critical mass. Taking this all into consideration, migration to ESP-IDF made sense as a next step.

What is ESP-IDF?

ESP-IDF is Espressif’s official development framework, providing direct access to the low-level capabilities of their chips. ESP-IDF's libraries are officially developed and supported by the chip manufacturer, guaranteeing access to the full functionality of their chips. Espressif's Arduino framework for ESP32/ESP8266 is built on top of ESP-IDF, enabling the use of familiar Arduino functions while still relying on ESP-IDF under the hood. Because of this, it is possible to build the Arduino compatibility as a component within ESP-IDF, where the project is officially using ESP-IDF, but still enabling the use of Arduino-style functions.

Why not stick with Arduino 2.x?

In my case, the main catalyst was a bug in the Arduino 2.x releases that caused the Bluetooth radio to freeze after a period of time. This made certain features—like Bluetooth temperature sensing—unreliable. This bug was fixed in the latest Arduino and ESP-IDF releases, but a fix was not back-ported to Arduino 2.x.

Even had this bug been fixed, however, committing to an outdated version of anything means that if future issues - including major security flaws - are discovered, we will potentially never be able to pull in the required fixes. As I am in the process of finishing the next major release for a number of my projects, now seemed like the ideal time to switch to using ESP-IDF with Arduino as a Component.

Converting a Platformio project to ESP-IDF

Changing to ESP-IDF without Arduino is straightforward. Simply update your platformio.ini to change the framework for your component to espidf :

[common]
framework = espidf
platform = espressif32

Make these changes, and Platformio will download and install the platform packages, and should create the relevant configuration files throughout your project. ESP-IDF includes its own component/library manager, as well as its own platform configuration tool. We will utilize both of these in the next step.

Leveraging Arduino as a Component

To install Arduino as a Component using ESP-IDF's component management tool, create a file at src/idf_component.yml with the following contents:

## IDF Component Manager Manifest File
dependencies:
  ## Required IDF version
  idf:
    version: '>=4.1.0'
  # # Put list of dependencies here
  # # For components maintained by Espressif:
  # component: "~1.0.0"
  # # For 3rd party components:
  # username/component: ">=1.0.0,<2.0.0"
  # username2/component2:
  #   version: "~1.0.0"
  #   # For transient dependencies `public` flag can be set.
  #   # `public` flag doesn't have an effect dependencies of the `main` component.
  #   # All dependencies of `main` are public by default.
  #   public: true
  espressif/arduino-esp32: ^3.1

With this, when you next hit "build" within Platformio, the ESP-IDF component manager will download the Arduino packages and install them to the managed_components directory. You'll probably want to add that directory - and dependencies.lock to your .gitignore file.

For Arduino to work, you will need to edit the default menuconfig options to match what Arduino expects. Create or open the sdkconfig.defaults file in the root of your project, and paste in the following:

CONFIG_AUTOSTART_ARDUINO=y
# CONFIG_WS2812_LED_ENABLE is not set
CONFIG_FREERTOS_HZ=1000
CONFIG_MBEDTLS_PSK_MODES=y
CONFIG_MBEDTLS_KEY_EXCHANGE_PSK=y
CONFIG_ESP_RMAKER_NO_CLAIM=y

After saving this, if any build-specific sdkconfig.xxxx files were created, delete them. They will be recreated incorporating the above defaults.

Before we can hit "compile", however, we need to tell Platformio to incorporate a handful of security certificates that Espressif libraries expect. Open up platformio.ini one more time, and add the following:

board_build.embed_txtfiles =
    managed_components/espressif__esp_insights/server_certs/https_server.crt
    managed_components/espressif__esp_rainmaker/server_certs/rmaker_mqtt_server.crt
    managed_components/espressif__esp_rainmaker/server_certs/rmaker_claim_service_server.crt
    managed_components/espressif__esp_rainmaker/server_certs/rmaker_ota_server.crt

(Note, the formatting here is a bit strange – the filenames need to be indented relative to the board_build.embed_txtfiles directive)

Other Menuconfig Options (Optional)

The above defaults are the minimum for an Arduino build, but will miss many features you may want, including Bluetooth support. The full suite of chipset features can be selectively enabled using ESP-IDF's menuconfig tool. To access this from within Platformio, go to the Platformio menu, and choose [Environment] > Platform > Run Menuconfig

The menuconfig utility will run inside Visual Studio's terminal, enabling you to select options for that specific environment.

Library Support

Although you now have a project with functioning Arduino support, most projects incorporate libraries from the Arduino ecosystem which Platformio can download. By default, Platformio has library compatibility checks which prevent the use of Arduino libraries when the platform is set to espidf. To disable these checks, open platformio.ini once again and add the following line:

lib_compat_mode = off

This will ensure that all libraries - including ones not tagged with either espidf or espressif32 support - will generally still attempt to download and compile.

Libraries that Don't Download

I say attempt as there are libraries that themselves specify a libCompatMode to platformio within their library.json file that cannot then be overridden. If you encounter a library like this, edit the library.json file to delete the following lines (ensuring that you end up with properly-formatted JSON):

  "build": {
    "libCompatMode": "strict"
  }

And you should be done!

With that, you should have a working project that uses ESP-IDF as the framework, but incorporates Arduino as a component. One thing I have noticed is that when running the project, oftentimes it seems to hang at Reading CMake configuration... – this appears to be hiding work that goes on with the Espressif tooling behind the scenes. Just wait, and you should be fine.

Final Thoughts

Unfortunately, with free, open source software you often get what you pay for, and paths to sustainability often require finding sources of revenue. In Platformio's case, unfortunately, it appears that the users are the ones caught in the middle – highlighted both by the stalled Espressif Arduino support as well as Platformio's refusal to merge community-built support for Raspberry Pi's RP2040. Here again, the barrier seems related to whether or not a hardware maker pays Platformio for official support rather than any technical or development-related reasons.

Open source software is rarely free of funding challenges, but the essence of open collaboration is that anyone can contribute and benefit. Platformio’s decision to prevent key framework support — unless microcontroller vendors pay for it — runs counter to this spirit - and is effectively shortsighted. By effectively blocking or refusing community-driven pull requests for newer microcontroller architectures, Platformio risks alienating the very developers who rely on its platform.

Third party IDEs thrive in an environment of transparency and mutual cooperation. When a single gatekeeper can unilaterally decline support for popular hardware—even when the community is willing to do the work—it undermines user confidence. No microcontroller manufacturer wants its users left in limbo because a third-party toolchain has locked down vital updates behind a paywall. Meanwhile, developers themselves—whether hobbyists or professionals—don’t want to be stuck juggling multiple environments just to get a modern toolchain working. In the long run, such barriers discourage people from committing to Platformio and can fracture the broader open source ecosystem.

It’s a shame to see these tensions overshadow what once made Platformio so appealing: cross-platform convenience, an active community, and the potential for frictionless embedded development. For the open source world to remain vibrant, collaboration and openness must take priority—otherwise, everyone loses out.

The Future - Moving away from Platformio?

One final benefit of shifting to ESP-IDF is that it opens up migrating to Espressif's official ESP-IDF Visual Studio solution. Although I would count the user experience of the official add-in as lacking in comparison to Platformio, using the manufacturer's toolchain means that the economic incentives align by default with future support.