I work for Red Hat and I have been noticing all the desks around me filling up with AI new-hires. I enjoyed hearing about how AI development has moved from "prompts" to "loops" and how agents now drive the software development cycle. I kept saying "that is cool but firmware development for an IoT device is different". When I thought about the difference, it was that the AI agents who were orchestrating the development could not observe the IoT device in the same way it could build tests for pure software development.
Then, I thought about what it would look like to build a closed loop system to develop firmware for my Particle devices. I came up with an approach that I am fairly happy with. I thought I would share it here in the hope that: 1) Someone could tell me how I could make this better and 2) Someone might find this useful.
I don't want put too much detail into this post before I know if anyone would find it useful. So, here is a high level view:
Chip and the architect decide what to build and write a small, budgeted work order into the repo.
Claude Code dispatches the work: Copilot implements, Codex investigates and verifies, and Claude Code builds, tests and opens the PR.
Chip approves, merges and releases firmware to the devices under test.
The sensor exerciser drives the devices with known inputs, so we know exactly what values to expect.
Three data feeds flow into AWS: serial logs from the Raspberry Pi forwarder, Particle logs from the Particle API, and Particle Ledger values, also from the Particle API. Each lands in DynamoDB for fast queries and in S3 as the full source of truth.
Actual results are compared with expected values, and the findings feed the next work order.
Why the loop matters
Every question about firmware performance now has a source of truth. The exerciser tells us what the right answer should be, and the data in AWS shows what the firmware actually read and did. Planning decisions rest on that evidence, not on guesses.
I would be happy to share more: the exerciser, the serial log forwarder and the design of the AWS back-end but I wanted to see if there is interest first. It could be that I have just solved my own problem, I don't want to assume others might be looking at the same problem as well.
The test bench part in the gray box is basically how we test Device OS. There are a number of controllers around the world. Each controller can flash firmware and monitor USB CDC serial output for a number of devices. The orchestration and data storage is different, but it's how new feature and regression testing is done across the whole product line.
@gusgonnet - Thank you for sharing. Watching this gave me a couple of ideas to improve my setup.
Not sure if it is possible but using Github actions to compile seems like a great area to look into.
If that works, then I could have my Raspberry Pi flash the devices on my serial log forwarder via USB. This would give me another route to get firmware onto the devices under test beside the current cloud flash.
if you have a Tachyon around, that could do it too.
even an old Mac - or a second hand Mac that you can buy cheap if the screen broke. You ssh into it and you are flying.
What do you use as a test harness for driving the integration tests on the whole setup?
In the past, I did great things with Gauge from Thoughtworks but on the repo we read now "Gauge's official sponsorship ended in 2021". Sad. You could write test cases in markdown/plain english while the steps could be implemented in a number of languages (python, javascript, ...)
Cucumber looks similar and interesting. It uses a tighter way of writing tests but in the same way as Gauge you can write the steps in the language you want (or almost).
If python were your cup of tea, you could use pytest.
Of course, and luckily, there are trillions of options here, even hand rolled ones.
@rickkas7 - Thank you for the pointer to the GitHub Actions docs. That turned out to be exactly what I needed. We have now moved to a process where Claude Code commits to GitHub, which kicks off a CI build. The compiled firmware is uploaded to the Particle product, flashed to the test devices and, only after testing passes, released to the fleet. The big win is provenance: every binary on a device traces back to one commit and one pipeline run, and the binary that passes on the bench is the same one that goes to the fleet. It was also reassuring to hear that the test bench looks a lot like how you test Device OS.
@gusgonnet - Thank you for the video and the ideas. The Tachyon and the old Mac are both good thoughts for the flashing side. For now, I am using the Particle OTA firmware capability as my devices are typically sleeping so USB flashing did not work. I could connect Pi GPIO pins to the Boron's reset and mode buttons but will wait to see if this is needed.
To your question about the test harness: honestly, we are building a custom test suite rather than using an off-the-shelf framework. The key piece is the sensor exerciser. It acts as the "known good and known constant" input, so for any test we know exactly what the device should report. The devices' actual output comes back through three feeds (serial logs from the Pi, Particle logs and Ledger values from the Particle API) into AWS. A test passes when what the firmware actually read and did matches what the exerciser says it should have done.
So the exerciser is the oracle, and AWS is the record. What we haven't done yet is formalize the test cases themselves, and that is where your suggestions are interesting. Writing scenarios in plain language (Gauge or Cucumber style) and having pytest drive the exerciser and check the AWS results could make the suite easier to read, extend and hand to the AI agents. I'll take a look.
I've attached an updated diagram that shows the new build pipeline.
