I'm experiencing significant performance degradation with builds created recently compared to builds from mid-April. The LCD rendering is noticeably slower, and the binary size has increased from 125KB to 127KB despite no code changes. This affects a Particle P1 device running Device OS 2.0.1.
Environment Details
Device: Particle P1
Device OS: 2.0.1
Particle CLI: Recently updated to 3.35.7 (problem persisted after downgrading)
Particle Workbench: Updated to 1.16.37 on April 16, 2025 (problem persisted after downgrading)
Build system: Using VS Code Particle Workbench plugin
Build scripts: Multiple versions in environment (1.10.0, 1.10.1, 1.15.0)
What changes in the build system would simultaneously cause performance degradation and increased binary size with identical source code?
Have there been changes to default compiler settings (optimisation flags, debug info, etc.) in recent build tools that would affect performance?
Could recent updates to the build system be implicitly enabling debug mode or reducing optimisation levels?
Are there known issues with the Particle Workbench 1.16.37 update from April 16 affecting build performance?
Additional Information
I'm willing to provide any logs, binary comparisons, or other information that might help diagnose this issue. The performance degradation is consistent across multiple builds with no code changes.
I presume you are using third-party libraries. How are you including them?
If you are using local build, presumably you have included them in your lib directory. Did you also remove the library from project.properties? If you did not, then cloud builds will use the version in the cloud, not your local version.
Cloud builds are not changed on new releases. A build pack is built when the version of Device OS is released, and is never touched again. It's a Docker image with that version of Device OS and all of the compiler and other dependencies. Those components are never upgraded. Are you able to reproduce the problem with a CLI build? That just calls the cloud build directly via the Particle CLI.
The Workbench compiler uses the same tool set as cloud builds, but because it's not running in a container it can't be guaranteed to be exactly the same. However, when new versions of Device OS are released and a new toolset (such as compiler version) is upgraded, it will only affect that release and future versions. Building for older versions of Device OS still uses the older toolset.
The build system should never cause a change like that unless you change Device OS versions.
Compiler flags are never updated for previous versions, and Device OS for Gen 2 hasn't been touched since 3.3.1, November 10, 2022. The flags are stored in the Device OS source that you are targeting, not in Workbench.
Cloud build flags never change for previous builds because they are stored in the Device OS source, which is frozen in the build pack Docker image.
Thanks for your insights. I've further investigated based on your suggestions:
Third-party libraries: Yes, we use 7 third-party libraries, they are included in the lib directory (and added to git). I found that there was an outdated project.properties file referencing 2 of these libraries (with the same version numbers). I removed these references, but the performance issues and increased binary size persist.
Cloud vs. Local builds: I've tried cloud compile via Workbench, and they show identical results to local builds. I attempted cloud compile via CLI, but encountered errors finding library files.
Multiple projects affected: This issue is affecting another separate Particle product too, where the binary size increased to 130KB, making it too large for the device. This will require extensive refactoring, but we don't understand how the size increased without code or library changes.
Code reduction test: In an attempt to identify the cause, I stripped out 80% of the code and all libraries except the LCD library. The new binary size is 50KB, but the LCD performance is still slow compared to pre-April builds.
What's particularly puzzling is that this occurred around April 16 across multiple projects despite your confirmation that build environments shouldn't change for existing Device OS versions. We have more than 80 firmware revisions dating back to 2018 where this issue was not present.
Do you have any suggestions for diagnostic tools we could use to compare the binaries at a deeper level to identify what sections have grown?
At this point, we're facing both performance degradation and size issues that are impacting our products without any clear explanation. Any additional guidance would be greatly appreciated.
It's not at all clear what could be causing that behavior, especially with cloud build because the build environment never changes after the target build pack is built.
However, since you have a smaller app that just contains the display code that is experiencing the slowness, I would start by examining that. Since you have the libraries downloaded and in source control there shouldn't be any changes to the library, but I would take a look at the code anyway. If the display is SPI, take a look at the code path to make sure it's running the DMA code path, which may be controlled by a #define. Also make sure that SPI is running at the correct speed, which wouldn't affect the code size, but worth checking anyway. If you have multiple SPI devices, make sure all of them have SPI transactions enabled.
Finally, take a close look at anything that uses a global allocated object, especially one that does anything non-trivial in the constructor. The order that global constructors are called is not defined by the compiler, and seemingly innocuous changes can cause the order to change, which can have unexpected side effects sometimes.
@rickkas7: Thanks for the suggestions. I've investigated several angles:
Interface Configuration: The LCD uses an 8-bit parallel interface (DB0-DB7 pins) with control signals (WR, RD, CS, RS). The library uses direct pin manipulation (pinSetFast/pinResetFast) for performance, which could be sensitive to compiler optimizations.
Minimal Test Case: I've created a minimal test case that demonstrates the issue with just the LCD library and basic drawing functions: GitHub - markoImake/particle-lcd-test
@marko, have you tried DeviceOS 3.3.1? I find it very reliable and what I use with my RGBMatrixLED library (which uses fast I/O and fast hardware timer interrupts). Also, any reason you are not setting SYSTEM_THREAD(ENABLED)?