AssetOTA Struggles to Flash Locally

Firmware Libraries
1

Hello,

I am experimenting with AssetOTA with a b402 using device os 5.5.0. I have a single asset sized about 700 KBs and I usually cant flash it locally. Its really slow (taking minutes) and dies more often than not. Oddly enough, flashing over the air works every-time.

image
image2778×336 39.7 KB

2

hey @radek, what was the command you used for flashing?
Have you tried via workbench?

3

This is with workbench

4

It tends to be more successful if I load in a blank firmware first

5

Thanks Radek!
We are struggling to replicate this at our end. We tried to create a few apps with 700 KB assets and didn't see the same issues. It also flashes very quickly so not sure what's going on here.

  1. Would it be possible for you to send us the asset firmware for testing? I can move this post into an internal ticket so you can send us the data there.
  2. Is the blank firmware test always flashes without any issues or are you seeing the same error in that instance as well?
  3. Can you send us the output of particle serial inspect
  4. Also the output of particle binary inspect for the user app.bin file.
6
  1. I will send it privately.
  2. I have not been able to cause the blank firmware to fail. I have also added a wait loop at the beginning if Serial is not connected, so my application code does not run if I disconnect serial. This also does not fail.
  3. The command fails to run: Could not get inspect device: Unexpected number in JSON at position 1. I can send you the som id privately. Particle CLI version is 3.17.3

Particle.bin
CRC is ok (7a563429)
Compiled for bsom
This is an application module number 2 at version 6
It is firmware for product id 23 at version 56
It depends on a system module number 1 at version 5501
It depends on assets:
Update_ESP_v001_003_001.bin (hash 1add9e247cb9c36044c9bc709be74d270142c1ea6f136f7860fe00885f00389b)

I have a theory. Seeing how there is no issue using a blank firmware, there must be some interference with my application code. I am not sure whether the application firmware is running when the asset is transferring. The device does get switched to normal mode and the RGB led blinks blue, before the asset is flashed, according to the flashing process. I am not sure what normal mode entails.

If the application code is running, it could have something to do with the PosixQueue lib we are using, since that is accessing the onboard file system at the same time that asset writing is taking place. They may be competing and I am not sure if there is any locking to prevent same time access.

It can also be anything else we do in our code.

I timed the flashing process with my real firmware in the wait loop for serial and it flashed in 2:00 with the asset. After connecting serial, I try flashing, it fails at 86% after 4:18.

I repeated this experiment using a blank firmware + the same asset, flashing went from 1:22 to 1:52 with serial open.

I guess having serial connected will also slow down the flashing process, since now the flashing computer may also have to compete for the serial connection to write the asset. That being said, the serial connection itself is not the sole culprit here, since I can successfully flash a large asset with it open as long as the firmware isn't doing anything. This makes me think that application code is in some way running during asset flashing.

There could also be a hard limit of ~4:20 for flashing in the cli.

7

Are you using SYSTEM_THREAD(ENABLED)? This is recommended.

Is there any condition that can prevent setup or loop from returning? You mentioned blocking until serial is connected. What is in that loop, just a waitFor?

8

System thread is enabled.

It is possible that my code would try to System.reset() if there is an error with an IC, for instance, if the SD card is not slotted in. But I am confident that is not occurring.

The loop is:
while(!Serial.isConnected())
delay(100);

AssetOTA is only successful if I am in this loop, so the rest of my code doesn't run. It might be worth trying to comment out various sections and see if I can pinpoint the issue.

Somethings that go on in my code:
SPI/SPI1/Serial1/I2C communication
Serial Logging
Multiple threads
Posix queue
Particle publishing

9

I'd just try commenting out parts of the code to see where the problem likely is. There should be no problem with any of the things in your code in general, but there could be something specific to exactly how it's implemented that is causing issues.

10

I found the issue. I went thru and commented every section of code and commented them in piece by piece as they flashed successfully. That left me with something I didn't expect to cause an issue.

We were using a delay(5000) for debugging in our loop function. Commenting this out completely fixed the issue.

It's an easy fix, but it does seem a little strange that this would impact flashing the asset.

11

Normally, the loop function executes 1000 times per second, sometimes much more. There are some system thing that run in the application thread in between calls to loop. Most things are in the system thread, but certain things, especially things that have callbacks into user code, usually run in the application thread for thread safety reasons.

The thing about delay() is that it only allows system processing in the application thread to occur once per second, which would explain why things run so slowly.

2 Likes