Rogue firmware app

Troubleshooting
1

I don’t even know where to begin with this one.

I have application firmware that runs fine on tens of devices. I tried setting up a new device today. I compiled and flashed my firmware from Particle Dev

My firmware runs in SEMI_AUTOMATIC and does a bit of setup before calling WiFi.connect. One of the setup tasks is flashing D7 so I know my firmware is in control of the Photon.

All is good and working as expected until one second after going to breathing cyan. The RGB LED flashes red as if more firmware is being loaded. And it is. When I go into console, my Photon is running app firmware in Spanish. Four functions and four variables are shown in Spanish.

I have never written any firmware in Spanish.

Besides crazy, any ideas where to go from here?

I wrote a tiny program that just flashes D7 and it works, but the second it finishes flashing D7, the rogue firmware is again flashed OTA.

// =============================================================
void setup() {

  // Open debugging
  Serial.begin(115200);
  blink(D7,50);
}

// =============================================================
void loop() {

  
}

// =============================================================
void blink(int pin, int times) {
    //WF_log("blink()",0);
    pinMode(pin, OUTPUT);

    for (int i=0;i<times;i++){
      delay(50);
      digitalWrite(pin,HIGH);
      delay(50);
      digitalWrite(pin,LOW);
    }
}
2

Is it red or magenta (red+blue)?
Also what Device OS version is on the device? If your Device OS isn't of same or greater version than your application target, then the device will try to update the Device OS to fit the need.

BTW, where did you get that device from?
Could this have been a refurbished device which has been part of a product in the past?
Your symptoms look very much like an automatic product firmware reflash.

You can file a support ticket at support.particle.io where you also attach the Device ID and a link to this thread.
If this device is/was supposed to be part of a product you may also be asked to provide proof that you are the legitimate owner of this device for development and not using a 3rd party product device as your own.

For the meantime you could put your device in Listening Mode and run particle serial inspect and post the output to have some more info about your device's current state - particularly after the "rogue" firmware has been reflashed.

1 Like
3

It almost sounds like someone has added your device ID to their product or something? Do you have a product with your devices added to it already?

1 Like
4

@ScruffR Thanks for the help!

Yes, magenta, sorry.

Here is particle serial inspect:

Platform: 6 - Photon
Modules
  Bootloader module #0 - version 101, main location, 16384 bytes max size
    Integrity: PASS
    Address Range: PASS
    Platform: PASS
    Dependencies: PASS
  System module #1 - version 207, main location, 262144 bytes max size
    Integrity: PASS
    Address Range: PASS
    Platform: PASS
    Dependencies: PASS
  System module #2 - version 207, main location, 262144 bytes max size
    Integrity: PASS
    Address Range: PASS
    Platform: PASS
    Dependencies: PASS
      System module #1 - version 207
      Bootloader module #0 - version 101
  User module #1 - version 2, main location, 131072 bytes max size
    UUID: 0102030405060708090A0B0C0D0E0F101112131415161718191A1B1C1D1E1F20
    Integrity: PASS
    Address Range: PASS
    Platform: PASS
    Dependencies: PASS
      System module #2 - version 1
  empty - factory location, 131072 bytes max size

…I have no idea what it all means.

All of our Photons have been directly purchased from Particle in batches of 100 except for a small batch of cores and first order of 20 Photons.

We froze our firmware years ago at 0.5.1 and all has been working fine. Just started to set up a batch of new devices and nothing is going right. I’m flashing both Particle 0.5.1 and Our Application using particle flash --usb and DFU mode which worked flawlessly last time we made a batch.

@HEng We are not using products. Can one add just any random device to their product? Doesn’t the product ID need to be baked into the Application?

5

@steelydev, maybe you could PM @rickkas7 your Device ID so that he could have a look at the logs what’s going on with the flashing.
Maybe some other product creator has inadvertedly added your device to his list of devices before you were able to claim it (if that’s even possible).

But @rickkas7 should be able to tell a bit more.

6

The symptoms sound like a product creator accidentally added your device ID to their product before you claimed it. If this happens, your device will be flashed with their firmware immediately upon connecting to the cloud for the first time.

Just create a support ticket with the output from the particle identify command, the email address you claimed the device to, and information about which order it came from, if possible.

1 Like
7

Thanks. I’ll create ticket.

Seems there should be some safeguard against this, even though it must be a rarity.

8

I fixed it for you. It’s kind of a hard problem to solve in general, because product creators need to be able to mark the devices that should be part of the product so the opposite thing, people making their devices into a product device, can’t occur. The situation you encountered is rare.

2 Likes
9

The product manufacturer should have physical possession of the Particle device if they are to add it to their fleet.

Maybe

particle product add --usb <product>

with the device in listen mode. The CLI will be logged in, the product ID will be verified and the device ID will come directly from the Particle device.

10

That's difficult to argue with product creators that have their devices manufactured and sold by 3rd party services - manufacturer and "creator" don't need to be the same entity.
The "creator" may never actually see their product from production to sale (think tens of thousands of devices). They only see the list of device IDs but never actually need to have the physical unit.

11

Does this mean that an attacker could set up a script to place device IDs into their product list, which would never be noticed until the attacker decided to reflash all units in their product grouping?

I know this requires a malicious actor with ill intent, but it seems like a powerful backdoor in the product if I can commandeer whole fleets of units in the wild.

12

This part is the key

13

To manufacture anything with a Particle, that Particle MUST be plugged in during manufacture at some point.

There HAS to be testing or flashing or something bordering on QA – putting it on a bed of nails to see if it is alive before shipping.

That is the point where it must be added to a product.

14

That can be done with a mere test firmware that hasn't got anything to do with the final product and does not require claiming. Testing can be done fully offline but claiming would add a lot of extra time to the testing process.

Sure you can propose such change in the design of the whole scheme but there is no reason that would make it an inescapable need, hence I'd not put my money on it.

15

QA is generally a good thing to have, but that doesn’t mandate you add the device to your product there and then, nor does it have to be flashed with your final version of code.
(Correct me if I’m wrong, which I could very well be)
To my understanding you can add a list of device IDs to your products in batch, like the ones you got from wholesale (or individual for testing). Then, when the customer gets it online, it will register itself as part of the product, seeing as the ID is in there, after which the current firmware version will be flashed automatically, along with the necessary system firmware.

16

I think you over-estimate the amount of testing/test engineering that people are often willing to pay for.

And also the difference in cost/engineering effort required to do a simple plug-in test vs a full interactive activation process on the line.

1 Like
17

@rickkas7 One potential solution to this would be to hash(DeviceID+salt) and require this number to register it. This essentially just increases the DeviceID space and randomizes it. So 6 of one, half a dozen, etc.

You would ship the device with the DeviceID & hash in the flash memory, so you can register it when you have physical access. You would also want to ship that combination with the Device ID list when you ship a reel.

But it probably doesn’t come up often enough to matter.

18

Once it’s on the bed the entire process can be automated.

19

That costs money!

20

So are you saying people are manufacturing products with zero testing not even plugging them in to see if they work? Because all it would take is plugging in to read the device ID.