SOLVED - Blinking Cyan Bursting Red Flicker

Troubleshooting
1

Hey folks, I just wanted to let you know that I had an issue that took up a lot of time this weekend. I knew it had to do with keys issues but was really struggling to get back online.

ISSUE
Device was working for several days and then all of a sudden, stopped connecting to the cloud. The device LED was flashing Cyan with bursts of red. This happened seemingly out of nowhere.

Troubleshooting Steps
I had gone to this post realizing that it was somewhat different but assuming it was a similar issue. These steps were overly complicated and did not work. I believe these steps were relevant with an earlier Particle SDK/version but now things seem to have been greatly simplified. THANKS PARTICLE.

RESOLUTION
Contacted support (Awesome btw, thanks Particle) - Resolution was to connect device to PC (already have node.js installed and particle cli), put the device in DFU mode and run the following commands.

particle keys server
particle keys doctor <device_ID>

All fixed up.

3 Likes
2

Are there any reasons why an electron lost its keys? In my case, my electrons runs for serval days, some electrons for weeks. One day, they suddenly lost their keys and I have to run the commands

particle keys server
particle keys doctor <device_ID>
particle flash --serial firmware.bin

In the development stage, this is no problem but in production mode not. Have anyone experience to avoid the lost of keys?

  • Firmware: 0.6.1
  • Electron 2G with Particle SIM
  • Full LiPo battery
  • Device Mode: Semi automatic
  • No Strings, only char[]
  • Reset button dosn’t help

Basis code:

SYSTEM_MODE(SEMI_AUTOMATIC);
STARTUP(Cellular.off());

void setup() {
    delay(3000);
    Cellular.setListenTimeout(30);
}

void loop(void) {
    Cellular.on();
    Cellular.connect();
    waitUntil(Cellular.ready);
    Particle.connect();
    waitUntil(Particle.connected);
    Particle.process();
    Particle.publish("event", pubData, PRIVATE);
    Particle.publish("event2", pubData2, PRIVATE);
    Particle.process();
    delay(10000); 
    delay(1000);
    System.sleep(SLEEP_MODE_DEEP, timeToSleep - (millis() / 1000));
}
3

I had the same thing happen Monday night. I was making changes and flashing. Then after a flashing it can not connect to the Cloud and I get the burst of Red sequence. It did take me three days to find out that was the problem and be given those commands to fix it.

4

Solving the problem is not the problem. The problem is, that this happens again and again in undefined intervals. The strange is that the key error doesn’t happen on each Electron. I have many Electrons running the same application code. At some devices the key error appears after hours/days/weeks, at some never until now. Sadly, after fixing the issue with the key doctor, the issue appears after some time again (sometimes).

I would be very happy to know what is the reason of the lost of keys and what can we do to avoid the issue. Maybe someone who has never had this issue and have the same basis code could post his code. So we can compare it with our code. But in general I don’t believe in a code issue because on some electrons with the same settings, the key issue never appears.

Update:
My current tests result that I don’t need to reflash the firmware after running the two other commands. The next insight is, that only running the keys doctor don’t fix the issue. I have to run particle keys server after particle keys doctor <device_id>.
Next steps are: Test whether only running the keys server command fix the issue or do I have to run both - and in which order. Currently I run doctor first and then server.

5

Update2:
Running only particle keys server (without particle login) seems not to solve the issue. The output of keys server is:

Found DFU device 2b04:d00a
spawning dfu-util -d 2b04:d00a -a 1 -i 0 -s 3298 -D C:\Users\Niklas\AppData\Roaming\npm\node_modules\particle-cli\keys\ec.pub.der
dfu-util 0.8

Copyright 2005-2009 Weston Schmidt, Harald Welte and OpenMoko Inc.
Copyright 2010-2014 Tormod Volden and Stefan Schmidt
This program is Free Software and has ABSOLUTELY NO WARRANTY
Please report bugs to dfu-util@lists.gnumonks.org

Opening DFU capable USB device...
ID 2b04:d00a
Run-time device DFU version 011a
Claiming USB DFU Interface...
Setting Alternate Setting #1 ...
Determining device status: state = dfuIDLE, status = 0
dfuIDLE, continuing
DFU mode device DFU version 011a
Device returned transfer size 4096
DfuSe interface name: "DCT Flash   "
Downloading to address = 0x00000ce2, size = 320
Download        [=========================] 100%          320 bytes
Download done.
File downloaded successfully
Invalid DFU suffix signature
A valid DFU suffix will be required in a future dfu-util release!!!
Okay!  New keys in place, your device will not restart.

After a reset (exit the DFU mode) the device starts green flashing and the the cyan blinking (slow and fast) with the red flickers.
Then I put the electron in DFU mode again and run the keys doctor with the following output:

Found DFU device 2b04:d00a
Found DFU device 2b04:d00a
New Key Created!
Found DFU device 2b04:d00a
This file already exists, please specify a different file, or use the --force flag.
Continuing...
spawning dfu-util -d 2b04:d00a -a 1 -i 0 -s 3106:leave -D <device_id>_ec_new.der
dfu-util 0.8

Copyright 2005-2009 Weston Schmidt, Harald Welte and OpenMoko Inc.
Copyright 2010-2014 Tormod Volden and Stefan Schmidt
This program is Free Software and has ABSOLUTELY NO WARRANTY
Please report bugs to dfu-util@lists.gnumonks.org

Opening DFU capable USB device...
ID 2b04:d00a
Run-time device DFU version 011a
Claiming USB DFU Interface...
Setting Alternate Setting #1 ...
Determining device status: state = dfuIDLE, status = 0
dfuIDLE, continuing
DFU mode device DFU version 011a
Device returned transfer size 4096
DfuSe interface name: "DCT Flash   "
Downloading to address = 0x00000c22, size = 121
Download        [=========================] 100%          121 bytes
Download done.
File downloaded successfully
Invalid DFU suffix signature
A valid DFU suffix will be required in a future dfu-util release!!!
Saved!
attempting to add a new public key for device <device_id>
submitting public key succeeded!
Okay!  New keys in place, your device should restart.

This also doesn’t help. So to conclude I did this order (all with permanent power supply over LiPo and USB):

  1. particle keys server
  2. reset button pressed
  3. let it try to connect -> cyan/red fail
  4. particle keys doctor XXX
  5. restart automatically -> cyan/red fail
  6. particle keys doctor XXX
  7. restart automatically -> cyan/red fail
  8. particle keys server
  9. Still stands in DFU after keys server
  10. particle keys doctor XXX
  11. restart automatically -> cyan/red fail -> WTF, until now steps 8.-11. helps against the issue
  12. remove all power supply (LiPo and USB) and wait for one minute
  13. repeat 8.-11. -> cyan/red fail

I’m not sure why my solution path doesn’t work anymore.

6

Are you sure you are using the device ID of this very device and not an ID recently collected from another device of yours?
Just to avoid confusion, you should always first acquire the device ID via particle identify and use that.

You may also check this message

If that file already exists but has some now invalid key stored uploading the invalid keys will not cure anything.
You may want to search and delete the respective file in C:\Users\Niklas\AppData\Roaming\npm\node_modules\particle-cli\keys\

BTW, these messages look puzzling

Unless you have redacted these messages to obscure your actual device ID these <device_id> entries should actually show a valid 24 HEX digit number :wink:

1 Like
7

@ScruffR: thanks for your response.

The device id is correct. The <device_id> also shows this id as 24 HEX number :blush:

Next strange is, that I have let my Electron alone for a few hours and now it doesn’t show the cyan/red issue anymore. I read at the community, that sometimes the connection works with wrong keys (wrong public key). So I believe it’s just a coincidence that I works and the issue is not solved.

In […]\node_modules\particle-cli\keys are this two files:

  • ec.pub.der
  • rsa.pub.der

There is no <device_id>_ec_new.der file like the line spawning dfu-util -d 2b04:d00a -a 1 -i 0 -s 3106:leave -D <device_id>_ec_new.der says. I found these key files in another directory (the location in cmd where I run the command :wink: ) Is this a problem? If yes, how to configure the cli to store the key files at the right directory?

I deleted these two files in \node_modules\particle-cli\keys and run keys doctor. This works. Running keys server command fails with the line:

Found DFU device 2b04:d00a
Make sure your device is in DFU mode (blinking yellow), and is connected to your computer
Error - Error: ENOENT: no such file or directory, stat 'C:\Users\Niklas\AppData\Roaming\npm\node_modules\particle-cli\keys\ec.pub.der'

So these two files are part of the server public key? I think so. Could they be deprecated? How to update them?

So I restore the backup of the public server keys back to its location. particle keys server works again.

After cleaning up the wrong key directory (where all the key files are stored in past) and running the keys doctor in the particle-cli/keys dir, the cyan/red issue is disapeared and the key files are at the right dir. (“Disapeared” is wrong because the issue was not there before because it goes away alone like described above - but comparing to my previous posts, the cyan/red issue appears after the keys doctor. So it’s a good progress :wink:)

So now it is clear how to get the correct keys on the electron. The problem why it lost it keys after some time is still there. Maybe their is a releation between these two problems. If in the past, the key doctor didn’t transfer any keys to the devices + some random … maybe.

8

Did anyone determine the cause for this and how to prevent it?

While the fix is ‘easy’ if we have the item at home. If I have these items at client sites often in different cities - I’m keen to avoid this!

1 Like