Photon behaving very strangely - can't put in into safe mode - Lost Public Key

Hi everyone,
I have 11 photons all running the same code for some weeks now, they are all mounted in the ceiling and not touched.
Today one has lost its wifi credentials and was flashing blue? (my code does not do anything with credentials). How can this occur?

After removing from the ceiling i popped it into listening mode and used the android app to setup the credentials which appeared to have worked. However I am pretty sure the WiFi listening SSID is different from when it was first setup, can this be?

The device continued to not run as expected, for example it wont go into safe mode? It flashes cyan, then very fast red.

So I then tried particle_firmware_manager-v0.6.0-osx.app which fails to complete (it works fine with another photon)

finally I have tried

AGMBP:~ andygraves2$ particle flash --usb tinker
Found DFU device 2b04:d006
checking file /usr/local/lib/node_modules/particle-cli/binaries/photon_tinker.bin
spawning dfu-util -d 2b04:d006 -a 0 -i 0 -s 0x080A0000:leave -D /usr/local/lib/node_modules/particle-cli/binaries/photon_tinker.bin
dfu-util 0.9

Copyright 2005-2009 Weston Schmidt, Harald Welte and OpenMoko Inc.
Copyright 2010-2016 Tormod Volden and Stefan Schmidt
This program is Free Software and has ABSOLUTELY NO WARRANTY
Please report bugs to dfu-util / Tickets

dfu-util: Invalid DFU suffix signature
dfu-util: A valid DFU suffix will be required in a future dfu-util release!!!
Deducing device DFU version from functional descriptor length
Opening DFU capable USB device...
ID 2b04:d006
Run-time device DFU version 011a
Claiming USB DFU Interface...
Setting Alternate Setting #0 ...
Determining device status: state = dfuIDLE, status = 0
dfuIDLE, continuing
DFU mode device DFU version 011a
Device returned transfer size 4096
DfuSe interface name: "Internal Flash "
Downloading to address = 0x080a0000, size = 3952
Download [=========================] 100% 3952 bytes
Download done.
File downloaded successfully

Flash success!

But tinker is not installed?

HELP!

Thanks

Andy

Is this thread anyuse?

:grinning:

1 Like

Hi Julian,
Thanks for the pointer, Public Key Reset solved it.
However does anyone know how such an issue arrises?
Loosing credentials randomly will be a big problem.
Thanks
Andy

1 Like

One possible cause for such situation might be a brownout condition or a too slow power-up on the regualtor, sending the device into a rapid on/off/on cycle which might corrupt some of the flash memory.
How do you power your devices?

1 Like

Hi,
MC7805ABTG supply is 24V

Hi ScruffR,
I posted the PSU diag, do you think this should be OK?
Thanks
Andy

It does look reasonable.
But if the 24V fails in in a undue manner that might still cause trouble.

1 Like

Good morning,
To be honest i did not sleep all night…
Loosing public keys or credentials when with a customer is a disaster :scream:
What can I do to avoid this?
Does anyone have issues like this, or is it unique to me (PSU) etc.
thanks
andy

I have never knowingly had this issue but depending on the local power quality I can see how it would be. Would this be of any use?

Other power shields are available. :smile:

That issue is one of the puzzles Particle wants to understand and solve for a long time, but reproducing it isn't easy and hence it's hard to do something against it.
Although this is mainly about the Electron, I have the feeling that the underlying problem is the same or at least similar

We can ping @BDub

2 Likes

I’m almost positive this (the brown-out) is not what happened. We had a partial cloud degradation around the same time @andyg seemed to be experiencing this issue. It manifested as a cyan->red flash pattern which is what @andyg saw. The device should have eventually recovered within a few minutes.

I’d like to better understand the following issues:

  • The lost WiFi credentials
  • The firmware manager failing to update.

@andyg Do you have any info on what version of system firmware you were running on the device at the time?

Hi there,
All 11 photons under test on 0.6.0
Its important for you to note that i don’t feel the initial issue (flashing blue) has anything to do with the cloud as they were not on the cloud.
11 photons, same code, running for a couple of weeks, on WiFi LAN only (no cloud connection physically or in code) only one of them looses WiFi credentials and flashes blue. This surely sounds like a local corruption/in the device however caused (code/hardware/power)?
thanks
andy

Perhaps @mdma can comment: Any idea how WiFi credentials could be lost or listening mode could suddenly be entered?

Andy, did you try simply resetting the unit that went into listening mode to see if it was put into listening mode (but perhaps still had stored credentials)? If it still had stored credentials it should have tried to reconnect after a reset

Firstly, our prototype system (11 photons) was power cycled a few times.

The photon in question did not recover.

We then removed it from the ceiling found it was flashing blue and tried to bring it back to life, as described at the beginning of this thread.

1 Like

Is there anyone who has lots of photons (or P1s) deployed in the field who could offer some advice on possible loss of credentials?

I think a way to mitigate these two issues would be to build in your WiFi credentials into firmware with WiFi.setCredentials() if WiFi.hasCredentials() is false. Obviously not a great hack, but it should offer some comfort. https://docs.particle.io/reference/firmware/photon/#hascredentials-

The Public Server Key can be replaced in a similar way, but I think it would be best to include this code as an add-on library. If someone would like to tackle that it would be appreciated, otherwise I have it on my list to get to at some point. Please let me know if you beat me to it though :smile:

2 Likes

I’ve also had similar issues to you some bench testing. Unfortunately, it has been long enough ago that my memory of exactly what happened has faded. It did leave me with a “bricked” Photon that I’ve never been able to recover trying to follow all of the recipes I’ve seen in the forums. I seem to remember that it was consistently complaining about key issues that I could never resolve.

Given that I was working on proving out some other things, I put another Photon on the breadboard and proceeded with a mental note thinking all the things you mentioned…This could be a killer for field applications with customers to have modules randomly go offline. I would like to help or see this get fixed before I can build a Photon into my sensor applications.

1 Like

Hi there,
Sorry i went quiet, was away…
Will try this,
thanks
andy

1 Like