I have 8 photons installed in a remote location. The photons are running with system thread enabled, and have been running for two weeks. All are claimed and were connected to the cloud.
They have identical code that pings a remote server and reports data. There is an application watchdog running which resets the photon if the application code has trouble.
After two weeks, two of the photons were flashing cyan rapidly with an intermittent red flash, and not connected to the cloud. The other six are working smoothly and are breathing cyan. All are on the same WiFi network. No app updates were being sent to the devices on the day they went offline.
I had one of the faulty devices sent to me.
What could be causing this to happen, and are there any changes I can make in code to prevent this from recurring?
My main concern isn’t fixing the faulty devices, but preventing this fault from developing in other photons that are or will be installed in the future , especially since I need them running smoothly in a far away location for weeks.
That made the photon connect to the cloud.
It now shows a hard fault SOS.
I can put it in safe mode and flash it, but I'd like to know what has caused this so that it doesn't recur in any of the other devices.
This particular device was working for days without a hitch, and the other photons are still running. No Red flash SOS were seen in this time.
Any ideas on what could have caused the Public Key to go bad and cause the initial cyan-red flash?
And what I could do to prevent it.
Thanks a lot!
I'm using a HTTP Client like the one in this thread,
Not sure what was the cause in your case, but since we see this happening more frequently in the last few days it might be in connenction with things like the Raspberry Pi Beta tests and the connected "donation" of extra resources and/or the release of system version 0.6.0 and/or maintenance and/or server issue and any combination thereof
Also these reasons
There also was some problem with load balancing yesterday, which should have been cured already.
We just have 2 of our photons with the same problem within the window of 2 weeks. That is the first time we see this problem in almost a couple of years using photon. We have a couple of hundred devices with clients and about a thousand waiting on some dealers shelves.
@ScruffR, do you think that something may have changed with system version 0.6.0?
We are quite sure that it wasn’t a key expiration. We rename the hotspot’s SSID to our company name and after the problem they returned to show “Photon-XXX” for the SSID name.
Is there any way to remotely fix the problem if/when it happens again? I’m pretty sure there is none…
I had just finished soldering a custom circuit board last night, and I went to test it with a 5V bench power supply to Vin. However, my Photon blinked as if it had lost my WiFi credentials. I used the Particle app to set the Photon back up, but the app says that the Photon connected to my WiFi but failed to connect to the cloud. Now, the Photon is blinking Cyan rapidly with intermittent Red/Green flashes. It might be important to note that my Photon was working just fine 4 days ago.
I have been reading other discussions about brownouts corrupting the WiFi credentials and Public Keys, but mostly in battery applications. Is it possible that turning off my power supply resulted in a brownout instead of power off?
I find it amazing that @bacichetti has the same problem as me within such a close time period, after his Photons working fine for years as mine has worked fine for some time.
Has anyone been able to resolve this issue? I appreciate any help.
Thanks, I was able to repair my Photon for now with your instructions and those above.
However, what I meant in my question was more like "how can I prevent this from happening in a product that I may sell one day?" And like @bacichetti asked,
Also, will this problem also happen if someone unplugs a wall power supply that is powering my project? I would test it, but I want to ask before possibly breaking my Photon again.
The circumstances why and how this happens are not yet fully understood. As soon they are, the issue will be fixed.
But a sudden power loss - like unplugging - has never harmed any of my devices.
It’s more of a slow disappearance or noisy power to cause issues like the “dim blue LED” issue or the less severe out of sync keys.
The out of sync keys might also happen when your device doesn’t check in on the cloud for a very long time (e.g. months).
Till then there is no reliable way to do this remotely.
Since your device can’t make it to the cloud and you’d need a USB DFU connection, there is little to do from the distance.
But this is a problem that needs to be tackled by Particle (@will?)
@JDvorak, I did more tests using both system 0.6.0 and 0.5.3 and i was able the break the keys twice, both time happened on the photon running system 0.5.3. That's kind of a relieve, because it's unlikely all me clients will start calling me. Looks more a case of statistics, despite the fact that we have photons running for 2 years, they are just a few dozens. Most of our clients have their systems running for a few months.
The problem, and that's my concern, is that there is about nothing I can do but wait until Particle came with a better protection to their flash AND/OR implement a way to restore the keys in the case they get erased. Both times the keys ware erased I was playing really hard with the VCC input voltage. It never fail during connection/disconnection of power.
I can confirm what @ScruffR said about why/how the keys get erased,
@bacichetti, it would be great to understand what you mean by “playing really hard with the Vcc” so others can try to replicate. Having repeatable failures would be great for debugging.
@peekay123, I connected a power supply to Photon’s Vcc pin and manually changed the output voltage between 1-5V volts for several minutes. I would be better to use a signal generator with a noise output, but I don’t have one at this moment, specially one capable of supplying 200 mA.
I forget to comment that at least one of my clients recall having a brownout on his house the weekend when this photon stop working.
@bacichetti, were you changing the voltage quickly or slowly (approx rate of change between 1v and 5v). Did you do this while the Photon was running an app? How did you observe the failure?
@peekay123, quite fast for manual oscillation, 1-5 Hz I would say, leaving some time to Cloud connection every a couple of cycles. Yes, photon running my full application. It’s very easy to see when it fails, it will start flashing red/cyan.
Thanks for the info! If anyone tries this again and can video the supply and RGB led that would be helpful. I’ve personally done some automated testing for 1000’s of cycles and have not seen this issue repeat itself, but I know it occurs.
@bacichetti when varying the supply, be careful not to exceed 5.5V which is the max input on VIN. It is the VIN pin you are using, correct?
@BDub, yes, I was using VIN, your right! I actually damage one photon later on because the 5.5V limit, that’s the problem not automating the test.
There is something I notice during the tests, the operation threshold using system 0.6.0 (2.4-2.5V or lower) is lower than with the system 0.5.3 (2.6-2.7V). I swapped boards and firmware and the result is consistent. Could it be a source cause of people seen more incidence of the problem with the new system version? My tests on suggest the problem can happen with older systems, but it say nothing about how frequent it actually will happen in real use cases.
I’ll try a few more devices over the next days and give you some feedback.
That's interesting, I don't believe there should be any reason for that to be true though. It sounds like a BOR (Brown Out Reset) setting, but that would be the same for both firmware versions. I'll have to try this as well with a lab supply.
, especially since I need them running smoothly in a far away location for weeks.