Long Term Photon Connection Stability

Hi jasonp,

Never did have a photon lose it wifi setup again. Never was able to figure out the root cause.

chipMonk

I didn’t find this thread until recently, but I too am now having the blinking cyan problem. Below, you will find my original post from about a week ago:

I have a Photon that is several years old. I recently deployed it on a newer wifi network where it seems to run flawlessly for several hours…before it loses wifi connectivity and begins blinking cyan. Hitting the reset button always seems to correct the issue - it blinks green and then re-connects to the wifi and breathes cyan. Several hours later - the problem reoccurs.

I have searched the forums but wasn’t able to locate a post that dealt with the specific problem I was having (there were several posts that discussed DFU mode, and occasional red bursts, but that’s not what I am seeing).

The problem unit has firmware 0.8.0 rc10. I have other photons working great with firmware 0.7.0 rc2.

Any thoughts? Thanks."

You already have a long discussion in that other thread, reposting the original without the link would mean same replies may be made resulting in little new information but “waste” of time for respondents

So if you want to refere to another thread always add a link.

1 Like

You are right - sorry about that and thanks for including the link.

UPDATE (October 19, 2018)
After running several test scenario’s I have come to the conclusion that the issue I was having was related to the system firmware that I had installed on my photons. I was using 0.8.0-RC.10 and the units would invariably go into the “flashing cyan” mode after several hours; it would only reconnect to the cloud after I reset my photon.

Earlier this week I “downgraded” the firmware to version 0.7.0 and my photons have been working fine ever since. It’s been 5 days and counting. My photons do lose connection from time to time, but they always recover on their own.

I thought I was done with this, but recently this problem has reappeared on my Photon weather station. It will run fine for one or two days and then suddenly lose its wi-fi connection. It sits there fast-blinking cyan forever and never reconnects. I have to go out in the back yard and power-cycle it. Then it reconnects instantly - for another day or two.

Oddly enough though the web IDE seems to think it’s online (it shows a slow pulsing cyan). This seems to have cropped up again after updating to f/w 1.5.0. Nothing in the code has changed.

I have

#if defined (PARTICLE)
SYSTEM_THREAD(ENABLED)
#endif

I’ve read this entire thread but it’s so old I don’t know if anything there is still relevant. Is there a workaround to force a Photon wi-fi reconnect these days?

@Michele I have personally noticed a big issue with memory (RAM) usage with OS 1.5.X - Particle know about this and are deciding what to do. Your application may be using a lot of RAM and therefore might be struggling. Another thing to consider is that the latest diagnostics / vitals and other changes that have been made in the Particle Cloud software have changed some of the behaviours that worked before. It would help if you mentioned which OS you were on before when it was working? Also, are you using antenna? What is the Wifi signal strength and quality like? and what mode are you using. Perhaps share your code?

I never found a software workaround that worked. And I tried LOTS of things.
Not even watchdog reboot. I gave up for a while.
What fixed mine was buying a good external antenna, and switching the mode to that.
I have had zero problem for years now. Same old software and firmware.

My wild guess:
It is not a problem with internet or router going up/down, or even software on the Photon.
It is a problem with a flaky wifi signal causing something in the wifi radio (baseband processor) firmware to lock up.

That’s possible. The program has grown fiarly large.

Output of arm-none-eabi-size:

|text|data|bss|dec|hex|
| --- | --- | --- | --- | --- |
|47648|256|12824|60728|NaN|

In a nutshell:
Flash used 47904 bytes
RAM used 13080 bytes

I didn’t note what f/w it was on before. I think it was 1.4 - whatever was popular late last year (2019). I’m using a high gain yagi antenna (they claim 12 dB) and I monitor the RSSI. It’s normally between -56 and -62 dB so I don’t think signal strength is an issue. The antenna mode is specifically set to external. I can share the code but first I have to redact a bunch of API keys in it.

Oh dear. I’ve already done that Too bad the watchdog reboot didn’t work beause that was going to be my next step. Maybe I’ll try that anyway. I have no idea what else to do.

Here’s a question - is it possible to downgrade the firmware on a Photon? It was running fine before 1.5.

Sounds like a different problem from mine.
If it is inside the Photon, then the watchdog reset likely would work.

Pruning memory usage seems like a good idea to try first.

I know someone else put an external watchdog hard reset device on their board as a last resort.

Any suggestions for some good watchdog code? Over here https://community.particle.io/t/my-photons-are-experiencing-inconsistent-wi-fi-connection-behavior/31647/15 I see this:

void checkWiFi() {
	IPAddress ip = WiFi.resolve("www.google.com");

	if (ip[0] == 0) {
		USBSerial1.printf("Resetting WiFi\n");	
		
		WiFi.off();
		delay(5000);
		WiFi.on();
		delay(1000);
		WiFi.connect();	
	}
}

Or is there some better way to do it?

Worth a try.
There is also a watchdog reset inside the photon.
You enable it, then have to write a keep-alive register periodically, and if you don’t it hard-resets the photon.
But that low level one is a bit of a pain.

First - Flash usage is way below the limit 128KB. RAM usage looks OK as well.

The easiest way to check for Cloud connection is Particle.connected() and WiFi connection WiFi.ready().

Your ‘watchdog’ could simply be checking every 60 seconds that Particle.connected() is true.

One thing I would say is that Cloud/Internet connection can come and go - it is never 100% even with an ethernet cable and certainly not with WiFi. If you are chasing 100% then don’t waste your time. I have a fleet of nearly 1000 photons and aside from the fact that a good number of them powersave by sleeping hence are offline, the stats from the rest show all sorts of drop-offs. The trick is to write a system that handles it.

2 Likes

That’s quite true. I don’t demand 100% connectivity. The probem I’m having is that once the connection drops, it never reconnects. I’ve let it sit for over 24 hours and it just fast-blinks cyan. If I power-cycle it, it reconnects instantly.

Right now I’m running the code above, checking every six minutes. It’s been up for four hours so far. We’ll see what happens over the next few days.

If it doesn’t work, I’ll try your suggestion next. Thanks!

Apologies if I have missed this in the thread but what system mode are you using?

I use SYSTEM_MODE(SEMI_AUTOMATIC); this will once connected try to reconnect if it loses connection. You mustn’t call Particle.connect() more than once, best done in setup(). I don’t have a process or function that retries to connect. On the Ethernet Featherwing there was a problem - now resolved with dropping and not reconnecting to the Cloud that needed a workaround.

Hmm, I’m not calling either SYSTEM_MODE or Particle.connect at all. I guess that makes it automatic?

So far the weather station has been up for 24 hours without losing its connection.

UPDATE (one hour later)

Well that didn’t work. After 25 hours, the Photon went offline again and wouldn’t reconnect (and I mean completely offline - I couldn’t even ping it). I power-cycled it and it was fine. Checking my fing logs I see that it also dropped wi-fi last night for two minutes but then automatically did reconnect. But not today. So apparently my explicit call to WiFi.connect () is not working.

I also checked my router in case it was a DHCP lease issue, but that’s set to “unlimited”.

Am using DeviceOS 1.5.2 and suffering the same “long term” issue of flashing cyan and not recovering connectivity.

Is there a definitive reason for this ongoing and frustrating issue?

Dare I say this is my last problem to solve before I am comfortable going into production…

The number of contributors and contributions to this post is a testament to how bad this issue is.

IMHO it is really poor that after all these years that it has not been sorted by official Particle advice, and/or DeviceOS fix.

1 Like

I’ve checked internally and our Photon test fleet runs reliably for weeks on end - it is always running the latest DeviceOS.

I would like to assist you in debugging this issue - could you please create a ticket on support.particle.io and provide me with example devices and the specific AP model you are seeing this issue on.

This thread will now be closed and we encourage anyone experiencing similar issues to either start a new thread for up to date information regarding the latest DeviceOS and Wifi or to create a support ticket at support.particle.io so that we can help you gather logs and figure out where the block is.
If you want to grab logs yourself, please see https://github.com/rickkas7/photon-clouddebug for instructions.

Engineering have looked into the issue and the consensus is that it is likely an incompatibility between the Broadcom WICED WiFi driver and the particular AP, or something in the user application that is creating a problem.