Long Term Photon Connection Stability

I was using Automatic mode. I would just leave the device plugged in 24/7 and only send an update every few mins. That’s all.

1 Like

We have about 120 photons running in 120 different gas stations in various states. Most of them have been deployed for about 2 years now. I would guess that about 100 of them have NEVER been touched and have gone through about 10 OTA firmware updates, power outages, internet service outages etc. and always have recovered. A few have been replaced due to OTA firmware update failures (about 4). There are about 15 devices that occasionally need to be reset (like once every 3 months). There are currently two locations where we are having to reboot the wifi router about every 3 days. Our stats would suggest this is some sort of environmental issue. These two locations that are having issues use have the same wifi routers as 60 of the other locations. The 60 locations have a standardized networking stack. The other locations we deal with many different routers and networking stacks which seem to be no problem.

An odd thing is that all of our connectivity issues have always been in New Mexico, and Nevada. I’m not sure if generally they have bad service down there which trickles down to re-connectivity issues to the wifi router? The devices in Idaho, Wyoming, Utah, Montana, just never go down.

I know thats a lot of different numbers there but the gist of it is we are pretty successful and happy with the photons. Honestly we haven’t done a major update to the app for a long long time and we are very rarely in the code. We use semi-automatic mode and declare the use of an external antenna. I’m sure there are other good experiences out there.

// This #include statement was automatically added by the Spark IDE.
#include "HttpClient.h"

// This #include statement was automatically added by the Spark IDE.
#include "application.h"

PRODUCT_ID(115);
PRODUCT_VERSION(25);

STARTUP(WiFi.selectAntenna(ANT_EXTERNAL));

// we will handle wifi manually
SYSTEM_MODE(SEMI_AUTOMATIC);

Thanks particle team!

-Jason

3 Likes

@chipmonk, I think it’s possible that we may have seen this before. I do think there was one time (out of 120 devices over 2 years) where I had to have the customer re-enter the wifi credentials. The customer swore that credentials never changed but who knows?

Did you ever find the cause of that? Did you experience it again?

Hi jasonp,

Never did have a photon lose it wifi setup again. Never was able to figure out the root cause.

chipMonk

I didn’t find this thread until recently, but I too am now having the blinking cyan problem. Below, you will find my original post from about a week ago:

I have a Photon that is several years old. I recently deployed it on a newer wifi network where it seems to run flawlessly for several hours…before it loses wifi connectivity and begins blinking cyan. Hitting the reset button always seems to correct the issue - it blinks green and then re-connects to the wifi and breathes cyan. Several hours later - the problem reoccurs.

I have searched the forums but wasn’t able to locate a post that dealt with the specific problem I was having (there were several posts that discussed DFU mode, and occasional red bursts, but that’s not what I am seeing).

The problem unit has firmware 0.8.0 rc10. I have other photons working great with firmware 0.7.0 rc2.

Any thoughts? Thanks."

You already have a long discussion in that other thread, reposting the original without the link would mean same replies may be made resulting in little new information but "waste" of time for respondents

So if you want to refere to another thread always add a link.

1 Like

You are right - sorry about that and thanks for including the link.

UPDATE (October 19, 2018)
After running several test scenario’s I have come to the conclusion that the issue I was having was related to the system firmware that I had installed on my photons. I was using 0.8.0-RC.10 and the units would invariably go into the “flashing cyan” mode after several hours; it would only reconnect to the cloud after I reset my photon.

Earlier this week I “downgraded” the firmware to version 0.7.0 and my photons have been working fine ever since. It’s been 5 days and counting. My photons do lose connection from time to time, but they always recover on their own.

I thought I was done with this, but recently this problem has reappeared on my Photon weather station. It will run fine for one or two days and then suddenly lose its wi-fi connection. It sits there fast-blinking cyan forever and never reconnects. I have to go out in the back yard and power-cycle it. Then it reconnects instantly - for another day or two.

Oddly enough though the web IDE seems to think it’s online (it shows a slow pulsing cyan). This seems to have cropped up again after updating to f/w 1.5.0. Nothing in the code has changed.

I have

#if defined (PARTICLE)
SYSTEM_THREAD(ENABLED)
#endif

I’ve read this entire thread but it’s so old I don’t know if anything there is still relevant. Is there a workaround to force a Photon wi-fi reconnect these days?

@Michele I have personally noticed a big issue with memory (RAM) usage with OS 1.5.X - Particle know about this and are deciding what to do. Your application may be using a lot of RAM and therefore might be struggling. Another thing to consider is that the latest diagnostics / vitals and other changes that have been made in the Particle Cloud software have changed some of the behaviours that worked before. It would help if you mentioned which OS you were on before when it was working? Also, are you using antenna? What is the Wifi signal strength and quality like? and what mode are you using. Perhaps share your code?

I never found a software workaround that worked. And I tried LOTS of things.
Not even watchdog reboot. I gave up for a while.
What fixed mine was buying a good external antenna, and switching the mode to that.
I have had zero problem for years now. Same old software and firmware.

My wild guess:
It is not a problem with internet or router going up/down, or even software on the Photon.
It is a problem with a flaky wifi signal causing something in the wifi radio (baseband processor) firmware to lock up.

That’s possible. The program has grown fiarly large.

Output of arm-none-eabi-size:

|text|data|bss|dec|hex|
| --- | --- | --- | --- | --- |
|47648|256|12824|60728|NaN|

In a nutshell:
Flash used 47904 bytes
RAM used 13080 bytes

I didn’t note what f/w it was on before. I think it was 1.4 - whatever was popular late last year (2019). I’m using a high gain yagi antenna (they claim 12 dB) and I monitor the RSSI. It’s normally between -56 and -62 dB so I don’t think signal strength is an issue. The antenna mode is specifically set to external. I can share the code but first I have to redact a bunch of API keys in it.

Oh dear. I've already done that Too bad the watchdog reboot didn't work beause that was going to be my next step. Maybe I'll try that anyway. I have no idea what else to do.

Here's a question - is it possible to downgrade the firmware on a Photon? It was running fine before 1.5.

Sounds like a different problem from mine.
If it is inside the Photon, then the watchdog reset likely would work.

Pruning memory usage seems like a good idea to try first.

I know someone else put an external watchdog hard reset device on their board as a last resort.

Any suggestions for some good watchdog code? Over here https://community.particle.io/t/my-photons-are-experiencing-inconsistent-wi-fi-connection-behavior/31647/15 I see this:

void checkWiFi() {
	IPAddress ip = WiFi.resolve("www.google.com");

	if (ip[0] == 0) {
		USBSerial1.printf("Resetting WiFi\n");	
		
		WiFi.off();
		delay(5000);
		WiFi.on();
		delay(1000);
		WiFi.connect();	
	}
}

Or is there some better way to do it?

Worth a try.
There is also a watchdog reset inside the photon.
You enable it, then have to write a keep-alive register periodically, and if you don’t it hard-resets the photon.
But that low level one is a bit of a pain.

First - Flash usage is way below the limit 128KB. RAM usage looks OK as well.

The easiest way to check for Cloud connection is Particle.connected() and WiFi connection WiFi.ready().

Your ‘watchdog’ could simply be checking every 60 seconds that Particle.connected() is true.

One thing I would say is that Cloud/Internet connection can come and go - it is never 100% even with an ethernet cable and certainly not with WiFi. If you are chasing 100% then don’t waste your time. I have a fleet of nearly 1000 photons and aside from the fact that a good number of them powersave by sleeping hence are offline, the stats from the rest show all sorts of drop-offs. The trick is to write a system that handles it.

2 Likes

That's quite true. I don't demand 100% connectivity. The probem I'm having is that once the connection drops, it never reconnects. I've let it sit for over 24 hours and it just fast-blinks cyan. If I power-cycle it, it reconnects instantly.

Right now I'm running the code above, checking every six minutes. It's been up for four hours so far. We'll see what happens over the next few days.

If it doesn't work, I'll try your suggestion next. Thanks!

Apologies if I have missed this in the thread but what system mode are you using?

I use SYSTEM_MODE(SEMI_AUTOMATIC); this will once connected try to reconnect if it loses connection. You mustn’t call Particle.connect() more than once, best done in setup(). I don’t have a process or function that retries to connect. On the Ethernet Featherwing there was a problem - now resolved with dropping and not reconnecting to the Cloud that needed a workaround.

Hmm, I’m not calling either SYSTEM_MODE or Particle.connect at all. I guess that makes it automatic?

So far the weather station has been up for 24 hours without losing its connection.

UPDATE (one hour later)

Well that didn’t work. After 25 hours, the Photon went offline again and wouldn’t reconnect (and I mean completely offline - I couldn’t even ping it). I power-cycled it and it was fine. Checking my fing logs I see that it also dropped wi-fi last night for two minutes but then automatically did reconnect. But not today. So apparently my explicit call to WiFi.connect () is not working.

I also checked my router in case it was a DHCP lease issue, but that’s set to “unlimited”.

Am using DeviceOS 1.5.2 and suffering the same “long term” issue of flashing cyan and not recovering connectivity.

Is there a definitive reason for this ongoing and frustrating issue?

Dare I say this is my last problem to solve before I am comfortable going into production…

The number of contributors and contributions to this post is a testament to how bad this issue is.

IMHO it is really poor that after all these years that it has not been sorted by official Particle advice, and/or DeviceOS fix.

1 Like