I can’t communicate with my Electron today. The console and IDE report it as being on-line, but I cannot call any cloud functions nor read any cloud variables from it. I have previously noted that the cloud can report the device to be online even though the 3G service to it is down. Is this the case today?
I had a number of devices go offline this morning. From past experience, it seems likely that something happened to the network briefly and the devices have not yet recovered.
The device status in the tools aren’t reliable due to the protocol used (UDP). As such, the only real way to know if a device is online is to ping it using a function or variable.
3G going down is also not something Particle is responsible for, considering they’re using someone else’s network. Asking whether “3G is down” is also a bit vague, since there’s global coverage. Where exactly would 3G be “down”?
@Moors7: I understand what you are saying exactly. However, the 3G supplier is under contract to Particle and I know of no tools available to me to further quantify the issue. When I asked if it was down, it was due to failure of “pings”, as you have indicated. I did say, in my original post, that I could not read cloud variables or execute cloud functions on the device. The Particle console and IDE both show the device as being online, and I can talk to the cloud and find out about the device (what variables and functions it exposes), but the device is, apparently, not communicating with the cloud.
The only other data point that I can offer is that the 3G billing record shows up as no data used since Sept 11. This is absolutely incorrect, as I was communicating with the device several times per day every day up until this morning. So it certainly seems that the 3G provider is having problems. I think that @hwestbrook is seeing this as well.
My Electron is at a user location and is not easily accessible to me. It would be nice to have some way to verify that 3G is down (not the device itself) somehow from my office, and even better if I know that the outage is being attended to.
This post was not intended as a complaint. It was intended as a question. It is not convenient nor easy for me to go to the location of the Electron to see what its LED is saying.
We’re having a similar issue on several devices today as well in several different locations across the country (Milwaukee, Philadelphia, NYC). Interestingly, other devices within a 1/4 mile of the devices having trouble are online just fine.
We got eyes on one of the devices, and it was rapidly blinking cyan. What does this code mean? Its not documented anywhere AFAICT.
Thanks!
3 of my 7 Electrons stopped reporting around 6:45 central time this morning. Even if it was caused by a problem with the cell network, the fact that they were not able to reconnect on their own seems like a bug in the firmware to me.
I suspect that flashing cyan means the same on Electron as on Photon, which is that the device cannot connect to the Particle Cloud.
I will ping @KyleG here to see if he can do some investigation, but a support request might also be called for if you are still having problems.
As to self-recovering @ih57452, that depends largely on the code that you have added to the Electron. If all of your Electrons are running Tinker (the default app) then that is in Particle’s hands, but if you are running your own code then the situation can be different and there are things your code can do to block the cloud connection.
Hey all I got @bkos ping let me see what I can find out.
Kyle
1 Like
Based upon the issues reported by @hwestbrook, @best and @ih57452 in response to my original post, I traveled to the Electron site and I also observed very rapidly flashing cyan (or white? hard to tell). I tried pressing the reset button. The Electron blinked green a few times and then went back to fast flashing cyan. After several tries to make it connect - all unsuccessful - I took my entire hub device (Electron with LiPO and all) back to my office for further investigation. But, I did not get there. While en-route I entered a different cell and suddenly I noticed flashing green and then breathing cyan. The Electron was still powered by the LiPO battery (please, no comments about my driving!). Since it had recovered, I turned back to the installation site and the connection remained OK. I reinstalled the Electron and it is working now. It remains unclear if entering a new cell zone was needed to get the Electron back online to the cloud or whether this was just a time coincidence.
@bko: based upon my being unable to restore connection by pressing the reset button, there should not have been any impact of my code on an inability of the Electron to reestablish a cloud connection. Perhaps the timing is just a coincidence but perhaps there is some recovery state issue between the Electron/SIM card data and the cellular provider.
P.S.: Reading data from this Electron has been rather slow over the past few days/weeks. Now that it is running again, it is back to running faster again. An indication of issues with the cell carrier perhaps?
Hey everyone, thanks for this info — we’re looking into it right now. Our telco partner was in fact performing some planned maintenance today, and they stated that connectivity would not be impacted. But, there may have been some degradation nonetheless; if there was, it appears to be intermittent and not widespread.
We’re reaching out to our telco partner now to see if there were any unexpected issues with their maintenance window. In the meantime, it sounds like connectivity has resumed for your particular device, @BobG?
2 Likes
This is not really perfectly true in the face of threaded system mode, the STARTUP macro with retained variables, and custom cellular modem commands coupled with cell site interaction but if you are not using these somewhat exotic features, then you are right. Simple code should not be able to block the cloud connection, but complex code certainly can.
But it sounds like @jonlogan is on the case now and I will step aside.
1 Like
Thanks, @jonlogan for hopping on this. Yes, my Electron-based system is up and running. However, it only recovered under the circumstances that I mentioned previously. If cell service was briefly interrupted, my Electron did not recover on its own. On the other hand, it did recover “on its own” while I was driving it back to my office. My take on all of this is that the issue was not a localized cell outage but in fact some sort of account verification issue by the provider and that it took most of the day to get it fixed. Of course, I have no direct evidence to support this, only the indirect observations that I have described.
@bko: thanks, as well. It is good to know this. I am not using threaded system mode, retained variables through startup, or custom cellular commands at this time. So I don’t think that anything on the Electron was blocking cellular connection to the cloud. Something the cellular provider did blocked the connection. That’s OK - glitches happen. But it is helpful to understand what happened so that we can understand how to recover quickly if it happens again.
To the Particle.io folks: we who are developing Particle-based systems to put into remote customer facilities need a few tools and capabilities to help us when a glitch happens at a customer site. In particular, we need a remote reset command of some sort that will reset our devices if they crash for any reason. This includes crashing of our software, as well as resetting under conditions as @bko indicated. It would also be nice to be able to read the state indicated by the color LED remotely, so that we can understand what has happened without having to have someone on site to observe the device. Obviously, none of this can work if communication is down, but these tools would nonetheless be welcome additions to the console.
2 Likes
That would somehow be tricky since remote requires an active connection which was not present at that time, as I understood.
Howeverthere already is a solution available: Watchdog.
It's not remotely triggerable, but is a means for a devices to monitor itself via inbuilt hardware.
You might like to have a look at this implementation of @rickkas7
BTW, in addition to @bko's possible ways to impact cell reconnection in your own code, there are also some less "esoteric" ways. When using SYSTEM_MODE(SEMI_AUTOMATIC) or SYSTEM_MODE() manual a lot of the Cellular.xxxx() and Particle.xxxx() functions can interfere when used "wrong".
2 Likes
Is there a better way than the forums to report this to Particle? If I knew what kind of data you needed and how to report it, I would be happy to provide it, especially if it resulted in more reliable connections.
Throughout yesterday I had a number of devices that were offline for 1 hour plus. In the same area, I had devices that were online. This issue seems similar to what happened earlier this year, where devices were not being allowed onto the network (at least I think that is what happened). This time, though, my devices recovered through the WD.
You could use the support portal over here:
https://docs.particle.io/support/support-and-fulfillment/menu-base/
@hwestbrook and @BobG can you get me ICC IDs for devices that were showing wonky behavior yesterday? (Please send them to me in a DM). Please tell me the specific timeframes during which they were having connectivity issues. We can do some research into their behavior.
Also, +1 on the suggestion of making use of a watchdog timer!
And +1 on using the support portal. That’s our channel for technical support. For handy reference, we also alias it from support.particle.io.
2 Likes