Electron online but not publishing / dropouts after certain time of usage

Troubleshooting
22

@ScruffR @RWB @rickkas7 - I think I have found out what has caused (and is causing) the issue. A few days ago it was very hot outside (yeah summer! :D) and the solar panel was so hot that I could not even touch it. The Electron was in a sealed box directly behind the panel and in direct contact to it so that there was a very strong heat inside that box.

I came to this conclusion because I’m currently running two Electrons in parallel (the one with the issue and a new one) - both are running the same software and both are using a third party SIM card. The Issue keeps coming back on the one that was inside that box - on the other Electron there is no issue at all.

And also both log dumps are totally identically - there is no difference at all!

So I think I might have baked it in the sun so that this is causing the issue …

Sorry for flooding the forums but I had absolutely no idea what could possibly lead to that failure …

I keep on running both Electrons here on my desk and if the issue does come back I will get back here …

23

It happened to me in cold weather while sitting outside in a small clear front Pelican case in direct sunlight.

I can do more testing also now that it’s getting up into the 80’s again.

24

Just saw something somewhat similar and I don’t think it’s caused by heat. Particle
Electron is running deep sleep every 10 minutes waking up to take a reading. Ran for about five days straight until about 1 AM last night stopped sending data. Hit the reset button and it was still running the sketch and taking the readings every 10 minutes, but the publishes were not coming through. Had to disconnect the battery before it started working again . Any recommendations to troubleshoot if it happen again? Particle app did not show it as being seen, even though it looked like it was connecting.

1 Like
25

This sounds like it’s the same issue I am having with one of my Electrons. Currently I am running a second one with same code in same hardware - until now it works for almost three days continuously - no dropout at all!

If I however use the other one - the one with that issue - even two days later it just takes about a few hours and the publishes are not coming through.

But as I said - both devices have same code / hardware / SIM Card provider (but two different cards of course) - AND mostly important - Both (!!) produce same log output!!!

But one fails - one not … (at least until now)

@codeandcircuit Would be awesome if you’d share the code you are running - just to have a comparison to my particular code …

Somehow this is totally strange @RWB @ScruffR even if it would be code (since assumed firstly) - Before the Issue appeared the code and device ran for days - on my side and on @codeandcircuit side - as far as we know …

26

It wouldn’t seem to have anything to do with the code. It’s just taking a 3 readings from sensors and publishing it, waiting a sec, and then going deep sleep for 600 seconds. And the code seemed like it was doing it’s thing just fine; I could see it starting up, triggering the reading (sensor makes a beep sound), and going back to sleep. Just no sign of it online.

27

@simmikolon Well you checking the debug logs against another Electron running the same sketch without any connection problems is a step forward towards finding a solution.

@BDub Considering the Electron that sends data successfully to the Particle Cloud and the Electron that does not send data successfully have the same debug output data can you think of any way to catch when the Particle Publish does show up on Particles end and correct for this issue via a Modem Rest which seems to be the only solution for the problem?

I figure subscribing to the Published event and then triggering a Modem Reset if we do not receive a reply from the Published event would work but that’s going to increase our Data consumption rates and overall cost to of operations per unit.

Any ideas?

28

@simmikolon, this is an obscure issue and when I tagged @rickkas7 he didn’t seem to find anything in particular either.
So we are a bit stuck about reproducibility.
My device running your code also seems to have no issues up to now.

Could you swap the SIMs and also remove the NO_ACK from the publish?
Just for checks.

29

I would say this is not an obscure issue. I've experienced it many times in the past but just switched over to pushing data to Ubidots and it happened less frequently but still, the same thing would happen every now and then pushing data to Ubidots and by passing the Particle Cloud which leads me to believe it's an issue with the Cellular network somehow.

Leave your Electron running long enough and I'm pretty sure your going to experience the same thing.

30

It’s still obscure as we don’t yet know where it comes from and how to prevent (not circumvent) it.

Having alternatives is a good thing but just ignoring an issue since we can make do is not my prefered philosophy :wink:

31

I can confirm that this morning my second Electron dropped communication by console stopped receiving publishes. Before it ran three days without any dropout at all!

I don't believe it's an issue with cellular network at all, because:

  1. Taking a look at the system log, communication is happening, handshake or re-use is being made, Electron CAN connect to the cloud and is actually receiving data!

  2. Even when publish do not arrive at console. Sending a "Signal for 10 sec" is instantly executed by the Electron - so there is downward communication

  3. My third party cellular ISP says that there is connection being made - and - you can easily see the dropout and how it affects the data usage (See screenshot attached below)

  4. Before issue appears the Electron(s) can publish for days - but after it appeared - even after resetting the modem - it just takes a few hours for the issue to reappear - once even just a few minutes (!!)

Here's screenshot from my third party cellular ISP:

Bildschirmfoto 2017-05-22 um 07.48.44.png1772×1146 87.8 KB

I have also swapped SIM-Cards but that has made no difference. Currently I am going back to Particle.io SIM Card and will have it running.

And I also haven't tested what happens when not putting the device into sleep. But I can remember having done this once with keeping it alive for more than five days - without any dropouts at all.

Also I haven't yet removed NO_ACK - I just want to keep things same to make tests with original SIM Card.

2 Likes
32

Your right, I took obscure to mean rare.

Hopefully, we can figure out what is going on or at a minimum a way to avoid it.

33

Interesting data!

I’m going to put my Electron back outside with a 3w solar panel pushing battery status updates to Ubidots every 2 mins and see how long it can run without this dropout condition to happen.

@ScruffR Do you still have your Electron up and running to test this also on a 3rd part SIM?

@BDub Can you also run a test with an Electron to help try to gather more data on this also?

34

@RWB @ScruffR @rickkas7 @BDub

Let me give you the preliminary result:

  1. Electron No. 1 (where the issue first appeared) - Original Third-Party SIM Card - Removed NO_ACK - Currently No Dropouts for about 5 hours …

  2. Electron No. 2 (where dropout happened after three days) - Same Code but with NO_ACK still set but swapped SIM Card to original Particle.io SIM Card - Currently also no dropouts for about 5 hours …

There is also something very weird happening after a dropout happens:

  1. Before Dropout I can flash Electron using particle flash --serial firmware.bin

  2. After dropout - particle flash --serial firmware.bin leads to the following error: Error writing firmware - unknown message undefined - I can just remove this by flashing via DFU !!!

This is totally strange … And I haven’t yet really figured out what causes this …

I am having an idea what could possible cause the dropouts:

The Third Party SIM Provider is connecting all devices through a closed VPN Network. Devices aren’t directly connected to the Internet. This allows sending values via UDP or TCP without encryption - on server-side you can perform an UDP or TCP update to HTTPS - This allows keeping actual data usage low …

Electron uses UDP hole punching for keepalive - so maybe the fact that those Soracom (third party ISP) SIM Cards tunnel communication through VPN before getting to the internet is not very compatible with that UDP technique !?

1 Like
35

I just checked the cloud-side logs for your device (the original one). Individual publishes are not logged so sometimes it can be hard to figure out exactly what is happening, however. Under what appears to be normal circumstances, I can see the device come online every 4 minutes. However, sometimes long periods of time pass with no log entries at all. I presume these are the points where events are not being received by the cloud. I don’t know for sure why that’s happening, but it appears to be a low layer (cellular to Internet, or cloud API session) not at the publish level.

1 Like
36

I have setup my Electron to push Battery data directly to Ubidots every 2 mins directly without using the Particle Publish or Particle Clould.

I’m using a Particle SIM.

The dashboard link is below.

The goal is to compare the dropout frequency to the Particle Publish dropout frequency to determine if it’s just a Particle Publish thing or a cellular network issue.

In the past, I have seen the drop out when pushing data to Ubidots but not with the current code I’m running yet. With this current code I am able to push data for at least a month without seeing any issues but now I’m running the new 0.6.2 firmware so this is a new test on the new firmware.

1 Like
37

Yes - Absolutely agree! It has to be low level. Have you seen what I wrote about the UDP hole punching!?

Soracom is isolating their cellular connections from the Internet - and they are doing this in such a way that you can send pure and unencrypted data without having to worry about the man in the middle.

@RWB @ScruffR Currently it seems like removing NO_ACK from the Electron that is using third party SIM does not lead to the issue.

And also switching back to original Particle.io SIM Card on the other Electron (which still has NO_ACK set) could successfully stop the issue from coming back.

But that's just after 10 hours of running. First time the issue appears took three days (at least last time). But after it appeared it took just few hours. The Electron which at first showed the issue didn't even take one hour when still running on third party SIM - with Particle.io SIM Card however it seems like it is working seamlessly ... We'll see what the night brings ...

@RWB Thanks for sharing the code! Will watch it!

1 Like
38

It will probably happen even with the Particle SIM given enough time.

I think the only way to be sure a remote device does not get stuck in this condition is to have it subscribe to its own published event and if it does not receive a reply after you send the event then use code to trigger another publish and if still unsuccessful trigger a full reset of the Electron.

1 Like
39

@RWB @ScruffR @rickkas7 Let me give you an update. This morning I woke up and checked publishes right away. This is what happened:

One Electron is still publishing. The other Electron had dropped out communication.

The Electron which dropped out is:

  • Electron with Third Party SIM from Soracom.io
  • Runs code WITH Acknowledgement set

@rickkas7 When viewing the Electron with dropped out communication In the Particle.io Mobile Application I can see that the “Last heard” Field was set to “17 hours ago” - Which points back EXACTLY to the time the Electron was first switched on! But while publishes still arrived at console this field always had a value of the time passed after the last publish - so about five minutes. When communication drops this value points back to the time the Electron was first switched on (initial handshake!?) So what exactly does set that field? What determines the value of “last heard”?

I am pretty sure this is due to the fact that soracom.io encapsulates devices before bringing then into the internet.

Good news -> Electron with original Particle.io SIM Card is still publishing! And hopefully it keeps on!

1 Like
40

… Still running … :joy::blush:

41

@rickkas7 @BDub

I have an Electron pushing data directly to Ubidots for about 2 days now.

Today I start seeing the Electron wake up and sit flashing Green for mins before going back to sleep per my code. The data that is supposed to be sent does not get sent to Ubidots. I think this will self-correct but right now here is the Debug data the Electron spits out when it wakes up, flashes green and sometimes breathes green for a few and goes back to sleep.

Any ideas what the Debug data means?

I'm going to leave it running and see if it ever auto corrects or if reset or battery pull is required to fix this.

This is an orgianl Particle SIM

561.487 AT send 15 "AT+USOCTL=0,1\r\n"
571.497 AT send 15 "AT+USOCTL=1,1\r\n"
581.507 AT send 15 "AT+USOCTL=2,1\r\n"
591.517 AT send 15 "AT+USOCTL=3,1\r\n"
601.527 AT send 15 "AT+USOCTL=4,1\r\n"
611.537 AT send 15 "AT+USOCTL=5,1\r\n"
621.547 AT send 15 "AT+USOCTL=6,1\r\n"
socketSendTo(-1,50.23.124.66,9010,,120)
ERROR
642.321 AT send 15 "AT+USOCTL=0,1\r\n"
652.331 AT send 15 "AT+USOCTL=1,1\r\n"
662.341 AT send 15 "AT+USOCTL=2,1\r\n"
672.351 AT send 15 "AT+USOCTL=3,1\r\n"
682.361 AT send 15 "AT+USOCTL=4,1\r\n"
692.371 AT send 15 "AT+USOCTL=5,1\r\n"
702.381 AT send 15 "AT+USOCTL=6,1\r\n"
socketSendTo(-1,50.23.124.66,9010,,120)
ERROR
723.155 AT send 15 "AT+USOCTL=0,1\r\n"
733.165 AT send 15 "AT+USOCTL=1,1\r\n"
743.175 AT send 15 "AT+USOCTL=2,1\r\n"
753.185 AT send 15 "AT+USOCTL=3,1\r\n"
763.195 AT send 15 "AT+USOCTL=4,1\r\n"
773.205 AT send 15 "AT+USOCTL=5,1\r\n"
783.215 AT send 15 "AT+USOCTL=6,1\r\n"
socketSendTo(-1,50.23.124.66,9010,,120)
ERROR
803.989 AT send 15 "AT+USOCTL=0,1\r\n"
813.999 AT send 15 "AT+USOCTL=1,1\r\n"
824.009 AT send 15 "AT+USOCTL=2,1\r\n"
834.019 AT send 15 "AT+USOCTL=3,1\r\n"
844.029 AT send 15 "AT+USOCTL=4,1\r\n"
854.039 AT send 15 "AT+USOCTL=5,1\r\n"
864.049 AT send 15 "AT+USOCTL=6,1\r\n"
socketSendTo(-1,50.23.124.66,9010,,120)
ERROR

image.png1944×1218 81.1 KB

1 Like