Electron online but not publishing / dropouts after certain time of usage

Troubleshooting
1

I am having a solar / battery driven Electron which is put into deep sleep for four minutes, wakes up, is awake for one minute to collect sensor data and goes back to deep sleep.

Code is set to work in semiautomatic mode with turning on cellular and connecting to particle after waking up from deep sleep.

The whole things works for just about a day +/- a few hours but then suddenly the console stops receiving events but Electron just works as normal - wakes up - goes back to sleep, etc. It also acquires cellular network as fast as usual.

One thing I notice is that in the mobile particle.io app there is a "last heared" field which accurately matches the time the Electron was awake and send off a publish.

However when console (and every subscribe) stops to receive messages the "least heared" drops to a value of "6 hours" and increases with the time no messages are sent.

The Electron however continues to wake up and go back to deep sleep - When it's awak I can even use the "Signal for 10 sec"-Function and the Electron instantly begins to signal via LED.

Here's my code:

To fix this I have to completely remove the Electron from power supply or battery. A reset via Push-Button won't fix the issue.

Could this be a problem with deep sleep and session re-use so that the session is expired but the Electron is still trying to make a re-use!? That would explain why this issues always appears after the same amount of time ...

EDIT:

I have activated system logging and here's the dump from console when device wakes up but the publish never arrives in console or api client:

As you can see, the cloud connection has successfully been made and also the publish is being send. But then some strange things happen which I cannot identify.

In contrast to this here's dump from a wake up when everything works normally:

@RWB - @ScruffR - Any advices here!?

EDIT: 22nd of May:

I just can confirm that the issue appeared this morning on another Electron running same software. It ran for three days without any issue at all. See detailed info in an answer below:

2

I have a hunch, but I’d like to get some life data for that first.

I know we talked about SLEEP_NETWORK_STANDBY before for another reason, but does the same happen with that option on?

3

Yes I know we have talked about that before - Haven’t tried it yet with SLEEP_NETWORK_STANDBY but will try tomorrow.

It’s kind of strange that not even a full reset can fix this. Only removing from power fixes it. Would make sense if not deep sleeping but pausing the Microcontroller - but as far as I understand - waking from deep sleep is like a reset for the MCU - all the code is run from beginning - or does this just affect the user code!?

I am not pretty sure but I can remember that when not using third party SIM it ran for a few days - but I had running a different code and often made changes at that time so this is not a reliable information.

In my post I mentioned that it might has to do with session re-use but that also makes no sense because when communication drops out the “signal for 10 sec” function works normally.

You said you have a hunch!?

EDIT:

What I CAN say is that when not sleeping at all the communication never drops out - tested it for weeks …

4

It might be just superstition but cell providers don’t like devices to go on-/offline too frequently and may blacklist your device for some time.
Keeping the connection intact should either cure the problem or rule this theory out.

5

Yes I just thought about that, too but communication from cloud to device just keeps working - also the RGB-LED indicator says that there has been successfully made a connection to the cloud. It’s just that communication from device to cloud drops. Not the other way round…

6

Currently I’m running this slightly altered code of yours on one of my 0.6.2 Electrons.

//STARTUP(Time.setTime(0));
//STARTUP(cellular_credentials_set("soracom.io", "sora", "sora", NULL));
SYSTEM_THREAD(ENABLED);
SYSTEM_MODE(SEMI_AUTOMATIC);

FuelGauge fuel;
PMIC pmic; 

uint32_t lastPublish = 0;
bool needsPublish = true;

void setup() {
    
    //int ret = Cellular.command("AT+UDCONF=20,1\r\n");
    
    //pmic.begin();
    pmic.setChargeCurrent(0,0,1,0,0,0);
    //pmic.setInputCurrentLimit(1500);
    
    Cellular.on();
    Cellular.connect();
    waitUntil(Cellular.ready);
    
    Particle.connect();
    waitUntil(Particle.connected);
    
    lastPublish = millis();
}

void loop() {
    
    if (!Particle.connected) {
        Particle.connect();
        waitFor(Particle.connected, 300000);
        lastPublish = millis();        
    } else {
        
        if (needsPublish) {
            char data[64];
            snprintf(data, sizeof(data), "v:%.2f,c:%.2f", fuel.getVCell(), fuel.getSoC());
            Particle.publish("B", data, PRIVATE, NO_ACK);
            needsPublish = false;
        }
    }
    
    if (millis() - lastPublish > 60000) {
        
        //pmic.begin();
        pmic.setChargeCurrent(0,0,1,0,0,0);
        //pmic.setInputCurrentLimit(1500);
        System.sleep(SLEEP_MODE_DEEP, 240); //, SLEEP_NETWORK_STANDBY);
    }    
}
1 Like
7

I've also had this happen to me when using the Particle Publish in the past.

If I remember correctly I also saw this in early code when pushing data to Ubidots but not with my recent code.

I also had to remove the battery since hitting the reset button would not fix the problem which was probably because the Modem does not reset when hitting the rest button.

SerialLogHandler logHandler(LOG_LEVEL_ALL); //This serial prints system process via USB incase you need to debug any problems you may be having with the system.

I added the serial print debug code to my app so if it did happen again I could see what was actually going on during the connection process but it never happened again so never got a chance to gather any data on it.

ApplicationWatchdog wd(660000, System.reset); //This Watchdog code will reset the processor if the dog is not kicked every 11 mins which gives time for 2 modem resets.

I also added the Watchdog code to try to trigger a reset should this cause a code lockup.

I'll look at your code and see if I notice where were doing something different.

8

That makes sense - But in my case the code is not locking up since it keeps on running - going into sleep - coming back and being responsive when messages arrive from cloud to device!

I am somehow thinking that the moden is not causing - or - that the hardware is not causing the issue at all - since - remember what I wrote - The communication from cloud to device keeps working - just device to cloud drops - and when this happens the "last heared" field instantly drops to "6 hours ago" ...

Ok so from a cloud-software / crypto perspective: When there's a timeout on session it has to be renewed and the messages arriving from device to cloud have wrong signature / key / token / whatever - but those messages coming from cloud do not have such a token since those are not being verified by the electrons system firmware ...

But that does not explain why a hardware-reset via button isn't able to bring communication back up ... ... ...

EDIT:

One thing I noticed is that when first connecting to cell tower AFTER removing battery / power - the connection takes some time - like 15-30s. But when coming back from deep sleep the reconnection to cell tower just takes about 2-4 seconds!!!

9

Because the Reset button does not reset the modem.

10

Sure! I meant why my Cloud-Software-Crypto-Theory cannot explain this - that there is no hardware reset is of course explaining it!

11

Oh wow! I was just about to check my devices before going to bed! It happened again - right now - Electron is not publishing but is responsive to messages coming from the cloud … So this time it took less than a whole day! It happened within 3 hours!!!

12

@simmikolon

I took some time and combined your code with the connection code I’m using to see if it works any better for you. One second and I’ll post it so you can test if overnight.

1 Like
13

I took a view at my third party’s SIM data volume usage dashboard and what I see is really weird:

IMG_0094.PNG1903Ă—1404 274 KB

Normally data usage is always centered around 3.9Kilobytes / hour - but those peaks do absolutely make no sense since I am not changing the code …

And since within those peaks there is no download (blue) I am pretty sure that there is something wrong with connection to cloud since it seems like the device performs retries since it does not get the desired data!

BTW I am not “pretty sure” - I simply “strongly believe” that this is a connection error - the device does transmit - even in those periods in which it seems that not publish is being send - but it somehow looses connection and performs retries …

14

You can give this code a try and see what happens.

Not sure it’s going to be any better but worth a shot.

https://go.particle.io/shared_apps/591f6d4c133ca11b2600148e

1 Like
15

I’ll give it a try tomorrow since currently I am already prepared for bed and I would have to walk a few meters in the dark cold outside to get my electron :laughing: … Thanks for preparing your code!

But really - have you seen my screenshot of the data volume dashboard - that’s something I haven’t seen before - those peaks are weird - but even more weird is the fact that there are periods with no download to device …

16

Get that Electron and let’s see if that code can work all night into the morning :smile:

I’m not using a 3rd party SIM so I have no access to data like that to compare it to.

17

On my way!

Well I have just that third party SIM but I can say that with using no sleep the data is constant - and I can say that the weird data usage corresponds to those dropouts …

18

In my code I have the debugging serial output so if if it happens again you can capture that debug data and we can have Bdub look at it.

20

@RWB @ScruffR Here’s log dump from a wake up after the console log / api client stops receiving publishes.

21

Could you private message me the Device ID of your Electron? I can check the cloud-side logs to see if there’s anything unusual. Since individual publishes are not logged I may not be able to see anything useful, but you never know. (You can also create a support ticket if you prefer.)

Rick