# Boron's gone offline

**URL:** <https://community.particle.io/t/borons-gone-offline/57253>\
**Category:** Firmware\
**Created:** [August 3, 2020, 12:34pm UTC](https://community.particle.io/t/borons-gone-offline/57253 "2020-08-03T12:34:07Z")\
**Posts on this page:** 16\
**Page:** 1

<div class="post-metadata">

**Author:** ![StngBo](https://sea2.discourse-cdn.com/flex026/user_avatar/community.particle.io/stngbo/32/28934_2.png) [@StngBo](https://community.particle.io/u/StngBo)\
**Post date:** [August 3, 2020, 12:34pm UTC](https://community.particle.io/t/borons-gone-offline/57253/1 "2020-08-03T12:34:07Z")

</div>

Hi I’m hoping someone on here can advise.

We have two borons in the field running apps on firmware 1.5.2. Both Borons normally operate while connected to the cloud. On the 01 Aug, both borons went offline, their system uptimes as per the last vitals were 526420 and 596673.

This morning 03 Aug, one of the devices connected long enough to send one set of vitals, uptime of 616708.

Obviously both devices are currently not connected to the cloud and also at the moment I have no way of remotely seeing what state the borons are in. Does anyone have any advice on how I can diagnose this any further? I can post vitals from the devices leading up to the disconnects, does anyone have any advice on how to interrogate vitals data and what warning signs to look out for? It’s just strange that they both went offline on the same day. Then stranger again that one has managed to send one set of vitals before disconnecting early this morning. Geographically, the devices are separated by a few hundred kilometres, but both running the same app.

Edit: I only mention uptimes, incase both apps are somehow leaking memory and the critical point is somewhere near the uptime marker of 600000 where they both went offline. Both apps are consistently using ~80% memory as per vitals prior to disconnect.

---

<div class="post-metadata">

**Author:** ![robc](https://avatars.discourse-cdn.com/v4/letter/r/e9a140/32.png) [@robc](https://community.particle.io/u/robc)\
**Post date:** [August 3, 2020, 2:44pm UTC](https://community.particle.io/t/borons-gone-offline/57253/2 "2020-08-03T14:44:54Z")

</div>

What is your keepAlive() setting?

// SYNTAX  
Particle.keepAlive(23 \* 60); // send a ping every 23 minutes

At first glance, it may be that the cloud did not acknowledge two default (23 minute) keepAlive() pings. So, the cloud may have timed your devices out and assigned them **offline** status.

[https://docs.particle.io/reference/device-os/firmware/boron/#particle-keepalive-](https://docs.particle.io/reference/device-os/firmware/boron/#particle-keepalive-)

If the above is what is happening to you, you may want to set your keepAlive() manually with a more frequent ping. Or, (what I do) send Particle.publish() occasionally with a NO\_ACK parameter. This last method reduces the overhead of data a bit.

Hopefully, you might monitor the console and find your Borons go back **online** (for X reason). Then you can access them OTA without a physical visit to reset.

Took me a while to find this:

> [@Detect cellular device offline within a short period of time](https://community.particle.io/t/detect-cellular-device-offline-within-a-short-period-of-time/56561/6):
>
> Now that I think about it, I think you might need to miss two keep-alive pings for the device to be marked offline. Since the offline indicator doesn’t know what keep-alive you’ve set, it assumes the default setting, so 46 minutes to an hour doesn’t seem unreasonable to me.
> 
> Because keep-alive pings use ACK mode, using NO\_ACK would reduce the data usage in half.

---

<div class="post-metadata">

**Author:** ![StngBo](https://sea2.discourse-cdn.com/flex026/user_avatar/community.particle.io/stngbo/32/28934_2.png) [@StngBo](https://community.particle.io/u/StngBo)\
**Post date:** [August 3, 2020, 3:25pm UTC](https://community.particle.io/t/borons-gone-offline/57253/3 "2020-08-03T15:25:49Z")

</div>

Thanks for the reply,

I don't set the Particle.keepAlive, so I assume it is set to the default 23 mins?

> At first glance, it may be that the cloud did not acknowledge two default (23 minute) keepAlive() pings. So, the cloud may have timed your devices out and assigned them **offline** status.

So if my devices have timed out, does the particle attempt to reconnect at certain intervals? If this was the case could it explain why one of the devices briefly came back online? Could it be getting stuck somewhere and if this was the case would a simple application watchdog with particle.reset guard against this(and by guard I mean reset device) ?

---

<div class="post-metadata">

**Author:** ![Paul\_M](https://avatars.discourse-cdn.com/v4/letter/p/91b2a8/32.png) [@Paul\_M](https://community.particle.io/u/Paul_M)\
**Post date:** [August 3, 2020, 3:37pm UTC](https://community.particle.io/t/borons-gone-offline/57253/4 "2020-08-03T15:37:05Z")

</div>

I am experiencing similar issues on 1.5.2 and the new 2.0.0 beta. Whereas, 1.3.1-rc1 seems to work perfectly. I won’t go into detail right here, but the totality of my experiences suggests that Particle has worsened the Boron connection stability through software updates. It has cost me a lot of time and money.

Just this morning, I power cycled a Boron with 2.0.0 beta on it which had totally lost connection for no reason at 1am and still not reconnected 10 hours later.  
One power cycle and, what do you know, 30 seconds later and instant internet/cloud connection. This proves it is a software issue. Highly frustrating and forcing me to consider alternatives to Particle for cellular connectivity.

I am downgrading my Borons to 1.3.1-rc1 to further test the theory. On the positive note, the two remote stations I have which _haven’t_ required me to drive hours to the site only for hopeless head-scratching at Particle’s unstable LTE connection code, which have been running and reconnecting perfectly for multiple months, have been both V1.3.1-rc-1.

---

<div class="post-metadata">

**Author:** ![robc](https://avatars.discourse-cdn.com/v4/letter/r/e9a140/32.png) [@robc](https://community.particle.io/u/robc)\
**Post date:** [August 3, 2020, 6:12pm UTC](https://community.particle.io/t/borons-gone-offline/57253/5 "2020-08-03T18:12:27Z")

</div>

> [@StngBo](#):
>
> I don’t set the Particle.keepAlive, so I assume it is set to the default 23 mins?

Yes. I assume as well after reading the docs and forum! 😉 So, recommend you shorten the time between keep.Alive()'s or send a periodic ping. I do both.

> [@StngBo](#):
>
> So if my devices have timed out, does the particle attempt to reconnect at certain intervals?

It varies. Sometimes I have seen the devices go online within an hour to 3. Other times - never.

> [@StngBo](#):
>
> Could it be getting stuck somewhere and if this was the case would a simple application watchdog with particle.reset guard against this(and by guard I mean reset device) ?

Yes. As Paul\_M was referring to - things seem to have changed. I use a watchdog to look at diagnostics and force a System.reset() before my device goes offline. You might already be using the library:

> [@Getting system diagnostic data from user firmware](https://community.particle.io/t/getting-system-diagnostic-data-from-user-firmware/43021):
>
> The system diagnostics feature in Particle Device OS 0.8.0 can make troubleshooting device problems easier. For example, when viewing a device in [the console](https://console.particle.io), you can view the device vitals: [devicevitals] This is periodically updated automatically so you don’t have to worry about it. You can also use the cloud API to [request a refresh of the device vitals](https://docs.particle.io/reference/api/#refresh-device-vitals). You can also monitor the [device vitals event](https://docs.particle.io/reference/api/#device-vitals-event) by subscribing to it. But what if you want to get at some of this information from your us…

Some example statements I use from the DiagnosticHelperRK library are:

g\_cloudConnectionStatus = DiagnosticsHelper::getValue(DIAG\_ID\_CLOUD\_CONNECTION\_STATUS);  
// connected = 2 connecting = ??????

and

g\_cloudConnectionErrorCode = DiagnosticsHelper::getValue(DIAG\_ID\_CLOUD\_CONNECTION\_ERROR\_CODE);  
// error code = 0 (no error) 10(?????) 17(?????)

One last option you might consider: [How to force a handshake for OTA updates - #30 by ScruffR](https://community.particle.io/t/how-to-force-a-handshake-for-ota-updates/36836/30)

Last, is an example I use while connected to cloud to force a new session (force a handshake):

Particle.publish("spark/device/session/end", "", PRIVATE);  
// will FORCE NEW SESSION AFTER DISCONNECT!!!!!!!!!  
softDelay(1000);

---

<div class="post-metadata">

**Author:** ![testboron](https://avatars.discourse-cdn.com/v4/letter/t/a88e4f/32.png) [@testboron](https://community.particle.io/u/testboron)\
**Post date:** [August 3, 2020, 6:43pm UTC](https://community.particle.io/t/borons-gone-offline/57253/6 "2020-08-03T18:43:57Z")

</div>

I am facing the same problem. Two of our LTE boron device lost connection to the cloud on August 1 and August 2nd which are in the field firmware version 1.5.1.

Both Borons normally operate while connected to the cloud. I don’t know how to recover it now. Device were publishing at the interval of 2 minutes. Nothing strange in the vitals.

App usage is ~74% memory as per vitals prior to disconnect. Is there way to recover it? Both device didn’t go into safe-mode due to app crash. What else could cause this problem?

---

<div class="post-metadata">

**Author:** ![robc](https://avatars.discourse-cdn.com/v4/letter/r/e9a140/32.png) [@robc](https://community.particle.io/u/robc)\
**Post date:** [August 3, 2020, 7:58pm UTC](https://community.particle.io/t/borons-gone-offline/57253/7 "2020-08-03T19:58:21Z")

</div>

If a device is presently publishing every 2 minutes and you can view this data in the console, when you ping the device is there a response of **connected**? Just curious. Sounds like it is connected, at least.

Try forcing a new session.

I am a Mac user. You can try to force a new session by using Terminal.

At console, for device in question:

Click on “View events from a terminal” icon. Using some scratchpad app, copy and paste 2 values into following curl statement:

curl -X PUT [https://api.particle.io/v1/devices/](https://api.particle.io/v1/devices/) **device\_id** /disconnect?access\_token= **latest\_access\_token**

device\_id & latest\_access\_token (make sure there are no extra characters present after your paste)

Launch Terminal.

Copy entire modified curl statement and paste into Terminal command line. Hit return.

Hopefully, a new handshake will occur (Force New Session).

Note: access\_token changes periodically so don’t try to use same token for more than 15 minutes

---

<div class="post-metadata">

**Author:** ![testboron](https://avatars.discourse-cdn.com/v4/letter/t/a88e4f/32.png) [@testboron](https://community.particle.io/u/testboron)\
**Post date:** [August 3, 2020, 8:04pm UTC](https://community.particle.io/t/borons-gone-offline/57253/8 "2020-08-03T20:04:54Z")

</div>

Thanks robc. Two devices are offline as of August 1st 9.00 pm and August 2nd 7.45 am and its not connected to particle cloud.

Will new session help to recover the unit which are offline?

---

<div class="post-metadata">

**Author:** ![robc](https://avatars.discourse-cdn.com/v4/letter/r/e9a140/32.png) [@robc](https://community.particle.io/u/robc)\
**Post date:** [August 3, 2020, 9:15pm UTC](https://community.particle.io/t/borons-gone-offline/57253/9 "2020-08-03T21:15:46Z")

</div>

Well, I don't think fns will do any good if they are truly offline.

I am sorry. I misread your post. I don't have any other suggestions.

> [@testboron](#):
>
> Device were publishing at the interval of 2 minutes. Nothing strange in the vitals.

My brain said "are" instead of "were".

---

<div class="post-metadata">

**Author:** ![StngBo](https://sea2.discourse-cdn.com/flex026/user_avatar/community.particle.io/stngbo/32/28934_2.png) [@StngBo](https://community.particle.io/u/StngBo)\
**Post date:** [August 4, 2020, 2:02pm UTC](https://community.particle.io/t/borons-gone-offline/57253/10 "2020-08-04T14:02:39Z")

</div>

Hey rob,

Thanks for your replies. Can I ask what do you use those statements for from the _DiagnosticHelperRK_ lib? Do you the post those variables using a publish statement?

Also

> I use a watchdog to look at diagnostics and force a System.reset() before my device goes offline.

What are you watching with your watchdog and what thresholds are you using to decide to reset? Just the above two? Could you show me how you structured your watchdog, no worries if you can't.

EDIT: One other thing I have noticed while looking into the diagnostics data is that the **device.cloud.coap.transmit** value seemed steady at around the ~30, but leading up to the disconnection on the 01 Aug this value increased to ~3500.

---

<div class="post-metadata">

**Author:** ![robc](https://avatars.discourse-cdn.com/v4/letter/r/e9a140/32.png) [@robc](https://community.particle.io/u/robc)\
**Post date:** [August 4, 2020, 6:03pm UTC](https://community.particle.io/t/borons-gone-offline/57253/11 "2020-08-04T18:03:43Z")

</div>

Sure. I posted something similar this morning so will link it here:

> [@Stable cellular reconnection newly ruined with Boron 2.0.0-rc1, and perhaps earlier](https://community.particle.io/t/stable-cellular-reconnection-newly-ruined-with-boron-2-0-0-rc1-and-perhaps-earlier/57264/2):
>
> I understand the frustration. I am using 1.5.2 on my customers Boron gateway to Xenon mesh network. With some proactive code to monitor the condition of the current Particle cloud connection on all my devices I am able to avoid (so far) long term lockouts from the cloud when a device goes offline. This solution ain’t perfect. roll_eyes (Please, from anyone, I need no further reminders of the deprecation of the mesh firmware. wink) I am occasionally reseting devices or forcing a new sessi…

I have posted the variables, in the past, using Particle.publish() so I could see events evolve in the Events console, for example. Now, I just use the two above statements in a small routine called every 3 minutes from loop(). If abnormal conditions persist, as in the above post link, for 3 minutes I typically call System.reset(). At present, my devices reset every 2 or 3 days on average.

Interesting note you made about the **device.cloud.coap.transmit** value. I will take a closer look at that, too. I might write a set of " **conditions**" to EEPROM so I can see what triggered a device last restart.

---

<div class="post-metadata">

**Author:** ![StngBo](https://sea2.discourse-cdn.com/flex026/user_avatar/community.particle.io/stngbo/32/28934_2.png) [@StngBo](https://community.particle.io/u/StngBo)\
**Post date:** [August 4, 2020, 6:08pm UTC](https://community.particle.io/t/borons-gone-offline/57253/12 "2020-08-04T18:08:11Z")

</div>

Thanks for being so helpful robc, I will let you know how the above treats me. I will be visiting the device on site probably on Friday so I will revert and let you know what sate the machines seems to be in when I arrive.

---

<div class="post-metadata">

**Author:** ![StngBo](https://sea2.discourse-cdn.com/flex026/user_avatar/community.particle.io/stngbo/32/28934_2.png) [@StngBo](https://community.particle.io/u/StngBo)\
**Post date:** [August 6, 2020, 1:14pm UTC](https://community.particle.io/t/borons-gone-offline/57253/13 "2020-08-06T13:14:50Z")

</div>

Hi robc

I was thinking of using something like this and just calling it every x mins to check for an error. Do you think it might work well? Do you usually check for both conditions, ie. && vs || ?

Thanks again for your help.

```auto
void cloudWatchDog(){
    if( millis()-lastChecked > checkPeriod){
        int prev_g_status = g_status;
        int prev_g_errorCode = g_errorCode;
        // values of diag variables from last time checked
        
        g_status = DiagnosticsHelper::getValue(DIAG_ID_CLOUD_CONNECTION_STATUS);             
        // **connected** = 2 connecting = ??????
        g_errorCode = DiagnosticsHelper::getValue(DIAG_ID_CLOUD_CONNECTION_ERROR_CODE);      
        // error code = 0 ( **no error** ) 10(?????) 17(?????)
        
        currentConnErrExists = (g_status!=2 && g_errorCode!=0); 
        // TRUE if cloud is not connected and there is an error code
        errorExistedLastCheck = ((prev_g_status==g_status) && (prev_g_errorCode==g_errorCode));
        // TRUE if errors existed last time they were checked. 
        
        if (currentConnErrExists && errorExistedLastCheck){
            System.reset();
        } else {
            lastChecked=millis();
        }
    }
}

```

**EDIT:** Fixed copy and paste errors as identified below

---

<div class="post-metadata">

**Author:** ![robc](https://avatars.discourse-cdn.com/v4/letter/r/e9a140/32.png) [@robc](https://community.particle.io/u/robc)\
**Post date:** [August 6, 2020, 2:21pm UTC](https://community.particle.io/t/borons-gone-offline/57253/14 "2020-08-06T14:21:14Z")

</div>

Glad you asked. I was thinking about what I posted the other day and checked my actual code. I am presently checking **only** DIAG\_ID\_CLOUD\_CONNECTION\_ERROR\_CODE. I will probably change the code similar to what you are proposing when I get a chance. I am presently hunting down the GitHub codes for the mysterious DIAG\_ID\_CLOUD\_CONNECTION\_ERROR\_CODE just for the heck of it. So, checking for

`g_status!=2 && g_errorCode!=0`

is a good idea.

Two evenings ago, there was reported system degradation which caused all my Argon devices to rapid flash green. Once the system recovered I noticed DIAG\_ID\_CLOUD\_CONNECTION\_STATUS was a value of “1” for connecting

`// DISCONNECTED = 0 CONNECTING = 1 connected = 2 DISCONNECTING = 3`

even though I was having no trouble interacting with my devices. I tried to force a new session (handshake) but no change. I issued a Particle.function() command for a device to System.reset() which cleared up codes to what we would expect to receive after restart: **connected = 2** && **error code = 0**. Your method would have cleaned up this issue for me right away. So, thank you!

You probably have already found it but I think there is an error near the top of cloudWatchDog().

`int prev_g_errorCode = g_status;`

should be

`int prev_g_errorCode = g_errorCode;`

then, further down:

`errorExistedLastCheck = ((prev_g_status==g_status) && (prev_g_errorCode==g_status));`

should be

`errorExistedLastCheck = ((prev_g_status==g_status) && (prev_g_errorCode==g_errorCode));`

---

<div class="post-metadata">

**Author:** ![StngBo](https://sea2.discourse-cdn.com/flex026/user_avatar/community.particle.io/stngbo/32/28934_2.png) [@StngBo](https://community.particle.io/u/StngBo)\
**Post date:** [August 6, 2020, 2:33pm UTC](https://community.particle.io/t/borons-gone-offline/57253/15 "2020-08-06T14:33:13Z")

</div>

Haha, yeah caught one of these copy and paste errors but missed the other one, think I need to get away from the screen for a while. Thanks for your help, well spotted. I’ll do some testing and let you know

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex026/uploads/particle/original/3X/3/2/3291e3c9351789fe8620ae5bcec8d3fec67ff221.png) [@system](https://community.particle.io/u/system)\
**Post date:** [February 5, 2021, 2:33am UTC](https://community.particle.io/t/borons-gone-offline/57253/16 "2021-02-05T02:33:16Z")

</div>

Closed on February 5, 2021, 2:33am UTC

This topic was automatically closed 182 days after the last reply. New replies are no longer allowed.