Boron LTE Stuck Flashing Green

boron
Tags: #<Tag:0x00007fe21f36b9c0>

#41

Likely an issue with the bootloader needing to be downgraded as well. Sometimes a bootloader mismatch can cause issues.

All device OS releases on the Github page have a bootloader binary for each device. ( https://github.com/particle-iot/device-os/releases ). You can download the .bin from there and flash it to your device while it is in listening mode using the CLI command particle flash --serial <FILENAME>. while your terminal is in the same working directory as where you downloaded it.

Sounds great. Excited to hear updates on this matter.

CLI setup will be a bit more reliable here. I definitely recommend this.

Pressing reset is not the same as pulling power, I’m afraid. Reset does not cause the modem to reset.


#42

We are not even going to try boron right now due to us having problems with the e-series boards not staying connected to the cloud. We are about to scrap everything. The only devices that we have that stay online is our very old electrons. Cell connectivity and cloud connectivity should be number one priority over anything else.


#43

@darkstar2002 Thanks for sharing. It is alarming to know this is such a widespread issue and confusing that there is not even more activity on this forum about it.

My guess is that most forum posters here either aren’t using LTE or aren’t actually deploying an LTE Gen 3 device where it needs to be operational and have reliable connection 23.5/7/365. I say “23.5” because it is more than acceptable for my Boron site to lose cell connection here-and-there as it inevitably will: but it MUST return automatically without a manual intervention (power reset).

I am confident the attentive, competent professionals at Particle (see above) are actively working on this issue. While it is discomforting that there is not a timeline for a solution (for the very good reasons which Particle reps have articulated above), I am confident they are doing everything they possibly can to make sure your investment and mine in Gen 3 LTE boards is not for nothing, as it would be in its current status.


#44

Certainly agreed. Reliability is absolutely a concern for us and I am ensuring all the feedback provided here is being relayed to our internal teams.

With regards to the specific issues you are seeing, it may be helpful to provide symptoms and, if applicable, cloud-debug logs for your particular devices having issues so that we can look into them further and better understand what the causes may be.

Just to confirm, the E Series issues you are seeing are with the E Series LTE, specifically?

Feel free to provide that information, along with device IDs in a support ticket and we’ll be certain to get you taken care of.

This issue is absolutely a concern. As mentioned, there are some concerns with regards to ublox specifically that we wish to take special care of–but Particle is also engaging resources in parallel to better understand the issues still and pursue workarounds to help unblock our customers in the interim while we’re able to get definitive, concrete solutions from ublox and determine appropriate means in which to distribute them.

As I have more information from engineering as they continue their investigation, I’ll be certain to relay the findings and developments.


#45

Is this a issue with all Boron LTE devices or just Boron LTE devices used outside the USA?


#46

To my knowledge, the issue can occur with Boron LTEs inside the US as well.


#47

Unfortunately those devices are about an hour away from us running on water wells. We do not even go out to reset them anymore. Our customers go out there and have to do a complete power pull. We just get notified from them when they do it. Usually what happens is something with cell goes out for a sec or two then they never reconnect to the cloud. You can not even hit the reset button. It does nothing for the connection.

Yes these are E series LTE. We only have a few of them while we where waiting for the borons to mature.


#48

As we progress in our Boron LTE investigation, I will ensure the E Series LTE is take into consideration along the way with regards to the issues you are seeing.

When pressing the reset button, the modem is not always re-initialized. As such, it’s likely why power needs to be fully cut.


#49

I’ve been experiencing all these same issues running three Borons with OS 1.1.0, all in rural Minnesota. The funny thing for me is thattwo are less than 100 yards from each other out in a field - the one to the south is able to establish a connection/stay connected for ~12 hours before the perpetual disconnect that only a manual reset can fix (for another ~12 hours) - and the one to the north has been able to establish a connection consistently over the past 6 days, even after missing scheduled publish events. They are effectively running the same firmware, with the only difference being a string saying which part of the web hook to use.

The third Boron has been out for a couple months, and has been manually reset several times, only to go into the perpetual green blink within 36 hours or so. I also have an Electron on 1.1.0 in Wisconsin with these same symptoms.

When I get the chance, I will update the Borons to 1.2.0 and hope for the best. I sure hope this gets resolved soon - I am a researcher and I decided to go with the Borons for data collection instead of a more traditional arduino with an SD card. So far, it has only meant missing out on data (and a lot of troubleshooting). With that said, I appreciate this thread and all the responses from Particle to get this issue resolved.


#50

Well I have successfully updated the Ublox R410M-02B LTE modem firmware to L0.0.00.00.05.08. At this stage it has not solved my connection loss as I was hoping it would. I am still seeing 5 - 6 minute dropouts. I have one boron running 1.2.1-rc.3 and the other running 1.3.0-rc.1 as there are some changes in there for the modem update I see.

To update the first modem I soldered the test points and did it that way, the second time I made up a little jig to make it easier. I’ll re do the jig when I have time but still works good and no soldering to the boron.

The 6 min drop outs are killing me especially when there is over 20 a day. That’s 2 hours offline!
I could settle for 20 x 2 min drop outs. Just got to figure out a easy way to say ping google every minute and if it times out then reset the modem. (Bandaid fix).

I am hoping for a fix soon. Maybe the talks between Ublox and the particle engineers something will click and before we know it the problem will get solved.

I’ll attach some pictures of the first test jig, the second version I have in mind will be better and even easier than this one, although it works fine.

Also attached is a screen shot of my sim portal showing just how bad the drop outs are.


#51

Hello @ric_hard,

Do you happen to know if the disconnects are just between your devices and Particle Cloud?
Or do you still have TCP/UDP connectivity to other non Particle servers?


#52

The Boron disconnects from from everything, including dropping from the cell tower.


#53

Hey folks – wanted to jump back in to let you know that the Particle team is watching this thread closely and is actively working to triage these symptoms into multiple root cause issues with the help of AT&T and u-blox.

In the meantime, if you are experiencing this issue we’d love to have your help capturing cloud debug logs that we will use to categorize and resolve these issues. You can find instructions for doing so here:

We will be a little slower to respond than usual due to the July 4 holiday, but want to reassure everyone that this is a top priority for our team. We’re committed to continuing to improve the reliability of LTE M1 connectivity and look forward to squashing some nasty :bug:s with your help over the upcoming weeks.


#54

PS - @ric_hard I’m really appreciate of the effort you’ve gone through already to add detail to what’s going on with your device. Super impressed by the u-blox updater that you made!

It looks like the device is using a 3rd party SIM – is that correct?


#55

Correct. Using a 3rd party sim TELSTRA.


#56

@ric_hard I love the modem update jig! Can you share some details on how you made it? Is the board CNC routed? What did you use for the pins?


#57

Hey mate, in order not to hijack or steer this post in the wrong direction, i’ll post all the details on this post Ublox modem firmware update that way this one can stay on track with the bigger connectivity LTE issues.


#58

@mstanley @will @Paul_M
Let me just start with, I have evidence to back this theory up :slight_smile:

So yesterday I had to drive 500km away (approx 7hrs). I decided to take one of my Boron’s with me and left one at home.
Home Boron running 1.3.0 powered by 12v.
Vehicle Boron running 1.1.1 powered by about 7v on battery.
Both are running a very similar version of code except for a few minor changes (only about what they post and when). Both are running updated Ublox firmware.

I left home about 12:30pm yesterday. (lunch time)
The Home Boron was junk - dropping its connection randomly however there were period for hours where it wouldn’t drop. A total of 57 dropouts from when I left to when I got home.

Vehicle Boron had a dropout or two before I left. On the 7hr drive it dropped out a few times, I think about 12 or so times. There was not always reception and I was travelling 100kph and it was in the boot, so I’m not too worried about that. I got into town about 7:30pm and arrived at the motel about 9pm. The Boron did not drop out once from 7:30pm to 10:30am (15 hrs) which for me is unheard of. At 10:30am I was on the road and dropping in and out of reception. On the 7 hour drive the vehicle boron dropped out about 11 times. Pretty consistent with the trip up.

Here is the weird part. I arrived home at 4:30pm to where both Boron’s are within range of each other.

The vehicle Boron is now dropping out randomly up to 35 dropouts, whilst the home Boron has been solid for the last 5.5hours still at 57 dropouts.

So… is there a possibility that the mesh is causing issues with the Boron’s? I generally have 2 Boron’s testing at most times… In my code I have called Mesh.off(); in the setup.

Sorry its long winded. I’m going to post this now and turn off the home Boron as my Final test and will update the results in the morning. In theory the vehicle Boron which is outside (but still in mesh range) should become good again… :-/

UPDATE :frowning: I guess my theory is a bust. It dropped 7 times over night.
I will try and take it to another location today and leave it there. I am now starting to wonder if it is tower related.?


#59

Hey folks,

I have some Electrons experiencing a similar issue after upgrade to 1.1.0: Frequent Cloud Disconnects, then flashing green for extended periods of time, eventually resetting after something like 1-3 hours (our firmware resets itself after 2 hours offline). Occasionally one will flash cyan for a moment, then revert to flashing green.

Our Firmware: SEMI-AUTOMATIC mode, SYSTEM_THREAD enabled.
Reconnect: every 20 minutes while connected (for possible sleep mode)
Reconnect: every 6 minutes while offline (!Particle.connected())
Modem-Reset: every hour, while offline
System-Reset: every two hours, while offline.
A system reset also resets the timer for these events.

We have the same code running on 0.6.4 with zero issues. Somewhere between 0.6.4 and 1.1.0 the connection was crippled. We’re tracking some unrelated issues with our firmware, just wanted to add that this doesn’t seem limited to LTE, seeing this with 3g as well.


#60

I will be firing up one of our boxed borons and will try I do some logging here in a few days. I want to say I think this is not isolated to the cell boards. I have several photons at home and work on different firmware versions and I can say that the newer the firmware the worse the cloud connection is. 2 of the photons are within 15ft of the AP with very strong signal readings. This might be a combination of device os and hardware issues for wifi and cell.