Boron LTE Stuck Flashing Green

boron
Tags: #<Tag:0x00007fe22b1d7dc8>

#61

Interesting. I’m now having a reconnection problem on my main development Boron LTE sitting on my desk. It has a high gain external antenna attached, and has been powered up and connected for about two weeks, including periodic resets. I pulled the USB connection to test battery life (2000mAH LiPo) and let ran it until the battery drained. It was offline for a day or so before I reconnected power. Now it will not connect with or without the battery attached. I’m using a Hologram SIM. I have tried firmware 1.2.1-rc.3 and 1.3.0-rc.1. Same behavior on both.

Behavior I’m seeing:

  • At startup AT commands may complete as expected, or they may take 30+ seconds (e.g. AT+CGDCONT=2,“IP”,“hologram”
  • The modem does not connect, and after several minutes my code resets the Boron to try again
  • I now see two items in the serial log. “ERROR: Failed to power off modem” and “ERROR: No response from NCP”. This occurs even after a full power cycle.

The location, antenna, circuitry, etc. did not change from the time it connected successfully to now.

I’m wondering if the modem has blacklisted my local towers for some reason. I don’t recall the command to clear the blacklist, but will look into it. Should I clear that on startup? I understand that will result in longer initial connection times, but if it will connect more reliably then it is worth it.

Any other suggestions?

EDIT: I just realized I reflashed 1.2.1-rc.3 instead of 1.3.0-rc.1. I’ll test with that version and update again.

EDIT 2: Same behavior with 1.3.0-rc.1.

EDIT 3: Here is the log from the Boron

...Reconnected to /dev/cu.usbmodem142301 ...
0000005491 [app] INFO: picsil Sense is running! ID e00fce6864c5f1a4f2c0606a
0000005492 [app] INFO: Firmware version 2.0.118, OS version 1.3.0-rc.1
0000005492 [app] INFO: Reset Reason: 0x8c
0000005493 [app] INFO: Free Memory 61416 bytes
0000005494 [app] INFO: Initializing Display
0000005647 [app] INFO: >> AT+CGDCONT=2,"IP","hologram"
0000011902 [hal] ERROR: Failed to power off modem
0000032903 [hal] ERROR: No response from NCP
0000103903 [app] INFO: >> AT+URAT=7
0000163903 [app] INFO: >> AT+UMNOPROF=0
0000223903 [app] INFO: >> AT+CFUN=1
0000283904 [app] INFO: Connecting to network...
0000386948 [gsm0710muxer] ERROR: The other end has not replied to keep alives (TESTs) 5 times, considering muxed connection dead
0000398599 [hal] ERROR: Failed to power off modem
0000419699 [hal] ERROR: No response from NCP
0000523904 [app] INFO: No cellular connection ...connection lost to /dev/cu.usbmodem142301 ...
...Reconnected to /dev/cu.usbmodem142301 ...
0000005487 [app] INFO: picsil Sense is running! ID e00fce6864c5f1a4f2c0606a
0000005488 [app] INFO: Firmware version 2.0.118, OS version 1.3.0-rc.1
0000005489 [app] INFO: Reset Reason: 0x8c
0000005490 [app] INFO: Free Memory 61480 bytes
0000005490 [app] INFO: Initializing Display
0000005644 [app] INFO: >> AT+CGDCONT=2,"IP","hologram"
0000011898 [hal] ERROR: Failed to power off modem
0000032899 [hal] ERROR: No response from NCP
0000103899 [app] INFO: >> AT+URAT=7
0000163899 [app] INFO: >> AT+UMNOPROF=0
0000223900 [app] INFO: >> AT+CFUN=1
0000283900 [app] INFO: Connecting to network...
0000396939 [gsm0710muxer] ERROR: The other end has not replied to keep alives (TESTs) 5 times, considering muxed connection dead
0000408590 [hal] ERROR: Failed to power off modem
0000429691 [hal] ERROR: No response from NCP
0000523900 [app] INFO: No cellular connection af...connection lost to /dev/cu.usbmodem142301 ...
...Reconnected to /dev/cu.usbmodem142301 ...
0000005487 [app] INFO: picsil Sense is running! ID e00fce6864c5f1a4f2c0606a
0000005488 [app] INFO: Firmware version 2.0.118, OS version 1.3.0-rc.1
0000005489 [app] INFO: Reset Reason: 0x8c
0000005490 [app] INFO: Free Memory 61480 bytes
0000005490 [app] INFO: Initializing Display
0000005644 [app] INFO: >> AT+CGDCONT=2,"IP","hologram"
0000011899 [hal] ERROR: Failed to power off modem
0000032900 [hal] ERROR: No response from NCP
0000103900 [app] INFO: >> AT+URAT=7

#62

Just wanted to thank everyone who is chiming in here!!! It has been really helpful to get you guys behind this thread and get it the attention it deserves. I am in contact with the Engineering Team at Particle, and for a couple weeks my Borons began to play nice. I wish I could say that was the end to this story.

The guys at Particle did send me a message on here which I must have missed with how busy this thread got, I was getting so many messages from this thread, it was hard to keep track of them in email. They offered to step up here with a refund, and I just replied this morning, as of course, in the last 72 hrs, these issues began to reappear for me overnight, while I was asleep.

I woke up to a bunch of green flashing lights. My initial suspicion this time, was a cell tower issue or the Particle backend was down, as ATT did have some reported outages in here in San Diego, but after waiting, and waiting some more, and then updating to from 1.2.1RC1 to 1.2.1GM, and then onto 1.3.0RC1. I can confirm that this issue has not been fixed in either one, and its been 3days, which is way too long to have these be dropping connection for in my book.

The issue does not appear to be exactly the same as it was originally for me though. Originally, my devices were stuck green and would never connect to the cloud. Now they will connect, but for very short periods of time, and then back to flashing green. I was barely able to get flashes to them before they disconnected again. As this issue is still occurring with a new bootloaders and TINKER installed, I know it is not related to my code. Just to note, as I saw someone list it above, I am using Semi Automatic and the System Thread is Enabled. Turning off Semi Automatic did not make a difference for me, just FYI.

While I hope the team at Particle, will take me up on my offer to work for them, even if they don’t, this is the sad truth to this story. If the team at Particle, cannot get the huge stability issues, fixed with their Borons, and their entire backend up to snuff, and I mean like RIGHT NOW, I will have to scrap their controllers and go another route, which will not only be a mess, but may also sink this product completely. I have been very patient, and continue to be, but at this point, I don’t care who you need to hire or fire, but this need to be fixed, NOW.

I am not going to expose the company I am putting my blood sweet and tears into, legally, for selling unreliable or defective devices, because I was shipped bad hardware or bad firmware updates, to good working hardware, as the same exact Boron hardware that started this thread, used to be rock solid in the locations I am having issues in now.

So, Mr. CEO, if you are keeping an eye on this thread, I don’t care, if it means that NFC and Bluetooth support, have to stay in beta a whole month longer, you absolutely, need to dedicate every single engineer you can possibly muster, to fixing these issues, as the quite literally, the fate of multiple companies is now in your hands, as it has been made abundantly clear in this thread, by not just me, but others too!!!

Please don’t take that for granted!!! If you want to market your business as helping creators, get products to market, and want people to sign contracts for priority service, then at least show us, that you can tackle the small stuff, like making your own stuff work right!!!

Thank you again, to everyone for jumping on this, to help confirm I’m not alone here!!! Best of luck with your own products and projects!!!
-Spencer


#63

Hi Spencer,

It sounds like you’ve got multiple threads going with the folks at Particle (myself included). Lets focus on digging into the root cause of what issues you’re seeing in those threads if that’s okay. What’s helpful in the forums are specific symptoms, device logs, code examples, schematics, and so on. I know in this case you mentioned you’re running tinker, and some system firmware versions which is helpful, thanks!

What signal strength are you seeing on your devices? How are they powered? Are they in a breadboard, or foam, or in a custom circuit board? Which antenna are you using? How is it mounted? When you mentioned waiting, what did you try? Did you reset or unplug / replug your devices?

It’s worth mentioning here that while we’re happy to help dig in, there are just so so so many variables that we need to understand in your case to help debug. Your device being offline for some amount of time could be your power supply, usb cable, battery, mounting, positioning, firmware, antenna selection, pcb, workstation, local carrier, local interference, carrier to internet, etc, etc. Your device could be operating outside the environmental or power conditions in the datasheet, etc, etc.

We have an awesome support team, and an awesome engineering team that try to make the best devices and firmware and experience, and we do keep an eye on threads where folks say things aren’t working. The more specific actionable information we can gather about something like this, the better we can understand the root cause and if there is something we can fix / change to make your experience better.

Thanks,
David


#64

Dave,
Did you receive the email from earlier today?

My setup has not changed since I began this thread, and ordered the additional Borons to confirm it was never the hardware, and was your firmware/bootloader/modem firmware, or issues with you carriers, and the issues is with all the borons in the exact same places they had been working fairly stabling for about a month, with only the occasional disconnect, maybe once a week, but they would reconnect on their own when I let them be, but this is still, nowhere near where it was from a stability standpoint, prior to 1.2.+ firmwares. My power did not go out. I was asleep and had not touched the firmware or changed any code, since this issue finally mysteriously went away with zero explanation from you guys a few weeks ago, again with zero changes it went away on my end and was fine, but now it’s back again. Right around the time you guys put out 1.3.0RC1 is when these issues began again, even though these devices where all still running 1.2.1RC1. It wasn’t until they has been offline for 3 days that I tried rebooting them and upgraded the firmware and bootloaders and started testing with only tinker, again to make sure it wasn’t anything i wrote, and to make sure it could not be blamed on me for anything i changed on my end.

I also have now explained this in 3 places, counting here and the first two are private, one earlier today, which is where these kinds of discussions should probably be had, I would really appreciate it, if you would just email me, as you wont call me, and also please touch base with Matt or have him copy you in on my conversation with him, so you have all the data. I know this is the weekend and I very much appreciate you reaching out, but I am getting different conflicting instructions from both you and Matt in different places too, so I am trying understand what I can do to help you make this better faster, as I feel I have given you all the data I can. If you need me to do logging I can send you them, I just need to know which one of you wants to take the lead on this, so I am not having two conversations about the same thing with two different people simultaneously. Again I appreciate the effort, but who should I send them to or be dealing with here to get the fastest results??

My only idea at this point, is to look into the modem firmware for some bugs or talk to your sim carriers providers, because this issue seems to primarily be impacting the LTE devices only right now. If you would like I can contact ATT, as I believe that is your sim provider on the builtin sim to inquire about outages, but at this point, I’m am having a really hard time believing this would last this long.

Help me to help you :grin:


#65

Hi @sdevo619

I provided more information on your concerns in private. In short, There are currently no known causative links to device OS and the issues you are seeing.

A general idea of impacted use cases is known. Exact circumstances leading to bad behavior is still unknown, but is currently being investigated. There is no blame being cast for the issues here. We acknowledge that issues exist and wish to only understand them.

Both Dave and I are happy to assist you moving forward. However, for sake of simplicity, feel free to consider me the primary contact on this matter moving forward. :slight_smile:

As a follow up to Dave’s input: Dave’s instructions are certainly useful. It is good to understand any changes that may have occurred along the way. As your setup has not changed much, I believe we may not need to collect much in terms of additional information at this time–at least for the Borons.

From our direct message conversation, I understand there are some concerns about the Electron as well. The issues observed here and what is known at this point does not suggest they are impacting the Electron. As such, Dave’s suggestions are recommended–as we should be considering this a different issue with no pre-existing context.

To my understanding of what is known at this time, the hardware is currently undergoing a deep dive by our engineering team. As hard as it is, I’m afraid the best option here is to wait patiently for updates. As we receive updates with actionable test points, we can test them as soon as possible to confirm progress has been made.