I would expect a power cycle to fix a memory leak issue, yes.
However, we are aware of more than one issue that can lead to blinking green. The case of where a device boots up fine and goes into a blinking green state is believed to be a memory leak. The state where it initializes in blinking green is known to have multiple causes and only some are understood. Some fixes to mitigate this are also in device OS v1.3.1-rc but may not mitigate all cases. It would be helpful to upgrade to v1.3.1-rc and see if the issues are still exhibited to understand how this bad state interacts with the latest bugfixes.
How long were your power cycles?
I have noticed that a brief cycle has minimal effect. <1 sec.
However, a 5 second seems to recover things.
I am wondering if certain registers in the modem do not reset to a working state unless a long enough powering off happens to the modem.
Hi ric. Preparation occurred today for the release. A few more tests need to be run but it’s my understanding is that if everything goes well, we expect a release tomorrow.
Hi folks, we have released device OS v1.3.1-rc.1 to tackle some of the issues causing difficulty to all the users here. Please refer to Status Update for cellular connectivity issues | 8/21/19 for instructions on how to provide feedback and debug logs. Thanks!
Just reading through this thread and it is like a bad familiar dream. Without doing a bunch of digging into my past data and back and forths with Particle… I started having issues last spring with my Boron LTE and 3G devices. Dropping service. Needing to have a complete power cycle or sometimes a reset to get out of blinking green.
My field season ended I had replaced some Borons with locally logging alternatives. I spent the rest of the summer on other tasks. I assumed my Borons would work this season. So far 3/5 Borons won’t connect and just flash green. I’ve gone through identical steps to load new 1.3.1 OS on all of them including the bootloader and the soft device. 2 of my Borons actually connect faster than they used to (my house has not the best service) the others just blink green.
I went back and forth with Particle quite a bit trying to understand what was happening. Apparently I wasn’t alone. Feel like the wind has been taken out of my sails… Any updates about how 1.3.1 is working from anyone else?
The Borons that fall into a blinking green state are a result of a muxer memory leak. Any devices that are able to successfully connect should now stay connected.
For the units that only initialize flashing green and are unable to ever come online–it is likely a separate issue. I encourage you to flash Boron cloud debug to your devices and provide the debug logs in a support ticket (either a new ticket, or an existing one for your devices) if you have not already. You may find Boron Cloud debug here at: https://github.com/rickkas7/boron-clouddebug
Feel free to provide me with the ticket IDs in a DM and I will be happy to personally take a look at the situation.
@mister@mstanley Anecdotally, I can report the 1.3.1 memory leak fix almost certainly solved the eventual-infinite-never-reconnect-green-flash-ad-infinitum issue that was the root of this thread, and which was the source of my troubles above, and which is different from the never-connects issue.
I have a 1.3.1 Boron in the field where there is highly variable signal where Borons used to always infinitely drop the connection. How long they went before dropping varied between a few hours and two months, but they ALWAYS invariably entered a never-reconnect infinite-green-flashing state.
Now, with 1.3.1, my Boron faithfully reconnects during the semi-regular bouts of time where cell service vanishes at this spot. Currently my Boron is at 23 disconnects, uploading rock-solid for days. Cell service went out for like a 36 hour period last week, and it faithfully reconnected.
Insofar as this thread is concerned, i.e. regarding that particular issue, I am very, VERY satisfied to say that the issue appears to be resolved (knock on wood), and I can resume using Particle Boron.
The only source of nervousness is how a true memory leak can be solved by making some task a higher priority in the operating system. That sounds like a band-aid, and not an actual solution to a memory leak, to me. But I trust the engineers fixed the issue and there will be NO permanently disabling memory leak no matter how long a disconnection state lasts and no matter how many disconnnects come. @mstanley, do you agree with that?
Hi Paul, glad you are getting back on track. Out of your 23 drop outs are you able to tell me how long the drop outs were for (basic average per drop out 5 - 6 min? )
My application requires real time switching and the 5 min drop outs are killing me.
I am not the one responsible for the implementations myself, so I cannot speak as an authority on the memory leak fix. However, it was my understanding that the memory leak was caused as a result of the task that was meant to clear the memory never receiving enough priority to execute. By increasing priority, it was given time to run and actually de-allocate the memory.
All customers impacted by the memory leak issue have reported the behavior no longer occurs post v1.3.1-rc.1. To my knowledge, the issue is now resolved.
@ric_hard Ric, happily I will share, however please know that the etiology of these drop-outs in my case is innocuous, unrelated to Particle software issues. My location itself receives variable cell signal (very strange, perhaps a combination of atmospheric and cell tower software effects).
My drop-outs seem to occur over concentrated and extended periods of 12 - 36 hours.
As I write this, the connection has been rock-solid for a week.
The point is Particle has FIXED this issue (horray!) as mstanley said, and the reconnections seem to be guaranteed once signal is restored.
The only thing Particle needs to do now is abandon these stupid, awful, terrible u.FL connectors and make a SMA connector for cell antenna. These stupid fragile u.FL connectors have caused their own fair-share of LTE failures in my experience.
Where would you fit that on the board? SMA is much larger and through-hole. There’s no way Particle could put an SMA connector on the boards and keep the same form factor. I agree the u.FL connectors are fragile and difficult for development purposes since they’re being moved and reconnected frequently, but in production, what’s the issue?
For development purposes, I just leave a u.FL to SMA pigtail cable attached to my boards so swapping antennas is not an issue.
I understand the size limitations. And I recognize that production needs are different. I’ve figured out how to make the u.FL connector work but it does still fail and it is the cause of about 5% of my field visits. During development I have had to scrap several Electrons due to damage to the u.FL connector. I do use a pigtail and that does help during development and production, but it does still fail for my application which is prone to vibration. I don’t consider the u.FL a deal breaker. I’m still happily using Particle boards.
Fixed!
