I'm trying to figure out possible causes, and remote fixes, for a couple of my Boron units. Unit A was previously placed at a remote location (about 100 miles from my work location) for monitoring and data gathering. It's been working great for several months. Unit B was developed with an updated version of the program and a new sensor. It was working great at my work location for the last couple of weeks of "final" testing for the new version.
Upon taking Unit B to the remote site, it wouldn't (and still isn't, for over a week) connect to cellular. It blinks green for a few minutes before giving up and rebooting. I assumed this was some quirk that would resolve itself, because it's happened before and takes a couple hours to sort out on the cellular network.
But then, upon getting Unit A back to my desk, it also won't connect anymore. It's been stable, it runs the same code as before, but the act of bringing it back has caused the same problem.
Both devices are now unable to connect, blink green, and reboot every few minutes. Each one was turned off for its trip, and I'm wondering if there's some kind of location/roaming limit to their LTE-M data plans. I also haven't tried anything locally on Unit A with the Device Doctor, because I'd really like to fix the "production" Unit B device remotely (and therefore want to use Unit A to find the fix).
Has anyone seen this before? Does anyone know why it would happen like this? I really didn't expect this to show up, and unfortunately could not be at the remote site long enough to ensure it was fixed after a day or two (or to use Device Doctor on it). I also didn't expect the returned one to break, which was very interesting and makes me think it's something with the subscription and network.
Thanks!
Even if the device had cellular connectivity problems, the device should never reboot on its own. It will power down the modem after 10 minutes to reset it but the device itself won't reset and it will stay blinking green the whole time.
Is it resetting with a SOS before reboot? That would indicate a software fault of some sort.
Is it resetting with no SOS pattern? That often means a power issue.
Do you have code to force a reset if you appear to be failing to connect? You should wait at least 10 minutes for the device to reset the modem itself first. If you do it at less than 5 minutes, the SIM could require an IMSI switch, which will never occur because the device is reset too soon, which will cause it to never get past blinking green.
If the device fails to connect, but does connect if you hit the reboot button, the causes are different.
Wow, thanks for the quick reply. That's very interesting the Boron isn't supposed to reboot like that. I've seen it do that across a couple of my little test programs. Which, since several of them look similar, might just be because I have the same mistake in all of them...
- There is no SOS pattern on reboot. It's doing the medium-speed green blink, then blips to white on reboot, then back to green for a few minutes.
- It could be a power issue, but the boards each have a charged battery (only about a year old) and use two different methods of power (AC-USB adapter and computer USB).
- I double-checked my code and I don't have anything explicitly calling for a reboot. I do have timeout-type things, but they are either soft timeouts that print a message or infinite loop timeouts if something is very wrong at startup (such as a required sensor missing).
Two additional notes and some data:
- When you say reboot button, I'm just making sure that's the same thing as the reset button? I have not tried hitting reset as my carrier boards have a power switch, and I've used that. But I can give it a go if it's the reset button. While it's not a remote fix, it is something easy I might be able to get a remote person to do for me.
- I have a status blink in my program loop to know if everything initialized and the main loop is running. The on-board blue LED is blinking in cycle, so that means the program itself thinks it's healthy. (Excluding any of the cloud bits, which are soft-fail timeouts and tries for just such an occasion.)
To grab some logs since it's apparently not supposed to do this, I have the formerly-known-to-be-working-one (Unit A) plugged in now. Here's what it's telling me via console, if it helps. The program is at INFO level, and since I'm currently trying to find a remote fix, I don't want to flash it to DEBUG yet. Though that did just give me an idea for one of my hardware buttons.
Serial connection closed. Attempting to reconnect...
Serial monitor opened successfully:
0000000901 [mux] INFO: Stopping GSM07.10 muxer
0000000902 [mux] INFO: Gracefully stopping GSM07.10 muxer
0000000902 [mux] INFO: Closing all muxed channels
0000000903 [mux] INFO: Muxed channel 1 already closed
0000000903 [mux] INFO: Muxed channel 2 already closed
0000000903 [mux] INFO: Muxed channel 3 already closed
0000000904 [mux] INFO: Muxed channel 4 already closed
0000000904 [mux] INFO: Sending CLD (multiplexer close down)
0000001805 [mux] INFO: Received response to CLD or timed out, exiting multiplexed mode
0000001855 [mux] INFO: GSM07.10 muxer thread exiting
0000001855 [mux] INFO: GSM07.10 muxer stopped
0000001856 [mux] INFO: Starting GSM07.10 muxer
0000001858 [mux] INFO: GSM07.10 muxer thread started
0000002157 [mux] INFO: Stopping GSM07.10 muxer
0000002157 [mux] INFO: Gracefully stopping GSM07.10 muxer
0000002158 [mux] INFO: Closing all muxed channels
0000002158 [mux] INFO: Muxed channel 1 already closed
0000002159 [mux] INFO: Muxed channel 2 already closed
0000002159 [mux] INFO: Muxed channel 3 already closed
0000002160 [mux] INFO: Muxed channel 4 already closed
0000002161 [mux] INFO: Sending CLD (multiplexer close down)
0000003061 [mux] INFO: Received response to CLD or timed out, exiting multiplexed mode
0000003111 [mux] INFO: GSM07.10 muxer thread exiting
0000003111 [mux] INFO: GSM07.10 muxer stopped
0000003129 [ncp.client] INFO: Using internal SIM card
0000005493 [mux] INFO: Starting GSM07.10 muxer
0000005494 [mux] INFO: Opening mux channel 0
0000005495 [mux] INFO: GSM07.10 muxer thread started
0000005498 [mux] INFO: Opening mux channel 1
0000006036 [app] INFO: Message failed to send to cloud.
Serial connection closed. Attempting to reconnect...
Serial monitor opened successfully:
0000000901 [mux] INFO: Stopping GSM07.10 muxer
<snip, etc.>
EDIT: Between things I did some very light poking, and the reboot cycle time is right around 60 seconds. I assume it IS 60 seconds but my measurements were bad.
Since the serial connection disconnects, the device is almost certainly resetting, but it's not clear why as it should not do that.
I haven't had much time to work on this further, especially after learning that it's not supposed to be acting this way. I think the next step for my local board is to run Device Doctor, see if I can get it flashed before it crashes out, and check whether it reconnects. Your idea about an IMSI switch seems very logical because of the significantly different location; while the carrier in one place should be available in the other, I suspect it's causing issues and the device isn't staying up long enough to trigger an automatic re-try/re-search for the alternate.
One thing I can confirm is the Reset button didn't change any behavior. The serial logs look the same, as does the 60-second reboot timing.
I have a handful of other dev boards available, and I plan to use them all as additional tests when I next travel to the location. (And possibly another site even farther away for additional behavioral information.)
I'll attempt to make posts to keep this thread alive a little longer so that any useful information can be shared with others for reference.