Boron LTE Fails to Connect

boron
Tags: #<Tag:0x00007fe21d3dac00>

#7

I was only able to flash (and have a functioning device) if I used system part 1.1.0-rc.2. Today I attempted to walk back one version at a time, starting with the bootloader and then the matching system part. I got back to 0.9.0 so that I could run the pre-built cloud debug tool.

The only thing that changed was that the modem gets hard reset and then power cycled after failing to respond (also no modem information is returned where it was previously). Then it seems to respond to AT commands but does not connect and later fails to respond to a power cycle. The hard reset is repeated and on it goes. I’ve attached the output in case you are interested.

Serial monitor opened successfully:
clouddebug: press letter corresponding to the command
a - enter APN for 3rd-party SIM card
k - set keep-alive value
c - show carriers at this location
t - run normal tests (occurs automatically after 10 seconds)
or tap the MODE button once to show carriers
c0000011679 [hal] ERROR: Failed to power off modem
starting tests...
turning cellular on...
0000036005 [hal] ERROR: No response from NCP
0000036006 [system.nm] INFO: State changed: DISABLED -> IFACE_DOWN
deviceID=<redacted>
manufacturer=
model=
firmware version=
ordering code=
IMEI=
IMSI=
ICCID=
0000117155 [app] INFO: enabling trace logging
attempting to connect to the cellular network...
0000117158 [system.nm] INFO: State changed: IFACE_DOWN -> IFACE_REQUEST_UP
0000117158 [system.nm] INFO: State changed: IFACE_DOWN -> IFACE_REQUEST_UP
0000117161 [hal] TRACE: PPP netif -> 8
0000117161 [net.ifapi] INFO: Netif pp3 state UP
0000117161 [net.ifapi] INFO: Netif pp3 state UP
0000117163 [system.nm] INFO: State changed: IFACE_REQUEST_UP -> IFACE_UP
0000117163 [system.nm] INFO: State changed: IFACE_REQUEST_UP -> IFACE_UP
0000117163 [hal] TRACE: Modem already on
0000117167 [hal] TRACE: Setting UART voltage translator state 1
0000118168 [ncp.at] TRACE: > AT
<repeated many times...>
0000137174 [ncp.at] TRACE: > AT
0000138175 [hal] ERROR: No response from NCP
0000138175 [hal] ERROR: No response from NCP
0000138176 [hal] TRACE: Setting UART voltage translator state 0
0000138177 [hal] TRACE: Hard resetting the modem
0000149177 [hal] TRACE: Powering modem on
0000149327 [hal] TRACE: Modem powered on
0000149328 [net.pppncp] TRACE: Failed to initialize ublox NCP client: -210
0000149329 [hal] TRACE: Setting UART voltage translator state 0
0000149330 [hal] TRACE: Powering modem off
0000149331 [hal] TRACE: Setting UART voltage translator state 0
0000160932 [hal] ERROR: Failed to power off modem
0000160932 [hal] ERROR: Failed to power off modem
0000161033 [hal] TRACE: Modem already on
0000161035 [hal] TRACE: Setting UART voltage translator state 1
0000162036 [ncp.at] TRACE: > AT
0000162039 [ncp.at] TRACE: < OK
0000163040 [hal] TRACE: NCP ready to accept AT commands
0000163041 [ncp.at] TRACE: > AT+UGPIOC?
0000163046 [ncp.at] TRACE: < +UGPIOC:
0000163047 [ncp.at] TRACE: < 16,255
0000163047 [ncp.at] TRACE: < 19,255
0000163048 [ncp.at] TRACE: < 23,0
0000163049 [ncp.at] TRACE: < 24,255
0000163050 [ncp.at] TRACE: < 25,255
0000163051 [ncp.at] TRACE: < 42,255
0000163052 [ncp.at] TRACE: < OK
0000163053 [ncp.at] TRACE: > AT+UGPIOR=23
0000163059 [ncp.at] TRACE: < +UGPIOR: 23,1
0000163059 [ncp.at] TRACE: < OK
0000163060 [hal] INFO: Using internal SIM card
0000163060 [hal] INFO: Using internal SIM card
0000163062 [ncp.at] TRACE: > AT+CPIN?
0000163067 [ncp.at] TRACE: < +CPIN: READY
0000163068 [ncp.at] TRACE: < OK
0000163069 [ncp.at] TRACE: > AT+CCID
0000163075 [ncp.at] TRACE: < +CCID: 89014103271226607248
0000163076 [ncp.at] TRACE: < OK
0000163077 [ncp.at] TRACE: > AT+COPS=2
0000163090 [ncp.at] TRACE: < OK
0000163091 [ncp.at] TRACE: > AT+CEDRXS=0
0000163096 [ncp.at] TRACE: < OK
0000163097 [ncp.at] TRACE: > AT+CPSMS=0
0000163101 [ncp.at] TRACE: < OK
0000163102 [ncp.at] TRACE: > AT+CEDRXS?
0000163107 [ncp.at] TRACE: < +CEDRXS:
0000163108 [ncp.at] TRACE: < OK
0000163109 [ncp.at] TRACE: > AT+CPSMS?
0000163126 [ncp.at] TRACE: < +CPSMS:0,,,"01100000","00000000"
0000163127 [ncp.at] TRACE: < OK
0000163128 [ncp.at] TRACE: > AT+CMUX=0,0,,1509,,,,,
0000163137 [ncp.at] TRACE: < OK
0000163138 [gsm0710muxer] INFO: Starting GSM07.10 muxer
0000163138 [gsm0710muxer] INFO: Starting GSM07.10 muxer
0000163140 [gsm0710muxer] INFO: Openning mux channel 0
0000163140 [gsm0710muxer] INFO: Openning mux channel 0
0000163140 [gsm0710muxer] INFO: GSM07.10 muxer thread started
0000163140 [gsm0710muxer] INFO: GSM07.10 muxer thread started
0000163195 [gsm0710muxer] INFO: Resuming channel 0
0000163195 [gsm0710muxer] INFO: Resuming channel 0
0000163196 [gsm0710muxer] INFO: Openning mux channel 1
0000163196 [gsm0710muxer] INFO: Openning mux channel 1
0000163297 [gsm0710muxer] INFO: Resuming channel 1
0000163297 [gsm0710muxer] INFO: Resuming channel 1
0000163299 [gsm0710muxer] INFO: Resuming channel 1
0000163299 [gsm0710muxer] INFO: Resuming channel 1
0000163301 [ncp.at] TRACE: > AT
0000163351 [ncp.at] TRACE: < OK
0000163352 [hal] TRACE: NCP state changed: 1
0000163353 [net.pppncp] TRACE: NCP event 1
0000163354 [hal] TRACE: Muxer AT channel live
0000163355 [hal] TRACE: PPP thread event LOWER_DOWN
0000163356 [hal] TRACE: PPP thread event ADM_DOWN
0000163358 [hal] TRACE: PPP thread event ADM_UP
0000163360 [hal] TRACE: State NONE -> READY
0000163360 [ncp.at] TRACE: > AT+CIMI
0000163401 [ncp.at] TRACE: < 310410122660724
0000163402 [ncp.at] TRACE: < OK
0000163403 [ncp.at] TRACE: > AT+CGDCONT=1,"IP","10569.mcs"
0000163451 [ncp.at] TRACE: < OK
0000163451 [ncp.at] TRACE: > AT+CEREG=2
0000163501 [ncp.at] TRACE: < OK
0000163501 [hal] TRACE: NCP connection state changed: 1
0000163502 [net.pppncp] TRACE: NCP event 2
0000163503 [net.pppncp] TRACE: State changed event: 1
0000163504 [ncp.at] TRACE: > AT+COPS=0
0000163504 [hal] TRACE: PPP thread event LOWER_DOWN
0000163551 [ncp.at] TRACE: < OK
0000163551 [ncp.at] TRACE: > AT+CEREG?
0000163601 [ncp.at] TRACE: < +CEREG: 2,3
0000163602 [ncp.at] TRACE: < OK
0000178703 [ncp.at] TRACE: > AT+CEREG?
0000178751 [ncp.at] TRACE: < +CEREG: 2,3
0000178752 [ncp.at] TRACE: < OK
0000193852 [ncp.at] TRACE: > AT+CEREG?
0000193901 [ncp.at] TRACE: < +CEREG: 2,3
0000193901 [ncp.at] TRACE: < OK
0000209002 [ncp.at] TRACE: > AT+CEREG?
0000209051 [ncp.at] TRACE: < +CEREG: 2,3
0000209051 [ncp.at] TRACE: < OK
0000224153 [ncp.at] TRACE: > AT+CEREG?
0000260850 [gsm0710muxer] ERROR: The other end has not replied to keep alives (TESTs) 5 times, considering muxed connection dead
0000260850 [gsm0710muxer] ERROR: The other end has not replied to keep alives (TESTs) 5 times, considering muxed connection dead
0000260855 [gsm0710muxer] INFO: Stopping GSM07.10 muxer
0000260855 [gsm0710muxer] INFO: Stopping GSM07.10 muxer
0000260905 [gsm0710muxer] INFO: GSM07.10 muxer thread exiting
0000260905 [gsm0710muxer] INFO: GSM07.10 muxer thread exiting
0000260908 [gsm0710muxer] INFO: GSM07.10 muxer stopped
0000260908 [gsm0710muxer] INFO: GSM07.10 muxer stopped
0000260910 [hal] TRACE: Setting UART voltage translator state 0
0000260911 [hal] TRACE: Powering modem off
0000260912 [hal] TRACE: Setting UART voltage translator state 0
0000272512 [hal] ERROR: Failed to power off modem
0000272512 [hal] ERROR: Failed to power off modem
0000272514 [hal] TRACE: NCP state changed: 0
0000272515 [net.pppncp] TRACE: NCP event 1
0000272615 [hal] TRACE: Modem already on
0000272617 [hal] TRACE: Setting UART voltage translator state 1
0000273618 [ncp.at] TRACE: > AT
<this is repeated many times...>
0000292622 [ncp.at] TRACE: > AT
0000293623 [hal] ERROR: No response from NCP
0000293623 [hal] ERROR: No response from NCP
0000293624 [hal] TRACE: Setting UART voltage translator state 0
0000293625 [hal] TRACE: Hard resetting the modem
^CSerial connection closed.

At this point, I’m going to wait for the replacement device and assume this one has a true hardware failure unless you think there are other things worth trying.


#8

In regard to the setup log, I did capture this and can see that everything succeeds until it starts waiting for the device to connect to the internet. As expected, it just cycles in this wait state because the device is not able to make a cellular connection. It finally times out.


#9

Hmm,

I’d like to get @BDub’s input on this one to see what he has to say.

If it’s nothing immediately obvious for him, we can see about having you file a support ticket and swapping this unit out, so that we can do a bit more hands on testing on this ourselves.


#10

Definitely looks like some kind of hardware failure and slightly resembling unactivated SIM. Can you put the device in listening mode and run particle serial inspect ? Let’s just make sure you have bootloader 301 and system 1101.


#11

Here is the inspect output. I noticed when running this another time a few days ago that the bootloader was 201. I don’t recall what system reported. Now it looks quite different.

Platform: 13 
Modules
  Bootloader module #0 - version 300, main location, 49152 bytes max size
    Integrity: PASS
    Address Range: PASS
    Platform: PASS
    Dependencies: PASS
  Monolithic module #0 - version 402, main location, 802816 bytes max size
    Integrity: PASS
    Address Range: PASS
    Platform: PASS
    Dependencies: PASS

#12

Thanks @trpropst, it looks like your Boron has v0.9.0-rc.3 monolithic firmware on it. Could you try upgrading to v1.1.0-rc.2 locally with the CLI and see if you get the versions I mentioned? You should be able to just flash this modular file v1.1.0-rc.2/boron-system-part1@1.1.0-rc.2.bin via CLI command particle flash --usb boron-system-part1@1.1.0-rc.2.bin

along with tinker boron-tinker@1.1.0-rc.2.bin via particle flash --usb boron-tinker@1.1.0-rc.2.bin

Unfortunately this will not let you see the logs since it’s not the monolithic version, but you should be able run particle serial inspect and try to let the device connect to the cloud.


#13

@BDub, I can do this today. I have not used the “som” builds as I thought they were for the B Series SoM parts specifically and I am using the Boron prototyping device. Do I have a fundamental misunderstanding about the firmware builds? Once you confirm the build to use I will do the update.

I have used the 1.1.0-rc.2 builds during this troubleshooting exercise, I just neglected to update prior to posting my inspect results. Here is the inspect output with the Boron 1.1.0-rc.2 firmware (not SoM).

Platform: 13 
Modules
  Bootloader module #0 - version 301, main location, 49152 bytes max size
    Integrity: PASS
    Address Range: PASS
    Platform: PASS
    Dependencies: PASS
  System module #1 - version 1101, main location, 671744 bytes max size
    Integrity: PASS
    Address Range: PASS
    Platform: PASS
    Dependencies: PASS
      Bootloader module #0 - version 201
  User module #1 - version 6, main location, 131072 bytes max size
    UUID: <long string redacted>
    Integrity: PASS
    Address Range: PASS
    Platform: PASS
    Dependencies: PASS
      System module #1 - version 1101

#14

Oops, yeah I added the -som by mistake :slight_smile: edited above

Ok that last particle serial inspect looks good. Does that still give you logs like before where the modem won’t turn off and claims it’s already on? If so I’d say we should proceed with a replacement device for you. cc: @mstanley


#15

If I flash the pre-built cloud debug tool at this point, the device goes into an error state, flashing red. I’m unable to get it in listen mode in this condition so I can no longer run the serial inspect. In order to recover, I have to flash the system part again for 1.1.0-rc.2. I assume the pre-built tool overwrites the system code with something that is incompatible with the bootloader. The pre-built tool is significantly larger than when I build it.

After recovering and flashing a version of the cloud debug tool built for 1.1.0-rc.2, I see the same log output.

deviceID=<redacted>
manufacturer=u-blox
model=SARA-R410M-02B
firmware version=L0.0.00.00.05.06 [Feb 03 2018 13:00:41]
ordering code=SARA-R410M-02B
IMEI=<redacted>
IMSI=u-blox
ICCID=<redacted>
0000021722 [app] INFO: enabling trace logging
attempting to connect to the cellular network...
0000138924 [gsm0710muxer] ERROR: The other end has not replied to keep alives (TESTs) 5 times, considering muxed connection dead
0000138924 [gsm0710muxer] ERROR: The other end has not replied to keep alives (TESTs) 5 times, considering muxed connection dead
0000150579 [hal] ERROR: Failed to power off modem
0000150579 [hal] ERROR: Failed to power off modem
0000171682 [hal] ERROR: No response from NCP
0000171682 [hal] ERROR: No response from NCP
0000305800 [gsm0710muxer] ERROR: The other end has not replied to keep alives (TESTs) 5 times, considering muxed connection dead
0000305800 [gsm0710muxer] ERROR: The other end has not replied to keep alives (TESTs) 5 times, considering muxed connection dead
0000317405 [hal] ERROR: Failed to power off modem
0000317405 [hal] ERROR: Failed to power off modem

With this configuration, I have the following additional issues:

  • The device can be put in listening mode but serial inspect usually times out.
  • The device will not go into safe mode but will go into DFU mode. When I attempt to go into safe mode the device just resets and begins running user code.

#16

To be honest I’m not sure why, but maybe that is compiled as monolithic and not compatible with the newer bootloader (which would be unusual). Maybe @rickkas7 can straighten us out.

When you flash modular system 1.1.0-rc.2 and tinker, I know you can’t see the logs but will the device connect at all?

This may likely be due to debugging interfering with the serial commands. This bug will be fixed in a future Device OS update.

Hmm, that is strange. When serial inspect doesn’t timeout, what does it look like in this mode? You might have to disable logging and recompile with that firmware to get serial inspect to work.


#17

No. This device has not connected under any circumstances since this began.

Are you asking if it will go into safe mode? I have not seen anything that allows it to go into safe mode.


#18

I’d say we’ve done our dew diligence here, thank you for attempting some soft fixes. I’ll let you and @mstanley take it from here on that replacement device.


#19

Thanks for checking into this @BDub :slight_smile:

And thank you Tom for taking the time to go through these steps.

If you’ve not already, please go ahead and file a ticket for a replacement, referencing this topic. We’ll be sending you a return label as well, as we’d like to get this unit into the hands of engineering to dig into this further and see what went wrong here.

Thanks everyone!

EDIT: As an added note, I’ll be updating the topic title to specify the Boron specifically, as I cannot say for certain this issue exists in our other R410M-02B devices. This will be helpful for others searching on this issue.


#20

@mstanley, I recently became aware of an issue with the R410M-02B that can render the device unusable. There is a FW update available from u-blox. Has Particle looked at the relevance of this issue to the Boron devices?


#21

We are aware of the issue described here. I’m uncertain of its relevance to the issue you are experiencing, but is something engineering will certainly investigate. Particle already has existing communications open with ublox in regards to known bugs and discussions on means of handling firmware update for our in-use ublox modules.


#22

Hi @mstanley,

I’ve got three devices with this problem. They blink green trying to connect to the cloud forever. They were running 1.1.0-rc.2 when encountering the issue. Updating them to 1.2.1 does not fix the issue.

clouddebug: press letter corresponding to the command
a - enter APN for 3rd-party SIM card
k - set keep-alive value
c - show carriers at this location
p - select MNO profile
t - run normal tests (occurs automatically after 10 seconds)
or tap the MODE button once to show carriers
starting tests...
turning cellular on...
0000015009 [system.nm] INFO: State changed: DISABLED -> IFACE_DOWN
0000036158 [hal] ERROR: No response from NCP
deviceID=e00fce68f378b52473295f7d
manufacturer=
model=
firmware version=
ordering code=
IMEI=
IMSI=
ICCID=
0000117160 [app] INFO: enabling trace logging
attempting to connect to the cellular network...
0000117164 [system.nm] INFO: State changed: IFACE_DOWN -> IFACE_REQUEST_UP
0000117164 [system.nm] INFO: State changed: IFACE_DOWN -> IFACE_REQUEST_UP
0000117168 [hal] TRACE: PPP netif -> 8
0000117169 [net.ifapi] INFO: Netif pp3 state UP
0000117169 [net.ifapi] INFO: Netif pp3 state UP
0000117171 [hal] TRACE: Powering modem on
0000117172 [system.nm] INFO: State changed: IFACE_REQUEST_UP -> IFACE_UP
0000117172 [system.nm] INFO: State changed: IFACE_REQUEST_UP -> IFACE_UP
0000117322 [hal] TRACE: Modem powered on
0000117323 [hal] TRACE: Setting UART voltage translator state 1
0000118325 [ncp.at] TRACE: > AT
0000119326 [ncp.at] TRACE: > AT
0000120327 [ncp.at] TRACE: > AT
0000121327 [ncp.at] TRACE: > AT
0000122328 [ncp.at] TRACE: > AT
0000123329 [ncp.at] TRACE: > AT
0000124330 [ncp.at] TRACE: > AT
0000125330 [ncp.at] TRACE: > AT
0000126330 [ncp.at] TRACE: > AT
0000127331 [ncp.at] TRACE: > AT
0000128332 [ncp.at] TRACE: > AT
0000129333 [ncp.at] TRACE: > AT
0000130334 [ncp.at] TRACE: > AT
0000131335 [ncp.at] TRACE: > AT
0000132335 [ncp.at] TRACE: > AT
0000133336 [ncp.at] TRACE: > AT
0000134337 [ncp.at] TRACE: > AT
0000135338 [ncp.at] TRACE: > AT
0000136338 [ncp.at] TRACE: > AT
0000137339 [ncp.at] TRACE: > AT
0000138340 [hal] ERROR: No response from NCP
0000138340 [hal] ERROR: No response from NCP
0000138342 [hal] TRACE: Setting UART voltage translator state 0
0000138344 [hal] TRACE: Hard resetting the modem
0000149345 [net.pppncp] TRACE: Failed to initialize ublox NCP client: -210
0000149347 [hal] TRACE: Setting UART voltage translator state 0
0000149349 [hal] TRACE: Modem already off
0000149450 [hal] TRACE: Powering modem on
0000149600 [hal] TRACE: Modem powered on
0000149601 [hal] TRACE: Setting UART voltage translator state 1
0000150603 [ncp.at] TRACE: > AT
0000151604 [ncp.at] TRACE: > AT
0000152605 [ncp.at] TRACE: > AT
0000153606 [ncp.at] TRACE: > AT
0000154607 [ncp.at] TRACE: > AT
0000155608 [ncp.at] TRACE: > AT
0000156609 [ncp.at] TRACE: > AT
0000157610 [ncp.at] TRACE: > AT

Serial Inspect output

Platform: 13 - Boron
Modules
  Bootloader module #0 - version 311, main location, 49152 bytes max size
    Integrity: PASS
    Address Range: PASS
    Platform: PASS
    Dependencies: PASS
  System module #1 - version 1213, main location, 671744 bytes max size
    Integrity: PASS
    Address Range: PASS
    Platform: PASS
    Dependencies: PASS
      Bootloader module #0 - version 311
  User module #1 - version 5, main location, 131072 bytes max size
    UUID: 362808E0DB24B4ED309227C9121675856A32E73D5D16D2C7922C49C9AEF353C2
    Integrity: PASS
    Address Range: PASS
    Platform: PASS
    Dependencies: PASS
      System module #1 - version 326

These three devices are in the field with 17 others for a R&D project we are running with Borons. All three experienced this issue after running completely out of battery. They were not powered by a LiPo battery, but were powered by a 12V SLA battery run through a regulator.

All three of these devices had this problem after running the 12V SLA completely out of power (down to ~6V). I think this may have something to do with that? Maybe a modem brownout?


#23

Hi Heath,

Those logs are fantastic, thanks so much for providing those! I have ensured that engineering will be taking a look at these.

We are still investigating this issue on our end. We have a device OS release coming sometime next week to mitigate some of the Boron LTE fail cases. As that comes out, we will be posting a new community topic for the release and instructions on how to capture additional debug logging so that we can continue to investigate connectivity concerns.


Boron LTE Stuck Flashing Green
#24

Not sure if this helps, but our carrier board can sit either a Boron or Electron and we have about 20 of each in the field. None of the Electrons have failed this way (or any other way).


#25

Hi Heath.

Certainly insightful. At the moment, our upcoming release candidate for v1.3.1 addresses a Boron specific issue. If you continue to notice this issue post v1.3.1-rc, it is helpful to know that this issue is still exclusively impacting Borons.


#26

Hi folks, we have released device OS v1.3.1-rc.1 to tackle some of the issues causing difficulty to all the users here. Please refer to Status Update for cellular connectivity issues | 8/21/19 for instructions on how to provide feedback and debug logs. Thanks!

@hwestbrook @trpropst