Consistent cellular data connection dropouts

[mattmak23412]

I have downloaded, setup and connected using a custom Esim and APN. The connection stays up and the internet can be reached for a period of time usually around 45 mins before the connection consistently drops. The modem still appears to be running? But the data connection is reported to be down and the internet unreachable.

This is a pretty major issue for my use-case to not have reliable cellular connections. I know it’s still early days but if there is any workarounds for this issue that would be great as our use-cases don’t use WiFi and Cellular was critical.

I also think the documentation for the cellular modem and Esim is pretty incomplete so far. For example I cant find out anywhere how to delete an Existing Esim profile?

Logs below for the modem status obtained from running command particle-tachyon-rild status the command wasnt documented anywhere but I managed to work it out and the logs are from just before and after the disconnection.

Sun Aug 03 07:20:59.592 AEST : Registration State: Registered, Home Network, RAT: 20
Sun Aug 03 07:21:00.600 AEST : Signal Strength: 23 (0-31) / -67dBm, RSRP: -99dBm, RSRQ: -16dB, RSSNR: 2.0dB
Sun Aug 03 07:21:00.808 AEST : MCCMNC: 50502, LAC: cb29, CID: 13c690a, ACT: 13 NR5G (5G), CARRIER: YES OPTUS YES OPTUS
Sun Aug 03 07:21:00.811 AEST : data_connection: up, status: 0, retry: -1, call_id: 0, type: IP, ifname: rmnet_data0
Sun Aug 03 07:21:00.811 AEST : addresses: 10.131.120.21/30, dnses: 198.142.152.162 198.142.152.163, gateways: 10.131.120.22, mtu: 1900
Sun Aug 03 07:21:00.816 AEST : Registration State: Registered, Home Network, RAT: 20
Sun Aug 03 07:21:01.823 AEST : Signal Strength: 24 (0-31) / -65dBm, RSRP: -98dBm, RSRQ: -15dB, RSSNR: 1.4dB
Sun Aug 03 07:21:02.024 AEST : MCCMNC: 50502, LAC: cb29, CID: 13c690a, ACT: 13 NR5G (5G), CARRIER: YES OPTUS YES OPTUS
Sun Aug 03 07:21:02.026 AEST : data_connection: up, status: 0, retry: -1, call_id: 0, type: IP, ifname: rmnet_data0
Sun Aug 03 07:21:02.026 AEST : addresses: 10.131.120.21/30, dnses: 198.142.152.162 198.142.152.163, gateways: 10.131.120.22, mtu: 1900
Sun Aug 03 07:21:02.031 AEST : Registration State: Registered, Home Network, CID: 013c690a, RAT: 14, TAC: 52009
Sun Aug 03 07:21:03.034 AEST : Signal Strength: 99 (0-31) / 85dBm, RSRP: -2147483647dBm, RSRQ: -2147483647dB, RSSNR: 214748368.0dB
Sun Aug 03 07:21:03.211 AEST : MCCMNC: 50502, LAC: cb29, CID: 13c690a, ACT: unknown, CARRIER: YES OPTUS YES OPTUS
Sun Aug 03 07:21:03.213 AEST : data_connection: up, status: 0, retry: -1, call_id: 0, type: IP, ifname: rmnet_data0
Sun Aug 03 07:21:03.213 AEST : addresses: 10.131.120.21/30, dnses: 198.142.152.162 198.142.152.163, gateways: 10.131.120.22, mtu: 1900
Sun Aug 03 07:21:03.217 AEST : Registration State: Registered, Home Network, RAT: 20
Sun Aug 03 07:21:04.225 AEST : Signal Strength: 99 (0-31) / 85dBm, RSRP: -2147483647dBm, RSRQ: -2147483647dB, RSSNR: 214748368.0dB
Sun Aug 03 07:21:04.427 AEST : MCCMNC: 50502, LAC: cb29, CID: 13c690a, ACT: unknown, CARRIER: YES OPTUS YES OPTUS
Sun Aug 03 07:21:04.431 AEST : data_connection: down
Sun Aug 03 07:21:04.441 AEST : Registration State: Registered, Home Network, RAT: 20
Sun Aug 03 07:21:05.449 AEST : Signal Strength: 99 (0-31) / 85dBm, RSRP: -2147483647dBm, RSRQ: -2147483647dB, RSSNR: 214748368.0dB
Sun Aug 03 07:21:05.647 AEST : MCCMNC: 50502, LAC: cb29, CID: 13c690a, ACT: unknown, CARRIER: YES OPTUS YES OPTUS
Sun Aug 03 07:21:05.650 AEST : data_connection: down
Sun Aug 03 07:21:05.654 AEST : Registration State: Registered, Home Network, RAT: 20
Sun Aug 03 07:21:06.663 AEST : Signal Strength: 99 (0-31) / 85dBm, RSRP: -2147483647dBm, RSRQ: -2147483647dB, RSSNR: 214748368.0dB
Sun Aug 03 07:21:06.850 AEST : MCCMNC: 50502, LAC: cb29, CID: 13c690a, ACT: unknown, CARRIER: YES OPTUS YES OPTUS
Sun Aug 03 07:21:06.854 AEST : data_connection: down
Sun Aug 03 07:21:06.860 AEST : Registration State: Registered, Home Network, RAT: 20
Sun Aug 03 07:21:07.867 AEST : Signal Strength: 99 (0-31) / 85dBm, RSRP: -2147483647dBm, RSRQ: -2147483647dB, RSSNR: 214748368.0dB
Sun Aug 03 07:21:08.075 AEST : MCCMNC: 50502, LAC: cb29, CID: 13c690a, ACT: unknown, CARRIER: YES OPTUS YES OPTUS
Sun Aug 03 07:21:08.078 AEST : data_connection: down

These logs are consistently the same every-time there is a disconnect.

Another issue maybe as well is that even when the cellular connection and the data connection is working the led on the board is breathing green instead of cyan like the docs would suggest?

For extra context I followed the instructions from here as best as I could eSIM Setup & Provisioning | Particle Developer but ran into some issues so the exact steps I ran are below.

1. particle-tachyon-ril-ctl disconnect
   particle-tachyon-ril-ctl power off
   particle-tachyon-ril-ctl power on
2. particle-tachyon-ril-ctl esim download '<rsp_url>'
3. particle-tachyon-rild connect <APN> connect sucessfully. Logs for that below

4. Sun Aug 03 08:02:44.849 AEST : Registering to Cellular Network...
   Sun Aug 03 08:02:44.902 AEST : ICCID currently active: 8961026025401557873 (Unknown)
   Sun Aug 03 08:02:44.903 AEST : Active SIM: 1 (INTERNAL)
   Sun Aug 03 08:02:44.903 AEST : Using APN: "yesinternet"
   Sun Aug 03 08:02:44.904 AEST : Command: AT+QNVR=5280,0, Response: 
   +QNVR: "0102000000000000"
   
   OK
   
   Sun Aug 03 08:02:44.952 AEST : Command: ATE0, Response: 
   OK
   
   Sun Aug 03 08:02:44.985 AEST : Command: AT+QNVFR="/nv/item_files/ims/IMS_enable", Response: +QNVFR: 
   00
   Sun Aug 03 08:02:45.023 AEST : Command: AT+QNVFR="/nv/item_files/modem/mmode/sms_only", Response: +QNVFR: 
   01
   Sun Aug 03 08:02:45.062 AEST : Command: AT+QNVFR="/nv/item_files/modem/mmode/ue_usage_setting", Response: +QNVFR: 
   01
   Sun Aug 03 08:02:45.102 AEST : Command: ATE1, Response: 
   OK
   
   
   Sun Aug 03 08:02:46.006 AEST : Get Preferred network type: 24 (NR5G TD-SCDMA LTE CDMA EVDO GSM WCDMA)
   Sun Aug 03 08:02:46.007 AEST : Command: AT+CGDCONT?, Response: 
   +CGDCONT: 1,"IP","yesinternet","0.0.0.0",0,0,0,0,,,,,,,,,"",,,,0
   
   OK
   
   Sun Aug 03 08:02:46.046 AEST : Command: AT+CEREG=2, Response: 
   OK
   
   Sun Aug 03 08:02:46.078 AEST : Command: AT+COPS?, Response: 
   +COPS: 0,0,"YES OPTUS YES OPTUS",13
   
   OK
   
   Sun Aug 03 08:02:46.156 AEST : Command: AT+CEREG?, Response: 
   +CEREG: 2,1,"CB29","13C690A",13
   
   OK
   
   Sun Aug 03 08:02:46.198 AEST : Network Registered!
   Sun Aug 03 08:02:46.198 AEST : Starting Data Call with RAT: 14 and APN: yesinternet
   Sun Aug 03 08:02:46.490 AEST : Setting DNS to "8.8.8.8 8.8.4.4"
   Sun Aug 03 08:02:46.534 AEST : Adding Route for rmnet_data0
   Sun Aug 03 08:02:46.604 AEST : Signal Strength: 21 (0-31) / -71dBm, RSRP: -103dBm, RSRQ: -14dB, RSSNR: 0.8dB
   Sun Aug 03 08:02:46.796 AEST : MCCMNC: 50502, LAC: cb29, CID: 13c690a, ACT: 13 NR5G (5G), CARRIER: YES OPTUS YES OPTUS
   Sun Aug 03 08:02:46.798 AEST : data_connection: up, status: 0, retry: -1, call_id: 0, type: IP, ifname: rmnet_data0
   Sun Aug 03 08:02:46.798 AEST : addresses: 10.176.94.206/30, dnses: 198.142.152.162 198.142.152.163, gateways: 10.176.94.205, mtu: 1900
   Sun Aug 03 08:02:46.803 AEST : Registration State: Registered, Home Network, CID: 013c690a, RAT: 19, TAC: 52009, PCI: 108, ECI: 20736266
   

5. After a period of time the data connection then drops?

Any help would be greatly appreciated. Thanks!

[zpm1066]

@mattmak23412

running command particle-tachyon-rild status

I get the following when command particle-tachyon-rild status is run.

particle-tachyon-ril already running in another process, waiting for it to finish...

Is your log output from journalctl particle-tachyon-rild?

[zpm1066]

@mrlambchop

Does the Tachyon Debugger module provide cellular monitoring, i.e. allow a live modem trace and capture to evaluate all communication (IP, LTE traffic) and view network parameters?

With other cellular modems, for packet & protocol details, I typically capture a modem trace and then export it to wireshark for analysis.

I would think such an essential troubleshooting tool would be available for the Tachyon, as perhaps part of the Quectel SG5600-NA development kit together with a wireshark plugin.

If not, How is Particle dev team troubleshooting cellular / IP issues at the protocol level?

Cheers!

[mrlambchop]

Hey mattmak23412

Before I type more, I did note that the eSIM docs were updated today! I think you will need to “apt update && apt upgrade” to get the latest packages for the RIL before using these instruction.

eSIM Setup & Provisioning | Particle Developer

Second, I think the docs are better now for debugging eSIM, but need a little more love still. I’ll work with Brett on what we can update here as he has coded a huge amount of magic already and update this shortly. The eSIM can be deleted - there are protections in there not to delete the Particle eSIMs however - will make sure we update!

Third, for this bug, can you (after running the apt upgrade above), run:

Tricks | Particle Developer

and send nick@particle.io the zip file of the logs? Posting here might share some secrets, but I’ll be able to grab them and take a look straight away.

Finally, we regard to the bug, I’m not sure - our eSIM has no timeout issues (so far!) - I think looking at the logs will be a huge help first.

Thanks!

Nick.

[mrlambchop]

Hey zpm1066

Yes! the USB connection can support the QCOM debug tools - however, QXDM requires a license from QCOM… If you happen to have a license, we can help turn this on!

Of course, you can use Wireshark + libpcap/tcpdump which is much easier:

sudo tcpdump -i rmnet_data0 -w capture.pcap

There are QMI tools as well, but lets hope this is much easier than exposing that!

Thanks

Nick.

[zpm1066]

@mrlambchop
Hi,

Thanks.

A QXDM licence is out of the question. Far too expensive and subscription based! Almost impossible to find out the cost.

Luckily there is an open source equivalent that should work, as Tachyon exposes a DM.