Weird Issue - Irregular E Series connection dropouts on customer side

Hi Community,

we have a serious problem with one of our E Series based devices in the field at the customer side.

The device is a industrial plc like controller and interfaces with flowmeters which monitor beer-flow in a very large pub. We have developed this solution for our customer and made intensive testing before we went into field.

Controller is placed inside the building in a cooling room under ground floor (cellar) and to get cellular signal we have attached a 6 meters long antenna cable + 2G/3G Antenna.

Problem is the following:

  • E Series Module connects to the cloud.
  • Handshake is successfully been made.
  • Some data is been send and module does respond to calls / request from the cloud.
  • After quite some time - we cannot really say what time since this is so random - the regular publishes drop and the E Series does not respond to any request (ping, function, etc.) from the cloud
  • Then after quite some time - sometimes just half an hour, sometimes almost six hours (!!) - the module comes back online with a publish saying device came online
  • There is not a panic or safeMode log
  • There is not even a device hash or reset message which could explain some sort of reset or re-handshake
  • Also during those non reachable phases the E Series module does continue to breathe cyan!

We have a debug interface build into our application firmware but unfortunately the unit is currently 800km away

What is interesting is that the time it takes for the E Series to come back online is longer during the nights but shorter during days

We have made some intensive testing and hired a professional firmware developer with high experience on ST Systems. Unfortunately I cannot share firmware since this is a commercial one.

We have also build a debug interface into the application firmware which can be programmed over the cloud. This also allowed us to send raw AT-Command to the uBlox Module.

Initially we were thinking of a problem with the cellular connection so we read out signal quality, cellular operator and what mode the module is in (GPRS,EDGE,UMTS,HSDPA,…) and got the following result:

  • The uBlox is connected in 2G mode using EDGE (not GPRS which we initially though could be a problem since being so slow)
  • The uBlox says that it has a signal quality of 100%

Now comes a strange thing:

In Germany Telefonica (Particle.io SIM Card Provider) works together with Cellular Carrier O2. But the uBlox says that it is connected with “E-Plus” - which isn’t O2.

So we initially thought that keepAlive might be a problem and lowered it - without any effect at all. Also UDP-Holepunching would also be done by any additional publish and once the device came back online we had a few minutes to set regular published to 10 seconds each.

But again after some time (this time it was just 14 min) communication dropped - but without any sign of error or notification.

For those of you who are asking - those are the AT Commands we have been using:

AT+UREG? which reads out the uBlox Module operation mode:

Response:
+UREG: 0,2

Last Digit means:
1 = GPRS
2 = EDGE
3 = WCDMA
4 = HSDPA
5 = HSUPA
6 = HSUPA+HSDPA

And the other command: AT+COPS=?which reads out available network providers and returns them asynchronically.

We also tried to reset the modem every time the E Series came back online (since a µC reset does not do reset the modem and only thing is power out) using this Command: AT+CFUN=16

But no result at all.

Currently I come to the thought that this 6 meter antenna cable + the e series being put in a room below ground floor might be causing the drop outs.

There is one last thing I am going to test (at the time the device hopefully comes back online). We have embedded a function that puts the E Series into safe mode.

But as I said we have made some intensive testing before but have never ever experienced this at all - also we have some additional units in the field - they all work as expected.

I would be so thankful for any advice or help - or just feedback - or even if someone of you has experienced a similar situation.

Best,

Simon

Let me ping someone that might be able to help, @rickkas7 or @ParticleD are you able to assist?