Electron "Extremely Slow" communications

[Jimmie]

I have developed an application which needs to send about 10KB of data to a server (as a single file). On a Photon, and using an FTP server, it took about 7 seconds which is too long for me. On the Electron, it took "for ever" and was not feasible.

On advice of @Scruffr, I rewrote the application to use a TCP server instead. The results on the Photon were extremely significant ( < 200 ms) which is of course a major improvement over 7 seconds for the FTP server.

On the Electron, the same file took TWO MINUTES!

Am I doing something wrong or is the Electron designed to just send one value at a time?

Using code from the @ScruffR post in:

Electron Signal Strength LED Algorithm

The easiest would be to map the range via tha map() function. Give this a shot int strength = map(RSSI, -131, -51, 0, 5); You may have to tweak the boundaries (-131/-51) a bit to get the mid point of each category right. And this is the code from the firmware repo void system_display_rssi() { /* signal strength (u-Blox Sara U2 and G3 modules) * 0: < -105 dBm * 1: < -93 dBm * 2: < -81 dBm * 3: < -69 dBm * 4: < - 57 dBm * 5: >= -57 dBm */…

I found that for WiFi, Strength = 4, and RSSI = -57
for Cellular Strength = 3, and RSSI = -81

So the cell signal does not look too bad .....

Thanks in advance for the community's thoughts.

[ScruffR]

How do you upload the data?
On Electrons you need to send chunks no bigger than 512 byte at a time.

[Jimmie]

I am sending chunks less than 100 bytes each time. (TCP)

When I was doing FTP, I was sending chunks of 512 bytes each time but the file never arrived even though I could see that the Electron hooked up to the FTP server.

Side note: I am not sure the Electron is reporting correct signal strength numbers because I know that cell service in my area is weak. My phone barely gets 1 bar so I am not putting too much faith in what the Electron is reporting …

[ScruffR]

How does the Electron behave when you try it with 512 byte at a time?
Can you post your code or a SHARE THIS REVISION link.
How are you receiving the data?

[Jimmie]

I have not tried sending fixed (512) chunks as the same code worked successfully with the Photon and takes well less than half a second to send the data.

I am receiving the data with my own TCP Server (written in .NET).

The code on the Electron (Photon) is below. I am sending a three-dimensional array of integers from memory.

void sendFromRAM(int tranType, int numS)
{
  if (tcpFiles == 1)
  {
  long now = millis();
  byte clientBuf[ftpChunk];
  int clientCount = 0;
  byte buf[4];
  int messageSize = 0;
  String myMessage;

  //Send data from RAM
  if (client.connect(myFTPAddress, tcpPort))
  {
    //-------------------------------------------------------------------
    //Send Header
    msgHeader = msgStart + myProfile + ",0," + msgFollow ;
    byte msgHeaderB[msgHeader.length()];
    msgHeader.getBytes(msgHeaderB, msgHeader.length() + 1);
    client.write(msgHeaderB, msgHeader.length());
    //----------------------------------------------
    if (tranType == 0) {
      myMessage = "Scans = " + String(numS) + ", H = " + String(maxH) + ", vert = " + String(vert) + ", Speed = " + String(vehS) + " mph" + ", Length:"  +  String(tLength) + " ft" + "\r\n" ;
      byte msg1[myMessage.length()];
      myMessage.getBytes(msg1, myMessage.length() + 1);
      client.write(msg1, myMessage.length());
    }
    //----------------------------------------------
    if ((tranType == 0) && (sendProfile >= 1))
    {  
      for (int j = 1; j <= numS; j++)   //Each Scan
      {
        myMessage = "Scan: " + String(j) + "\r\n" ;
        myMessage.getBytes(clientBuf, myMessage.length() + 1);
        client.write(clientBuf, myMessage.length());

        for (int i = 0; i < sS; i++)
        {
          //myMessage = (String(i) + "," + String(cloudP[j][i][0]) + "," + String(cloudP[j][i][1]) + "," + String(cloudP[j][i][2]) + "\r\n");
          myMessage = (String(i) + "," + String(cloudP[j][i][0]) + "," + String(cloudP[j][i][1]) + "\r\n");
          myMessage.getBytes(clientBuf, myMessage.length() + 1);
          client.write(clientBuf, myMessage.length());
        }
      }
    }  
    //------------------------------------------------------
    //Send End
    byte msgEndB[msgEnd.length()];
    msgEnd.getBytes(msgEndB, msgEnd.length() + 1);
    client.write(msgEndB, msgEnd.length());
    //------------------------------------------------------
    delay(2);
    client.flush();
    client.stop();
    sendTime = millis() - now;
    //-------------------
     if (goPublish == 1) publishManager.publish("TCP" , ("$$,55," + String(baseNum) + ","  + String(myProfile) + "," + String(sendTime) + " ms," + "^"));
     //------------------- 
  }  //client connect
  else
  {
    Particle.publish("TCP Error", "No connection to " + String(remote_addr));
    
  } //client connect
 }
}  //end

[ScruffR]

@Jimmie

with the Photon and takes well less than half a second to send the data.

You may be missing some important points between Photon and Electron.
Apart from the inherently slower and more "vulnerable" transport via cellular compared to WiFi, the communication between the STM32 and ublox cellular modem goes via relatively slow serial interface while on the Photon the communication goes via a considerably faster bus.
Also the Electron communicates with the cellular provider on a UDP based protocol while the Photon works natively on TCP.
Also the TX buffer of the Photon can send chunks of up to 1024 byte, the Electron only 512.

Consequently the max transfer rate on the Electron will undoubtedly be slower than on the Photon, but if the difference gets that severe you need to reduce as much of the impact caused by your own code as possible since you can't do anything about the inherent differences.

In your code I also see some problematic portions
e.g.

      byte msg1[myMessage.length()];
      myMessage.getBytes(msg1, myMessage.length() + 1);

here you are declaring msg1 one byte too short.
Although there is no need to actually copy the string into the byte array, you can just write it that way

myMessage.getBytes((const byte*)myMessage.c_str(), myMessage.length());

You may also have frequently read that we keep discouraging the use of String and rather suggest to use snprintf() to create formatted strings.

However, when you are sending your data to a self-written TCP server, why not just send the raw binary data and convert it on the receiving end?
That would cut down on transfer data and time tremendously.

[Jimmie]

@ScruffR

You may also have frequently read that we keep discouraging the use of String and rather suggest to use snprintf() to create formatted strings.

Thank you @ScruffR for the detailed response.

The reason my code is inefficient is because I know very little about C so I would really appreciate your help with those two simple questions so I can modify my code:

1. How does one use snprintf()? I need to send the following:

myMessage = (String(i) + "," + String(cloudP[j][i][0]) + "," + String(cloudP[j][i][1]) + "\r\n");

2. What do you mean by "..send the raw binary data "? How is this done on the Electron side?

The two questions above may be related but I am having trouble with them because I mainly program in VB.NET.

Thanks again for all your help and would appreciate any help on those questions.

[ScruffR]

I useful read in order to understand snprintf() is this
http://www.cplusplus.com/reference/cstdio/printf/
snprintf() uses the same logic, but "prints" its output to a character buffer with a given max size.

For your specific string you'd do it this way (assuming all your variables are int

char msg[128];
snprintf(msg, sizeof(msg), "%d,%d,%d\r\n", i, cloudP[j][i][0], cloudP[j][i][1]);

@Jimmie

What do you mean by "…send the raw binary data "? How is this done on the Electron side?

Lets suppose - since your code snippet does not reveal the actual info - you want to send your cloudP array which is a three dimentional array of uint16_t values not exceeding a total of 512 bytes, it would just be done this way

  client.write((const byte*)cloudP, sizeof(cloudP));

That's it.
On the receiving end you need to fill the incoming data into a struct that allows you to retrieve the respective values correctly, but that's a VB.Net question - it's doable there to but I do prefer C#.

[Jimmie]

Thank you @Scruffr. I am almost there.

1. When using

char msg[128];
snprintf(msg, sizeof(msg), "%d,%d,%d\r\n", i, cloudP[j][i][0], cloudP[j][i][1]);

then to write it,

client.write(msg, sizeof(msg));

I am getting a compile error "invalid conversion from 'char*' to 'const uint8_t* {aka const unsigned char*}' [-fpermissive]".

2. For my cloudP array, does the statement below send the entire array? The entire cloudP array is definitely > 512 bytes. So what to do in this case?

client.write((const byte*)cloudP, sizeof(cloudP));

Thanks again.

[ScruffR]

@Jimmie

client.write(msg, sizeof(msg));

With that you'd actually write

client.write((const uint8_t*)msg, strlen(msg)+1);

1. You need the typecast ((const uint8_t*)) to convince the compiler that the array you are passing to the function can in deed be considered as valid parameter despite it being declared differently.
2. Since you don't want to send 128 bytes (= sizeof(msg)) but rather only the portion of the array that contains the string, all the bytes after that should not be sent.

@Jimmie

For my cloudP array, does the statement below send the entire array? The entire cloudP array is definitely > 512 bytes. So what to do in this case?

That's why I so elaborately introduced my code snippet this way

@ScruffR

Lets suppose - since your code snippet does not reveal the actual info - you want to send your cloudP array which is a three dimentional array of uint16_t values not exceeding a total of 512 bytes, it would just be done this way

If your array is longer, you'd just need to "chop it up" into chunks and send them one-by-one, as I mentioned earlier.

[Jimmie]

Thank you @Scruffr. Will try it.

I am just wondering why would sending the raw array be faster since the byte conversion is done by the main processor before sending it to the modem (as you had explained earlier, sending to the blox modem is where the bottleneck is) …?

Thanks again for your time.,

[ScruffR]

@Jimmie

I am just wondering why would sending the raw array be faster since the byte conversion is done by the main processor before sending it to the modem

There is no conversion taking place at all - hence "raw".

Based on the assumption (since you still haven't reavealed the actual declaration of cloudP) that your array is an array of a numeric datatype and as such the byte count to transfer any arbitrary number in its binary form is statistically less than what the same number would take in its "printed" form (byte x = 123 takes one byte raw but 3+1 characters to print). You also won't add any extra bytes by injecting text or separators.
And exctly that reduction of bytes to transfer would inevitable also reduce the time to do so.

[Jimmie]

Thank you @ScruffR. It is clear now.

Sorry, I forgot to send the declaration Here it is:

unsigned long cloudP[302][15][1];

[ScruffR]

@Jimmie

unsigned long cloudP[302][15][1];

Judging by this statement of your code above, I assume something else may be amiss

myMessage = (String(i) + “,” + String(cloudP[j][i][0]) + “,” + String(cloudP[j][i][1]) + “\r\n”);

With the declaration of your array above, the max index for your third dimension would be [0] since you declare it to only have 1 element.
You don't declare the max index but the number of elements per dimension and the index always starts with 0.

When saving transfer "cost" what are your expected max values for each element in your field.
unsigned long takes up 32 bit, so you can store values in the range of 0 .. 4294967296. If you don't need that large numbers you may want to consider shorter datatypes (e.g. uint16_t max 65535) which again would reduce data consumption drastically.

[Jimmie]

Thank you @ScruffR for your time and patience. I have learned a lot.