TCP Client Write Function

Hi!

I’m using the built in tcp client to frequently write messages to and receive messages from a backend. However,
I just discovered that the write function seems to cause a leak.
Using programmer shield and debug output, I can see that my freeMemory sometimes decreases by about 1200 specifically during the call to that function. I print out freeMemory right before and right after, and that is where the drop happens, according to the freeMemory function.

The particle documentation says that there are two TCPClient::write methods:

  • client.write(val);
    where val is a single byte to be written

  • client.write(buf, len);
    where buf is an array of bytes and len is the length of the buffer.

In my code, I use client.write(buf), and hand it an array of integers. Though it doesn’t match either of the documented usages, it seems to work; the messages are being sent successfully, and I am getting the responses I expect. BUT, there’s this memory leak…

Am I calling the method wrong? Is the documentation out of date? What is going on? Thanks in advance!

EDIT: here is an example program that works in a similar way to my program, but with less timing control and without doing anything with the response. I do not see the memory leak here.

#include "application.h" //needed when compiling spark locally

#define HOST_NAME "posttestserver.com"
#define HOST_PORT_NUMBER 80
    
using namespace std;
    
SYSTEM_MODE(MANUAL);
    
short BUFF_SIZE;
short num_retrys;
short num_bytes_rcvd;
char* request_buff;
char* response_buff;
TCPClient _tcp_client;
char _state;
    
const char STATE_INIT = 0;
const char STATE_SEND_MESSAGE = 1;
const char STATE_RECEIVE_RESPONSE = 2;
const char STATE_FINISH_RESTART = 3;
  
bool write_msg(char*& msg)
{
    bool rslt = true;
    if(!_tcp_client.connected())
    {
        rslt = _tcp_client.connect(HOST_NAME, HOST_PORT_NUMBER);
    }
    if(rslt == false)
    {
        Serial.println("Can't connect to server!");  
    }
    else {
        Serial.println("Writing message:");
        Serial.println(msg);
        short msg_len = strlen(msg);
        Serial.print("freeMemory before write: ");
        Serial.println(System.freeMemory());
        short bytes_written = _tcp_client.write(msg);
        Serial.print("freeMemory after write: ");
        Serial.println(System.freeMemory());
        rslt = (rslt && bytes_written == msg_len);
        if(bytes_written != msg_len) 
        {
            Serial.println("Error, bytes_written != msg_len");
        }
    }
    return rslt;
} 
    
void setup()
{
    Serial.println("Main::Setup:: Waiting ...");
    delay(2000);
    BUFF_SIZE = 512;
    num_retrys = 0;
    request_buff = new char[BUFF_SIZE];
    strcpy(request_buff, "");
    response_buff = new char[2*BUFF_SIZE];
    strcpy(response_buff, "");
    _state = STATE_INIT;

    num_bytes_rcvd = 0;
   
    Serial.println("Turning on wifi module");
    WiFi.on();
    Serial.println("Attempting to connect to wifi");
    WiFi.connect();
    Serial.println("Checking connection");
    while(!WiFi.ready()){
        Serial.println("wifi connection failed, retrying ...");
        delay(3000);
        WiFi.connect();
    }
    Serial.println("WiFi connection succeeded!");
    
    Serial.println("Attempting to connect to host");
    _tcp_client.connect(HOST_NAME, HOST_PORT_NUMBER);
    while(!_tcp_client.connected()){
        Serial.println("host connection failed, retrying ... ");
        delay(1000);
        _tcp_client.connect(HOST_NAME, HOST_PORT_NUMBER);
    }
    Serial.println("Connected to host");
    delay(2000);
}
    
void loop()
{
    // Serial.printf("Free Memory: %d\n", System.freeMemory());
    if(!_tcp_client.connected())
    {
        Serial.println("Disconnected during state machine, retrying...");
        bool rslt = _tcp_client.connect(HOST_NAME, HOST_PORT_NUMBER);
        if(!rslt)
            return;
    }
 
    switch(_state){
        case STATE_INIT: {
            strcat(request_buff, "POST /post.php HTTP/1.1\r\n");
            strcat(request_buff, "Host: ");
            strcat(request_buff, HOST_NAME);
            strcat(request_buff, "\r\n");
            strcat(request_buff, "Connection: Keep-Alive\r\n");
            strcat(request_buff, "Content-Length: 27\r\n");
            strcat(request_buff, "\r\n");
            strcat(request_buff, "field1=value1&field2=value2\r\n");
            strcat(request_buff, "\r\n");
 
            _state = STATE_SEND_MESSAGE;
            break;
        }
    
        case STATE_SEND_MESSAGE: {
            if(write_msg(request_buff))
            {
                _state = STATE_RECEIVE_RESPONSE;
            }
            else
            {
                Serial.println("Message send failed, delay then retrying");
                num_retrys++;
                Serial.print("num_retrys = ");
                Serial.println(num_retrys);
                delay(2000);
            }
            break;
        }
  
        case STATE_RECEIVE_RESPONSE: {
            if(_tcp_client.available() > 0)
            {
                //get rid of anything we receive, don't care.
                response_buff[num_bytes_rcvd] = _tcp_client.read();
                num_bytes_rcvd++;
            }
            else {
                response_buff[num_bytes_rcvd] = 0;
                Serial.println("No more bytes to read from server");
                Serial.print("Number of bytes received = ");
                Serial.println(num_bytes_rcvd);
                Serial.println("Content from server: ");
                Serial.println(response_buff);
                _state = STATE_FINISH_RESTART;
            }
            break;
        }
 
        case STATE_FINISH_RESTART: {
            num_bytes_rcvd = 0;
            strcpy(request_buff, "");
            strcpy(response_buff, "");
            delay(1000);
            _state = STATE_INIT;
            break;
        }
    }
}

Why is that? Why would the memory drop specifically happen during the use of that write function in one program but not the other??

Could you please show your code, just to confirm that there are no other reasons that could cause the behaviour?

e.g. Use of String objects can cause “loss” of free space too.

Sure, I’m working on a minimal example right now.

I’m still seeing this issue very clearly. As I’m running through my program, I can clearly see the freeMemory drop over the call to TCPClient.write(). My example program (above) doesn’t show the same issue. Anyone have any advice?

Then it might be good to provide some code that does show the issue.

Ok, so. it appears that, because I have written this simple program that does not exhibit the same issue, the issue would be coming from somewhere else. BUT, it still shows itself during the tcp.write function. I don’t know why that would be or where that error is coming from, which is why I am now asking for some advice on how to proceed. What things could possibly cause the tcp client to leak when writing messages?

Usually you would call tcp.write() not without parameters, so it might have to do with your parameters or the way you provide them or ...
So seeing a code that does exhibit the erronous behaviour might help to narrow down the cause.

I understand. I’m just not sure what parts of the very complex system I’m pulling this from to place into sample code in order to cause this behavior.

If you’ve really narrowed it down to the one tcp.write() statement that causes the issue you could start by providing the actual code line along with the used parameters, their definition and the relevant assignments.
This way we could at least see if something pops into the eye.

Additionally you could upload your project to something like GitHub or Dropbox.

But best would be to write up a simple project that actually does show the issue.

1 Like

Alright! I finally got an app that displays what I am talking about: https://gist.github.com/bgenchel/b5018230a050ddfa320e

This app operates similarly to the real app and shows a smaller version of the same problem. It generally works by having messengers send messages to a webhandler that enques requests and sends them off in order. Like an operating system, this program works by allowing a certain amount of time for a state machine to run through it’s states.

In this simplified version, there are only two messengers that send off the same message over and over again to the handler. Memory drops by 100-1000 every once and a while, and through print statements I can see that it happens over the tcp.write function.

Just one thing I noted skimming over your code, not all code paths in (_msg)_state_machine do return a value. Maybe just add it for good measure.

But maybe @bko can see more there.

1 Like

Hi @bgenchel

You seem to be complaining about memory leaks on a small micro when you are using std::queue and std::unorderedmap and std::make_tuple. I would get rid of all that overhead and try again with simpler code and fully static arrays of data.

It is certainly possible that there are memory leaks in tcp.write, but it seems a lot more likely to me that using std libraries is causing you problems.

2 Likes

Interesting. So is there always a guarantee of memory leaking when using the standard libraries? In the actual program, we rely on these data structures to implement a system that we feel needs to be at this level of complexity in order to carry out what we need it to do.

Using static arrays to implement that, we would need to constantly shift all the elements in it over, which would cause a tremendous slow down.

Also, do you have any inkling of why that drop in freeMemory would occur over the write function?

Hi @bgenchel

No, it is not guaranteed that std libraries will leak memory, but if I were betting, I would place my bet there in this case. A particularly likely suspect is memory fragmentation, where all the allocation and deallocation of RAM ends up leaving many small non-contiguous spaces and eventually a malloc or new call will fail. Using static memory does not mean that you have to give up on your abstractions like queue and map, you may just need to manage them yourself instead of relying on a library.

You have to understand that this is a small platform without the luxury of swap space or the other niceties of an OS, so memory management falls more on the programmer’s shoulders than on the environment. If you need these libraries to do your work, I would say that you need to ensure that they work reliably with limited memory impacts, as you intend to use them. That should be fairly easy to test by say replacing the TCP write call with something like buf = malloc(1024); free(buf);

While I do not have a specific reason why, it certainly could be that tcp.write() is allocating temporary storage (and hopefully deallocating correctly later) and that causes the memory management code to recognize that it cannot allocate a chuck of the needed size and so the free memory goes down. Perhaps @mdma can lend an opinion here at some point since I don’t know much about the free memory reporting code he put in.

If I did write them myself, wouldn’t there be similar risk of fragmentation? I just imagine that whoever wrote the standard libraries would do a better job than I would haha.

The System.freeMemory() isn’t accurate - it represents the minimum amount of free memory available, not the actual amount.

For memory leak testing, you’ll see this decrease to 0 if there is a true memory leak. At present, it’s not possible to do micro-benchmarks, allocating a buffer, then freeing and checking the value of System.freeMemory() before and after.

1 Like

Sure, but our memory does reach 0 over time at which point our app crashes. Why would it always drop in the same place if there wasn’t any link between the action and the memory?

Seems there is a memory leak then! I’ll add investigating this to our todo list.

2 Likes

Have there been any updates on this issue?

I am currently stuck with the exact same issue and it is becoming a very big problem. The devices work to about 12-24 hours ( The device uptime is probably dependant on how much the application uses TCPClient connections ), while 1260 bytes of memory is consumed by ‘TCPClient::write’ every once in a while. Once the free memory stat shows <1500bytes of memory the device goes into an undefined state, where it is not reachable through the network. From there on in about 30 minutes the device comes up again (Im guessing it hard faults or smth).

I have been using firmware 0.4.7.

Thank you very much for that test app. What kind of server should I run this against?