TCPClient reliability problems [SOLVED]

We are working to finalize a product release and are running into problems with using

TCPClient.write(_buf, _len);

They don’t seem to work properly. However when we use

TCPClient.print(_buf);

we get better results, but it’s very slow and can take as long as 30 seconds to write 200 bytes

We are writing to a db service so we don’t have any access to the debug log on the server we are sending data to.

We can test the remote server with Google Chrome Postman. It takes the requests as fast as we can send them and it reports 250ms to send the data.

Is there any information known about problems with TCPClient.write()?

I have tried to break down the _buf lengths into smaller chunks, and it will appear to send faster, however it never seems to reach the destination server.

EDIT: Even with TCPClient.print() we are still missing data on the receiving end. Out of 50 separate POSTs, 8-10 don’t get there. It gets worse with longer delays between trying to send data.

I should mention that the Spark Core has a WiFi connection and I don’t have any problems with connecting to the cloud. It’s only when I try and establish a connection on my own using the TCPClient object. Also It has the lastest CC3000 deep update patch.

Hi @mtnscott

You don’t say how big your buffer is, and that might be good to know. Is it thousands of bytes? Hundreds?

The problem and slowness with TCPClient.print() is that it sends multiple packets–one for each character–on both Spark and Arduino so that takes a lot more time.

Are you checking the return values from TCPClient.write()? It can return negative values on error.

Hi @bko The total length of bytes I’m trying to send it under 200. When I break it down I try and send anywhere from 20 to 50 bytes at a time.

I figured code would be the best way to demonstrate this problem.

This example uses the Phant library on the IDE. If you setup a local listener on a system in your environment you will see that after a while the Spark will fail to send the complete message. It will appear to have send the entire message but in reality is stops part way thru and will only recover after two more send messages are written.

Here is the example code:

#include "application.h"
#include "phant.h"

Phant::Stream stream1("192.168.1.140", "4JwJOg8xCw63f8wGM8Zp", "b5y58rrFo07xYmyY0rEl", 1010, PHANT_POST_METHOD);

int _ret;
int _retry;
int _loopcount;
unsigned long _ms;
float _f;

void setup() {
    Serial.begin(9600);

    while(!Serial.available()) {
	Serial.println("Press any key to begin");
	delay(1000);
    }

    Serial.println("PHANT test");

    //Phant intitialization
    _ms = millis();
    stream1.begin();

    //clearing previous stream values
    while (_retry < 5 && !(_ret = stream1.clearStream())) { delay(500); _retry++; }
    _f = (millis() - _ms) / 1000.0;
    if (_ret)
    	Serial.print("Stream successfully cleared");
    else
    	Serial.print("Stream could not be cleared (connection failed)");
    if (_retry) {
	Serial.print(" Retries = "); Serial.print(_retry);
    }
    Serial.print(" - time = "); Serial.print(_f, 1); Serial.println("s");

}

void loop() {
    //Adding a int field to the stream
    stream1.add("loop",++_loopcount);

    _retry = 0;
    _ret = 0;
    _ms = millis();
    while (_retry < 5 && !(_ret = stream1.sendData())) { delay(500); _retry++; }
    _f = (millis() - _ms) / 1000.0;

    Serial.print("Post ["); Serial.print(_loopcount); Serial.print("] ");
    if (_ret)
      Serial.print("successfully sent");
    else
      Serial.print("could not be sent (connection failed)");
    if (_retry) {
	Serial.print(" retries = "); Serial.print(_retry);
    }

    Serial.print(" - time = "); Serial.print(_f, 1); Serial.println("s");
    delay(1000);
}

Here is the serial output -

Press any key to begin
Press any key to begin
PHANT test
Stream successfully cleared - time = 0.4s
Post [1] successfully sent - time = 0.6s
Post [2] successfully sent - time = 0.6s
Post [3] successfully sent - time = 0.6s
Post [4] successfully sent - time = 0.6s
Post [5] successfully sent - time = 0.6s
Post [6] successfully sent - time = 0.6s
Post [7] successfully sent - time = 0.6s
Post [8] successfully sent - time = 0.6s
Post [9] successfully sent - time = 0.6s
Post [10] successfully sent - time = 0.5s
Post [11] successfully sent - time = 0.6s
Post [12] successfully sent - time = 0.6s
Post [13] successfully sent - time = 0.6s
Post [14] successfully sent - time = 0.6s
Post [15] successfully sent - time = 0.6s
Post [16] successfully sent - time = 0.6s
Post [17] successfully sent - time = 0.6s
Post [18] successfully sent - time = 14.5s
Post [19] successfully sent - time = 0.6s
Post [20] successfully sent - time = 0.6s
Post [21] successfully sent - time = 0.6s
Post [22] successfully sent - time = 0.5s
Post [23] successfully sent - time = 0.6s
Post [24] successfully sent - time = 0.6s
Post [25] successfully sent - time = 0.6s
Post [26] successfully sent - time = 0.6s
Post [27] successfully sent - time = 0.6s

Post 18, the one with the large time is the one that fails. Post 18 never completed and Post 19 is the beginning of a good post.

Here is the detail of what was received on the other end -

POST /input/4JwJOg8xV9CLG4wGM8Zp HTTP/1.1
Host: 192.168.1.140
Connection: close
Phant-Private-Key: b5y58raboNC5YmyY0rEl
Content-Type: application/x-www-form-urlencoded
Content-Length: 7

loop=17
POST /input/4JwJOg8xV9CLG4wGM8Zp HTTP/1.1
HoPOST /input/4JwJOg8xV9CLG4wGM8Zp HTTP/1.1
Host: 192.168.1.140
Connection: close
Phant-Private-Key: b5y58raboNC5YmyY0rEl
Content-Type: application/x-www-form-urlencoded
Content-Length: 7

loop=19

The Phant library phant.cpp and phant.h are available in the IDE Here is the actual code that does the sending -

int Phant::Stream::sendData() {
	int length = _host.length()+1;
	char charBuffer[length];
	_host.toCharArray(charBuffer,length);
	if(_client.connect(charBuffer,_port)) {
		if (_method == PHANT_POST_METHOD)
			_client.println(post());
		else //if(_method == PHANT_GET_METHOD)
			_client.println(get());
		delay(150);
		_client.flush();
		_client.stop();
		_params = "";
		return 1;
	}
	return 0;
}

Well… I just noticed this library has changed since I ran this test. The new updated library works for this small example, so I will try and use the updated library in my larger test.

If this works in the larger test, then I will delete this post.

Hi @mtnscott

I don’t know much about the library you are using–I think you should contact the library’s author. He does say in the readme: “Success or failure is not currently validated.” by which he apparently means that the library does not look at the returned bytes from the server.

This library also appears to have a very primitive method of disposing of returned data by waiting 150ms and flushing the buffer. This is not sufficient. A better method is to wait for returned data to be available and read until there is no more data available, and then finally flush and stop the client.

I suspect you are running out of TI CC3000 packet buffers but it is difficult to know.

This library also can do a lot of Arduino String concatenation creating many temporary Strings and then destroying them. This can cause memory to be fragmented and cause out of memory errors, but I don’t think that is happening your case.

@bko Thanks for your input. The library was taken from SparkFun’s arduino library. So the string usage is part of the original library, I will probably re-write the library as I don’t like the way it uses them anyway.

For now I will add your suggestion of waiting for the return data and pass the results upstream. As for the TI CC3000 packet buffers, can you explain that in more detail, or point me to some information where I can learn more about them.

http://www.ti.com/product/cc3000

There is a lot of information on the TI site as well as their forum.

The best thing you can do is be very defensive in your programming: Check all return values, close and retry on error, avoid dynamic memory allocation intensive operations, etc. This is especially true when you are trying to achieve very high reliability.

@bko, thanks, looks like I have a stable library. Do you know if there are any plans to have the TCP library updated to handle transfers larger than a byte at a time? I would think the STM32 would be able to support it.

I noticed that the TCPClient likes being called less often with more bytes than more often with fewer bytes. It seems the more calls to TCPClient - the less reliable the connection.

Thanks for your help!

If you use the TCPClient.write(uint8_t *buffer, size_t nbytes) method, you get one packet. That is the best way right now. If you have an Arduino String object like I believe you do, you can do:

client.write(myStr.c_str(), myStr.length());  //sends one packet

There is a github issue to make the print method do a better job at this, but as I said earlier, Spark inherited this from Arduino which also has this problem.

1 Like

@bko Thanks, I already have it as char array so it will be trivial to use write instead of print.

1 Like

So can we mark this one solved then?

Just a bit more - I need to convert my code to use write instead of print. I’m currently working on trying to get my black core to work as the white ones do. Can’t seem to figure that out (I know it is not part of this thread, but the white is running long term tests and I was hoping to use the black to test the switch from print to write)

@bko - works like a champ with TCPClient.write! Go ahead and close this thread. Much appreciated!

1 Like

Would you mind posting some of your new code so I may use it as a basis? I am having this same issue and would like to see how you are checking for returned data and deciding when to stop and flush.

Thank you.

2 Likes