[SOLVED] Using TCPClient for downloading large amounts of data

I’m receiving a large HTTP response (18KB) and it takes way longer than it should (200s). So the speed is only 90b/s.

Code is a slight modification of the TCPClient example:

#include "Particle.h"
#include "spark_wiring_tcpclient.h"

TCPClient client;

static uint32_t timer_start;

void setup(void) {
    Serial1.begin(115200);
    while (!Serial1) Particle.process();

    Serial1.println("Connecting...");

    if (client.connect("YYYY", 8001)) {
        Serial1.println("Connected");

        client.println("GET /ZZZZ HTTP/1.1");
        client.println("Host: YYYY");
        client.println("Authorization: Basic XXXXXXXXXXXXXXXXXX==");
        client.println("Content-Type: application/json");
        client.println("Content-Length: 0");
        client.println("Connection: keep-alive");
        client.println();

        timer_start = millis();
    } else {
        Serial1.println("Couldn't connect.");
    }
}

static char buffer[20000];
static uint32_t buffer_idx;

void loop(void) {
    if (client.available()) {
        if (buffer_idx < sizeof(buffer)) {
            buffer[buffer_idx++] = client.read();
        }
    }

    if (!client.connected()) {
        uint32_t timer_end = millis();

        Serial1.println();
        Serial1.println("Disconnecting.");

        client.stop();

        uint32_t seconds = (timer_end - timer_start) / (float)1E3;
        Serial1.printlnf("Transfer took %lu seconds.", seconds);

        Serial1.println("buffer:");
        Serial1.println(buffer);

        while (true);
    }
}

EDIT: Doing this in MANUAL mode is quicker (123s), but still too long.

You seem to be reading only one byte per iteration of loop() (which takes 1ms each in AUTOMATIC mode).
How about tightening that a bit to this

  while (client.available() && buffer_idx < sizeof(buffer)) {
    buffer[buffer_idx++] = client.read();
  }

BTW, there is no need for #include "spark_wiring_tcpclient.h" - Particle.h already takes care of that.

And for additional ways to read from TCPClient, you can have a look at the functions provided by the ancestor class Stream
https://docs.particle.io/reference/firmware/photon/#stream-class

Using client.read(buffer, bufferSize) will get a huge speed improvement over reading a character at a time.

The test program below doesn’t exactly do HTTP, but the concept is the same. It opens up a connection to a server then receives a bunch of data. The server was configured to send 1 Mbyte of data, then close the connection.

// Test app for receiving large amounts of data over a TCP connection

#include "Particle.h"

SYSTEM_THREAD(ENABLED);

// Finite state machine states
enum State { STATE_CONNECT, STATE_READ, STATE_RETRY_WAIT };

// Various constants

// bufSize is the number of bytes we make in a typical write call. Making this 2048 or
// larger can cause data corruption on the Photon. 1024 seems experimentally to be ideal;
// if you make it smaller it works without errors but the data rate drops.
const size_t bufSize = 1024;

// Various timeout values.
const unsigned long retryWaitTimeMs = 5000;
const unsigned long sendTimeoutMs = 60000;

// Set to the IP address of the server to connect to
IPAddress serverAddr(192,168,2,4);
const int serverPort = 7123;

// Global variables
State state = STATE_CONNECT;
TCPClient client;
unsigned long stateTime = 0;
uint8_t buf[bufSize];
unsigned long totalRead;
unsigned char expectedChar;
unsigned long startTime;

void setup() {
	Serial.begin(9600);
}

void loop() {
	switch(state) {
	case STATE_CONNECT:
		Serial.printlnf("** starting millis=%lu", millis());

		if (!client.connect(serverAddr, serverPort)) {
			// Connection failed
			Serial.println("** failed to connect");
			stateTime = millis();
			state = STATE_RETRY_WAIT;
			break;
		}
		totalRead = 0;
		expectedChar = 0;
		startTime = millis();
		state = STATE_READ;
		break;

	case STATE_READ:
		if (client.connected()) {
			int count = client.read(buf, bufSize);
			if (count > 0) {
				for(size_t ii = 0; ii < (size_t)count; ii++) {
					if (buf[ii] != expectedChar) {
						Serial.printlnf("** data mismatch expected=%u got=%u index=%u totalRead=%lu millis=%lu",
								expectedChar, buf[ii], ii, totalRead, millis());
						client.stop();
						stateTime = millis();
						state = STATE_RETRY_WAIT;
					}
					expectedChar++;
				}
				totalRead += count;
			}
		}
		else {
			unsigned long elapsed = millis() - startTime;

			Serial.printlnf("** connection closed totalRead=%lu elapsed=%lu", totalRead, elapsed );
			client.stop();
			stateTime = millis();
			state = STATE_RETRY_WAIT;
		}
		break;

	case STATE_RETRY_WAIT:
		if (millis() - stateTime > retryWaitTimeMs) {
			// Wait a few seconds before retrying
			state = STATE_CONNECT;
			break;
		}
		break;
	}
}

I consistently can receive 1 Mbyte of data in under a second, usually around 950 milliseconds.

** connection closed totalRead=1048576 elapsed=950
4 Likes

Also, try changing:

client.println("Connection: keep-alive");

to

client.println("Connection: close");

as that might trim a few seconds off as well. Your program runs until the tcp connection is closed, but keep-alive is asking the web server to keep the connection open after the http response has been sent in case you were going to send another http request.

@andrey have you been able to get your speed to improve?

Thank you all for suggestions. I’ll have a chance to test them soon.

Initial version of the code was based on the example and the fact that as I understand, there’s a possibility that the TCPClient.available() is called quicker than the data is received so the loop breaks before the full response is received. Is that correct?

Also, it seems like the method TCPClient.read(buffer, size) isn’t documented in the Particle Firmware Reference?

Yes, this seems undocumented, but as I indicated there’d be a function readByte(buffer, length) that’s inherited from the Stream class where it is documented too.
https://docs.particle.io/reference/firmware/photon/#readbytes-

Yes that is possible but would not matter since buffer_index is global and if you fall out of my while() before the response has been fully read, you’ll pick up just there on next iteration of loop(), but if you always ever only read one character at a time, you’ll be wasting a lot of time while the buffer is already nicely filled to read a lot more than one byte.

So, while @rickkas7’s code is a best practice example, I’d still recommend to get the code you readily understand working first, before you dive into his slightly more involved code - IMHO


BTW, I’m currently adding TCPClient::read(buffer, length) to the docs :wink:
Should be online soon
Is online already
https://docs.particle.io/reference/firmware/photon/#read--2

3 Likes

Just replacing the if (client.available()) with while (client.available()) didn’t do much much at all.

I’ve tried doing it this way inside setup() to get a tighter loop:

while (client.connected()) {
    while (client.available() && buffer_idx < sizeof(buffer)) {
        buffer[buffer_idx++] = client.read();
    }
}

Again, no change.

I’ve also tried the ‘new’ read method, but it hasn’t given significant speed increase:

uint8_t *buffer_ptr = &buffer[0];
int32_t buffer_len_remaining = sizeof(buffer);
while (client.connected() && buffer_len_remaining > 0) {
    int32_t read;
    if ((read = client.read(buffer_ptr, buffer_len_remaining)) > 0) {
        buffer_remaining_len -= read;
        buffer_ptr           += read;
    }
}

Turns out that each call to read ends up receiving only 1 byte at a time.

I haven’t yet tried to adapt @rickkas7’s code, but it seems do be doing mostly the same thing.

EDIT: @timx was absolutely right, that was the ultimate root of the problem, it takes only 8 seconds to read ~19KB now. :ok_hand:

1 Like

this solved my problem - i wasn’t seeing the entire message, reading TCPclient.read() char-by-char. thanks