Using client.read(buffer, bufferSize) will get a huge speed improvement over reading a character at a time.
The test program below doesn’t exactly do HTTP, but the concept is the same. It opens up a connection to a server then receives a bunch of data. The server was configured to send 1 Mbyte of data, then close the connection.
// Test app for receiving large amounts of data over a TCP connection
#include "Particle.h"
SYSTEM_THREAD(ENABLED);
// Finite state machine states
enum State { STATE_CONNECT, STATE_READ, STATE_RETRY_WAIT };
// Various constants
// bufSize is the number of bytes we make in a typical write call. Making this 2048 or
// larger can cause data corruption on the Photon. 1024 seems experimentally to be ideal;
// if you make it smaller it works without errors but the data rate drops.
const size_t bufSize = 1024;
// Various timeout values.
const unsigned long retryWaitTimeMs = 5000;
const unsigned long sendTimeoutMs = 60000;
// Set to the IP address of the server to connect to
IPAddress serverAddr(192,168,2,4);
const int serverPort = 7123;
// Global variables
State state = STATE_CONNECT;
TCPClient client;
unsigned long stateTime = 0;
uint8_t buf[bufSize];
unsigned long totalRead;
unsigned char expectedChar;
unsigned long startTime;
void setup() {
Serial.begin(9600);
}
void loop() {
switch(state) {
case STATE_CONNECT:
Serial.printlnf("** starting millis=%lu", millis());
if (!client.connect(serverAddr, serverPort)) {
// Connection failed
Serial.println("** failed to connect");
stateTime = millis();
state = STATE_RETRY_WAIT;
break;
}
totalRead = 0;
expectedChar = 0;
startTime = millis();
state = STATE_READ;
break;
case STATE_READ:
if (client.connected()) {
int count = client.read(buf, bufSize);
if (count > 0) {
for(size_t ii = 0; ii < (size_t)count; ii++) {
if (buf[ii] != expectedChar) {
Serial.printlnf("** data mismatch expected=%u got=%u index=%u totalRead=%lu millis=%lu",
expectedChar, buf[ii], ii, totalRead, millis());
client.stop();
stateTime = millis();
state = STATE_RETRY_WAIT;
}
expectedChar++;
}
totalRead += count;
}
}
else {
unsigned long elapsed = millis() - startTime;
Serial.printlnf("** connection closed totalRead=%lu elapsed=%lu", totalRead, elapsed );
client.stop();
stateTime = millis();
state = STATE_RETRY_WAIT;
}
break;
case STATE_RETRY_WAIT:
if (millis() - stateTime > retryWaitTimeMs) {
// Wait a few seconds before retrying
state = STATE_CONNECT;
break;
}
break;
}
}
I consistently can receive 1 Mbyte of data in under a second, usually around 950 milliseconds.
as that might trim a few seconds off as well. Your program runs until the tcp connection is closed, but keep-alive is asking the web server to keep the connection open after the http response has been sent in case you were going to send another http request.
Thank you all for suggestions. I’ll have a chance to test them soon.
Initial version of the code was based on the example and the fact that as I understand, there’s a possibility that the TCPClient.available() is called quicker than the data is received so the loop breaks before the full response is received. Is that correct?
Also, it seems like the method TCPClient.read(buffer, size) isn’t documented in the Particle Firmware Reference?
Yes that is possible but would not matter since buffer_index is global and if you fall out of my while() before the response has been fully read, you'll pick up just there on next iteration of loop(), but if you always ever only read one character at a time, you'll be wasting a lot of time while the buffer is already nicely filled to read a lot more than one byte.
So, while @rickkas7's code is a best practice example, I'd still recommend to get the code you readily understand working first, before you dive into his slightly more involved code - IMHO
