Hi guys !
I am currently using this library: https://github.com/nmattisson/HttpClient to make a request to an JSON page. My problem is the time taken by this process. It can take up to 7 seconds before the next request is complete. If any of you had a idea to reduce time, It would be welcome !
Thanks
William
Hi,
Using the sample provided the Call is really slow, mainly because of the Server they use (www.timeapi.org). Maybe they are slowing them down internally.
I use the same libary and the same code without timing problems.
So, if the sample is running slow, thats not a bug!
I’m in the process of updating that library to use the write instead of print, which should make a few seconds difference…
Hi,
I’m using my own server for the request (witch the request time on my browser is less than half a second).
Hootie81, when do you think the new library is going to be available ?
I was wondering, what happen during time before the Spark start print stuff on the serial monitor ?
thats where the issue is, before each serial.print in the library does client.print to the TCPclient. sometimes this takes a few seconds! creating a buffer and sending all at once fixes this problem.
Im working on the updated library now, for the next hour or so.. then ill have to go to bed as i have an early flight tomorrow. hopefully i can get it working in that time.. but dont expect it to be tested!
@tchinou1 I have made some big changes to the library, it compiles and i have quickly tested and its working well for me and nice and fast on the send side of things.
Probably best if you create a new app, and manually add a set of tabs called HttpClient.h and HttpClient.cpp and copy the code below into them. then copy and paste the code you have working with the old httpclient into the .ino tab. if you try and add tabs with the same name to your existing code it will give you loads of compile errors (as i just found out).
HttpClient.h
#ifndef __HTTP_CLIENT_H_
#define __HTTP_CLIENT_H_
#include "spark_wiring_string.h"
#include "spark_wiring_tcpclient.h"
#include "spark_wiring_usbserial.h"
/**
* Defines for the HTTP methods.
*/
static const char* HTTP_METHOD_GET = "GET";
static const char* HTTP_METHOD_POST = "POST";
static const char* HTTP_METHOD_PUT = "PUT";
static const char* HTTP_METHOD_DELETE = "DELETE";
/**
* This struct is used to pass additional HTTP headers such as API-keys.
* Normally you pass this as an array. The last entry must have NULL as key.
*/
typedef struct
{
const char* header;
const char* value;
} http_header_t;
/**
* HTTP Request struct.
* hostname request host
* path request path
* port request port
* body request body
*/
typedef struct
{
String hostname;
IPAddress ip;
String path;
// TODO: Look at setting the port by default.
//int port = 80;
int port;
String body;
} http_request_t;
/**
* HTTP Response struct.
* status response status code.
* body response body
*/
typedef struct
{
int status;
String body;
} http_response_t;
class HttpClient {
public:
/**
* Public references to variables.
*/
TCPClient client;
char buffer[1024];
/**
* Constructor.
*/
HttpClient(void);
/**
* HTTP request methods.
* Can't use 'delete' as name since it's a C++ keyword.
*/
void get(http_request_t &aRequest, http_response_t &aResponse)
{
request(aRequest, aResponse, NULL, HTTP_METHOD_GET);
}
void post(http_request_t &aRequest, http_response_t &aResponse)
{
request(aRequest, aResponse, NULL, HTTP_METHOD_POST);
}
void put(http_request_t &aRequest, http_response_t &aResponse)
{
request(aRequest, aResponse, NULL, HTTP_METHOD_PUT);
}
void del(http_request_t &aRequest, http_response_t &aResponse)
{
request(aRequest, aResponse, NULL, HTTP_METHOD_DELETE);
}
void get(http_request_t &aRequest, http_response_t &aResponse, http_header_t headers[])
{
request(aRequest, aResponse, headers, HTTP_METHOD_GET);
}
void post(http_request_t &aRequest, http_response_t &aResponse, http_header_t headers[])
{
request(aRequest, aResponse, headers, HTTP_METHOD_POST);
}
void put(http_request_t &aRequest, http_response_t &aResponse, http_header_t headers[])
{
request(aRequest, aResponse, headers, HTTP_METHOD_PUT);
}
void del(http_request_t &aRequest, http_response_t &aResponse, http_header_t headers[])
{
request(aRequest, aResponse, headers, HTTP_METHOD_DELETE);
}
private:
/**
* Underlying HTTP methods.
*/
void out(const char *s);
void request(http_request_t &aRequest, http_response_t &aResponse, http_header_t headers[], const char* aHttpMethod);
void sendRequest(const char* aHttpMethod, const char* aRequestPath);
void sendBody(const char* aRequestBody);
void sendHeader(const char* aHeaderName, const char* aHeaderValue);
void sendHeader(const char* aHeaderName, const int aHeaderValue);
void sendHeader(const char* aHeaderName);
};
#endif /* __HTTP_CLIENT_H_ */
HttpClient.cpp
#include "HttpClient.h"
#define LOGGING
static const uint16_t TIMEOUT = 5000; // Allow maximum 5s between data packets.
/**
* Constructor.
*/
HttpClient::HttpClient()
{
}
void HttpClient::out(const char *s) {
client.write( (const uint8_t*)s, strlen(s) );
#ifdef LOGGING
Serial.write( (const uint8_t*)s, strlen(s) );
#endif
}
/**
* Method to send a header, should only be called from within the class.
*/
void HttpClient::sendRequest(const char* aHttpMethod, const char* aRequestPath)
{
sprintf(buffer, "%s %s HTTP/1.1\r\n", aHttpMethod, aRequestPath);
out(buffer);
}
void HttpClient::sendBody(const char* aRequestBody)
{
out(aRequestBody);
}
void HttpClient::sendHeader(const char* aHeaderName, const char* aHeaderValue)
{
sprintf(buffer, "%s: %s\r\n", aHeaderName, aHeaderValue);
out(buffer);
}
void HttpClient::sendHeader(const char* aHeaderName, const int aHeaderValue)
{
sprintf(buffer, "%s: %d\r\n", aHeaderName, aHeaderValue);
out(buffer);
}
void HttpClient::sendHeader(const char* aHeaderName)
{
sprintf(buffer, "%s\r\n", aHeaderName);
out(buffer);
}
/**
* Method to send an HTTP Request. Allocate variables in your application code
* in the aResponse struct and set the headers and the options in the aRequest
* struct.
*/
void HttpClient::request(http_request_t &aRequest, http_response_t &aResponse, http_header_t headers[], const char* aHttpMethod)
{
// If a proper response code isn't received it will be set to -1.
aResponse.status = -1;
// NOTE: The default port tertiary statement is unpredictable if the request structure is not initialised
// http_request_t request = {0} or memset(&request, 0, sizeof(http_request_t)) should be used
// to ensure all fields are zero
bool connected = false;
if(aRequest.hostname!=NULL) {
connected = client.connect(aRequest.hostname.c_str(), (aRequest.port) ? aRequest.port : 80 );
} else {
connected = client.connect(aRequest.ip, aRequest.port);
}
#ifdef LOGGING
if (connected) {
if(aRequest.hostname!=NULL) {
Serial.print("HttpClient>\tConnecting to: ");
Serial.print(aRequest.hostname);
} else {
Serial.print("HttpClient>\tConnecting to IP: ");
Serial.print(aRequest.ip);
}
Serial.print(":");
Serial.println(aRequest.port);
} else {
Serial.println("HttpClient>\tConnection failed.");
}
#endif
if (!connected) {
client.stop();
// If TCP Client can't connect to host, exit here.
return;
}
//
// Send HTTP Headers
//
// Send initial headers (only HTTP 1.0 is supported for now).
#ifdef LOGGING
Serial.println("HttpClient>\tStart of HTTP Request.");
#endif
sendRequest(aHttpMethod, aRequest.path.c_str());
// Send General and Request Headers.
sendHeader("Connection", "close"); // Not supporting keep-alive for now.
if(aRequest.hostname!=NULL) {
sendHeader("HOST", aRequest.hostname.c_str());
}
//Send Entity Headers
// TODO: Check the standard, currently sending Content-Length : 0 for empty
// POST requests, and no content-length for other types.
if (aRequest.body != NULL) {
sendHeader("Content-Length", (aRequest.body).length());
} else if (strcmp(aHttpMethod, HTTP_METHOD_POST) == 0) { //Check to see if its a Post method.
sendHeader("Content-Length", 0);
}
if (headers != NULL)
{
int i = 0;
while (headers[i].header != NULL)
{
if (headers[i].value != NULL) {
sendHeader(headers[i].header, headers[i].value);
} else {
sendHeader(headers[i].header);
}
i++;
}
}
// Empty line to finish headers
out("\r\n");
client.flush();
//
// Send HTTP Request Body
//
if (aRequest.body != NULL) {
sendBody(aRequest.body.c_str());
}
#ifdef LOGGING
Serial.println("HttpClient>\tEnd of HTTP Request.");
#endif
//
// Receive HTTP Response
//
// The first value of client.available() might not represent the
// whole response, so after the first chunk of data is received instead
// of terminating the connection there is a delay and another attempt
// to read data.
// The loop exits when the connection is closed, or if there is a
// timeout or an error.
// clear response buffer
memset(&buffer[0], 0, sizeof(buffer));
unsigned int bufferPosition = 0;
unsigned long lastRead = millis();
unsigned long firstRead = millis();
bool error = false;
bool timeout = false;
do {
#ifdef LOGGING
int bytes = client.available();
if(bytes) {
Serial.print("\r\nHttpClient>\tReceiving TCP transaction of ");
Serial.print(bytes);
Serial.println(" bytes.");
}
#endif
while (client.available()) {
char c = client.read();
#ifdef LOGGING
Serial.print(c);
#endif
lastRead = millis();
if (c == -1) {
error = true;
#ifdef LOGGING
Serial.println("HttpClient>\tError: No data available.");
#endif
break;
}
// Check that received character fits in buffer before storing.
if (bufferPosition < sizeof(buffer)-1) {
buffer[bufferPosition] = c;
} else if ((bufferPosition == sizeof(buffer)-1)) {
buffer[bufferPosition] = '\0'; // Null-terminate buffer
client.stop();
error = true;
#ifdef LOGGING
Serial.println("HttpClient>\tError: Response body larger than buffer.");
#endif
}
bufferPosition++;
}
#ifdef LOGGING
if (bytes) {
Serial.print("\r\nHttpClient>\tEnd of TCP transaction.");
}
#endif
// Check that there hasn't been more than 5s since last read.
timeout = millis() - lastRead > TIMEOUT;
// Unless there has been an error or timeout wait 200ms to allow server
// to respond or close connection.
if (!error && !timeout) {
delay(200);
}
} while (client.connected() && !timeout && !error);
#ifdef LOGGING
if (timeout) {
Serial.println("\r\nHttpClient>\tError: Timeout while reading response.");
}
Serial.print("\r\nHttpClient>\tEnd of HTTP Response (");
Serial.print(millis() - firstRead);
Serial.println("ms).");
#endif
client.stop();
String raw_response(buffer);
// Not super elegant way of finding the status code, but it works.
String statusCode = raw_response.substring(9,12);
#ifdef LOGGING
Serial.print("HttpClient>\tStatus Code: ");
Serial.println(statusCode);
#endif
int bodyPos = raw_response.indexOf("\r\n\r\n");
if (bodyPos == -1) {
#ifdef LOGGING
Serial.println("HttpClient>\tError: Can't find HTTP response body.");
#endif
return;
}
// Return the entire message body from bodyPos+4 till end.
aResponse.body = "";
aResponse.body += raw_response.substring(bodyPos+4);
aResponse.status = atoi(statusCode.c_str());
}
@Hootie81 Thanks a lot ! I’m going to try this. I was just wondering, when you were saying: on the send side of things. Were you talking about sending a request or sending data ? If you were talking about sending data, is it going to help the speed for getting data ?
Thanks again 
I did heaps of testing a while back and found most of the delays were caused by the core printing the request… it would print a couple of headers then wait about 2 seconds, then print a few more headers and wait again… and nearly always delay on the last linefeed! but the response is super quick, even on slow internet. I can fill a 4k buffer in less than a second with this code on a 3g connection, including sending the request! but i get a heap error… but that kinda expected with a buffer set to 4096 on a core
Having said that, i do plan on changing the receive side slightly too… but thats another day when i have recovered from a boys weekend in Bali this weekend!
Hey, did you had time to accelerate the request for getting data from a web page ? If yes, how do I update the library ?
Thanks !
@tchinou1 the code above increases the speed a fair amount. to use it you will have to create the .h and .cpp files by pushing the + in the top right of the web IDE, name the file HttpClient.h and it will automatically change the .cpp. it should add the correct #include for you in your .ino file. then cut and paste the code above into the files you just made…
the code is a drop in replacement for the httpclient you were originally using.
Let me know if you have any issues.
Seriously awesome man !
I was just wondering is I could get rid of this:
if (nextTime > millis()) {
return;
}
without getting any problem ?
Thanks !
I was using the original HTTP Library with my code, and after like 3-4 transmissions I would only get connection fails!
Now with your updated files and turning off logging, I get fast transmissions, but now whenever I get a connection failed, I do a small delay and then everything picks right back up! Thanks for working on the library!
Is that in the ino file? I don’t have my pc set up at the moment. . So I’m on my phone for the next couple of days.
You should be able to change it yes… I can’t remember what it’s for though… maybe have a look and see how nextTime is set and reduce the number that gets added to millis when set.
In the changes that you made for the HTTP Library is there a reason why these
lines:
http_request_t request = {0};
memset(&request, 0, sizeof(http_request_t));
are initially commented out in HTTP.cpp, line 64 ?
I’m looking into it now but the reason I’m bringing this up is because my json parser is having some issues due to the response body not being accurate. If the response body receives a a server response that is smaller than the last response body it keeps the extra characters in the body.
For example if my initial reponse body was {“Do that thing”:“200”}
Then my next server response is {“Do that task”:“200”}
The response body will look like {“Do that task”:“200”}} With the extra brace which messes up the parser. Should those two lines I mentioned above be uncommented in HTTP.cpp or should I put them in my own firmware every time I call response?
Cheers,
UST
I’ll fire up my computer and check but I don’t recall changing that part of the code…
No worries @Hootie81 
I might have found something maybe to do with buffer, but I’ll keep you posted and await your reply!
Hey @Hootie81
Adding this line:
memset(buffer, 0, sizeof(buffer));
In the section to receive the HTTP response fixed my issue, and everything seems to be working as normal !
I just checked the original HttpClient on GitHub, and there was a change made in October and the following was added
// clear response buffer
memset(&buffer[0], 0, sizeof(buffer));
i guess thats the reason why! Ill add it into the updated one above…
I apologize for my newbie question. Is there a way to include this library and disable the LOGGING at the same time? It'll be great to have it as an option when there are networking problem.
Thanks
Roge
@Roge, it is totally possible, just comment out the define by adding the double slashes like this
//#define LOGGING
that way each time the code gets to one of the #ifdef LOGGING points, LOGGING wont be defined so that section of code will be skipped.
Let me know if you have any issues, more than happy to help