Spark Core Http Client Library

@bko, @juano2310,

I committed the string fix and a spark.json file to the repo. Can you test it so I didn’t make any mistakes?
(I’m sans core atm)

Cheers,
Nils

Also added experimental support for connecting to IPs from @ryotsuke.

Would really appreciate if people would give it a go and give thumbs up or down on these latest changes.

1 Like

@nmattisson thanks a lot!
I tried to validate it on the IDE and I think that the application files should be in a separated directory called “examples”.
Thanks again!

Hi @nmattisson

Thank you! :heartpulse:

The example library with the “right” directory structure for the web IDE and the JSON example is here:

There is a firmware directory with examples and doc directories (can be empty). So application.cpp and .h can be put in the examples directory (and renamed if you like).

I will have a look at the changes tomorrow (it’s late here). Thanks again!

1 Like

I added doc and example directories according to the repo @bko sent.

I’m not using the web IDE myself, so I’m not sure what the paths should be. If things don’t work, please look at those first and let me know if it should be different.

Thanks for your help everyone!

2 Likes

Thanks @nmattisson again and @bko for all your contributions!
I think I was able to submit the library to the IDE. Please, it would be great if anyone can double check that the library is accessible.
Thanks again!
Juan

Hi @nmattisson,

Thank you so much for your contribution! I was able to push data from my Spark Core to Ubidots using your library.

Here’s the tutorial on how to connect Spark to Ubidots.

3 Likes

@aguspg Looks great! Thanks for letting me know.

2 Likes

This is indeed a nice library. Thanks for sharing!

Just one question. Is there any reason why buffer was set to not much larger number than 1024, like even 20 times more.

There’s alway limited RAM and you need to consider saving some for other user code and not maximize whatever you can :smiley:

20x would be all the RAM the Spark Core has. :wink:

2 Likes

Hi @nmattisson!

I’ve been using the Httpclient library for a few weeks now to send data to a custom cloud.

In my setup, I’m sampling sensor data at 15 Hz and each sample has 12 bytes (which is why I use a priv. cloud :smiley: ). So I was wondering if I could do a POST every second with all this raw data.

That was too much data for one request body string, so I tweaked with the library and made String body[30]; in the request structure. Consequently, I changed the client.println(...) and a few others in the .cpp.

The code compiles flawlessly but almost always gives me a 400 Bad Request, with the reason being bad JSON formatting. Occasionally, I get the 200 OK, which is worth the same.

Could you help me sort this out? Are there more people in a similar situation?

Thanks!

Hi @LIGHThouse,

Can you post the code you’ve changed and an example of the JSON that is causing the issues?

Cheers,
Nils

I just wanted to point out that after researching it for another post, TCPClient.print() of an Arduino String has a particularly bad implementation right now that will limit performance. You can work around this by using

  client.write(myString.c_str());
4 Likes

Hi @nmattisson,

I’m restarting the experiment from scratch to eliminate some variables. Let me see how it goes.

Here’s the modified .cpp file:

#include "HttpClient.h"

#define LOGGING
static const uint16_t TIMEOUT = 5000; // Allow maximum 5s between data packets.

unsigned int x = 0, lgth = 0, chckNULL = 0;

/**
* Constructor.
*/
HttpClient::HttpClient()
{

}

/**
* Method to send a header, should only be called from within the class.
*/
void HttpClient::sendHeader(const char* aHeaderName, const char* aHeaderValue)
{
    client.print(aHeaderName);
    client.print(": ");
    client.println(aHeaderValue);

    #ifdef LOGGING
    Serial.print(aHeaderName);
    Serial.print(": ");
    Serial.println(aHeaderValue);
    #endif
}

void HttpClient::sendHeader(const char* aHeaderName, const int aHeaderValue)
{
    client.print(aHeaderName);
    client.print(": ");
    client.println(aHeaderValue);

    #ifdef LOGGING
    Serial.print(aHeaderName);
    Serial.print(": ");
    Serial.println(aHeaderValue);
    #endif
}

void HttpClient::sendHeader(const char* aHeaderName)
{
    client.println(aHeaderName);

    #ifdef LOGGING
    Serial.println(aHeaderName);
    #endif
}

/**
* Method to send an HTTP Request. Allocate variables in your application code
* in the aResponse struct and set the headers and the options in the aRequest
* struct.
*/
void HttpClient::request(http_request_t &aRequest, http_response_t &aResponse, http_header_t headers[], const char* aHttpMethod)
{
    // If a proper response code isn't received it will be set to -1.
    aResponse.status = -1;

    // NOTE: The default port tertiary statement is unpredictable if the request structure is not initialised
    // http_request_t request = {0} or memset(&request, 0, sizeof(http_request_t)) should be used
    // to ensure all fields are zero
    bool connected = false;
    if(aRequest.hostname!=NULL) {
        connected = client.connect(aRequest.hostname.c_str(), (aRequest.port) ? aRequest.port : 80 );
    }   else {
        connected = client.connect(aRequest.ip, aRequest.port);
    }

    #ifdef LOGGING
    if (connected) {
        if(aRequest.hostname!=NULL) {
            Serial.print("HttpClient>\tConnecting to: ");
            Serial.print(aRequest.hostname);
        } else {
            Serial.print("HttpClient>\tConnecting to IP: ");
            Serial.print(aRequest.ip);
        }
        Serial.print(":");
        Serial.println(aRequest.port);
    } else {
        Serial.println("HttpClient>\tConnection failed.");
    }
    #endif

    if (!connected) {
        client.stop();
        // If TCP Client can't connect to host, exit here.
        return;
    }

    //
    // Send HTTP Headers
    //

    // Send initial headers (only HTTP 1.0 is supported for now).
    client.print(aHttpMethod);
    client.print(" ");
    client.print(aRequest.path);
    client.print(" HTTP/1.0\r\n");

    #ifdef LOGGING
    Serial.println("HttpClient>\tStart of HTTP Request.");
    Serial.print(aHttpMethod);
    Serial.print(" ");
    Serial.print(aRequest.path);
    Serial.print(" HTTP/1.0\r\n");
    #endif

    // Send General and Request Headers.
    sendHeader("Connection", "close"); // Not supporting keep-alive for now.
    if(aRequest.hostname!=NULL) {
        sendHeader("HOST", aRequest.hostname.c_str());
    }

    //Send Entity Headers
    // TODO: Check the standard, currently sending Content-Length : 0 for empty
    // POST requests, and no content-length for other types.
    //if (aRequest.body != NULL) {
        //sendHeader("Content-Length", (aRequest.body).length());
    //} else if (strcmp(aHttpMethod, HTTP_METHOD_POST) == 0) { //Check to see if its a Post method.
        //sendHeader("Content-Length", 0);
    //}
	chckNULL = 0;
	lgth = 0;
	for(x=0;x<2;++x)
	{
		if(aRequest.body[x] == NULL)
		{
			++chckNULL;
		}
		else
		{
			lgth += (aRequest.body[x]).length();
		}			
	}
	if(chckNULL != 2)
	{
		sendHeader("Content-Length", lgth);
	}
	else if (strcmp(aHttpMethod, HTTP_METHOD_POST) == 0)
	{
		sendHeader("Content-Length", 0);
	}

    if (headers != NULL)
    {
        int i = 0;
        while (headers[i].header != NULL)
        {
            if (headers[i].value != NULL) {
                sendHeader(headers[i].header, headers[i].value);
            } else {
                sendHeader(headers[i].header);
            }
            i++;
        }
    }

    // Empty line to finish headers
    client.println();
    client.flush();

    //
    // Send HTTP Request Body
    //

    //if (aRequest.body != NULL) {
        //client.println(aRequest.body);

        //#ifdef LOGGING
        //Serial.println(aRequest.body);
        //#endif
    //}
	
	for(x=0;x<2;++x)
	{
		if(aRequest.body[x] != NULL)
		{
			client.println(aRequest.body[x]);
		}
		
		#ifdef LOGGING
		Serial.println(aRequest.body[x]);
		#endif
	}

    #ifdef LOGGING
    Serial.println("HttpClient>\tEnd of HTTP Request.");
    #endif

    //
    // Receive HTTP Response
    //
    // The first value of client.available() might not represent the
    // whole response, so after the first chunk of data is received instead
    // of terminating the connection there is a delay and another attempt
    // to read data.
    // The loop exits when the connection is closed, or if there is a
    // timeout or an error.

    unsigned int bufferPosition = 0;
    unsigned long lastRead = millis();
    unsigned long firstRead = millis();
    bool error = false;
    bool timeout = false;

    do {
        #ifdef LOGGING
        int bytes = client.available();
        if(bytes) {
            Serial.print("\r\nHttpClient>\tReceiving TCP transaction of ");
            Serial.print(bytes);
            Serial.println(" bytes.");
        }
        #endif

        while (client.available()) {
            char c = client.read();
            #ifdef LOGGING
            Serial.print(c);
            #endif
            lastRead = millis();

            if (c == -1) {
                error = true;

                #ifdef LOGGING
                Serial.println("HttpClient>\tError: No data available.");
                #endif

                break;
            }

            // Check that received character fits in buffer before storing.
            if (bufferPosition < sizeof(buffer)-1) {
                buffer[bufferPosition] = c;
            } else if ((bufferPosition == sizeof(buffer)-1)) {
                buffer[bufferPosition] = '\0'; // Null-terminate buffer
                client.stop();
                error = true;

                #ifdef LOGGING
                Serial.println("HttpClient>\tError: Response body larger than buffer.");
                #endif
            }
            bufferPosition++;
        }

        #ifdef LOGGING
        if (bytes) {
            Serial.print("\r\nHttpClient>\tEnd of TCP transaction.");
        }
        #endif

        // Check that there hasn't been more than 5s since last read.
        timeout = millis() - lastRead > TIMEOUT;

        // Unless there has been an error or timeout wait 200ms to allow server
        // to respond or close connection.
        if (!error && !timeout) {
            delay(200);
        }
    } while (client.connected() && !timeout && !error);

    #ifdef LOGGING
    if (timeout) {
        Serial.println("\r\nHttpClient>\tError: Timeout while reading response.");
    }
    Serial.print("\r\nHttpClient>\tEnd of HTTP Response (");
    Serial.print(millis() - firstRead);
    Serial.println("ms).");
    #endif
    client.stop();

    String raw_response(buffer);

    // Not super elegant way of finding the status code, but it works.
    String statusCode = raw_response.substring(9,12);

    #ifdef LOGGING
    Serial.print("HttpClient>\tStatus Code: ");
    Serial.println(statusCode);
    #endif

    int bodyPos = raw_response.indexOf("\r\n\r\n");
    if (bodyPos == -1) {
        #ifdef LOGGING
        Serial.println("HttpClient>\tError: Can't find HTTP response body.");
        #endif

        return;
    }
    // Return the entire message body from bodyPos+4 till end.
    aResponse.body = "";
    aResponse.body += raw_response.substring(bodyPos+4);
    aResponse.status = atoi(statusCode.c_str());
}

Here are the JSON objects I’m using. There are \ slashes because I’m using sprintf()
This one’s in response.body[0]:

[{\"a\":\"0\",\"b\":\"0\"},

This one’s in response.body[1]:

{\"a\":\"0\",\"b\":\"0\"},

This one’s in the last response.body[n-1]:

{\"a\":\"0\",\"b\":\"0\"}]

Thanks!

For anyone using the web IDE and getting a long, ugly error with the example code, I discovered that you need to edit the second #include statement. The top two lines should be:

#include "application.h"
#include "HttpClient/HttpClient.h"

That made the code work for me!

Does anyone have experience, or an example, of setting and using cookies with the httpclient lib?

Hey All,

I just got my sparkcore and got it up and running on Spark Build IDE. I tried your example for connecting to TimeAPI, and the code compiles and uploads fine. The serial print outs keep reading “HttpClient> Connection Failed”.

I did not change the example code at all, and I’m just wondering if anyone has had a similar problem or a solution for this.

Thanks in advance!

There is a couple of things… try changing the page it tries to open… i think its set to time.api or something default but i couldnt get that to work. try getting google or something more common.

if it still doesnt work try this

Is the TCPClient.print() still an issue? Should I replace them all in a pull request?