Spark Core Http Client Library

Hello,

I’m just starting out with the Spark Core and found myself needing an HTTP Client Library to interact with my REST API. Unfortunately the libraries I found all seemed to rely on the Arduino Ethernet Library which made them unsuitable for the Spark Core. So today I started working on my own library, which you can now find on Github.

Before this I’ve never touched C++ so the code is probably both badly architected and in poor style, but it’s a start and it will improve as I’m learning. Contributions are very welcome and any feedback is appreciated! Don’t hesitate to get in touch.

Cheers,
Nils

6 Likes

Nice! Thank you for sharing your library! Definitely something a lot of people have a need for. :slight_smile:

Thanks @Dave!

While adding support for custom HTTP headers today, I noticed that there is an extra character being sent along with the intended string when using the TCP Client. Either the TCP Client is adding an extra character to the beginning of strings being sent or (quite possibly) the reason for that character has something to do with C++ strings and might not have anything to do with the Spark firmware. I thought I’d share it here just in case, maybe the explanation is easy.

I send the HTTP request body like this:

Application.cpp:

  request.body = "{\"key\":\"value\"}";

HttpClient.cpp

  // Empty line to finish HTTP Headers
  client.println();
  client.flush();

  // Send HTTP Request body.
  client.println(aRequest->body);

And for some reason jsontest.com responds with an error, saying there is a character before the first curly brace in the string:
“error”: “A JSONObject text must begin with ‘{’ at 1 [character 2 line 1]”

I haven’t looked too deeply in the TCP Library, but since that is used for many things I’d assume it rather bug-free.

It’s not a big deal, any JSON parser will probably strip leading and trailing whitespace anyway, but I’m guessing it shouldn’t be there.

I wonder if it’s expecting a json content type header, or a \r\n mismatch or something? I would want to do a capture with something like WireShark and see what the extra character is, definitely. I probably won’t get a chance to test that today, but if I do I’ll post my results here.

Thanks! The library works for me in some cases, but i have one url where I have trouble:

I am trying to make a get request to the following url, but it fails.

http://query.yahooapis.com/v1/public/yql?q=select%20%20item.forecast.text%2C%20item.forecast.high%2C%20item.forecast.low%20from%20weather.forecast%20where%20u%3D'c'%20and%20woeid%3D%20781788&format=json&callback=

(returns json with weather forecast)

Can somebody try to load it? On my installation it stops after sending the request.

I get an error using curl:

<error xmlns:yahoo="http://www.yahooapis.com/v1/base.rng" yahoo:lang="en-US"><description>Invalid identfier c. me AND me.ip are the only supported identifier in this context</description></error>

So I tried replacing the ‘c’ with %27c%27 and that worked better.

on my spark, even the basic url http://query.yahooapis.com/v1/public/yql does not work.

Hi @Coffee

Are you sure your GET request has a Host: query.yahooapis.com line? This has tripped me up before with edge routers.

I am using this successfully to get and parse an RSS feed. They are easy to parse!


const char weatherServer[] = "weather.yahooapis.com";
const char weatherURLtoGet[] = "/forecastrss?w=2367105";

and the function:

void sendGetRequest(const char * server, const char * url)
{
    if (myTCP.connect(server, 80)) {
        //Serial1.print("Connected to Server");
        //digitalWrite(D7,HIGH);
        myTCP.print("GET ");
        myTCP.print(url);
        myTCP.println(" HTTP/1.0");
        myTCP.println("Connection: close");
        myTCP.print("Host: ");
        myTCP.println(server);
        myTCP.println("Accept: text/html, text/plain");
        myTCP.println();
        myTCP.flush();
    } 
}

Maybe you could try this simpler way as a way to help debug the library?

Hi @Coffee,

Like @bko is saying, you need to split the URL between the host (“query.yahooapis.com”) and the path (“v1/public/yql”) — the former is set when you open the connection and the latter when you make a request.

It would be helpful if you posted your code and we can see if the problem is there or with the library!

Cheers,
Nils

1 Like

Hi guys, thanks for the hints.

here is my code, i removed non-relevant staff, i hope it still represents my problem:

#include "HttpClient.h"

HttpClient* client;

void setup() {
	Serial.begin(9600);
	client = new HttpClient("query.yahooapis.com", 80);
}

void loop() {

	Serial.println("loading weather");
	http_request_t request;
	request.path =
			"http://query.yahooapis.com/v1/public/yql?q=select%20%20item.forecast.text%2C%20item.forecast.high%2C%20item.forecast.low%20from%20weather.forecast%20where%20u%3D'c'%20and%20woeid%3D%20781788&format=json&callback=";
	request.body = "";
	Serial.println(request.path);

	http_response_t* response = client->get(&request);
	if (response->status == 200) {
		Serial.print("weather response: ");
		Serial.println(response->body);
	} else {
		Serial.print("weather request failed ");
		Serial.println(response->body);
	}
}

@Coffee,

This line is wrong:

You need to only have the path in the request, not the full URL (and not the HTTP marker). Also, '%20 ' in paths typically denotes spaces and are generally not to be recommended. Are you sure the request is formatted correctly?

EDIT: I tested it, and it actually works despite the strange formatting so I'm supposing they just designed their API that way. I got a pretty big JSON reply though, so you might also have to look at the TCP buffer size so that the response fits.

I made a bunch of updates to this library today: https://github.com/nmattisson/HttpClient/commits/master

If anyone is using it with large JSON responses, let me know how it works for you!

3 Likes

This library is coming along, and I think it’s getting ready for more widespread use. Most things have been changed since I originally posted it, and the current usage looks like this:

request.hostname = "www.timeapi.org";
request.port = 80;
request.path = "/utc/now";

// Get request
http.get(response, request, headers);
Serial.print("Application>\tResponse status: ");
Serial.println(response.status);
Serial.print("Application>\tHTTP Response Body: ");
Serial.println(response.body);

A lot of memory problems were fixed, and the overall footprint is smaller now. Feel free to give it a try — I hope it’s quite beginner friendly now.

2 Likes

@nmattisson Unfortunately I didn’t find your library before porting the simple RestClient for Arduino.

Posted info about it here: https://community.spark.io/t/rest-client-library-for-spark/4114

Maybe there is something interesting in it for you. I’m digging into your library to understand the capabilities…

Example code is not working for me when using in WEB ide
I’ve pasted code of cpp and h files in web IDE

#include "HttpClient.h"
/**
* Declaring the variables.
*/
unsigned int nextTime = 0;    // Next time to contact the server
HttpClient http;

// Headers currently need to be set at init, useful for API keys etc.
http_header_t headers[] = {
    //  { "Content-Type", "application/json" },
    //  { "Accept" , "application/json" },
    { "Accept" , "*/*"},
    { NULL, NULL } // NOTE: Always terminate headers will NULL
};

http_request_t request;
http_response_t response;


void setup() {
    Serial.begin(9600);
    Serial1.begin(57600);
    Serial1.print("\n");
    Spark.function("cmd", fnCmd);
}

int fnCmd(String command) {
        Serial1.print(command + "\n");
        return 1;
}


void loop() {
    
    
      if (nextTime > millis()) {
        return;
    }

    Serial.println();
    Serial.println("Application>\tStart of Loop.");
    // Request path and body can be set at runtime or at setup.
    request.hostname = "192.168.1.12";
    request.port = 9605;
    request.path = "/cmd";

    // The library also supports sending a body with your request:
    //request.body = "{\"key\":\"value\"}";

    // Get request
    http.get(response, request, headers);
    Serial.print("Application>\tResponse status: ");
    Serial.println(response.status);

    Serial.print("Application>\tHTTP Response Body: ");
    Serial.println(response.body);

    nextTime = millis() + 500;
   
}

Getting error http://pastebin.com/y5iZpuhM

Apparently you’ve updated code but not updated readme - responce and request should be switched places in http.get
Though still error: http://pastebin.com/9ywUSgmF

Managed to get it compiled with following workaround http://screenshots.ryotsuke.ru/scr_f0f168dc8eb9.png

And got to RED BLINKING core after that :frowning:
Any help? Code to reproduce is above

8 blinks = Out of heap memory

1 Like

Hi @ryotsuke,

I’ve never tried using the web ide, and have no plans to, so no guarantees there. Maybe try putting your code in a local build environment, and see if you encounter the same problems?

Cheers,
Nils

1 Like

Locally it doesn’t build at all :-/ http://screenshots.ryotsuke.ru/scr_75b38be56cef.png

In function `HttpClient::get(http_request_t&, http_response_t&, http_header_t*)':
N:\Spark\core-firmware\build/../src/HttpClient.h:93: undefined reference to `HttpClient::request(http_request_t&, http_response_t&, http_header_t*, char const*)'
obj/src/application.o: In function `__static_initialization_and_destruction_0':
N:\Spark\core-firmware\build/../src/application.cpp:8: undefined reference to `HttpClient::HttpClient()'
collect2.exe: error: ld returned 1 exit status

P.S. builded after adding cpp file to build.mk
Cant test if works as new firmware does not want to connect to Wifi

@nmattisson
I’ve patched HttpClient to support IP addresses:
Don’t know how to do pull requests, so:
http://pastebin.com/zLRgBFzK
http://pastebin.com/8J3EBPUe

Changes:

  1. Do not return from connect code before logging that connect has failed
  2. Add field IPAddress ip to Request struct
  3. Use request.ip if request.hostname is unavailable

Sample usage:

// Request path and body can be set at runtime or at setup.
request.ip = {192,168,1,12};
request.port = 9605;
request.path = "/cmd";

// The library also supports sending a body with your request:
//request.body = "{\"key\":\"value\"}";

// Get request
http.get(request, response, headers);
Serial.print("Application>\tResponse status: ");
Serial.println(response.status);

Serial.print("Application>\tHTTP Response Body: ");
Serial.println(response.body);

@Dave could you take a look why this library produces 8RED errors when using in Web Ide?

1 Like