[Solved] Httpclient gets stuck - incomplete data

Troubleshooting
1

I am having a problem that will have an obvious solution when someone points out what I am missing, but until then I am stuck.

I am accessing weather data from the following webpage:
http://api.worldweatheronline.com/free/v2/weather.ashx?q=76112&format=xml&num_of_days=1&date=today&cc=no&tp=24&key=abcdefg1234567890

It works fine in a browser.

But the following simple code truncates the data after just a few characters received:

#include "HttpClient.h"
HttpClient http;
// Headers currently need to be set at init, useful for API keys etc.
http_header_t headers[] = {
	//    { "Content-Type", "application/json" },
	//  { "Accept" , "application/json" },
	{ "Accept" , "*/*"},
    { NULL, NULL } // NOTE: Always terminate headers will NULL
};

http_request_t request;
http_response_t response;

void setup()
{
  delay (10000); // Allow 10 seconds to get terminal going.
  Serial.begin(9600);
  Serial.println();
  Serial.println("Today weather query");
  request.hostname = "api.worldweatheronline.com";
  request.port = 80;
  request.path = "/free/v2/weather.ashx?q=76112&format=xml&num_of_days=1&date=today&cc=no&tp=24&key=abcdefghijkl1234567";
  http.get(request, response, headers);
  Serial.println (response.body);
  Serial.println("End of data reached succesfully");
}

void loop()
{
}

Help?!?!

2

What type of device are you using?

3

A Photon.

4

Looking at how that HttpClient library works, it’s pretty naive. What’s happening is this:

The client code uses a char buffer[1024] for reading the response. When the buffer fills up, it just stops reading.

One big problem is that this buffer is used for both the headers and the body of the response. Then, it saves the HTTP response code, saves the body (everything after a \r\n\r\n pair), and tosses the response headers away. And this site is sending over 900 bytes of headers, and a 1700 byte body.

So yeah, you’re probably getting less than 100 bytes of the actual response body.

There’s another HttpClient library on Github that might work better(?):

EDIT: But if you want a quick fix, you can just manually copy the original library code into your project and change the buffer size to 4096, or something.

2 Likes
5

@dougal AHA! So it’s the headers! I was wondering why other websites would show me more data before truncating!
I am keeping a close eye on memory use of my project, and so far it’s very small, so a bigger buffer size should do the job.

(I knew there was a simple explanation… my brain just needed a shove in the right direction!)

THANKS!