Faster HTTP request

Sounds like you’re running into a TCPClient timeout? I’m using the develop branch and seeing responses of 500ms. Working on webhooks at the moment and also finding they’re too unreliable which is a real shame.

2 Likes

What I don’t understand is that before, the response time that I had was something like 500ms for me too. But, now for an unclear reason it’s a couple of second…

@JumpMaster
Can you point me to the dev branch that you are using? What I’m doing is fairly simple, it is worth a try.

I meant the firmware develop branch for compiling offline. Actually once the system1 and system2 is updated tcpclient works as expected and you can compile code in the cloud…

@JumpMaster I am not sure to understand, can I use the develop branch ? If yes, how ?

Take a look at this topic…

Interesting, but for I want to share my project with a couple of friends and I don’t want them to have to compile locally. I don’t get it, according to the web IDE HttpClient is the most used library and can’t find any topic about the slow response time… Have I miss something ?

The problem is in firmware 0.4.4 and earlier on the Photon. The TCPClient used in HttpClient is not detecting the server disconnecting so instead the connection is closed after a 5 second timeout. This is fixed in the next firmware release. To work around the issue for now copy HttpClient.cpp and HttpClient.h adding them to your program. Change line 5 as below and reduce the timeout. It need to be long enough that your request completes, try 2000 and then 1000.

static const uint16_t TIMEOUT = 5000; // Allow maximum 5s between data packets.

I fixed this in HttpClient by parsing the content length of the response into an integer and exited my loop after the response body buffer was equal to the content length. I can post the code if anyone is interested.

2 Likes

@rac146, Yes it can be nice to have !

Here it is… I added a getHeaderInfo function to the HttpClient. Basically it begins reading the response until it finds 1. Content Length and 2. Start of response body. Once the start of the response body is found it will return the content length as an int. You can then use this content length to read the rest of the response and know when the response body is fully read. Just a disclaimer, my C++ skills aren’t the best and I’m sure there are better ways to do this ;).

int HttpClient::getHeaderInfo()
{
    unsigned int bufferPosition = 0;
    bool contentLengthFound = false;
    bool contentLengthComplete = false;
    char contentLength[10];
    unsigned int contentLengthCount = 0;
    unsigned int contentLengthInt = -1;
    bool complete = false;
    char bufferHeader[512];

  do {
     if(client.available())
    {
       //get next char
        char c = client.read();

        bufferHeader[bufferPosition] = c;
        bufferPosition++;

        //2
        //if we know the content length header is being read then the next characters
        //are the actual numbers of the content length.
        //once we reach a newline, we have the entire content length.
        if(contentLengthFound && c != '\n')
        {
            contentLength[contentLengthCount] = c;
            contentLengthCount++;
        }
        else if(contentLengthFound && c == '\n')
        {
            contentLengthFound = false;
            contentLengthComplete = true;

            String raw_contentLength(contentLength);

            contentLengthInt = atoi(raw_contentLength.c_str());
        }

        //1
        //we're looking for the content length first
        //every time we get a character, convert entire buffer to String
        //and look for 'Content-Length: '
        if(!contentLengthFound)
        {
            String raw_response(bufferHeader);

            int contentLengthPosition = raw_response.indexOf("Content-Length: ");

            if (!contentLengthFound && contentLengthPosition != -1) {
                contentLengthFound = true;
            }
        }

        //3
        //once the content length is read completetly we want to find the 
        //response body. This is represented by '\r\n\r\n'
        if(contentLengthComplete)
        {
          String raw_response(bufferHeader);

          int bodyPos = raw_response.indexOf("\r\n\r\n");
          if (bodyPos != -1) {

            complete = true;

          }
        }

    }

  } while(!complete && client.connected());

   memset(&bufferHeader[0], 0, sizeof(bufferHeader));

  return contentLengthInt;
}

You want to make that getHeaderInfo call at line 162 of the existing HttpClient.cpp. Now in the while loop (starting at line 190), create a count to compare the response body to the content length:

(Pseudocode)

    int currentResponseCount = 0;
    while (client.available()) {
      char c = client.read();
      buffer[bufferPosition] = c;
      currentResponseCount++;
    
      if(currentResponseCount == contentLengthCount)
      {
        buffer[bufferPosition] = '\0';
        client.stop();
        //this will stop the loop
        error = true;
      }
      else
      {
        bufferPosition++;
      }
  }

Thanks @rac146 I’ll take a look a this this weekend ! @JumpMaster do you know when the next firmware is supposed to be available ?

No idea and the recent commits have reintroduced the bug.

I’ve nearly finished porting the Arduino http client library which is more sophisticated then the one we’re using. I’m also combining the previous changes to use client.write rather than client.print. Will have a version on GitHub later tonight (UK time) if testing goes well.

2 Likes

@JumpMaster I just want to say thank you for taking over and helping Particle with this rather important (fundamental?) library. You work is greatly appreciated!

1 Like

Thanks @Awake. Hopefully it will be something the community can build upon.

1 Like

Here is the ported Arduino library. I haven’t been able to do a massive amount of testing but HTTP GET with IP or hostname seem to work.

Feedback good or bad is welcome! :smile:

Nice job thanks !
If you are porting Arduino library to Spark, maybe you could be interesting in that one: http://drcharlesbell.blogspot.ca/2013/04/introducing-mysql-connectorarduino_6.html

I used this library to communicate with MySQL database and it was working pretty well. :smile:

@JumpMaster I just the library that you ported and the response time is pretty good ! Great Work ! However, I tried to get a long string of data and I couldn’t get it all.

String:

[{"1":"00aeff","2":"00aeff","3":"00d9ff","4":"00d9ff","5":"03fabc","6":"03fabc","7":"00fab3","8":"00fab3","9":"00ffb7","10":"00ffb7","11": "00ff3c","12": "00ff3c","13":"00ff33","14":"00ff33","15":"00ff1e","16":"00ff1e","17":"15ff00","18":"15ff00","19":"55ff00","20":"55ff00","21":"1","22":"1","23":"0","24":"0","25":"100","26":"0","27":"3000","28":"1","29":"2","30":"2"}]

String receive:

[{"1":"00aeff","2":"00aeff","3":"00d9ff","4":"00d9ff","5":"03fabc","6":"03fabc","7":"00fab3","8":"00fab3","9":"00ffb7

Is the website something that’s public so I can test? How long did it say the Content length was?

I’ve updated the .ino file with a workaround for when no Content-Length is supplied. I believe all websites should when HTTP/1.0 is requested but not all do.

Working pretty well right now, Thanks !