Reading a xml using http 1.1 in chunks

I’m trying to read the state.xml file from this device:

I’ve ended up manually connecting to it and can send the get command for the state.xml file no problem with authentication.

I’m currently reading it using client.read() and writing it one byte at a time to an array as per the http client library.

The issue is I miss the end of the file. My read takes 240ms and gets the 638 characters only, the buffer is set to 1024.

So i get:
��������HTTP/1.1 200 OK Cache-Control: no-store Content-Type: text/xml Content-length: 705 <?xml version=“1.0” encoding=“utf-8” ?><datavalues><input1state>0</input1state><input2state>0</input2state><input3state>0</input3state><input4state>1</input4state><relay1state>0</relay1state><relay2state>0</relay2state><relay3state>0</relay3state><relay4state>0</relay4state><units>C</units><sensor1>x.x</sensor1><sensor2>x.x</sensor2><sensor3>x.x</sensor3><sensor4>x.x</sensor4><s1Alrm>0</s1Alrm><s2Alrm>0</s2Alrm><s3Alrm>0</s3Alrm><s4Alrm>0</s4Alrm><count1>0.000</count1><count2>0.000</count2><extvar0>0.00</extvar0><extvar

which is the truncated version of:

HTTP/1.1 200 OK Cache-Control: no-store Content-Type: text/xml Content-length: 705 &lt<?xml version=“1.0” encoding=“utf-8”>00010000Cx.xx.xx.xx.x00000.0000.0000.000.000.000.000.043200:0C:C8:03:59:C11563655166`

My relevant bit of code is below:

unsigned int bufferPosition = 0;
 char buffer[1024];
    TCPClient client;
    client.connect("192.168.1.2", 80);
    client.print("GET /state.xml HTTP/1.1\r\n\r\n");
    startfetch = millis();
    while (bufferPosition<1023) {
	char c = client.read();
            if (bufferPosition < sizeof(buffer)-1) {
            buffer[bufferPosition] = c;
            } 
         bufferPosition++;
    
    }
 endfetch = millis();
buffer[bufferPosition] = '\0'; // Null-terminate buffer

I was previously using while(client.connected() for the read loop however this was quite a bit slower.

Changing the buffer size doesn’t change the data I get.

I’m not sure if the server is ditching the connection before I read all the data or if there is some other issue going on - there is other code on the device.

I cannot for the life of me understand how to use the chunk version of client.read() with the uint8_t buffer - if possible could someone give me a working example.

What exactly is the stumbling block for you with this function?
https://docs.particle.io/reference/device-os/firmware/photon/#read--2

bytesRead = client.read(buffer, length);

There isn’t really more to an example than that.

However, I (have to) guess you get an error message complaining that your buffer is not uint8_t*, right?
But that doesn’t matter since char[] and uint8_t* are both pointers to a memory space of same layout. Hence you can just use - standard C, not specific to TCPClient - typecasting

bytesRead = client.read((uint8_t*)buffer, length);

Another thing worth noting with TCPClient is this
image

This means, all Stream methods can be used too.

BTW, I can’t see which platform you are using - Photon and Argon use WiFi an have a max TCP chunk size of 1024 but cellular devices like Electron and Boron only allow max 512.

1 Like

Thank you so much for replying.

I’m using a photon, with the length set to 1024.

I’ve got further, however when I try to read back the buffer I get only one or two characters, both using the syntax you suggested, or declaring a uint8_t variable then trying to get it back to a string using String strout = (char*)buffer;

Ultimately I need to parse the result so I can then act when the variables change - so it needs to be a string? How do I get the string out of the uint8_t buffer?

This is where the code is now

    const size_t bufSize = 1024;
    char buffer[1024];
    TCPClient client;
    client.connect("192.168.1.2", 80);
    client.print("GET /state.xml HTTP/1.1\r\nAuthorization: Basic <removed>\r\n\r\n");
    startfetch = millis();
    if (client.connected()) {
	    client.read((uint8_t*)buffer,bufSize);
        } 
    buffer[1024] = '\0'; // Null-terminate buffer
 endfetch = millis();
String out = buffer;
 Particle.publish("xmlout",out,PRIVATE);

Particle.publish() is limited to 622 bytes
https://docs.particle.io/reference/device-os/firmware/photon/#particle-publish-

Also, when requesting data, you may need to allow for the response to come in.
As the example above shows, client.read(buf, len) will return the number of bytes actually read. If that tells you that you not yet got the number of bytes you expect, you should probably wait for more data to come in and read again (appending to the data you already read). You can also use client.available() to first inquire how much data you should expect to be readable at the time.

Since you are issuing a HTTP GET request, you could see if the HTTPClient library does the chunked read for you.

1 Like