[Resolved]- Firmware upgrade via SoftAP!

Troubleshooting
1

My application is offline but I want to take advantage of the Wifi on the photon. I’ll be using the SoftAP to display values and I want to be able to do a firmware upgrade of my product via softap because this way it’s easy for the customer and he can do it with a cellphone after downloading the file from my website.

So the first step to do that is to be able to read a file from the soft ap. Right now I’m using a text file but the issue is the same with the binary file.

The html code look like this:

<p>Select file for firmware upgrade</p>
 <form id="uploadbanner" enctype="application/x-www-form-urlencoded" method="post" action="upgrade">
   <input id="fileupload" name="myfile" type="file" />
   <p><input type="submit" value="Begin upgrade" id="submit" />
</form>

The decoding look like this. Right now I’m simply sending everything to the serial port for debugging.

void myPage(const char* url, ResponseCallback* cb, void* cbArg, Reader* body, Writer* result, void* reserved)
{
    String urlString = String(url);
    Serial.printlnf("handling page %s", url);
    
    if (strcmp(url,"/index")==0) {
        Serial.println("sending redirect");
        Header h("Location: /index.html\r\n");
        cb(cbArg, 0, 301, "text/plain", &h);
        return;
    }

    if (strcmp(url, "/upgrade") == 0)
    {
        // Here is the output I should get
        /*
        POST Data: ------WebKitFormBoundaryEEbsS0VLVz50qOeT
        Content-Disposition: form-data; name="myfile"; filename="binfile"
        Content-Type: application/octet-stream

        content file here...
        */
        
        uint8_t contentData[513];
        int dataPos;
        int datalength = body->read(contentData, sizeof(contentData)-1);
        contentData[513] = '\0';
            ;
        String contentString = String((const char*)contentData);
        Serial.print((const char*)&contentData[0]);
        Serial.println("");

        dataPos = contentString.indexOf("filename=");
        Serial.printlnf("Index of filename: %d", dataPos);

        if(dataPos != -1)
        {
            // Get filename here if we want
            String filename = contentString.substring(dataPos + 10, contentString.indexOf("Content-Type")-1);
            Serial.printlnf("Filename: %s", filename.c_str());
        }
        
//         dataPos = contentString.indexOf("octet-stream");
//         Serial.printlnf("Index of octet-stream: %d", dataPos);
        dataPos = contentString.indexOf("\r\n\r\n");
        if(dataPos != -1)
        {
            dataPos += 4;            
            Serial.println("Data start here:");
            datalength -= dataPos;

            while (body->bytes_left)
            {
                Serial.printf("Write data:%s", &contentData[dataPos]);

                dataPos = datalength;
                if (dataPos == datalength)
                {
                    do
                    {
                        if(datalength == -1)
                            Serial.println("Datalength -1");
                        //delay(50);
                        datalength = body->read(contentData, sizeof(contentData)-1);
                        contentData[513] = '\0';
                        //if (datalength == 0)
                        {
                            Serial.printlnf("Length = %d, Bytes left: %d", datalength, body->bytes_left);
                        }
                    } while (datalength <= 0 && body->bytes_left);
                    dataPos = 0;
                }
            }

            Serial.println("");
            Serial.printlnf("Bytes left: %d", body->bytes_left);

            delay(20);
            
        }
    }

    int8_t idx = 0;
    for (;; idx++) {
        Page& p = myPages[idx];
        if (!p.url) {
            idx = -1;
            break;
        }
        else if (strcmp(url, p.url) == 0) {
            break;
        }
    }

    if (idx == -1) {
        cb(cbArg, 0, 404, nullptr, nullptr);
    }
    else {
        cb(cbArg, 0, 200, myPages[idx].mime_type, nullptr);
        result->write(myPages[idx].data);
    }
}

Right now I’m able to get all the file but there are part missing… the debug output look like this. Check line number 77

Data start here:
Write data:This is a test file to figure out if all the data is sent.
00
01
02
03
04
05
06
07
08
09
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
7@Length = 512, Bytes left: 11019
Write data:
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
@Length = 512, Bytes left: 10507
Write data:441
442
443
444
445
446

And it continues like this to the end of the file.

Why is the read function not returning all the data. Is it because I’m too slow to process the data? What else can I do?
Thanks!

1 Like
2

Hi Suprazz,
I can’t answer your direct question, but do see something else which might cause a problem at some point. I believe you are indexing off of an array (twice).

You defined contentData array to be 513 characters in length. Therefore the last valid index would be 512. And 0 is the first value. In two places you have the following line of code: contentData[513] = ‘\0’;

3

the contentData[513] = ‘\0’; was only for the printf to be sure that I dont output more that the buffer content.
In reality it’s useless and I can remove it.

4

I understand the intent, but you should use contentData[512] = ‘\0’; instead. By using 513, you are actually setting the contents of whatever is in memory AFTER your array. This next byte could be an unused memory location or it could be a loop variable. You don’t know. Bad things can happen and be almost impossible to track down.

For example, looking at your code, it might be your “dataPos” variable. You declare it after contentData, so it might be located in memory just after your array. Not for certain, but it could be there.

5

yes you’re right! brain fart here!

6

Even if I do 2 read one after the other, there is an hole in the data:

        int datalength = body->read(contentData, sizeof(contentData)-1);
        contentData[sizeof(contentData)-1] = '\0';
        
        int datalength2 = body->read(contentData2, sizeof(contentData)-1);
        contentData2[datalength2] = '\0';    
        Serial.printf("Write data:%s", &contentData[dataPos]);
        Serial.printf("Write data2:%s", &contentData2[0]);

And output is like this:

...
73
74
75
76
77
Write data2:
211
212
213
214
215
...
7

My app does something similar: It sends “a lot” of data over the serial channel. We found that the channel can accept only some large number of characters to print. You are printing 2 times 512 times roughly 4 chars. 4K characters. Maybe that’s too many and the serial channel will barf.

We block ours into small records to send a chunk at a time. We don’t really use printf directly; instead a local bufferer. Once we have a buffer composed of many small records, our background process in “loop()” unpacks the big buffer and sends the small records one at a time.

Just a thought. Maybe you are overflowing the capacity of the serial channel. (Not sure if channel is the right word.)

Similarly, “loop()” should not take “too long” to execute before getting back to the firmware controlling the chip. But I doubt your program is taking THAT long. I think the time value is something like 10 seconds or so.

8

Here is what I found this morning while playing with the buffer size…
No matter the size of the buffer, if it’s under 1152 bytes the data of the second read is always at the same position in my file (line 211).

So, If I read 2 times 256 bytes and I print both, first print will begin at the beginning of the data, and second print will output data at line 211 of my file and up. So there is an hole in the data.
If I do the same thing with 512 bytes buffers, same thing. Second print will also start a line 211
If I use buffers bigger than 1152 bytes, then the second read is offseted at line 441.

If I read exactly 1152 bytes in each body->read(), I have all the data and no holes in the data!

It look like the data is read in chunk and if you have less data than the chunk size, everything is fine, but as soon as the data that you need to read is bigger than the chunk size you cannot do 2 consecutive read in the same “chunk”.

and BTW, 1152 is a number used directly used or a multiple used in the TCP firmware functions!

@ScruffR and @KyleG lease read this. I dont know who to contact to open a ticket regarding this issue.

2 Likes
9

@Suprazz, I’m not sure but I don’t think this has really to do with the Particle firmware but might be in connection with the protocol/transport layer or your network settings (e.g. MTU).

I think @rickkas7 or @bko had a good background post somewhere on here, but I’m unable to locate it again. Maybe they can chime in.

On the other hand https://support.particle.io is the place to post a support request.

1 Like
10

I’m using the functionalities provided by the “firmware” of the particle photon so the problem is clearly not in my application.

Thanks for the quick answer I’ll wait for updates.

11

Hi @Suprazz

I am sorry but the problem is likely to be in your code, not in the system firmware. @rickkas7 in particular has done extensive stress testing on TCP connections, transferring megabytes of data. Your earlier code did in fact write past the end of two arrays which made it buggy for sure.

With the Arduino-compatible interface provided on Particle devices, you are not dealing with a high-level PC networking stack that buffers all the data and gives it to you at your leisure. Instead you have a low-level, just above the packet level in fact, interface that you must manage and sometime buffer in your code if you want performance.

Soft-AP mode may have other performance challenges since it comes from code in the WICED software.

The serial port is not as fast as the network connection typically and so dumping everything to serial puts an extra slow down in your code. Again, the serial port data is buffered with a small buffer but there is no giant PC OS to buffer all the data for you--you have do that yourself.

Why don't you write code that tests if you are getting the data you think are sending directly on the Photon and just report success or failure over the serial port? That way your performance will not be limited by the small serial port buffer.

1 Like
12

I can create an other example to demonstrate the issue but my last post clearly explain the issue and what is happening. I can remove the serial print statement to confirm that the issue is still there without it but the issue is at getting the data, not writing it to the serial port.

The issue happen when requesting the data via the body->read function, before I write the info to the serial port. And this is why I did everything with 2 buffers instead of only one, to demonstrate that 2 consecutive read dont return consecutive data if the buffer size is different than 1152 bytes long.

This code sample output all the data without anything missing

        uint8_t contentData[1152];
	uint8_t contentData2[1152];
	int datalength = body->read(contentData, sizeof(contentData));
	int datalength2 = body->read(contentData2, sizeof(contentData));
	Serial.write(&contentData[0], datalength);
	Serial.write(&contentData2[0], datalength2);

But this code return non continuous data. Data is lost between the two reads… and body->bytes_left demonstrate that because after processing all the data, body->read will return 0 while body->bytes_left still return a value bigger than 0.

uint8_t contentData[512];
uint8_t contentData2[512];
int datalength = body->read(contentData, sizeof(contentData));
int datalength2 = body->read(contentData2, sizeof(contentData));
Serial.write(&contentData[0], datalength);
Serial.write(&contentData2[0], datalength2);
13

OK, that is a good test case. I think the WICED http server is expecting you to read entire MSS packets when you call the body->read function. There appears to be a bytes_left value there as well, so you could check that but I don’t see the need particularly. The call to the WICED TCP reader eventually uses an uint16 as the size, so there is a fundamental limit of 65535 bytes in one call.

If want someone to look at this, I would suggest that you create a github issue with the two code samples above as a test case and explain that is only applies to soft-ap:

When I search the github source for 1152, I only find one reference where it is defined as the TCP_MSS (maximum segment size), which is in turn used to define some buffer sizes.

Is this somehow hard for you to work-around? If you just read in TCP_MSS sized chunks it works OK, right?

14

The issue is on github now: https://github.com/spark/firmware/issues/1131

Yes I’'ll work with 1152 bytes buffers and it will be fine for now but I think this issue should be solved or documented.

Thanks

1 Like
15

@Suprazz

I see from this thread that the issue is now on github which hopefully will be fixed soon.

Like ScruffR linked to, to file a support ticket that can be done at the bottom of this page, and looks like the attached screen shot.

Feel free to PM me if there is anything I can assist with.

Kyle

Screen Shot 2016-10-01 at 11.26.11 AM.png1156×572 23.8 KB

16

Great! Thanks for the support!

I was finally able to to the firmware upgrade of the particle photon over the Soft AP.

The upgrade cannot be done directly in the page handler of the soft ap, especially if the SYSTEM_THREAD(ENABLED);

So I saved the data to an external memory and I did the upgrade in the main loop.

I also created this so I can refer later and it can help others: https://github.com/Suprazz/ParticleUpgradeExample

I’ll try to make a complete example of the upgrade code via Soft AP.

4 Likes
17

Here is the code to upgrade via SoftAP.

Right now, it works with a laptop (windows 10) and Android.

There is an issue with the Iphone I have. The last body->read doesnt return all the data (I dont know why) and because of the previous bug I discovered, it’s not possible to get the few bytes left).

Here is the debug log… the issue of doing 2 consecutive read is know. The issue of not reading all the available data is unknow:

handling page /upgrade
Length = 1152, Bytes left: 2863
Webkitform length = 46
Length = 1152, Bytes left: 1711
Length = 1152, Bytes left: 559
Length = 402, Bytes left: 157
Length = 0, Bytes left: 157
body->readerr

The upgrade file is stored to a SPI flash (or it could be on a SD card) because doing the upgrade directly in the myPage handler cause problems and always return upgrade failed no matter what I do.

2 Likes
18

New bug found!

If the content of the data sent over SoftAP (including headers) is over 65536 bytes data is lost again.

body->bytes_left is defined as a size_t. Is this a uint16? If not, is it a Stream class limitation?

19

Yes, there is a limit of 2^16-1 as noted above.

1 Like
20

Thanks for the fast answer.

Do you have a suggestion of what I could do to avoid this issue? Because right now it work well (if under 65k) but with this limitation, I cannot do anything.

I could split the file in 2 parts but tha’ts not really solution…