Http Download library

This isn’t in Spark Library format yet, but will be soon. Until then, check the repo:

The library makes it really easy to download large files from HTTP servers and saving them to an SD card. It handles HTTP Range headers to work around the CC3000 packet size limitation. This adds a lot of overhead to the download, but adds immense reliability where non existed before. It will also retry specific packets if they fail.

If anyone else has been struggling with the 1460 byte packet size limitation, this could be a model for you to get it working, as well.

Lastly, my C++ is a bit rusty. Don’t judge :smile:

9 Likes

Great !!! How about upload function, you will work on your library?

Great work on the lib, especially if you’ve managed to get things working where others have had problems! Could you tell me a bit about how you worked around the issues?

Cheers :-),
mat.

Easy workaround: I just ask the server for the data in chunks using the HTTP Range header. An example request header is:
Range: bytes=0-1024

That will tell any HTTP 1.1 compliant server that I only want 1024 bytes of the file. Adding in the HTTP response headers overhead, the total size of the TCP packet stays under 1460 bytes and the CC3000 stays responsive.

The downside is that this is clearly a hack. I’m opening, sending, receiving, and closing a new TCP connection each time. This is super reliable because I can retry individual packets, but also very, very slow. My benchmarks are roughly 1K / second. It works great if you need to download something in the background, but obviously won’t let you stream HD video :wink:

I’m working now to eliminate some of the overhead by using HTTP persistent connections. That should eliminate the Connect/Disconnect/Wait overhead by doing all of the communication on a persistent socket.

Stay tuned,
ryan

3 Likes

Looks like your really cleaned up my chunk download code. Looks fantastic! I look forward to issuing a pull request to download the file to flash

1 Like

I hadn’t actually seen your code before, but just took a look. I LOVE the way you traverse the HTTP headers with client.find(). I didn’t know that method existed! It’s not documented, but I’m totally going to use that. Right now, I’m just reading chunks as they’re available and storing them so I can strstr them later.

1 Like

Hi @rsteckler, good work with the library! Just wondering does this library work with the Photon?

@rsteckler also, can’t see where you define any pins in the example… am I missing something?

@rsteckler @ScruffR @harrisonhjones Wondering if I could tap into your minds real quick about this kinda older library that seems to do exactly what I’m wanting to do which is download via HTTP directly to an SD card.

I tried to compile this using Particle DEV and the CLI with both throwing compile errors. Not sure why since others have not complained about this when trying out the library.

Can you take a look and see if it’s something simple or not? Or maybe it’s just that the library is too old?

The first thing to do is rearrange the files to fit the new file structure - look at a v2.0 lib and copy the structure.

@ScruffR Ok, I took some time to look into the V2 libraries and I think changed the library file structure to match. I think I did it right? :confused:

All that seemed to do is make the SD.h folder not available when compiling.

If I take the files out of the “src” folder and move them to the main folder it no longer complains that the SD.h file is missing and I’m right back to where I started with 16 compile errors. Not sure why I get all these compile errors now when I figured they would have shown up for others.

The library in what I think is the V2 format is here:

@andrewa Built off this same library without problems but he is using Eclipse to compile so I’m not sure what the deal is.

I get all these errors:




My guess is that if this worked before then something must have changed that is breaking all these functions? Or maybe I just still do not have the v2 library file sturcture right?

I’ll give it a try


@RWB, give this a try

1 Like

@ScruffR Thank you for taking the time to work on this.

I just gave it a go on a Photon with uSD card attached.

The loop runs just fine but it never stores a file on the SD card.

Did you give the library a try? Did it work for you?

I haven’t tried it, I just got the project structure sorted and replaced the old SD library for SdFat. The problem might be some difference in behaviour between the two libraries.

But with this version you might be able to add some debug logs to find the culprit.

Got it.

He had added a wrapper to the SD library which is probably the reason it’s not pushing data to the SD card.

I wanted to ask you if it was working for you before trying to mess with anything else.

Trying to incentivize @rickkas7 to create us an HTTP library optimized around the Photon if/when he has the time :slight_smile: Until then this is the only option I have to work with which is better than nothing :slight_smile:

Thank you for your help with this.

I did learn somethign about the 2.0 version of the Particle libraries sturcture.

Intermingling libraries is never a good thing :wink:
From a (admittetly rough) look at the HttpDownload.cpp I didn’t see any need for special treatment in the SD library so I didn’t even look at that at all.
And I still see no point in messing inside two libraries to achieve one goal.

1 Like

Hey I know this is a long time ago - did you ever get this httpdownload library working to an SD card?

I think the only one I got working was a FTP to upload and download to a SD Card.

I have made a small amount of progress on this.

The example .ino given did not start the SD card controller - maybe I missed this in the source. Anyhow, fixed that and tried it.

Initially I got http response of 301 - The 301 redirect is considered a best practice for upgrading users from HTTP to HTTPS! As expected there was a file on the SD card, however, it was empty - figures.

I checked the URL for the wikimedia .jpg file and this was indeed an https: URL and the hostname and path appeared to be different. I have changed these as follows:

    request.hostname = "commons.wikimedia.org";
    request.port = 80;
    request.path = "https://wiki/Burning_Yellow_Sunset.jpg";

Now I am getting more bytes read and something in the .jpg file but both examples (simple and complex) are ending with a 404 error. Log below. Any ideas why this is the case - TCP on Photon does not support HTTPS?

**** Simple Example
SD File Opened for Write: sunset-s.jpg
Connecting to: commons.wikimedia.org:80
Start of HTTP Request.
GET https://wiki/Burning_Yellow_Sunset.jpg HTTP/1.1
Connection: close
Range: bytes=0-999
HOST: commons.wikimedia.org
End of HTTP Request.
Waiting for response
Reading response
Bytes available: 128
Processing headers
Bytes available: 128
Processing headers
Bytes available: 128
Processing headers
Bytes available: 128
Processing headers
Bytes available: 128
Processing headers
Bytes available: 128
Processing headers
Bytes available: 128
Processing headers
Bytes available: 128
Processing headers
Bytes available: 128
Bytes available: 128
Bytes available: 128
Bytes available: 128
Bytes available: 128
Bytes available: 128
Bytes available: 47
Bytes read: 1839
Connection closed
SD File Closed
Response status: 404

1 Like

Real quick answer because I’m on the go.

Your path should only contain the part of the url after the domain. It shouldn’t contain the protocol (https).

For SSL, you should only change the port to 443. There might also be another parameter that specifies a secure connection, but don’t change the path or domain.

1 Like