Http Download library


#1

This isn’t in Spark Library format yet, but will be soon. Until then, check the repo:

The library makes it really easy to download large files from HTTP servers and saving them to an SD card. It handles HTTP Range headers to work around the CC3000 packet size limitation. This adds a lot of overhead to the download, but adds immense reliability where non existed before. It will also retry specific packets if they fail.

If anyone else has been struggling with the 1460 byte packet size limitation, this could be a model for you to get it working, as well.

Lastly, my C++ is a bit rusty. Don’t judge :smile:


[Project Advice] Download bmp to an SD card
Best Method? - Download Large Files to SD using Photon
Upload download mp3 file to SD card
TCPClient read() data missed
#2

Great !!! How about upload function, you will work on your library?


#3

Great work on the lib, especially if you’ve managed to get things working where others have had problems! Could you tell me a bit about how you worked around the issues?

Cheers :-),
mat.


#4

Easy workaround: I just ask the server for the data in chunks using the HTTP Range header. An example request header is:
Range: bytes=0-1024

That will tell any HTTP 1.1 compliant server that I only want 1024 bytes of the file. Adding in the HTTP response headers overhead, the total size of the TCP packet stays under 1460 bytes and the CC3000 stays responsive.

The downside is that this is clearly a hack. I’m opening, sending, receiving, and closing a new TCP connection each time. This is super reliable because I can retry individual packets, but also very, very slow. My benchmarks are roughly 1K / second. It works great if you need to download something in the background, but obviously won’t let you stream HD video :wink:

I’m working now to eliminate some of the overhead by using HTTP persistent connections. That should eliminate the Connect/Disconnect/Wait overhead by doing all of the communication on a persistent socket.

Stay tuned,
ryan


#5

Looks like your really cleaned up my chunk download code. Looks fantastic! I look forward to issuing a pull request to download the file to flash


#6

I hadn’t actually seen your code before, but just took a look. I LOVE the way you traverse the HTTP headers with client.find(). I didn’t know that method existed! It’s not documented, but I’m totally going to use that. Right now, I’m just reading chunks as they’re available and storing them so I can strstr them later.


#7

Hi @rsteckler, good work with the library! Just wondering does this library work with the Photon?


#8

@rsteckler also, can’t see where you define any pins in the example… am I missing something?


#9

@rsteckler @ScruffR @harrisonhjones Wondering if I could tap into your minds real quick about this kinda older library that seems to do exactly what I’m wanting to do which is download via HTTP directly to an SD card.

I tried to compile this using Particle DEV and the CLI with both throwing compile errors. Not sure why since others have not complained about this when trying out the library.

Can you take a look and see if it’s something simple or not? Or maybe it’s just that the library is too old?


#10

The first thing to do is rearrange the files to fit the new file structure - look at a v2.0 lib and copy the structure.


#11

@ScruffR Ok, I took some time to look into the V2 libraries and I think changed the library file structure to match. I think I did it right? :confused:

All that seemed to do is make the SD.h folder not available when compiling.

If I take the files out of the “src” folder and move them to the main folder it no longer complains that the SD.h file is missing and I’m right back to where I started with 16 compile errors. Not sure why I get all these compile errors now when I figured they would have shown up for others.

The library in what I think is the V2 format is here:

@andrewa Built off this same library without problems but he is using Eclipse to compile so I’m not sure what the deal is.

I get all these errors:




My guess is that if this worked before then something must have changed that is breaking all these functions? Or maybe I just still do not have the v2 library file sturcture right?


#12

I’ll give it a try


@RWB, give this a try


#13

@ScruffR Thank you for taking the time to work on this.

I just gave it a go on a Photon with uSD card attached.

The loop runs just fine but it never stores a file on the SD card.

Did you give the library a try? Did it work for you?


#14

I haven’t tried it, I just got the project structure sorted and replaced the old SD library for SdFat. The problem might be some difference in behaviour between the two libraries.

But with this version you might be able to add some debug logs to find the culprit.


#15

Got it.

He had added a wrapper to the SD library which is probably the reason it’s not pushing data to the SD card.

I wanted to ask you if it was working for you before trying to mess with anything else.

Trying to incentivize @rickkas7 to create us an HTTP library optimized around the Photon if/when he has the time :slight_smile: Until then this is the only option I have to work with which is better than nothing :slight_smile:

Thank you for your help with this.

I did learn somethign about the 2.0 version of the Particle libraries sturcture.


#16

Intermingling libraries is never a good thing :wink:
From a (admittetly rough) look at the HttpDownload.cpp I didn’t see any need for special treatment in the SD library so I didn’t even look at that at all.
And I still see no point in messing inside two libraries to achieve one goal.