This isn’t in Spark Library format yet, but will be soon. Until then, check the repo:
The library makes it really easy to download large files from HTTP servers and saving them to an SD card. It handles HTTP Range headers to work around the CC3000 packet size limitation. This adds a lot of overhead to the download, but adds immense reliability where non existed before. It will also retry specific packets if they fail.
If anyone else has been struggling with the 1460 byte packet size limitation, this could be a model for you to get it working, as well.
Great work on the lib, especially if you’ve managed to get things working where others have had problems! Could you tell me a bit about how you worked around the issues?
Easy workaround: I just ask the server for the data in chunks using the HTTP Range header. An example request header is:
Range: bytes=0-1024
That will tell any HTTP 1.1 compliant server that I only want 1024 bytes of the file. Adding in the HTTP response headers overhead, the total size of the TCP packet stays under 1460 bytes and the CC3000 stays responsive.
The downside is that this is clearly a hack. I’m opening, sending, receiving, and closing a new TCP connection each time. This is super reliable because I can retry individual packets, but also very, very slow. My benchmarks are roughly 1K / second. It works great if you need to download something in the background, but obviously won’t let you stream HD video
I’m working now to eliminate some of the overhead by using HTTP persistent connections. That should eliminate the Connect/Disconnect/Wait overhead by doing all of the communication on a persistent socket.
I hadn’t actually seen your code before, but just took a look. I LOVE the way you traverse the HTTP headers with client.find(). I didn’t know that method existed! It’s not documented, but I’m totally going to use that. Right now, I’m just reading chunks as they’re available and storing them so I can strstr them later.
@rsteckler@ScruffR@harrisonhjones Wondering if I could tap into your minds real quick about this kinda older library that seems to do exactly what I’m wanting to do which is download via HTTP directly to an SD card.
I tried to compile this using Particle DEV and the CLI with both throwing compile errors. Not sure why since others have not complained about this when trying out the library.
Can you take a look and see if it’s something simple or not? Or maybe it’s just that the library is too old?
@ScruffR Ok, I took some time to look into the V2 libraries and I think changed the library file structure to match. I think I did it right?
All that seemed to do is make the SD.h folder not available when compiling.
If I take the files out of the “src” folder and move them to the main folder it no longer complains that the SD.h file is missing and I’m right back to where I started with 16 compile errors. Not sure why I get all these compile errors now when I figured they would have shown up for others.
The library in what I think is the V2 format is here:
@andrewa Built off this same library without problems but he is using Eclipse to compile so I’m not sure what the deal is.
My guess is that if this worked before then something must have changed that is breaking all these functions? Or maybe I just still do not have the v2 library file sturcture right?
I haven’t tried it, I just got the project structure sorted and replaced the old SD library for SdFat. The problem might be some difference in behaviour between the two libraries.
But with this version you might be able to add some debug logs to find the culprit.
He had added a wrapper to the SD library which is probably the reason it’s not pushing data to the SD card.
I wanted to ask you if it was working for you before trying to mess with anything else.
Trying to incentivize @rickkas7 to create us an HTTP library optimized around the Photon if/when he has the time Until then this is the only option I have to work with which is better than nothing
Thank you for your help with this.
I did learn somethign about the 2.0 version of the Particle libraries sturcture.
Intermingling libraries is never a good thing
From a (admittetly rough) look at the HttpDownload.cpp I didn't see any need for special treatment in the SD library so I didn't even look at that at all.
And I still see no point in messing inside two libraries to achieve one goal.
The example .ino given did not start the SD card controller - maybe I missed this in the source. Anyhow, fixed that and tried it.
Initially I got http response of 301 - The 301 redirect is considered a best practice for upgrading users from HTTP to HTTPS! As expected there was a file on the SD card, however, it was empty - figures.
I checked the URL for the wikimedia .jpg file and this was indeed an https: URL and the hostname and path appeared to be different. I have changed these as follows:
Now I am getting more bytes read and something in the .jpg file but both examples (simple and complex) are ending with a 404 error. Log below. Any ideas why this is the case - TCP on Photon does not support HTTPS?
Your path should only contain the part of the url after the domain. It shouldn’t contain the protocol (https).
For SSL, you should only change the port to 443. There might also be another parameter that specifies a secure connection, but don’t change the path or domain.
Until then this is the only option I have to work with which is better than nothing 
