Are large downloads feasible?

Getting Started
1

I’m working on an idea for a new weekend project, and it looks like the Photon may be perfect for it.

One part of the project requires downloading objects of size around 50-500kB from S3 / Google Drive / Dropbox / something. It’s ok if it takes several tens of seconds to finish. (Some of this is going to be over https.)

It looks like the usual way to relay requests between Photon and general web services is through Webhooks. However, there’s a note at the top of the sample responses at https://docs.particle.io/guide/tools-and-features/webhooks/#handling-web-responses:

# website responses are cut into chunks of 512 bytes
# they're sent at a rate of up to 4 per second.
# The first 100KB of the request will be sent, and the rest will be dropped.

Is this saying the request would be truncated at 100KB, or the response? It says “request”, but it’s in a section talking about the response. It’s perfectly fine if the request is cut off at 100KB; I’m just sending small metadata updates out from the device. But a limit of 100KB on the response would be a problem.

One obvious alternative would be to use a TCP socket to download the data directly to the device, instead of going through the web. This would cut me off from some APIs, since I can’t easily use SSL on the Photon (is that correct?). Another alternative would be to deploy a middle layer that breaks large objects into 100KB chunks that can be requested separately. This is for a weekend project, so I’m trying to keep it really simple.

Any thoughts on the matter?

2

@Piquan, the REQUEST is truncated to 100KB (which in your case I suspect is excessive). The RESPONSE will be chunked in 512 byte chunks so you will receive multiple responses to your subscription after firing the webhook. This approach uses the Publish/Subscribe model coupled with a webhook. This gives you the HTTPS access coupled with COAP-wrapped responses for security. Large data throughput, however, is compromised.

There is also an HTTPS library available here:

Using TCP directly will give you the most flexibility in your approach but with no security (per se). In some applications where security isn't important, I actually run an FTP client and pull the data from a server directly to microSD (and vice versa).

Finally, the proxy is another alternative. I can tell you that some of the top priorities for the Particle Team after the Electron is released is to work on supporting HTTPS and IPV6. Have fun! :smiley:

4 Likes