How to send large file from Particle to server?

Cloud
21

The TCP stack in the Electron is different than the Photon. In particular, you can only call TCP from the main loop on the Electron. The Photon is lets you call it from other threads, including software timers, but you cannot do that on the Electron.

Also, don’t send exactly 1024 bytes. You can send 1023, or 512, but I believe there is a bug sending exactly 1024 bytes on the Electron.

22

OK - got the DMA code done.

Writing and reading 32768 block (32k)

Took 18764us to write and 17508us to read
By my calculations that would mean transferring 16 x 32k blocks which would be 1/2 Megabytes would take 300,224Us (1/3rd of a second). (plus a little bit of overhead!).

Transferring 1byte at a time took about 13.7 seconds!

According to @peekay123 (See Post by Paul) Based on my results, is not a lot of speed difference between FRAM and SD Card! I believe I am pushing data out at full SPI speed - and I would suspect that mode 0 on sdfat is also pushing data at that rate - so the actual limiting factor is getting the data to the device!

It has been an interesting experiment however. I will tidy up my library and make it available.

3 Likes
23

Hello hello,

We would like to send the audio via a secured connection for privacy purposes… for example HTTPS. I am looking into using the glowfish library.

Does anyone have suggestions on how to stream customer-sensitive data using the Photon?

Thank you!

24

@rickkas7

I started working on this again, and have a couple of questions for you regarding the DMA ADC (for audio) in your example here: https://github.com/rickkas7/photonAudio/tree/master/audio3

I am trying to reformat the ADC data and then send it out as a POST request to an AWS ec2 instance. E.G.

POST ...
...
(float of ADC value)\n
(float of ADC value)\n
...
End\n

I’ve had to put in short delay(50) so that the TCP buffer doesn’t overflow when sending… So, my questions are:

  1. What happens to the ADC samples if the samples-buffer overflows? Are they lost (circular buffer)? I’m thinking this will happen at high sample rates & slow TCP connections…

  2. Can I just make the sample buffer bigger (maybe 8192) and send out 512 samples at a time?

  3. Is there a min/max on the sampling rate of the DMA-ADC?

  4. Could you provide a link for the DMA documentation you referenced in the source (cortex_mx_stm32 example)?

Thank you so much!!

25

@rickkas7 :point_left: poke

26

The audio3 code uses double buffers. The ADC reads the buffer until it’s full, then switched to the other buffer. When a buffer is full, the loop code sends it out using TCP. If the network was really slow, it’s possible that the send would not complete before the DMA code switches back to that buffer. The sample code does not detect this, and instead just sends out corrupted data.

The optimal size for writing to TCP is 1024 bytes in most cases. You can make it 512 bytes, but contrary to what you might think, making it larger makes the performance worse, so there is no reason to make it larger than 1024 bytes.

Also, there is never any reason to put a delay in when writing to TCP. With the Photon, if write returns -16 you must just send the same buffer again. That will allow you to send at the maximum rate possible.

The reason buffers larger than 1024 bytes are not helpful is that there is an internal buffer in the TCP stack. I think it’s about 3K. In order for write to succeed, there must be enough empty space in the buffer to write the whole thing. If you try to do large writes, it has to wait so long for there to be enough free space in the buffer than the buffer usually empties out before it can be refilled. The -16 return code means there wasn’t enough free space.

The same thing happens when you use really small buffers, say 100 bytes. The reason is that the loop only runs 1000 times per second, and you can’t keep the buffer full.

Since you can only have 2 DMA buffers, if you want to make the buffers bigger you should split the TCP writes up into 1024 byte writes for best performance.

Tha maximum ADC sample rate is 6 M samples/sec but it’s really hard to use at that rate because it generates data faster than you can reasonably consume it. There’s probably a minimum rate, but it depends on what prescaler value and timer you use. With a large prescaler it probably could sample quite slowly, like seconds per sample.

Here are the documents that I often reference:

STM32F205xx Data Sheet

STM32F2xx Standard Library

STM32F205 Programming Manual

STM32F205 Reference Manual

4 Likes