Best way to transfer large data from Photon

Firmware
21

@peekay123 & @RWB - the code @peekay123 posted can also get the data from FTP to SD card. I removed that part of the code since I need SD to FTP transfer only. It must work though.

BTW: The transfer takes such a long time as reported before since the delay(2) was added after every read! It should be only if 64 bytes have been read! So the code should be:

while(fh.available())
  {
    clientBuf[clientCount] = fh.read();
    clientCount++;

    if(clientCount > 63)
    {
      dclient.write(clientBuf,64);
      clientCount = 0;
      delay(2);
    }
  }
1 Like
22

This is exactly what I needed, and itā€™s such a nice, compact bit of code.

I can confirm transfers from FTP to SD work fine. The only change I had to made was commenting out Serial.write(c); hereā€¦

#else
  while(dclient.connected())
  {
    while(dclient.available())
    {
      char c = dclient.read();
      fh.write(c);
      Serial.write(c);
    }
  }
#endif

The print works fine with text files but it froze up if I downloaded even a small binary file.

1 Like
23

@Jerware Just wanted to confirm that @peekay123 's FTP to SD library is still working for you for downloading files from the web to an SD card attached to a Photon.

What was the largest file size you downloaded successfully using this code?

Iā€™m wanting to download some potentially large files to update a 4D systems LCD display when needed and Iā€™m hoping this may do the job.

@peekay123 Have you played with our updated this library over the last year since this thread was started? I know you recommened using TCP for transferring larger amonts of data like this.

24

@RWB, I havenā€™t had a lot of bandwidth these days to test. If I get a chance, Iā€™ll review this weekend.

1 Like
25

Still working for me. I just use it to transfer a small binary file thatā€™s around 768 bytes.

1 Like
26

I can test it I just wanted to confirm that the library was working for others as far as downloading to an SD card is concerned before wiring it up and starting down this path.

Sounds like others have had no problem pushing data from the Photon SD card to a server which is nice to hear since it should allow us to upload photo images that a camera puts on the SD card.

@Vitesze You can also try this library for pushing camera images to a server.

27

@Jerware Good to hear!

I just took a look at the SDFAT library linked above by @pteja and I see that that library is specifically for using SD cards which makes sense :slight_smile:

The coded needed to download a file from the web looks to be a separate library that just pushes the downloaded data to the SD card using the SDFat library functions. Is that right?

Can you share the code needed to download files from the web and then stored them on the SD card?

Trying to see how this handles downloading larger files.

Thanks! Any help is apperciated!

28

I just download from an FTP server using the code above by @peekay123

29

@Jerware Cool!

I got it working for Uploading and Downloading!

Testing with the same Quick n Easy FTP Lite software @peekay123 used with success as long as I set the IP address to something other than the default 127.0.0.1

Transferring images up and down worked just fine with the file being rather small.

Testing a 1gig video file now. The first attempt stopped at 5 MB for some reason, the 2nd attempt stopped at 130 MB. Iā€™m attempting the 3rd download with System Threaded enabled to keep the Photon from breathing green and disconnecting from the cloud.

Pretty cool library for getting large files on and off the Photon!

This should be great for a Photon + Camera that stored itā€™s pictured on the SD card, just push the camera pictures to your FTP server and you can view them anywhere.

Thanks, @peekay123 for the FTP library port also Thanks to @pteja & @Jerware for the two code tweaks that made this library work on my first attempt which is always a nice thing.

I had to add SYSTEM_THREAD(ENABLED); to the top of the sketch to keep the Photon from breathing Green while the download was running.

30

@peekay123 Sorry to bring up old threads but I had questions regarding the following section:

if(!eRcv()) return 0;
client.println("USER particle");
if(!eRcv()) return 0;
client.println("PASS photon");
if(!eRcv()) return 0;
client.println("SYST");
if(!eRcv()) return 0;
client.println("Type I");
if(!eRcv()) return 0;
client.println("PASV");
if(!eRcv()) return 0;

If I'm using an FTP server with login credentials, would I replace USER particle with USER (myUserName)?

I'm trying to use a remote server like driveHQ to send/receive audio (.ogg) files between photons in different regions. Unless you have another suggestion, as I'm a bit unfamiliar with these communication protocols.

31

@Vomitcomet, yes, you would replace USER and PASS for your specific server. FTP is not the fastest protocol but it should work for reasonably sized files.

The only other way to transfer data would be via HTTP or direct TCP to a server. You may find some examples from @rickkas7 in the forum.

32

@peekay123 Thank you for your response! Iā€™ll try testing it out tomorrow, iā€™m assuming port 21 is the default FTP for most routers?

33

@peekay123, yes, port 21 is the ā€œclassicā€ FTP port.

34

let us know what you get working.

I was able to transfer smaller files to a SD via FTP using some of Rickkas code but largeer over 5mb files would never fully transfer a lot of the time.

35

@peekay123 @RWB I managed to get it working reasonably well for uploading/downloading to remote driveHQ server. Had to use:

client.println(ā€œCWD someDirectoryNameā€)

to get access to the folders I wanted. Right now trying to find a work around for the 8.3 naming convention (not sure how to implement long file names) and reading/creating timestamps for files. Wanted to have a file name like dev1d090219t1730, but doesn't seem like it can be done with long file name convention.

1 Like
36

Iā€™ll try uploading larger files than just a minute or two of 16khz audio, just to see how the speed is. Right now for about 1 min 30 (342 kb) itā€™s about 15-20ish seconds (my speed can also be limited due to the server iā€™m using).

37

Great so no file transfer errors on the smaller files during your testing?

What are you using to record the Audio files to SD? What type of microphone setup?

38

@RWB

20190210_171233
20190210_1712333456Ɨ4608 622 KB

No errors so far for transferring. Apologies for the messy setup, currently using a VS1053 breakout board for record/playback, MAX9814 microphone breakout (set to lowest gain, DC decoupling capacitor in line with output into MIC+ so that I can use Line-In), and only reading at 16khz sample rate right now. Probably will eventually try and consolidate those plus a few extra things (charging IC, regulator possibly, accelerometer, tri-colour LED) onto one PCB.

39

Sweet!

Are the audio recordings good quality?

40

@RWB They're about cell phone quality at 16 khz sample rate (64 byte buffer size), little bit of electrical clicking from the breadboard and not using shielded wires, and with most of the background noise removed from reducing the gain.

It would be much better quality if I got 44.1 khz working but it doesn't store any audio during recording, but cell phone quality is sufficient for me considering storage space. If you end up using the VS1053 breakout, the header file (Adafruit_VS1053_Photon.h) hardcoded the DREQ pin so I did local edits to my library so it would work.

34 #define VS1053_FILEPLAYER_PIN_INT WKP