Best way to transfer large data from Photon

I am using this code to transfer data from SD card to an FTP server. It works partially.

  1. The file gets transferred only partially. The source is a text file having 838,736 bytes out of which only 203,968 get transferred. Then the connection closes normally.
  2. There are some errors in the file. Some characters are missing :weary:

How can I debug this? What part of the code? Any hints welcome!!

Here is the result on the serial port:

@peekay123 - just noticed this in the eRcv() function:

while(!client.available()) Spark.process();

Shouldn’t this be particle.process()?

Will try anyway and post.

@pteja, both will work but it should be Particle.process(). Which SD library are you using in your code?

SD lib from greiman

@pteja, is that the newest SDFat library? If not, I highly recommend using that one. It uses DMA and is written by the original SDFat author.

It is the latest lib. The issue is not the SD library. The data is getting stored properly.

It is the FTP part that has issues. Just rechecked with change to particle.process. Same result.

Only 204,672 bytes got transferred and missing characters again in the FTPed file on server :unamused:

Here is the link to SDFat library.

Here is a topic related to SD library.

@pteja, a small delay may need to be added between 64 byte outgoing chunks to allow the background code to catch up:

  while(fh.available())
  {
    clientBuf[clientCount] = fh.read();
    clientCount++;

    if(clientCount > 63)
    {
      dclient.write(clientBuf,64);
      clientCount = 0;
    }
    delay(2);
  }

You may need to try different values to see if it helps or not.

1 Like

It works. But takes a VERY long time - I aborted twice thinking the transfer had stopped and then added a serial.print where delay is to view progress. It took several minutes for a 800kb file.

Need to fine-tune the transfer timing by adjusting the delay to 1ms.

The (seemingly) strange thing is nothing happens on the FTP server till the entire transfer is done. Any idea why?

Will the delay need to be changed if the available bandwidth is lower?

Many thanks @peekay123

@pteja, the bottleneck is at the tcp stack. You can reduce the delay but also increase the clientBuf size. Make it progressively larger, adjusting the clientCount threshold accordingly. You could try making it 1024 bytes to see how well that works and tuning the delay as well.

1 Like

@peekay123 Can this code be used to dump data from a FTP server to the SD card that is connected to the Photon? Or is it only the other way around?

@RWB yup. I’m not sure I posted the entire code that supports that though.

UDPATE: I did not post the entire code so that part is missing.

Your mainly using it to offload data on the Photon to the FTP server right?

Have you sent data to the SD card connected to the Photon successfully?

@RWB, I’ve only test with sd to ftp, not ftp to sd. I will work on that this weekend.

@peekay123 Sweet! Even if you don’t get to it this weekend please do let me know how it goes when you do get to it. :wink:

1 Like

@peekay123 & @RWB - the code @peekay123 posted can also get the data from FTP to SD card. I removed that part of the code since I need SD to FTP transfer only. It must work though.

BTW: The transfer takes such a long time as reported before since the delay(2) was added after every read! It should be only if 64 bytes have been read! So the code should be:

while(fh.available())
  {
    clientBuf[clientCount] = fh.read();
    clientCount++;

    if(clientCount > 63)
    {
      dclient.write(clientBuf,64);
      clientCount = 0;
      delay(2);
    }
  }
1 Like

This is exactly what I needed, and it’s such a nice, compact bit of code.

I can confirm transfers from FTP to SD work fine. The only change I had to made was commenting out Serial.write(c); here…

#else
  while(dclient.connected())
  {
    while(dclient.available())
    {
      char c = dclient.read();
      fh.write(c);
      Serial.write(c);
    }
  }
#endif

The print works fine with text files but it froze up if I downloaded even a small binary file.

1 Like

@Jerware Just wanted to confirm that @peekay123 's FTP to SD library is still working for you for downloading files from the web to an SD card attached to a Photon.

What was the largest file size you downloaded successfully using this code?

I’m wanting to download some potentially large files to update a 4D systems LCD display when needed and I’m hoping this may do the job.

@peekay123 Have you played with our updated this library over the last year since this thread was started? I know you recommened using TCP for transferring larger amonts of data like this.

@RWB, I haven’t had a lot of bandwidth these days to test. If I get a chance, I’ll review this weekend.

1 Like

Still working for me. I just use it to transfer a small binary file that’s around 768 bytes.

1 Like

I can test it I just wanted to confirm that the library was working for others as far as downloading to an SD card is concerned before wiring it up and starting down this path.

Sounds like others have had no problem pushing data from the Photon SD card to a server which is nice to hear since it should allow us to upload photo images that a camera puts on the SD card.

@Vitesze You can also try this library for pushing camera images to a server.