Communicating With a SD Card - "delete" Content From Text File? [Solved]

Firmware
22

Hmm, I seem to have one last hurdle to pass before this is finally complete.

If the file is very large, roughly 720 “lines” with characters the same as the ones in the code I posted earlier, it takes a considerable amount of time using the two pointer method to delete characters.

The method is still great, but now I’m thinking what I could do is that every time a file gets to a particular size (I can check using File.size()) I create a new file, auto generate a name using a base name like “test#.txt” and do the two-pointer delete method normally.

However what I wanted to do in conjunction with creating these new files, is re-name the old text file to “test2.txt”, and the new file would be “test1.txt” that way when I do the two pointer delete method, it will check for files in order of “1 , 2 , 3 … 10 (for example)” where 10 is the oldest data.

I’ve looked online, and there seems to be some newer Arduino library that has SdFat.rename() and SdFile.rename(), but these don’t seem to be apart of the Spark SD library. Any ideas on how to rename a text file, @peekay123 @bko ? I was thinking SdFile.make83Name() might be of use, but it seems to be a private class and only usable for directory files.

EDIT: I found your thread @peekay123 with @mdma about porting over SdFat, I’m guessing this repo is not quite spark compatible yet? Is there still another method to rename text files outside this?

As always any direction is very much appreciated :slight_smile:

23

@UST, I think it may be time to review what you are trying to achieve with the files and look at the entire process instead of just the delete line portion. Can you elaborate on how/why the file is used? Understanding the “whole” may lead to a better or more efficient solution. :smile:

24

You are right @peekay123 I'll explain below (I'll do it in points, that way if anything is unclear just address that point(s) :)' )

  1. I'm collecting data and saving the data as a "line" on a text file every minute.
  2. After saving the data, the Core sends the "line of data" off via WiFi
  3. If the transmission IS successful the Core performs the two-pointer-delete method and removes that line so that the next line can be sent.
  4. If unsuccessful the core just attempts to send the same line again.
  5. If it's 50 sec into a new minute, things interrupt (after deleting line if it was done) and data is collected for the new minute.

So #3 is where my last hurdle seems to be, with this whole text file business. If my file gets too large, the delete method takes a while to go through. So it could happen that I miss a new minute of data, which I don't want. So I was looking into this whole process if the original file gets too big:

25

@UST, I am assuming you need to create this “line” queue because you don’t always have cloud connectivity, correct?

  • Lines have to be sent in order?
  • If a line fails to send, all subsequent lines will wait until that send succeeds?
  • If the “lines” are not sent right away (every minute) and are queued, at what interval do you send them? That is, how fast are you clearing out the queued lines?
  • How are you sending the “lines” - Spark.publish(), TCPClient?
  • How long do you anticipate your worst cloud “black out” period to be?

I have an idea for another approach but it requires answers to be above first. :smile:

1 Like
26

@peekay123 I’m excited to hear!

  • Yes Lines should be sent in order, but I guess they don’t need to be now that I think about it, so either or works !
  • If a line fails to send, all the subsequent lines don’t even get read by File.read(), so yes in essence they wait until the first most is sent, then deleted, so that the next in line gets sent :wink:
  • Clearing them out as fast as possible. The less backed up data the better. So at the minute mark ( Time.second() = 0 ), after data has been saved to the SD, a line is transmitted and if done successfully , the line is immediately deleted via the two-pointer method. This process repeats until Time.second() is between 50 - 58 seconds (AND after all deleting has been completed, so the file doesn’t get corrupted) which at that time the process halts. This gives time to collect data for the new minute.
  • The “lines” are being sent via TCPClient
  • The worst black out period could be anywhere from a hour to a week, to even possibly two weeks! But I’d rather not have a “max”, and maybe just a large capacity SD card to account for even the largest down times. My SD card I’m using now is 32GB and I don’t even think I’ve dented it yet :stuck_out_tongue:
27

@UST,

  1. Order is not critical which means they are not timestamped?
  2. What is the worst case single line length?
  3. Do you open and close the file when you add a new line every minute?
  4. After sending lines, do you close the file BEFORE the sampling every minute?

:smile:

28

@peekay123,

  1. They are timestamped , but the only reason I say it’s not critical cause, eventually the data will all get to where it needs to go and get organized from there. But if order could prioritized then that’s a bonus.
  2. Worst case is 58 bytes in length! That is including “\r” and “\n” characters!
  3. Yes when the new minute begins the file is opened, data is written, then it is closed!
  4. Yes the file (which is a File. object) is closed before sampling. Maybe it could be left open to make things faster b/n samples, but it needs to be closed when doing the two-pointer delete method, because I create a SdFile object that associates with the same file in order to use truncate so the two objects can’t be opening the same file . . . at least that was my understanding :slight_smile: ?
29

@UST, so from my calculations:

  • 58 bytes/minute x 60 mins/hr * 24 hrs = 83,520 bytes/day
  • Let’s say a formatted 32GB card can hold 28x10e9 bytes
  • You could store about 335,000 days of data or 919 years!!!

So storage is not the issue. It now comes down to what you do with the queued data. You could simply let the file grow and just treat it like a queue with tail and head pointers. You shove data in at the head and take things out at the tail. When the cloud is connected, the tail always “catches up” to the head. When the cloud is down, the head gets further from the tail and when the cloud connects again, you get a flurry of send activity until the tail catches up to the head. So, basically, you never erase the file!

Another approach is to do the same but once your tail catches up to the head AND the file is bigger than a certain size, you truncate the file to the beginning (not erase) and start over again.

The idea is that you don’t do this crazy line delete stuff. You don’t have a storage issue, the SD performance will not change because you increase the size of the file since you are doing very simple operations. Finally, if size is an issue ( :stuck_out_tongue: ) you could just truncate the file and start over. Keep It Simple, er, Dude! :smile:

2 Likes
30

Woah ! @peekay123 thanks for doing those calculations for me; definitely gives me more information to use!

Okay, so let me digest all of this . . . Also this is the first thing that came to mind on how to implement your mentioned method!

So lets say I have a text file like this, I would have my tail pointer “T” as such, and H wouldn’t be really needed, cause I’ll always write at the end of the text file.

Ttesting. 1, 2, 3.
Ttesting. 4, 5, 6.
Ttesting. 7, 8, 9.

So on the minute mark things would go down like this:

myFile = SD.open("test.txt", O_RDWR); //Since I'm at the end of the             
                                  //test.txt, no need to use seek() to find "H"
if(myFile.available())
{ 
     //for example this is the acquired data
    myFile.println("testing. 1, 2, 3.");           
}
myFile.seek(0); // go to the start
while (myFile.available())
{
  char X = myFile.peek();
  if (X == T)  unsigned long posit = myFile.position(); 
}
//---- then do the reading and sending ----// 
//if success, file cursor SHOULD be at the start of the next "line" 
unsigned long nextposit = myFile.position();
myFile.seek(posit); 
myFile.write('X'); 
posit = nextposit; 
myFile.seek(posit); 
//--then read again and send again--//
//-- if success again , repeat the above five lines --// 
//------ rinse and repeat this until 50<=Time.second() AND the 5 above lines have finished running ------//
//--  THEN we do: --//
 myFile.close(); 
 // ------------ on the NEXT minute , we look for the "T" and repeat the process -------//

So is it correct-ish, or is there a simple,er, way :stuck_out_tongue: :slight_smile: ?

Doing it with this method I guess size doesn’t matter ( :stuck_out_tongue: ) If you’re saying SD speed, reading/writing, won’t be affected. Which I guess it wouldn’t be. But when I read through the file to find “T”, if the file is REALLY large won’t it take longer to look for it? My guess is your saying it might but it won’t be anything that affects performance such as missing the 50 sec timeout. And if by testing and such I find that a file can get too large, I can truncate it as you suggested!

Edit: I realize the code isn’t 100% few lines I left out here and there, BUT the general idea is there I believe

31

@UST, close but not quite. The head and tail pointers need to be file indexes so you read from the tail until your end of line char and reset the tail to the next line. So when the head and tail are the same value, you have no new lines to send. No searching,just seeking to file index. No fuss, no muss. :smile:

1 Like
32

@peekay123 Oh okay! That makes more sense, and it’ll be simpler too then what I had written!

But in the event of a power outage these indexes will be lost in my variables, so maybe I’ll truncate every time the tail and header are the same to avoid re-sending of data! :wink: Thanks again for all the help!

33

@UST, now you’re cooking with propane!!! :stuck_out_tongue: Will you be sharing you code once it’s working?

34

Yeah I think I’ll cook something up, I’ll keep posted ! :stuck_out_tongue:

1 Like
35

H’okay, so this seems to be doing the trick!

#include "application.h"
#include "sd-card-library.h"

// SOFTWARE SPI pin configuration - modify as required
// The default pins are the same as HARDWARE SPI
const uint8_t chipSelect = A2;    // Also used for HARDWARE SPI setup
const uint8_t mosiPin = A5;
const uint8_t misoPin = A4;
const uint8_t clockPin = A3;

unsigned long tail = 0;
unsigned long nextTail = 0;
unsigned long head = 0;

SdFile root;
Sd2Card card;
SdVolume volume;
SdFile file;
File myFile;

SYSTEM_MODE(SEMI_AUTOMATIC);

void setup()
{
  Serial.begin(115200);
  while (!Serial.available()) SPARK_WLAN_Loop();
  Serial.print("Initializing SD card...");
  // Initialize SOFTWARE SPI
  if (!card.init(mosiPin, misoPin, clockPin, chipSelect))
  {
    Serial.println("initialization failed! 1");
    return;
  }
  if (!SD.begin(mosiPin, misoPin, clockPin, chipSelect))
  {
    Serial.println("initialization failed! 2");
    return;
  }
  if (!volume.init(card))
  {
    Serial.println("Could not find FAT16/FAT32 partition.\nMake sure you've formatted the card");
    return;
  }
  Serial.println("initialization done.");
}

void loop()
{
  if (Time.second()%30 == 0)
  {
    root.openRoot(volume);
    root.ls(LS_R | LS_DATE | LS_SIZE);
    Serial.println();
    myFile = SD.open("delete.txt", O_RDWR);
    head = myFile.position();
    myFile.write("testing 1, delete.\r\n");
    myFile.write("testing 2, delete.\r\n");
    myFile.write("testing 3, delete.\r\n");
    myFile.write("testing 4, delete.\r\n");
    myFile.seek(tail);
    int r = random(10);   // simulate succesful transmission
    if(r%2 == 0)
    {
      Serial.println("Succesful Tranmission");
      while(myFile.available())
      {
          while(myFile.available())
          {
            char T = myFile.peek();
            if (T == '\n')
            {
              // Make String, to be able to send
              break;
            }
            myFile.read();
          }
          tail = myFile.position();
          if (tail == head)
          {
            myFile.close();
            file.open(root, "delete.txt");
            file.truncate(0);
            file.close();
            root.ls(LS_R | LS_DATE | LS_SIZE);
          }
      }
    }
    Serial.println("Unsuccesful Tranmission");
    myFile.close();
    root.close();
  }
}
2 Likes