Communicating With a SD Card - "delete" Content From Text File? [Solved]

UST · January 28, 2015, 3:38pm

Hmm, I seem to have one last hurdle to pass before this is finally complete.

If the file is very large, roughly 720 “lines” with characters the same as the ones in the code I posted earlier, it takes a considerable amount of time using the two pointer method to delete characters.

The method is still great, but now I’m thinking what I could do is that every time a file gets to a particular size (I can check using File.size()) I create a new file, auto generate a name using a base name like “test#.txt” and do the two-pointer delete method normally.

However what I wanted to do in conjunction with creating these new files, is re-name the old text file to “test2.txt”, and the new file would be “test1.txt” that way when I do the two pointer delete method, it will check for files in order of “1 , 2 , 3 … 10 (for example)” where 10 is the oldest data.

I’ve looked online, and there seems to be some newer Arduino library that has SdFat.rename() and SdFile.rename(), but these don’t seem to be apart of the Spark SD library. Any ideas on how to rename a text file, @peekay123 @bko ? I was thinking SdFile.make83Name() might be of use, but it seems to be a private class and only usable for directory files.

EDIT: I found your thread @peekay123 with @mdma about porting over SdFat, I’m guessing this repo is not quite spark compatible yet? Is there still another method to rename text files outside this?

As always any direction is very much appreciated

peekay123 · January 28, 2015, 4:06pm

@UST, I think it may be time to review what you are trying to achieve with the files and look at the entire process instead of just the delete line portion. Can you elaborate on how/why the file is used? Understanding the “whole” may lead to a better or more efficient solution.

UST · January 28, 2015, 4:24pm

You are right @peekay123 I'll explain below (I'll do it in points, that way if anything is unclear just address that point(s) :)' )

I'm collecting data and saving the data as a "line" on a text file every minute.
After saving the data, the Core sends the "line of data" off via WiFi
If the transmission IS successful the Core performs the two-pointer-delete method and removes that line so that the next line can be sent.
If unsuccessful the core just attempts to send the same line again.
If it's 50 sec into a new minute, things interrupt (after deleting line if it was done) and data is collected for the new minute.

So #3 is where my last hurdle seems to be, with this whole text file business. If my file gets too large, the delete method takes a while to go through. So it could happen that I miss a new minute of data, which I don't want. So I was looking into this whole process if the original file gets too big:

peekay123 · January 28, 2015, 4:58pm

@UST, I am assuming you need to create this “line” queue because you don’t always have cloud connectivity, correct?

Lines have to be sent in order?
If a line fails to send, all subsequent lines will wait until that send succeeds?
If the “lines” are not sent right away (every minute) and are queued, at what interval do you send them? That is, how fast are you clearing out the queued lines?
How are you sending the “lines” - Spark.publish(), TCPClient?
How long do you anticipate your worst cloud “black out” period to be?

I have an idea for another approach but it requires answers to be above first.

UST · January 28, 2015, 5:20pm

@peekay123 I’m excited to hear!

Yes Lines should be sent in order, but I guess they don’t need to be now that I think about it, so either or works !
If a line fails to send, all the subsequent lines don’t even get read by File.read(), so yes in essence they wait until the first most is sent, then deleted, so that the next in line gets sent
Clearing them out as fast as possible. The less backed up data the better. So at the minute mark ( Time.second() = 0 ), after data has been saved to the SD, a line is transmitted and if done successfully , the line is immediately deleted via the two-pointer method. This process repeats until Time.second() is between 50 - 58 seconds (AND after all deleting has been completed, so the file doesn’t get corrupted) which at that time the process halts. This gives time to collect data for the new minute.
The “lines” are being sent via TCPClient
The worst black out period could be anywhere from a hour to a week, to even possibly two weeks! But I’d rather not have a “max”, and maybe just a large capacity SD card to account for even the largest down times. My SD card I’m using now is 32GB and I don’t even think I’ve dented it yet

peekay123 · January 28, 2015, 6:02pm

@UST,

Order is not critical which means they are not timestamped?
What is the worst case single line length?
Do you open and close the file when you add a new line every minute?
After sending lines, do you close the file BEFORE the sampling every minute?

UST · January 28, 2015, 6:18pm

@peekay123,

They are timestamped , but the only reason I say it’s not critical cause, eventually the data will all get to where it needs to go and get organized from there. But if order could prioritized then that’s a bonus.
Worst case is 58 bytes in length! That is including “\r” and “\n” characters!
Yes when the new minute begins the file is opened, data is written, then it is closed!
Yes the file (which is a File. object) is closed before sampling. Maybe it could be left open to make things faster b/n samples, but it needs to be closed when doing the two-pointer delete method, because I create a SdFile object that associates with the same file in order to use truncate so the two objects can’t be opening the same file . . . at least that was my understanding ?

peekay123 · January 28, 2015, 6:44pm

@UST, so from my calculations:

58 bytes/minute x 60 mins/hr * 24 hrs = 83,520 bytes/day
Let’s say a formatted 32GB card can hold 28x10e9 bytes
You could store about 335,000 days of data or 919 years!!!

So storage is not the issue. It now comes down to what you do with the queued data. You could simply let the file grow and just treat it like a queue with tail and head pointers. You shove data in at the head and take things out at the tail. When the cloud is connected, the tail always “catches up” to the head. When the cloud is down, the head gets further from the tail and when the cloud connects again, you get a flurry of send activity until the tail catches up to the head. So, basically, you never erase the file!

Another approach is to do the same but once your tail catches up to the head AND the file is bigger than a certain size, you truncate the file to the beginning (not erase) and start over again.

The idea is that you don’t do this crazy line delete stuff. You don’t have a storage issue, the SD performance will not change because you increase the size of the file since you are doing very simple operations. Finally, if size is an issue ( ) you could just truncate the file and start over. Keep It Simple, er, Dude!

UST · January 28, 2015, 9:35pm

Woah ! @peekay123 thanks for doing those calculations for me; definitely gives me more information to use!

Okay, so let me digest all of this . . . Also this is the first thing that came to mind on how to implement your mentioned method!

So lets say I have a text file like this, I would have my tail pointer “T” as such, and H wouldn’t be really needed, cause I’ll always write at the end of the text file.

Ttesting. 1, 2, 3.
Ttesting. 4, 5, 6.
Ttesting. 7, 8, 9.

So on the minute mark things would go down like this:

myFile = SD.open("test.txt", O_RDWR); //Since I'm at the end of the             
                                  //test.txt, no need to use seek() to find "H"
if(myFile.available())
{ 
     //for example this is the acquired data
    myFile.println("testing. 1, 2, 3.");           
}
myFile.seek(0); // go to the start
while (myFile.available())
{
  char X = myFile.peek();
  if (X == T)  unsigned long posit = myFile.position(); 
}
//---- then do the reading and sending ----// 
//if success, file cursor SHOULD be at the start of the next "line" 
unsigned long nextposit = myFile.position();
myFile.seek(posit); 
myFile.write('X'); 
posit = nextposit; 
myFile.seek(posit); 
//--then read again and send again--//
//-- if success again , repeat the above five lines --// 
//------ rinse and repeat this until 50<=Time.second() AND the 5 above lines have finished running ------//
//--  THEN we do: --//
 myFile.close(); 
 // ------------ on the NEXT minute , we look for the "T" and repeat the process -------//

So is it correct-ish, or is there a simple,er, way ?

Doing it with this method I guess size doesn’t matter ( ) If you’re saying SD speed, reading/writing, won’t be affected. Which I guess it wouldn’t be. But when I read through the file to find “T”, if the file is REALLY large won’t it take longer to look for it? My guess is your saying it might but it won’t be anything that affects performance such as missing the 50 sec timeout. And if by testing and such I find that a file can get too large, I can truncate it as you suggested!

Edit: I realize the code isn’t 100% few lines I left out here and there, BUT the general idea is there I believe

peekay123 · January 29, 2015, 1:26am

@UST, close but not quite. The head and tail pointers need to be file indexes so you read from the tail until your end of line char and reset the tail to the next line. So when the head and tail are the same value, you have no new lines to send. No searching,just seeking to file index. No fuss, no muss.

UST · January 29, 2015, 1:28pm

@peekay123 Oh okay! That makes more sense, and it’ll be simpler too then what I had written!

But in the event of a power outage these indexes will be lost in my variables, so maybe I’ll truncate every time the tail and header are the same to avoid re-sending of data! Thanks again for all the help!

peekay123 · January 29, 2015, 1:42pm

@UST, now you’re cooking with propane!!! Will you be sharing you code once it’s working?

UST · January 29, 2015, 1:44pm

Yeah I think I’ll cook something up, I’ll keep posted !

UST · January 29, 2015, 3:17pm

H’okay, so this seems to be doing the trick!

#include "application.h"
#include "sd-card-library.h"

// SOFTWARE SPI pin configuration - modify as required
// The default pins are the same as HARDWARE SPI
const uint8_t chipSelect = A2;    // Also used for HARDWARE SPI setup
const uint8_t mosiPin = A5;
const uint8_t misoPin = A4;
const uint8_t clockPin = A3;

unsigned long tail = 0;
unsigned long nextTail = 0;
unsigned long head = 0;

SdFile root;
Sd2Card card;
SdVolume volume;
SdFile file;
File myFile;

SYSTEM_MODE(SEMI_AUTOMATIC);

void setup()
{
  Serial.begin(115200);
  while (!Serial.available()) SPARK_WLAN_Loop();
  Serial.print("Initializing SD card...");
  // Initialize SOFTWARE SPI
  if (!card.init(mosiPin, misoPin, clockPin, chipSelect))
  {
    Serial.println("initialization failed! 1");
    return;
  }
  if (!SD.begin(mosiPin, misoPin, clockPin, chipSelect))
  {
    Serial.println("initialization failed! 2");
    return;
  }
  if (!volume.init(card))
  {
    Serial.println("Could not find FAT16/FAT32 partition.\nMake sure you've formatted the card");
    return;
  }
  Serial.println("initialization done.");
}

void loop()
{
  if (Time.second()%30 == 0)
  {
    root.openRoot(volume);
    root.ls(LS_R | LS_DATE | LS_SIZE);
    Serial.println();
    myFile = SD.open("delete.txt", O_RDWR);
    head = myFile.position();
    myFile.write("testing 1, delete.\r\n");
    myFile.write("testing 2, delete.\r\n");
    myFile.write("testing 3, delete.\r\n");
    myFile.write("testing 4, delete.\r\n");
    myFile.seek(tail);
    int r = random(10);   // simulate succesful transmission
    if(r%2 == 0)
    {
      Serial.println("Succesful Tranmission");
      while(myFile.available())
      {
          while(myFile.available())
          {
            char T = myFile.peek();
            if (T == '\n')
            {
              // Make String, to be able to send
              break;
            }
            myFile.read();
          }
          tail = myFile.position();
          if (tail == head)
          {
            myFile.close();
            file.open(root, "delete.txt");
            file.truncate(0);
            file.close();
            root.ls(LS_R | LS_DATE | LS_SIZE);
          }
      }
    }
    Serial.println("Unsuccesful Tranmission");
    myFile.close();
    root.close();
  }
}

Topic		Replies	Views
Deleting Line(s) in Text File -- Redirected Thread in Description Firmware	2	1486	January 16, 2015
Delete line from a txt file on SD card (SdFat) Firmware	14	6207	July 4, 2019
Boron SOM SDfat Hardware boron	3	644	September 28, 2021
microSD board, help getting it to work with BORON Hardware boron	12	1027	November 27, 2021
Trouble communicating with SD Card [solved] Troubleshooting	8	10535	November 4, 2015

Communicating With a SD Card - "delete" Content From Text File? [Solved]

Related topics