Communicating With a SD Card - "delete" Content From Text File? [Solved]

Sorry for all the back and forth here’s the code:

#include "application.h"
#include "sd-card-library.h"

File myFile;

// SOFTWARE SPI pin configuration - modify as required
// The default pins are the same as HARDWARE SPI
const uint8_t chipSelect = A2;    // Also used for HARDWARE SPI setup
const uint8_t mosiPin = A5;
const uint8_t misoPin = A4;
const uint8_t clockPin = A3;

char OVERWRITE = 'O';
char check = 0;
unsigned long replace = 0;
unsigned long read = 0;
unsigned long replace2 = 0;
int i =0;

SYSTEM_MODE(SEMI_AUTOMATIC);

void setup()
{
  Serial.begin(115200);
  while (!Serial.available()) SPARK_WLAN_Loop();

  Serial.print("Initializing SD card...");
  // Initialize SOFTWARE SPI
  if (!SD.begin(mosiPin, misoPin, clockPin, chipSelect)) {
    Serial.println("initialization failed!");
    return;
}

  Serial.println("initialization done.");

  myFile = SD.open("test.txt", FILE_WRITE | O_RDWR);

  myFile.seek(0); // go to beginning position of the text file
  if (myFile)
  {     // if the file opened okay, write to it:
    Serial.print("Writing to test.txt...");
    for (int i=0; i<1; i++)
    {
      myFile.println("Testing 1, 2, x.");
      myFile.println("Testing 1, 2, z.");
      myFile.println("Testing 1, 2, g.");
    }
    //myFile.close();            // close the file: closing the file is what actually saves the data on the SD
    Serial.println("done.");   // ^^ you can .close() to save or .flush() to save
  } else {
    Serial.println("error opening test.txt");       // if the file didn't open, print an error:
  }

myFile.seek(0);

while (myFile.available()) Serial.write(myFile.read());


myFile.seek(0);     // go back to the beginning of the test.txt
myFile.write(OVERWRITE);  // does this move position from 0 to 1 ? if so then myFile.seek(0);
myFile.seek(0);
replace = myFile.position();
while (myFile.available())
{
   check = myFile.peek();
   if (check == 'T')
   {
    read = myFile.position();
    break;
   }
   myFile.read();
}


myFile.seek(read);
while (myFile.available())
{
  check = myFile.read();
  myFile.seek(replace);
  myFile.write(check); // or should I do myFile.print(check); instead?
  replace++;
  read++;
  myFile.seek(read);
  if (myFile.peek() == -1)
  {
    uint32_t newSize = myFile.size() - (read - replace);
    myFile.truncate(newSize);
    break;
  }
}


myFile.seek(0);
Serial.println();
while(myFile.available()) Serial.write(myFile.read());
myFile.close();
//SD.remove("test.txt");
}

void loop ()
{
  // just loop doe
}

EDIT: I Guess I should mention that I am using this in Spark Dev , if that makes any difference :frowning:

@UST, truncate is part of the class SDfile, not File. I will need to think about how to set this up.

1 Like

I saw that too, looking into it as well. Appreciate all the help!

EDIT: Looking into the SdFile class, it seems like I need to create a directory and then a file (or directory file) to put into that directory, had to do something else today, but still going to be plugging away at this.

EDIT: Can I use the SdFile class the same as the File class or are they fundamentally different ? Any ideas how to set this up @peekay123? :slight_smile:

Going to be tackling this today for most of the day, I’ll post once I find a solution.

But if anyone has any ideas just throw them here or PM me! :slight_smile: Thanks !

Alright,

So I made more headway than I thought I would, but now I reached a new road block. But I feel that I’m getting close.

The SdFile class has a “read” function in this form:

 * \param[out] buf Pointer to the location that will receive the data.
 * \param[in] nbyte Maximum number of bytes to read.

    int16_t SdFile::read(void* buf, uint16_t nbyte)

Now when I have this code:

while (x == 0) 
{
   char check = file.read();    //file is my SdFile object/instance
   Serial.print(check);           // I've used Serial.write as well
   if (check == EOF) break;
}

I am able to successfully read the “file” and everything in it, however, it just keeps reading. Doesn’t ever stop and the characters it prints don’t seem to be EOF (although they might be I’m not sure) the characters are fuzzy, but when copy and pasted into a word doc they are boxes with tiny “?” inside them.

So then I tried going back to the read function from the SdFile class and tried formatting it similar to that:

unsigned int size = sizeof(file);  // file is my SdFile object/instance
                                   // -- size is apparently = 40
void* bufferz[size];               //void is what threw me off the                       
              //  most, tried making it many other data types, didn't work. 
Serial.write(file.read(bufferz, size));

What comes out on the Serial Terminal (PuTTY) is “40”.

So two questions,

1.) SdFile class doesn’t have the “available” function like File class does, which makes it hard to know when I’m about to reach the end of the file. So how would I know that I’m reaching the end of my file when reading “file” similar to the way I read it in my code pasting?

2.) I know I must be doing somethign wrong in the way I’m using the read function from the SdFile class, what is it?

As always an insight or direction is always appreciated :smile:

Here is full working code, on how to delete a line using @peekay123’s 2-pointer method and using both the File and SdFile class from @BDub’s SD Card Library:

#include "application.h"
#include "sd-card-library.h"

// SOFTWARE SPI pin configuration - modify as required
// The default pins are the same as HARDWARE SPI
const uint8_t chipSelect = A2;    // Also used for HARDWARE SPI setup
const uint8_t mosiPin = A5;
const uint8_t misoPin = A4;
const uint8_t clockPin = A3;

char OVERWRITE = 'O';
uint32_t newSize = 0;
char check = 0;
unsigned long replace = 0;
unsigned long read = 0;

SdFile root;
Sd2Card card;
SdVolume volume;
SdFile file;
File myFile;

SYSTEM_MODE(SEMI_AUTOMATIC);

void setup()
{
  Serial.begin(115200);
  while (!Serial.available()) SPARK_WLAN_Loop();
  Serial.print("Initializing SD card...");
  // Initialize SOFTWARE SPI
  if (!card.init(mosiPin, misoPin, clockPin, chipSelect))
  {
    Serial.println("initialization failed! 1");
    return;
  }
  if (!SD.begin(mosiPin, misoPin, clockPin, chipSelect))
  {
    Serial.println("initialization failed! 2");
    return;
  }
  Serial.println("initialization done.");
  if (!volume.init(card))
  {
    Serial.println("Could not find FAT16/FAT32 partition.\nMake sure you've formatted the card");
    return;
  }

myFile = SD.open("delete.txt", O_RDWR);

Serial.print("myFile position is: "); Serial.println(myFile.position());
myFile.seek(0);
Serial.print("myFile position after seek is: "); Serial.println(myFile.position());

if (SD.exists("delete.txt"))
{
  Serial.println("Created file as File");
}

myFile.write("Testing 1, delete.\r\n");
myFile.write("Testing 2, delete.\r\n");
myFile.write("Testing 3, delete.\r\n");
myFile.write("Testing 4, delete.\r\n");
myFile.write("Testing 5, delete.\r\n");
myFile.write("Testing 1, 2, 3.\r\n");
myFile.write("Testing 4, 5, 6.\r\n");
myFile.write("Testing 7, 8, 9.\r\n");
myFile.write("Testing 10, 11, 12.\r\n");
myFile.write("Testing 13, 14, 15.\r\n");

myFile.seek(0);

while(myFile.available())
{
  check = myFile.read();
  Serial.write(check);
}

Serial.print("myFile position after read is: "); Serial.println(myFile.position());
myFile.seek(0);
myFile.write(OVERWRITE);  
myFile.seek(0);
replace = myFile.position();
while (myFile.available())
{
   check = myFile.peek();
   if (check == 'T')
   {
    read = myFile.position();
    break;
   }
   myFile.read();
}

myFile.seek(read);
while (myFile.available())
{
  check = myFile.read();
  myFile.seek(replace);
  myFile.write(check); 
  replace++;
  read++;
  myFile.seek(read);
  if (myFile.peek() == -1)
  {
    newSize = myFile.size() - (read - replace);
    break;
  }
}

myFile.close();

root.openRoot(volume);
boolean exists = file.open(root, "delete.txt", O_CREAT | O_WRITE | O_RDWR);
//file.seekSet(0);

if (exists)
{
Serial.println("Opened file as SdFile");
}

file.truncate(newSize);
file.close();

myFile = SD.open("delete.txt");
while(myFile.available())
{
  check = myFile.read();
  Serial.write(check);
}
Serial.println();
myFile.close();


root.ls(LS_R | LS_DATE | LS_SIZE);

}

void loop ()
{
  // just loop 
}

I used a lot of print statements for my own debugging purposes, those can be taken out obviously. Other than that, I found I needed everything else in order for it to work properly and consistently. I have it set up to just delete the first line of the text file, it works. Thanks again @peekay123 for your earlier help, if anyone has any suggestions let me know :slight_smile:

3 Likes

@UST, great work!! I am glad you figured this out for yourself since now you own that knowledge :stuck_out_tongue:

1 Like

Hmm, I seem to have one last hurdle to pass before this is finally complete.

If the file is very large, roughly 720 “lines” with characters the same as the ones in the code I posted earlier, it takes a considerable amount of time using the two pointer method to delete characters.

The method is still great, but now I’m thinking what I could do is that every time a file gets to a particular size (I can check using File.size()) I create a new file, auto generate a name using a base name like “test#.txt” and do the two-pointer delete method normally.

However what I wanted to do in conjunction with creating these new files, is re-name the old text file to “test2.txt”, and the new file would be “test1.txt” that way when I do the two pointer delete method, it will check for files in order of “1 , 2 , 3 … 10 (for example)” where 10 is the oldest data.

I’ve looked online, and there seems to be some newer Arduino library that has SdFat.rename() and SdFile.rename(), but these don’t seem to be apart of the Spark SD library. Any ideas on how to rename a text file, @peekay123 @bko ? I was thinking SdFile.make83Name() might be of use, but it seems to be a private class and only usable for directory files.

EDIT: I found your thread @peekay123 with @mdma about porting over SdFat, I’m guessing this repo is not quite spark compatible yet? Is there still another method to rename text files outside this?

As always any direction is very much appreciated :slight_smile:

@UST, I think it may be time to review what you are trying to achieve with the files and look at the entire process instead of just the delete line portion. Can you elaborate on how/why the file is used? Understanding the “whole” may lead to a better or more efficient solution. :smile:

You are right @peekay123 I’ll explain below (I’ll do it in points, that way if anything is unclear just address that point(s) :)’ )

  1. I’m collecting data and saving the data as a “line” on a text file every minute.
  2. After saving the data, the Core sends the “line of data” off via WiFi
  3. If the transmission IS successful the Core performs the two-pointer-delete method and removes that line so that the next line can be sent.
  4. If unsuccessful the core just attempts to send the same line again.
  5. If it’s 50 sec into a new minute, things interrupt (after deleting line if it was done) and data is collected for the new minute.

So #3 is where my last hurdle seems to be, with this whole text file business. If my file gets too large, the delete method takes a while to go through. So it could happen that I miss a new minute of data, which I don’t want. So I was looking into this whole process if the original file gets too big:

@UST, I am assuming you need to create this “line” queue because you don’t always have cloud connectivity, correct?

  • Lines have to be sent in order?
  • If a line fails to send, all subsequent lines will wait until that send succeeds?
  • If the “lines” are not sent right away (every minute) and are queued, at what interval do you send them? That is, how fast are you clearing out the queued lines?
  • How are you sending the “lines” - Spark.publish(), TCPClient?
  • How long do you anticipate your worst cloud “black out” period to be?

I have an idea for another approach but it requires answers to be above first. :smile:

1 Like

@peekay123 I’m excited to hear!

  • Yes Lines should be sent in order, but I guess they don’t need to be now that I think about it, so either or works !
  • If a line fails to send, all the subsequent lines don’t even get read by File.read(), so yes in essence they wait until the first most is sent, then deleted, so that the next in line gets sent :wink:
  • Clearing them out as fast as possible. The less backed up data the better. So at the minute mark ( Time.second() = 0 ), after data has been saved to the SD, a line is transmitted and if done successfully , the line is immediately deleted via the two-pointer method. This process repeats until Time.second() is between 50 - 58 seconds (AND after all deleting has been completed, so the file doesn’t get corrupted) which at that time the process halts. This gives time to collect data for the new minute.
  • The “lines” are being sent via TCPClient
  • The worst black out period could be anywhere from a hour to a week, to even possibly two weeks! But I’d rather not have a “max”, and maybe just a large capacity SD card to account for even the largest down times. My SD card I’m using now is 32GB and I don’t even think I’ve dented it yet :stuck_out_tongue:

@UST,

  1. Order is not critical which means they are not timestamped?
  2. What is the worst case single line length?
  3. Do you open and close the file when you add a new line every minute?
  4. After sending lines, do you close the file BEFORE the sampling every minute?

:smile:

@peekay123,

  1. They are timestamped , but the only reason I say it’s not critical cause, eventually the data will all get to where it needs to go and get organized from there. But if order could prioritized then that’s a bonus.
  2. Worst case is 58 bytes in length! That is including “\r” and “\n” characters!
  3. Yes when the new minute begins the file is opened, data is written, then it is closed!
  4. Yes the file (which is a File. object) is closed before sampling. Maybe it could be left open to make things faster b/n samples, but it needs to be closed when doing the two-pointer delete method, because I create a SdFile object that associates with the same file in order to use truncate so the two objects can’t be opening the same file . . . at least that was my understanding :slight_smile: ?

@UST, so from my calculations:

  • 58 bytes/minute x 60 mins/hr * 24 hrs = 83,520 bytes/day
  • Let’s say a formatted 32GB card can hold 28x10e9 bytes
  • You could store about 335,000 days of data or 919 years!!!

So storage is not the issue. It now comes down to what you do with the queued data. You could simply let the file grow and just treat it like a queue with tail and head pointers. You shove data in at the head and take things out at the tail. When the cloud is connected, the tail always “catches up” to the head. When the cloud is down, the head gets further from the tail and when the cloud connects again, you get a flurry of send activity until the tail catches up to the head. So, basically, you never erase the file!

Another approach is to do the same but once your tail catches up to the head AND the file is bigger than a certain size, you truncate the file to the beginning (not erase) and start over again.

The idea is that you don’t do this crazy line delete stuff. You don’t have a storage issue, the SD performance will not change because you increase the size of the file since you are doing very simple operations. Finally, if size is an issue ( :stuck_out_tongue: ) you could just truncate the file and start over. Keep It Simple, er, Dude! :smile:

2 Likes

Woah ! @peekay123 thanks for doing those calculations for me; definitely gives me more information to use!

Okay, so let me digest all of this . . . Also this is the first thing that came to mind on how to implement your mentioned method!

So lets say I have a text file like this, I would have my tail pointer “T” as such, and H wouldn’t be really needed, cause I’ll always write at the end of the text file.

Ttesting. 1, 2, 3.
Ttesting. 4, 5, 6.
Ttesting. 7, 8, 9.

So on the minute mark things would go down like this:

myFile = SD.open("test.txt", O_RDWR); //Since I'm at the end of the             
                                  //test.txt, no need to use seek() to find "H"
if(myFile.available())
{ 
     //for example this is the acquired data
    myFile.println("testing. 1, 2, 3.");           
}
myFile.seek(0); // go to the start
while (myFile.available())
{
  char X = myFile.peek();
  if (X == T)  unsigned long posit = myFile.position(); 
}
//---- then do the reading and sending ----// 
//if success, file cursor SHOULD be at the start of the next "line" 
unsigned long nextposit = myFile.position();
myFile.seek(posit); 
myFile.write('X'); 
posit = nextposit; 
myFile.seek(posit); 
//--then read again and send again--//
//-- if success again , repeat the above five lines --// 
//------ rinse and repeat this until 50<=Time.second() AND the 5 above lines have finished running ------//
//--  THEN we do: --//
 myFile.close(); 
 // ------------ on the NEXT minute , we look for the "T" and repeat the process -------//

So is it correct-ish, or is there a simple,er, way :stuck_out_tongue: :slight_smile: ?

Doing it with this method I guess size doesn’t matter ( :stuck_out_tongue: ) If you’re saying SD speed, reading/writing, won’t be affected. Which I guess it wouldn’t be. But when I read through the file to find “T”, if the file is REALLY large won’t it take longer to look for it? My guess is your saying it might but it won’t be anything that affects performance such as missing the 50 sec timeout. And if by testing and such I find that a file can get too large, I can truncate it as you suggested!

Edit: I realize the code isn’t 100% few lines I left out here and there, BUT the general idea is there I believe

@UST, close but not quite. The head and tail pointers need to be file indexes so you read from the tail until your end of line char and reset the tail to the next line. So when the head and tail are the same value, you have no new lines to send. No searching,just seeking to file index. No fuss, no muss. :smile:

1 Like

@peekay123 Oh okay! That makes more sense, and it’ll be simpler too then what I had written!

But in the event of a power outage these indexes will be lost in my variables, so maybe I’ll truncate every time the tail and header are the same to avoid re-sending of data! :wink: Thanks again for all the help!

@UST, now you’re cooking with propane!!! :stuck_out_tongue: Will you be sharing you code once it’s working?

Yeah I think I’ll cook something up, I’ll keep posted ! :stuck_out_tongue:

1 Like