I am using an SD card as a queue for events that aren’t sent during periods of poor cell service, and am trying to figure out how to delete data/lines from the file efficiently. I have tried following the solved issue here, but I can’t seem to follow it to the point of getting it to work…
As of now, I am writing each failed event to a new line on a file separated by '\n'. This way, I can fairly easily iterate over each line and then publish the event when cell service resumes. My issue is that I can’t seem to overwrite or delete the line from the file. I have tried (see below) to simply overwrite that part of the file with an empty string of the same number of characters, but it doesn’t seem to work.
Any thoughts/hints/strategies would be greatly appreciated!
int get_line_count() {
int line_count = 0;
if (SD_INIT) {
if (LF.open(QUEUEFILE, O_RDWR)) {
char inputString[255] = "";
char inputChar;
int idx = 0;
int str_size = 0;
while (LF.available()){
inputChar = LF.read();
if (inputChar != '\n') {
inputString[idx] = inputChar;
idx++;
} else {
line_count++;
idx = 0;
inputString[0] = '\0';
}
}
LF.close();
}
}
return line_count;
}
// loop() runs over and over again, as quickly as it can execute.
void loop() {
// add 25 lines to a file
delay(2500);
Serial.println("\nAnother cycle...");
if (SD_INIT){
int lines = get_line_count();
Serial.println(String::format("Lines left: %d", lines));
// Delete the first line from the SD card
if (SD_INIT) {
if (LF.open(QUEUEFILE, O_RDWR)) {
char inputString[255] = "";
char inputChar;
int idx = 0;
int start_idx, end_idx;
int str_size = 0;
// go to the start of the file!
LF.seek(0);
start_idx = LF.position();
// Get the first line
while (LF.available()){
inputChar = LF.read();
if (inputChar != '\n') {
inputString[idx] = inputChar;
idx++;
} else {
// we have a complete line
end_idx = LF.position();
str_size = end_idx - start_idx;
char replaceStr[str_size] = "";
//String replaceStr = String(str_size);
Serial.println(String::format("Start=%d, End=%d, len=%d", start_idx, end_idx, str_size));
Serial.println(String::format("Replace %s with %s", inputString, replaceStr));
// Overwrite the String
LF.seek(start_idx);
LF.write(replaceStr);
LF.close();
break;
} // end else
} // end LF.available
LF.close();
} // end LF.open
} // end SD_INIT
}
As for your original question, however: It’s really inefficient to remove a line from a file because it requires rewriting the file.
Instead, what I’d do is create a file, one per event, with a sequential 8-digit number in an events directory. That way, you publish the lowest numbered file, then if it publishes successfully, you delete it.
An alternative to @rickkas7’s suggestion would be to just mark any particular line as “deleted” (e.g. by replacing the first byte with a “magic” byte, like some file systems do when deleting files from a directory) and only act on unmarked lines.
If you can’t find any unmarked lines in the file, it’s safe to delete the entire file.
Which version suits your use-case best depends on number of possible active lines, likelyness to have lines that will stay for long periods and how long is and similar factors.
I am looking for a solution to the exact same problem. I am writing some sensor data into a file on my SD card while the device is offline.
Once the device comes back online, it opens the file, reads a line and publishes the data to the cloud.
I am struggling to delete the read/published line before moving onto the next line in the file.
Solutions I tried:
Replacing the characters that have already been read - If the device goes offline before it is able to read and publish all the data from the text file on the SD, the file size is going to keep increasing (because it is never deleting the read characters) and I’m going to run out of free space soon.
truncate - truncate deletes data only from the end of file. Which means that I have to read through the entire file, find the last line (don’t know how I’d find the start of the last line), read/publish the line and then truncate the file to (original_filesize-read_LineSize). This seems like a very expensive process, especially when it is repetitive.
@archr, did you read the entire thread? Some suggestions were pointed out such as:
Create more files (groups of 10, hourly, ect.) that you can read/publish in its entirety and delete entirely
Mark read/published entries with a “marker” as the first character indicating the line is to be ignored. Once all the lines are sent, you can delete the entire file.
Create another file and copy line-by-line only the outstanding lines then delete the file and work from the copy.
Personally, I would go with one of the first two suggestions.
@arch, for your option 2, if you have a fixed length line, you can use a seek instruction which positions the read cursor at exactly the position you want (e.g. n * len_of_line would allow you to read the n-th line).
For option 1 (similar to my marker-suggestion) you don’t need to replace all bytes and if it should happen that your app crashes exactly that split second after publishing and before setting the one byte - how big is the damage in resending that one line?
If this should really happen repeatedly, I’d rather look for the flaw in your code that causes a “persistent” issue of that kind
For people who are still looking for alternatives, I found a couple of solutions that work:
Solution 1 - Works for me
I create a file each time I sample sensor data- so that’s one file, containing a single line of sensor data. Once the device is back online I use sdFat’s “openNext” function to open a file, publish its contents, and then rename it to “delete.txt”. So delete.txt is overwritten with the published file’s data. Since I don’t really care about delete.txt, I remove it periodically!
This ensures I never run out of memory (assuming my device is online during some part of the day)
Solution 2- Retained Memory (risky)
In this solution, I create only 1 file. I initialize 3 retained variables: line_number, head[], tail[]. Each time I write to the file, I save the line number and the corresponding head and tail pointer positions. This way I have the start and end position of each line. I can simply use the line_number to navigate to the head and tail positions of the last line in the file, read/ publish it, and finally, truncate the file.
Cons : Limited memory, loss of retained data in cases of power outage
Follow-up Question:
Is there a way to judge exactly how expensive, in terms of POWER consumption, solution 1 is?
I ran into an issue with using openNext in O_RDWR mode. I am using software SPI and openNext works when I open a file in read mode (sdFile.openNext(sd.vwd(),O_READ)). But for renaming the file after read, I need to open it in read-write (RDWR) mode. When I do that, I am not even able to read the file.
Also, I’m using openNext because I don’t have the file names.
@rickkas7 I have a question about using your library:
When the buffer fills up, which in my case happens after about 10-15 events, it pops off the last event to make room for the most recent one (at least that’s how I interpret your docs). Is there an easy way to do something else with that event (i.e. write it to file) rather than dump it?
