Writing Struct data in file

princeerss · June 20, 2022, 5:25am

Could you help me on writing Struct data in Particle file system as text/csv file?

typedef struct {
  int humidity;
  time_t timestamp;
  const char * date_time;
  String device_id;
} SensorData;

ScruffR · June 20, 2022, 11:39am

Your date_time would need to be a character array, and not a mere pointer to some memory which will definitely not hold the same date once you retrieve that pointer

However, a UNIX epoch timestamp can be held in an int32_t much more compact and can be used for calculations much easier too - in order to format such a numeric timestamp you can use Time.format() anytime - hence I’m not sure why you want to store timestamp and date_time in the same struct, when both are virtually the same piece of information.

As for the device_id you’d also need a char[25] as a String for the same reasons as above.

princeerss · June 23, 2022, 6:42am

I am using these 2 functions for file read and write. Passing my data as comma separated string as
fileString = “DEVICE_001,27,80,1655966444” but after writing I called the read file function but it prints empty values. Could you suggest ?

void writeFile(String fileString) {
  int fd = open("/rssi.txt", O_RDWR | O_CREAT | O_TRUNC | O_APPEND);
  if (fd != -1) { 
    write(fd, fileString.c_str(), fileString.length());
    close(fd);
  }   
}

void readFile() {
  struct stat statbuf;
  int sd = stat("/rssi.txt", &statbuf);

  if (sd != 0) {
    return;
  }

  int fd = open("/rssi.txt", O_RDONLY);
  Log.info("fd %d", fd);

  if (fd != -1) {
    void *buffer;
    int rd = read(fd, buffer, 6);
    char* content = static_cast<char*> (buffer);  
    Log.info("rd %d", rd);

    close(fd);
    Log.info("got a msg buffer");
    Log.info(content);
  }
}

dreamER · June 23, 2022, 6:08pm

Not sure how you even were able to compile this ?
don't think that stat(); function even exist for Particle dev and SD card.
Which Lib for SD card are you using ? (I'm assuming that you want to read/write from/to SD card)
You can't do this:

you have to initialize the File class first e.g: File myFile; or SdFile csvFile; not int myFile; as is not going to work 100% what will happen when you will try
myFile.println("TESTUS...."); compiler will give up believe me
i suggest to start from here and look for: examples/LowLatencyLoggerADXL345/LowLatencyLogger.ino there's some info how to write/read from/to struct named block_t
good luck

princeerss · June 24, 2022, 3:41am

This is the entire code I am using. I am trying to write on Particle’s internal file system not on SD card. I just wanted to confirm my data is stored in file or not that’s why I printed content. In real time I post that file content via MQTT.

#include "Particle.h"
#include <fcntl.h>

SYSTEM_MODE(MANUAL);

SerialLogHandler logHandler(LOG_LEVEL_TRACE);

#define SCAN_INTERVAL 5000
#define ONE_DAY_MILLIS (24 * 60 * 60 * 1000)
unsigned long lastSync;

//Function Declarations
void syncCloudTime();
void readFile();
void writeFile();

typedef struct{
  String device_id;
  unsigned long timestamp;
  float sensorReading;
} SensorData;

void setup() {
    (void)logHandler; // Does nothing, just to eliminate warning for unused variable
    //DynamicJsonDocument deviceList(1024);
    syncCloudTime();
}

void loop() {
    if (millis() - lastSync >= ONE_DAY_MILLIS) {
        syncCloudTime();
    }
   
    SensorData data;
    data.device_id = "SENSOR_01";
    data.timestamp = Time.now();
    data.sensorReading = 22.0;
      
    String fileString = String(data.device_id) + "," 
                      + String(data.timestamp) + "," 
                      + String(data.sensorReading);
    writeFile(fileString);
    delay(250);
    readFile();
    delay(5000);
}

void syncCloudTime() { // Request time synchronization from the Particle Device Cloud
    WiFi.on();
    WiFi.connect();
    waitFor(WiFi.ready,60000);
    Particle.connect();
    Particle.syncTime();
    waitFor(Time.isValid, 60000);
    lastSync = millis();
    Particle.disconnect();
    WiFi.disconnect();
    WiFi.off();
}

void writeFile(String fileString) {
  int fd = open("/data.txt", O_RDWR | O_CREAT | O_TRUNC | O_APPEND);
  if (fd != -1) { 
    write(fd, fileString.c_str(), fileString.length());
    close(fd);
  }   
}

void readFile() {
  struct stat statbuf;
  int sd = stat("/data.txt", &statbuf);

  if (sd != 0) {
    return;
  }

  // this always returns 1823 for me, event if I change the file contents

  // read the file
  int fd = open("/rssi.txt", O_RDONLY);

  Log.info("fd %d", fd);

  if (fd != -1) {
    void *buffer;
    int rd = read(fd, buffer, 6);
    char* content = static_cast<char*> (buffer);  

    Log.info("rd %d", rd);
    close(fd);

    Log.info("got a msg buffer");

    // THIS IS JUST EMPTY, I would expect it to be "foobar"
    Log.info(content);
  }
}

ScruffR · June 24, 2022, 5:02am

This only creates a pointer. It does not reserve any memory to read the data into.

I'd also do away with String and rather use a character array
e.g. instead of

princeerss:

    String fileString = String(data.device_id) + "," 
                      + String(data.timestamp) + "," 
                      + String(data.sensorReading);
    writeFile(fileString);
...
void writeFile(String fileString) {
  ...
}

I'd do this

  char fileString[128];
  snprintf(fileString, sizeof(fileString)
          , "%s,%012lu,%.3f"
          , data.device_id   // where device_id should be declared as char[25] in the struct, not String
          , data.timestamp
          , data.sensorReading
          );
  writeFile(fileString, strlen(fileString)+1);
...
void writeFile(const char* fileString, size_t len) {
  ...
}

However, since you have a struct, why not write that as is and as your original question was indicating you wanted anyhow?

  write(fd, (const void*)&data, sizeof(data));
  // and
  read(fd, (void*)&data, sizeof(data));

(providing device_id is defined as char device_id[25])

princeerss · June 24, 2022, 7:50am

Sure I try this definitely . Thanks a lot

princeerss · July 14, 2022, 11:39am

I tried this way to read the file buffer but it reads only the first time written value and not showing remaining file contents. Could you help me to debug this ?

void readFile()
{
  struct stat statbuf;
  int sd = stat("/rssi.txt", &statbuf);

  Log.info("Size of RSSI File %d",statbuf.st_size);

  // read the file
  int fd = open("/rssi.txt", O_RDWR);

  Log.info("File open for read int value %d", fd);
  char* buffer;

  if (fd != -1) {
    buffer = (char*)malloc((statbuf.st_size + 1));
    int rd = read(fd, buffer, statbuf.st_size);

    Log.info("File Read int value %d", rd);

    Log.info("Got a msg buffer!!");
    Log.info(buffer);

    WiFi.on();
    WiFi.connect();
    waitFor(WiFi.ready,60000);
    client.connect("BLEPARTICLE_" + String(Time.now()));

    // publish/subscribe
    if (client.isConnected()) {
      client.publish("post/message",buffer);
    }
    lastMqttPost = millis();
    free(buffer);
    client.disconnect();
    Particle.disconnect();
    WiFi.disconnect();
    WiFi.off();
  }
  close(fd);
}

ScruffR · July 14, 2022, 11:49am

I’d suggest you write out the file contents byte by byte as I suspect your file to contain some non-printable characters (i.e. '\0' zero-terminator) which may terminate the Log output as well as the client.publish().

The overload you are using assumes a string and hence will stop as soon the first zero-terminator is enountered.

princeerss · July 14, 2022, 11:51am

Thanks for the suggestion. I will check that while writing

princeerss · July 14, 2022, 12:40pm

Removed the null character. Now working Fine. Thank you for the help

princeerss · July 14, 2022, 12:55pm

Getting this character now at end at the MQTT receiver
\xff\xff

Message received-> 2022-07-14 18:19:55.168990 post/message b’CA:4F:DA:6A:9E:07,BB-CA4FDA,1.0,1.0,1657802935,-58CA:4F:DA:6A:9E:07,BB-CA4FDA,1.0,1.0,1657802951,-64CA:4F:DA:6A:9E:07,BB-CA4FDA,1.0,1.0,1657802966,-65\xff\xff’

ScruffR · July 14, 2022, 2:06pm

Is there a problem with that?
When you know that’ll always indicate the “end of file/transmission” you may as well facilitate it.

If it messes up your receiving side, we’d need to know where the original data comes and how it’s stored in the file.

0xFFFF may come from an int16_t reading -1

princeerss · July 14, 2022, 2:12pm

This is the entire source code I am using to fetch data and storing in file and reading data at specific interval from file to send via MQTT

// This #include statement was automatically added by the Particle IDE.
#include <MQTT.h>

// This #include statement was automatically added by the Particle IDE.
#include <math.h>
#include "Particle.h"
#include <fcntl.h>
#include <sys/stat.h>



SYSTEM_MODE(MANUAL);

SerialLogHandler logHandler(LOG_LEVEL_TRACE);

#define SCAN_INTERVAL 10000
#define MEASURED_POWER -77
#define N 2
#define ONE_DAY_MILLIS (24 * 60 * 60 * 1000)
unsigned long lastSync;

#define MQTT_POST_INTERVAL (30 * 60 * 1000)
unsigned long lastMqttPost;



void callback(char* topic, byte* payload, unsigned int length) {
Log.info("Message Received From MQTT");
}

MQTT client("192.168.1.3", 1883, callback);

void setup() {
    (void)logHandler; // Does nothing, just to eliminate warning for unused variable
    //DynamicJsonDocument deviceList(1024);
    syncCloudTime();
    int dl = unlink("/rssi.txt");


}

void loop() {
    if (millis() - lastSync >= ONE_DAY_MILLIS) {
        syncCloudTime();

    }
    if (millis() - lastMqttPost >= MQTT_POST_INTERVAL) {
        readRssiFile();


    }
    Vector<BleScanResult> scanResults = BLE.scan();

    if (scanResults.size()) {
        Log.info("%d devices found", scanResults.size());

        for (int ii = 0; ii < scanResults.size(); ii++) {
            uint8_t buf[BLE_MAX_ADV_DATA_LEN];            
            size_t len;
            // For Device OS 2.x and earlier, use scanResults[ii].address[0], etc. without the ()
            Log.info("MAC: %02X:%02X:%02X:%02X:%02X:%02X | RSSI: %dBm",
                    scanResults[ii].address[0], scanResults[ii].address[1], scanResults[ii].address[2],
                    scanResults[ii].address[3], scanResults[ii].address[4], scanResults[ii].address[5], scanResults[ii].rssi);
            char name[25];
            strcpy(name,scanResults[ii].advertisingData.deviceName());

            len = scanResults[ii].scanResponse(buf, sizeof(buf));
            if (len == 14) {
                Serial.printf("Scan response data length: %02d\r\n", scanResults[ii].scanResponse.length());
            
                char fileString[51];
                snprintf(fileString, sizeof(fileString)
                , "%02X:%02X:%02X:%02X:%02X:%02X,%s,%s,%s,%lu,%d",
                scanResults[ii].address[5], scanResults[ii].address[4], scanResults[ii].address[3],
                scanResults[ii].address[2], scanResults[ii].address[1], scanResults[ii].address[0]
                ,name,
                "1.0",
                "1.0",
                Time.now(),
                scanResults[ii].rssi
                );
                Log.info(fileString);
                Log.info("FileString Size %d",sizeof(fileString));
                
                writeFile(fileString, strlen(fileString));

         
            }
        }
    }

    delay(SCAN_INTERVAL);
}


void syncCloudTime(){ // Request time synchronization from the Particle Device Cloud

  
    WiFi.on();
    WiFi.connect();
    waitFor(WiFi.ready,60000);
    Particle.connect();
    Particle.syncTime();
    waitFor(Time.isValid, 60000);
    lastSync = millis();
    Particle.disconnect();
    WiFi.disconnect();
    WiFi.off();

}




void writeFile(const char* fileString, size_t len) {
    int wd = open("/rssi.txt", O_RDWR | O_CREAT | O_APPEND);
   
    int w = write(wd, fileString,len);
    Log.info("write Response %d",w);

    close(wd);
    
}

void readRssiFile()
{

  struct stat statbuf;
  int sd = stat("/rssi.txt", &statbuf);

  Log.info("Size of RSSI File %d",statbuf.st_size);

  // this always returns 1823 for me, event if I change the file contents

  // read the file
  int fd = open("/rssi.txt", O_RDWR);

  Log.info("File open for read int value %d", fd);
  char* buffer;

  if (fd != -1) {
    buffer = (char*)malloc(statbuf.st_size+1);
    int rd = read(fd, buffer, statbuf.st_size);

    Log.info("File Read int value %d", rd);

    Log.info("Got a msg buffer!!");
    Log.info(buffer);
    


    WiFi.on();
    WiFi.connect();
    waitFor(WiFi.ready,60000);
    client.connect("BLEPARTICLE_" + String(Time.now()));

    // publish/subscribe
    if (client.isConnected()) {
    client.publish("post/message",buffer);
    }
    lastMqttPost = millis();
    free(buffer);
    client.disconnect();
    Particle.disconnect();
    WiFi.disconnect();
    WiFi.off();
  
  }
  close(fd);
   int dl = unlink("/rssi.txt");


  
}

ScruffR · July 14, 2022, 2:17pm

Maybe try to initialize your buffer to 0x00 before you read into it.

    buffer = (char*)malloc(statbuf.st_size+1);
    memset(buffer, 0, statbuf.st_size+1);
    int rd = read(fd, buffer, statbuf.st_size);

princeerss · July 14, 2022, 2:31pm

Great. It worked Finally !!

system · August 14, 2022, 12:32am

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Copy String to Struct - I am confused Getting Started	1	1238	January 1, 2020
[Solved]- Storing strings into an array - Basic programming question General	3	1372	March 17, 2017
Publishing Current Epoch Time using Particle Publish General	7	1890	May 10, 2016
Data transfer tips Tutorials	2	1035	February 17, 2018
Store/Read String from FileSystem Firmware argon	4	77	July 13, 2024

Writing Struct data in file

Related topics