Magnetometer Detection Over Cellular with Boron

I am working on a project where I want to use a magnetometer as a detection sensor to be used with the Boron 404X. The way it works is I have a magnetometer connected to the Boron 404X via I2C communication. I have successfully been able to transmit one magnetometer event reading (X, Y, and Z axes) over cellular using particle.publish() through a webhook to my phone as string text.

The main issue is that I would like to send multiple event readings (preferably as one JSON file) as one event to some server. As I understand it, particle.publish() uses UTF-8 characters, not JSON. Also, there is a data limit of 1024 bytes per event using this function, which is not enough for my purposes. I was wondering if there was some other way to communicate over cellular, allowing me to transmit a large JSON file.

I am not sure how large this JSON file would be, but here is the information I have gathered about the magnetometer data. The magnetometer collects data at 20Hz with a 16-bit output for the X, Y, and Z axes (48 bits total per reading). However, since each character is 1 byte for particle.publish(), and assuming 3 digits (characters) per axis all concatenated together, I would only be able to send 1024/(9x20) = 5.6 seconds of readings for one event. 5.6 seconds is not enough for my purposes, most likely 30 seconds of data per event would suffice.

In summary, I want to take a set magnetometer readings over a span of 30 seconds and store them as a JSON file (whether in SIM or internal memory, it doesn't matter). Then, I would like to transmit that JSON file to a server using cellular on the Boron 404X. It does not matter how long it would take to transmit through cellular, since I know that cellular data rates are low. I just need this operation to be doable. I do not know how to convert these readings to JSON nor do I know how to send a JSON file over cellular.

I have linked the magnetometer and the magnetometer's datasheet that I am using, as well as code that I used to transmit this data to my phone using the Pushover webhook. The code will first see if the magnetometer is connected to the Boron. If it is, it will collect data from the magnetometer, store that data into a variable as a string, and then send that data to my phone. Any help on this is appreciated.

Magnetometer: https://cdn-learn.adafruit.com/downloads/pdf/adafruit-lis2mdl-triple-axis-magnetometer.pdf

Magnetometer Datasheet: https://www.st.com/resource/en/datasheet/lis2mdl.pdf

// This #include statement was automatically added by the Particle IDE.
#include <Adafruit_BusIO_Register.h>

// This #include statement was automatically added by the Particle IDE.
#include <Adafruit_Sensor.h>

// This #include statement was automatically added by the Particle IDE.
#include <Adafruit_LIS2MDL.h>

#include <string>

using namespace std;

/* Assign a unique ID to this sensor at the same time */
Adafruit_LIS2MDL lis2mdl = Adafruit_LIS2MDL(12345);
#define LIS2MDL_CLK 13
#define LIS2MDL_MISO 12
#define LIS2MDL_MOSI 11
#define LIS2MDL_CS 10

String str1,str2;

unsigned long pwrCheckTimeStart; //to check readings every 20sec

void setup() {
    
    // Enable auto-gain
    lis2mdl.enableAutoRange(true);
    
    // Initialize the sensor
    if (lis2mdl.begin()) { // I2C mode
        str1 = "Mag There?";
        str2 = "Yes";
        sendData();
    }
    
    else {
        str1 = "Mag There?";
        str2 = "No";
        sendData();
        while (1) delay(10);
    }
    
       
    pwrCheckTimeStart = millis();
}

void loop() {
  // Loop every 20 sec
  if(millis()-pwrCheckTimeStart>20000) {
    pwrCheckTimeStart = millis();
    
    // Get new sensor event
    sensors_event_t event;
    lis2mdl.getEvent(&event);
    
    // How the magnetometer reading appears on my phone
    str1 = "MagReadings";
    str2 = "X: " + String(event.magnetic.x) + " Y: " + String(event.magnetic.y) + " Z: " + String(event.magnetic.z);
    sendData();
       }
    
      //********************
}

void sendData(){
     unsigned long startConnectTime = millis();
     char pushMessage[50], pushName[50];
     str1.toCharArray(pushName, str1.length() + 1);
     str2.toCharArray(pushMessage, str2.length() + 1);
     
     Serial.println(str1);
     Serial.println(str2);
     
     
     String pushoverPacket = "[{\"key\":\"title\", \"value\":\"";
     pushoverPacket.concat(str1);
     pushoverPacket.concat("\"},");
     pushoverPacket.concat("{\"key\":\"message\", \"value\":\"");
     pushoverPacket.concat(str2);
     pushoverPacket.concat("\"}]");
     Particle.publish("magread", pushoverPacket, PRIVATE);//then send to push safer so we get the notifications on our mobile devices

     Serial.print(millis() - startConnectTime);
     Serial.println("ms to connect");
}

JSON is ASCII, which is a subset of UTF-8, and is a popular format for publishing data.

However the length limitation still applies.

There are three alternatives:

Split the data into multiple publishes of up to 1024 bytes. This is the most feasible if you need to send data not in real time, and not continuously. You reassemble this data on a server. If the data is less than 16Kbytes, you could reassemble it using Logic and Ledger then send the reassembled version using a webhook.

If the updates are less than 16 Kbytes and are somewhat infrequent (such as once or day or possibly as much as one per hour) you can use device-side ledger coming soon in Device OS 5.9.0. This will allow you to save the data on the device in its flash memory and it will be synchronized to the cloud.

A bit farther out than that, a future version of Device OS will likely provide a way to send publishes up to 16K instead of 1K.

1 Like

@rickkas7 Thank you for the quick reply. In which case, I have some more questions.

Is there any other method of cellular communication through the Particle Boron that allows for a complete file transfer rather than the 1024-byte limit of publishing? I think it would make it easier for me to code as I have almost no experience in coding.

Another question I had is can the Boron transmit data while still store magnetometer readings into a file (parallel functionality). This is because I would like the detection to be as real-time as possible, and having the Boron stop collecting data when transmitting would also result in measurement losses.

If the Boron has no other way of exceeding the 1024-byte limit and it cannot transmit and collect data at the same time, then I believe the only solution is the one you explained, which is not ideal for my situation.

Using the first alternative, is there a way to read a JSON file (or array) size in bytes before transmission? I am not be sure how to go about splitting the file into 1024-byte chunks, and I was not able to find how to read file sizes in the documentation. I want to be able to control the size of each chunk for transmission, as I want the X, Y, and Z measurements of one reading together in one publish before the next one activates. Thank you for the help.

The 1024 byte limitation is not network-based, it's Particle cloud-based. While in theory you could bypass the cloud, we don't recommend doing so because it requires writing both client and server code, as well as dealing with properly securing the data. It also may become prohibitively expensive, as the Particle SIM was intended for low-data-rate applications and sending large amounts of data frequently may incur large numbers of block charges.

You can collect data and send it asynchronously, but because of the upload rate limit to the Particle cloud of approximately 1 Kbyte per second, you'll eventually hit a limit where you can't save the data anymore, or it would take a prohibitively long time to upload.

Hi, how many times per day will you want to send all this information of 30 sec readings to the cloud and eventually your server?

You mention saving in a JSON file, but what if you stored reading after reading on files of up to 1kb and then transmit them? You could choose to store the readings in a raw format (vs JSON).

Cheers

@pduong4, is the data being used for training an ML model? Ideally, the processing of the data, or at least the reduction of data is done on the Boron before being sent to the cloud. This reduces the amount of data being sent and becomes more event driven then just generating a data stream which the Boron is not really designed for.

1 Like

@rickkas7 Thank you for the information.

@rickkas7 @gusgonnet @peekay123 Thank you all for giving me these suggestions and let me explain my project as specific as possible so that everyone has a understanding of what I am trying to accomplish.

I am trying to make a solar-powered parking lot counter. The way this works is that a magnetometer will be placed at an entrance/exit of a parking lot, where cars will drive over it. As a car passes over it, this data will be collected by the Boron and upload it via cellular to my client's server. To reduce the power consumption of this setup, the plan was to establish a baseline magnetometer reading. As a car passes over the magnetometer, it will trigger a spike in the X, Y, and Z measurements. When this spike happens, the Boron will start storing data from the magnetometer until the readings go back to the baseline. Then, I would like to send that stored data via cellular to my client's server where it will be used for training a Machine Learning model. This is so that based on the measurements, my client can tell whether a car, motorcycle, golf cart, etc. has passed over the magnetometer, and in which direction (entering or exiting the lot). This means that the amount of data for each transmission is variable and the number of transmissions per day is also variable. Ideally, this is done in real-time, meaning that as soon as a car passes by and the readings are back to the baseline, then the client will receive the data via cellular.

In summary, I would like the Boron to transmit raw magnetometer data with variable size and variable frequency to my client's server so that my client and process the data on his end. This is because we believe that the Boron does not have the processing power to analyze the datapoints to correctly identify vehicles and their direction of motion. The Boron will also be connected to a LiPo battery and 5V 6W solar panel (connected to the microUSB port), meaning that PMIC must also be implemented.

Thank you so much for your time, and I hope that this clears any ambiguity in what I am looking for.