One of my Photon projects gathers data from a number of different sources based on a schedule. It then knits them into JSON strings which are exposed by Particle Variables. The JSON is then downloaded, parsed and displayed via HTML on my computer and cell phone. When the Photon reboots, It would take a while for the data to repopulate, and that was annoying. I also wanted to log the data locally.
I recently came across a solution that doesn’t have a write-life, and that can handle a large data set … so this is how things work today:
When my Photon updates a Particle Variable, it now publishes a MQTT message
where the MQTT topic is the Particle Variable’s name.
where the MQTT message is the JSON string.
where the MQTT retain flag was set true.
On a reboot.
my Photon subscribes to the same MQTT topics that it publishes
the MQTT broker automatically republishes the last message for each topic since the retain flags were set true.
the Photon receives the messages, refreshes the Particle Variables, and unsubscribes before it resumes normal operation.
an RPI on my home network subscribes to the same topics and logs them to a network drive.
Now, nothing is lost, I have logging, and I’ve added a control topic to make sure the Photon knows where it was in the collection schedule … in case of a long power outage.
Nice. Did you consider using a struct for the data and saving it to EEPROM? You could have the photon load the data into RAM after a reboot. The data would persist after a reboot or power loss.
I actually do use EEPROM to store two relatively static data classes. Other classes are too dynamic and/or too large for EEPROM. I did consider adding a battery to the Photon and then using either RAM or SD storage to preserve the data at the Photon, but …
Logging was my first priority. Publishing the JSON strings via MQTT was an easy way to log data to a hard-drive. Once that was in place I started thinking about how nice it would be if the Photon could simply ask for the data back. I discussed it with a friend, and he proposed the idea described above.
The really nice thing is this solution for my application was that the JSON strings already existed. It took very little code to connect to MQTT and publish the existing strings for logging. It took even less code to subscribe to the topics and copy the JSON strings back to the variables after a reboot. My original application logic was not impacted at all. No new JSON formatting or parsing was required.
Take a look at AWS IOT and their “Device Shadow” feature. You could easily create your own version of this. Essentially, you have a JSON object that represents your devices “state”. Both the device and the cloud have a copy. This state object / JSON document can be either:
sent in complete form as a “status” message
sent in partial form as an “update”, which updates the receivers status document by only sending the parts needing to be changed.
So here is how you use this for restoring state after reset:
The Photon resets, clearing all the state information to a defined default state.
The Photon sends a state “status” document to the cloud as soon as it is connected.
The cloud compares the received “status” document to it’s own latest status document, calculates the differences, and sends an “update” partial document with the required changes back to the device in response.
The Photon receives the update document, parses the differences, and changes both it’s internal variables and it’s state document to match.
Once you have a structure like this, you can actually use this idea to completely get rid of your Particle variables. This gives you an offline, cloud source of truth that should always match your device (assuming that every time a variable change, you use that MQTT message to also update your cloud state document with the contents).
Also the “ArduinoJson” library is handy for more complex JSON work if you aren’t already aware of it.
Also btw, you can usually configure your broker to log connection/disconnection events, and assuming you use the particle device_id as your client id (or can look it up), you can have a script listening to those events and updating the connection status of your device in the cloud-side state document as well. Presto, you always know if your devices are connected without having to check (well, within the resolution of your MQTT keep-alive interval, which is often 15 seconds, so still pretty good!). You could also configure a last will and testament MQTT message that updates the cloud state document with the disconnection state, and then update with a connected state directly from the device upon first connection.