Logging data and missed packets

General
1

I am using a Photon (the WiFi version and Cellular version) to log temperature/humidity (using a DHT11 sensor). What I am having trouble with - I am sending data to Particle cloud - working fine. I have also linked to a Ubidots account for front-end/UI view of the points collected. Although Ubidots is slow to respond sometimes…everything seems to work fine. What I need to know how to do, but am stuck at - what if I have a battery backup for my Photon (I do) and the power is on, the device still collects data, but the internet goes down.

Since I am logging data, I would want those missing points. Anyone know of any way to do this? Basically, here is the desired outcome:

  • The Photon streams collected sensor data to the cloud
  • There is a break in connection - either the power or internet…so the Photon continues to collect information.
  • Once re-connected to the cloud, there is a check to see what data is missing from the time of the last collected data reading to the current time.
  • The Photon pushes those missing packets to the cloud in the respected place and times.

As a side note, I bought the SD board, but have yet to figure that out (library/install) but have the hardware/wiring connected correctly. I am just confused on the library side of things…

I believe there is enough space on the Photon on-board storage for at least 24 hours of temp/humidity readings

What would I need to do to make this happen? Please advise - any assistance is GREATLY appreciated…thank you fellow makers

2

Which SD board?

Which onboard space are you referring to?

3

Thank you for responding!

I used this SD: https://www.sparkfun.com/products/544

I **attempted to make sense of and install this library: SDFAT https://github.com/greiman/SdFat-Particle

I put it on a breadboard like so (real image)

222220170511_095050.jpg1900×1069 1.66 MB
:

I have the pins correct - there were lots of example files in the DAFAT library - but for NuBz like me, I can’t make heads or tails of how to simply make all data points store to the card, until full, then recycle maybe earliest point - or something like that.

By “onboard storage” I meant the amount of space already existing on the Photon itself - since my current libraries (for the DHT11 and Ubidots) and base libraries don’t take up that much space…

Thank you so much for taking your time!!

Is what I hope to do too complex for the Photon?

4

So, in essence, what detailed steps would be required to: store data in a SD Card only if there is no internet connection, and once there is internet/cloud connection, transfer new data and stored SD Card data to my Particle cloud account?

Additionally, is there a way to then update a service like Ubidots with that archived/stored info appropriately?

To me, this sounds incredibly complex for the Photon, but from about ten hours of Googling, and frustration…I see a faint light at the end of that tunnel…but that information is so fractured and incomplete… :slight_smile:

5

Yes. Additional bookkeeping will be needed on the Photon end.

I am going to assume that if the Photon is online, you are just sending data points (without a timestamp)? This is likely using the POST method (adding new element inside the cloud).

The Ubidots API supports four HTTP methods:

GET - Used to retrieve information from the cloud
POST - Used to create a new element inside the cloud
PUT - Used to edit existing elements
DELETE - Used to delete existing elements

What you will need to do is now assign a timestamp to the data you are collecting on the Photon. By doing so, when you fetch data back from Ubidots using GET, you can compare the timestamps and send data for timestamps that are missing. Otherwise, if you are just sending data elements, Unidots will just assign the current timestamp to them. Then, you are left with a harder task matching up the data values.

It would be good to begin sharing code. Do not share any of the api or variable keys :wink:.

Which Ubidots library are you using on the Photon?

6

--Yes, you are correct. Right now, basically, I have my setup just streaming/sending the temp and humidity points to the cloud (Particle) and then to Ubidots from Particle. You are also correct, I would like each point time-stamped - which it seems like it is already doing server side - just not locally on the Photon (which is probably necessary) - but then I have to consider the issue that server response from the Photon to posting on Ubidots will vary from the local time stamp created by the Photon...not even sure it matters though.

Here is what I use to configure pretty much everything - basically, i created a separate account over at Ubidots, and just had to enter my credentials, Particle takes care of the handshake...:I'm also using the Adafruit_DHT library

============start=========================

#include "Adafruit_DHT.h"
#include "Ubidots.h"

// DHT parameters
#define DHTPIN 5
#define DHTTYPE DHT11

// Ubidots parameters
#define UBIDOTS_TOKEN "IgW20KNicolaTeslado4jfSGl" (not actual code)
#define UBIDOTS_DATASOURCE_NAME "2Di Cloud Prototype"

int temperature;
int humidity;

DHT dht(DHTPIN, DHTTYPE);
Ubidots ubidots(UBIDOTS_TOKEN);

void setup() {
  ubidots.setDatasourceName(UBIDOTS_DATASOURCE_NAME);
  dht.begin();
}

void loop() {
  temperature = dht.getTempFarenheit();
  humidity = dht.getHumidity();

  Particle.publish("temperature", String(temperature), PRIVATE);
  Particle.publish("humidity", String(humidity), PRIVATE);

  ubidots.add("temperature", temperature);
  ubidots.add("humidity", humidity);
  ubidots.sendAll();

  delay(5000);
}

=================end==============================

To recap - here is what I am looking to do:

1- Have a Particle (WiFi and Cellular) - I have both log temperature and humidity via a DHT11 or DHT22 sensor 24/7.
2- Data is sent to Particle cloud, and also to Ubidots for GUI/alerts for temperature/humidity vaiables
3- The Particle will timestamp the readings, and store all points for up to 30 days.
4- If there is a loss in internet/cell connection, the logger continues to log data via battery backup (I have this taken care of too)
5- Once back online, the Particle, in conjunction with Particle cloud, "knows" which points were missing from the time the Particle went offline to when it returned. - Those points/readings will be uploaded and populated accordingly.

Is this possible relatively easily - I don't mind getting my hands dirty - just need very detailed links, processes, theory, etc. I still don't even quite know why I can't write to my SD - I have it wired right...just don't know what to do with the huge library (SDFAT) :slight_smile:

Thank you so much everyone - you are all so helpful and nice!

7

@EBBQ, doing 2 publishes in a row violates the one-publish-per-second limit and will cause problems.

Does this mean your devices are powered continuously and won't sleep? I do hope you realize that with data being sent every 5 seconds, on the Electron this will represent a LOT of data (and possible high cost). The Photon and Electron are very different when it comes to connectivity (wifi vs cellular) so you will need to take this into consideration.

At 5sec intervals, that is 17,280 readings per 24hrs!

What do you mean by "via battery backup"?

The Particle Cloud does not store any data (it is transactional only) and has no concept of ordered data so it "knows" nothing. To do this, you will need your own server which "knows" order or stores the data in a database which can be ordered by timestamp.

As for the SDFat library, I suggest you get it working with the examples provided in the library before adding you own stuff. :smiley:

8

^^ I'm sorry, I made a mistake there, and completely left out that part of it. No, not continuous...I would like the Particle to sleep, and then wake up once every 5 minutes to send data, then return to sleep. So, in essence, it would send a total of 288 data packets each 24 hour period (60min per/hr div. by 5min intervals =12 wake/sends per hour multiplied by 24 hours = 288) Additionally, overall (and part of the reason for potentially needing the SD card) I would like to store 30 days of these readings (the once-every-5-minute readings, not all readings)

Also, What I meant by "I have both..." I meant I have a Photon in cellular version and a Photon in WiFi version. I want to accomplish the same thing with both, just have one transmit via cellular and one transmit via WiFi.

As for continuous power, I am adding a battery to both...the battery will be the primary source of power. I have thought about a "DEL 121672 3.7V 1100mAh" battery - primarily because I have about 30 of them brand new from another project (with a Raspberry Pi).

Is there an existing commercial solution for this which would not require writing a bunch of custom code (re-inventing the wheel) because I know there are lots of companies already doing this...some have developed their own cloud/storage/dashboards and others seem to use things like Amazon.

^^ This also may be a game changer, because without a acceptable cloud, even if I get the photon to work the way I want it to, I may be hindered by no place to push and access the data...I'll worry about that later though :slight_smile: I'm already committed!!

I looked at the library. I see the "example.ino" file(s) however, there is literally no explanation...where do I put the example? What after I compile/flash? How do I actually make it to where data is stored at intervals requested on the SD drive? How would I know if it is being stored? I know you are not the SDFAT library help-desk, just throwing that out there...I have no problem trying to figure it out...but I am a little slower...i am a Google wizard, but sometimes there sin't much to go on.

Thank you so much everyone for helping me and assisting me - I am trying to meet a prototype deadline for someone...and I feel so far away from the goal!

9

The Electron is cellular and the Photon is WiFi. The Electron comes with its own 2000ma (required for cellular operation) battery whereas the Photon does not. How do you propose to manage and/or charge the battery? How long do you expect either device to operate with the battery?

I am not aware of a cloud solution which "fixes" your out of sync data. By storing a timestamp as part of your database, the reordering of data becomes easy. Regardless, you will need to research possible solutions and I suspect code writing will be involved.

It seems you have not taken the time to learn how to compile and flash to your Particle device. The SDFat library contains multiple examples including one called "TryMeFirst.cpp". I suggest you take the time to read the Particle getting started documentation and get familiar with the tools.

I sense you are rushing your project to meet your deadline but you do not have the either the knowledge or experience to implement it. Forum members won't provide you with a solution but they will help along the way.

3 Likes
10

@EBBQ, when exactly is your deadline?

1 Like
11

I understand your sentiment. I appreciate you taking the time to let me know I know nothing. IF I knew as much as you, I wouldn't be posting unless I was (A) helping someone else or (B) bored out of my mind and trying to kill time.

I don't know a great deal about compiling on this platform. I am very new to this particular type of programming. I have never had formal education in any of this, and despite that, do try to learn and figure out everything on my own. I have a lot of self-taught experience with a Raspberry Pi (kinda the first "maker" arena) and have a patent on a device I created using that.

I am under a deadline...otherwise, I probably would figure everything out on my own; I have actually never posted in a forum for help before. Yes, unbelievable, but also true. I work for a very small company...myself and another person. The company has fallen on hard times; we are attending a small trade show and want to show "proof of concept" for a design we will ultimately build. This is the important bridge between us potentially staying solvent with new customers and current investors.

I know you don't know me - I am nobody to you. I didn't mean to sound deserving of help without figuring things out...honestly, I am down to the wire because I tried to figure everything out without asking for help early on. As I said, I do have my Particle Photon logging temp and RH every five minutes, and pushing to particle cloud, and integrated with Ubidots. I know, to you this is child's play. Again, this is completely new to me...I actually hate asking for help, and I love figuring things out for myself. I am one of those people.

Now, I feel even more vulnerable, and bad...about the project and asking for help. I know full and well you have no obligation to help me or anyone else...in fact, no one has to help anyone. However, I always repay a good deed, and never forget kindness. You have absolutely confirmed your suspicions and theory that I don't fully know what I am doing.

I really appreciate you (and everyone else's) time in offering feedback so far. I really do. I know in situations especially like this - communities like makers/tinkerers and general gurus in all things off the beaten path seem to come together and make great things. You are all doing that. IoT is really exciting, and has so much potential. I think the Particle team and those like you involved in it are making it great.

Again, i appreciate your help. I wish you well, and thank you again for your assistance. I will keep trying to sort all of this out as soon as I possible can...this Friday is a soft deadline to at least have something
resembling a final/workable product going...I have had worse deadlines before...it just makes me stronger :slight_smile:

12

@EBBQ, I can assure you @peekay123 didn't intend to come across demeaning or humiliating as he one of the first to jump in when it comes to supporting others and even porting libraries for them when they get stuck.

As I understand it, it's more the concern whether you haven't taken too big a bite to swallow espevially with a tight schedule. With very little time left even a seasoned Particle dev might struggle to take this off you as contract work, due to lots of open questions - even more so when doing this with the extra delays of doing this via a forum.

We are happy to help, but we can't stop time and this seems what you are running out of - you're not running out of willing supporters here.

Hence Paul's last question

BTW, there is nothing sleazy in honestly asking for help in a forum, so doing this as soon you get the feeling you are getting stuck might safe you precious time.

1 Like
13

@EBBQ, first things first. When you is your deadline and how much programming experience in C++ do you have? Because of your deadline, you need to use a platform you are familiar with. What have you developed on already?

A proof of concept doesn’t need to be a finished product so you need to consider the following:

  1. WiFi and Cellular reception can be tricky/non-existent in show room environments (ie trade show). Have you considered this?

  2. Perhaps logging to a Google Spreadsheet is the fastest way to capture data without having to worry about time-based ordering. It would also be easy to do graphs and basic data analysis. @ScruffR, any thoughts on that?

  3. You need to verify that the TryMeFirst.ino example in the SDFat library is working.
    This will validate your wiring and get you familiar with the SDFat library.

  4. If you want battery operation, sending data every 5 seconds will not provide any battery savings. You will need to consider sending data every 5 minutes or more to gain the benefits of deep sleep.

The basics of logging are not complicated. You create/open a log file on the SD card which will accumulate your data. Then:

  • Sample your sensors

  • Convert and store your data (including timestamp) as it will be sent in the publish. That is, in ASCII format with a terminator like CRLF or a NULL (a zero value). Ideally, all records are the same length to keep things simple.

  • You will maintain two file pointers. One for the last record in the file and another for the last record sent. When wifi is working, both of these will be the same. When wifi drops, the last record pointer will “freeze” and the last record in file pointer will keep going up.

  • The two pointers will need to be kept as retained values when the Photon goes to deep sleep. This will work as long as power is maintained to the Photon.

  • Remember that when the Photon wakes from deep sleep, it will act as if the RESET button was pushed. That means your code needs to be aware of what it WAS doing the last time it ran. Thus the use of retained values.

  • Store the data record in the file, increment the “last record” pointer then test if wifi is connected. If not, go to sleep (thus not changing the last sent pointer). If wifi is connected, send the data starting from the “last sent” pointer to the “last record” pointer, incrementing the “last sent” pointer for each record sent. When wifi never drops, you will only be sending one record. When wifi has been out for a while, that may be multiple records.

This is an over-simplified explanation so you understand what, in broad strokes, needs to be done. I addition to this, you will need to synchronize the Photon’s onboard RTC in order to maintain accurate timestamps. You will also need to manage the 30 day capture window. An easy trick is to create a filename with the capture start date and after 30 days, close it and start a new file with a new date. As you may have guessed, the filename should be kept in retained memory.

What also remains is how you monitor the battery so you don’t over-discharge it and cause the Photon to lose power and all its retained values.

There is a lot to think about here. I’ll look to see if I can find a similar project in the Arduino world that could help. :wink:

2 Likes
14

Thank you, and I apologize for sounding self-loathing. Honestly, I probably did bite off more than I could chew…you think I’d learn my lesson on that over the years… :grimacing:

I appreciate you feedback…and also, as I mentioned, the feedback and responses form everyone. You are all awesome -and moving forward, I hope to continue learning and helping others eventually too!

I appreciate you taking the time to respond. :slight_smile:

1 Like
15

Honestly, I am more of a compiler. I know a little C++, a little .Js, a little PHP, seasones HTML/CSS etc. i come from the web-based/frontend side of things. I have successfully though taken parts of many different projects and integrated them into one...like restoring a car. As far as sitting down and writing from scratch, I am at a loss. :dizzy_face:

If we have bad reception, I planned on tethering to my mobile WiFi. I have confirmed a strong Cellular signal.

This sounds really interesting, and would probably suffice until a later date where we would develop/plan something more proprietary for end users.

I am taking a stab at it this morning

I must have made a typing mistake - I only want it to wake up and sample every 5 minutes. So, ideally, the Photon/sensors would be asleep (deep or plain) until 5 minute intervals - wakes up, sends/pushes a reading, time stamps it, stores it on SD at same time, and goes back to sleep. IF there happened to be no data sent (i.e. internet/cell outage) the next time it woke up, somehow it would know those points were never sent/received and send those along with the current one. Then go back to sleep.

Everything after this makes me dizzy...if that is any indication of where I am at. Honestly though, as for the long term considerations (battery monitoring/backup, etc.) I can take time to figure that all out, slowly - methodically, intelligently - without the haste and sense of dire urgency I have for this right now :slight_smile:

I was thinking (and everyone knows what happens when I try to think) - that there would exist several less-complex projects/code/libraries, and when combined may accomplish what I am trying to do - at least in the short-term.

You have no idea how much I appreciate your help and expert guidance. I realize I should have asked for help LONG before this point, and that the scope of this is way larger than I originally thought it would be. It is my fault - out of time, out of money, and desperate. I hate feeling like this - that whole "Necessity is the mother of invention" thing is waaayyy over-rated :slight_smile:

Anyway, enough rambling, I appreciate you - and others chiming in. Thank you - thank you- thank you!!!

16

Hello, my 2cents, if you need to send a fixed timestamp you can insert it as fourth parameter inside the add() method, this line should do that:

timestamp = Time.now(); // Gets the timestamp from Particle cloud
float value = 1.0;
ubidots.add("test-var", value, NULL, timestamp); // Sends a value with a custom timestamp```

Regards
1 Like
17

Thank you!! Where exactly would I add this? If it helps, here is my current project:

Capture.PNG1578×825 79.5 KB

18

I am actually wondering if I would even need the SD card - IF I only save missed points in the instance of an outage…and my rate of sample/wake is 5 minutes? If that is the case, i can stop trying to get the SD card and SDFAT library from working…is that doable?

19

Hello, you would add this to line 31:

unsigned long timestamp = Time.now(); // Gets the timestamp from Particle cloud

and at lines 32 and 33 you would add your timestamp in this way:

   ubidots.add("temperature", temperature, NULL, timestamp);
   ubidots.add("humidity", humidity, NULL, timestamp);

You can give a try using a date i.e from 2016, if you need to get a custom timestamp you can get it here. Remember that your timestamp should be in seconds and not in milliseconds, by default Time.now() from particle’s firmware returns a timestamp in seconds.

1 Like
20

Just a side note Time.local() would give you the timestamp including the Time.zone() setting (if you need that instead of UTC timestamps)

3 Likes