Cloud local storage - 100% fail-safe delivery from sensor to Particle cloud, to storage cloud

@will
I put a lot of effort in developing a sensor (prototype at this stage) so that I have no blocking measurements (with FreeRTOS threads and queues). So when there is no cloud connection, the measurements get bufferend on the sensor itself (and also timestamped by a battery backed-up RTC).

But when the cloud connection is back, data gets pushed (by calling a function so that values are delivered with the correct timestamp). But now I run into the next problem. I can have a trigger when the functions called but what if in the cloud

  • The Webhooks fails
  • There is no connection to the storage service
  • There is a problem in the Particle cloud

Is there a way (not that I see in any documentation at this point in time) to have that data stored in the Particle cloud until it is confirmed the data got processed by the web service at the other hand of the Webhook? Or will there be in the future?

It is nice to have real-time values, but we need the full chain from sensor to end cloud to have a 100% delivery guarantee (even if it is delivered later).

In our case we want to store it in AWS and have afterwards data analysis and in a second stage real-time forecasts on it so we can send commands back to the sensor or the customer.

But to achieve a 100% fail-safe chain we must have the ability to have (temporary) storage in the particle cloud.

How should this be addressed?

Thank you for any reply on this important topic to us. This is actually for us the last hurdle to take in being able to use Particle in an industrial production ready set-up (first one was to have non-blocking measurements and buffering in the device).

Webhooks do expect a response from the target server and that response would be forwarded to you via a dediacted hook-response/... message your device can/should subscribe to.
If you don’t get the expected result back the chain is broken somewhere and a resend may be in order.

@ScruffR OK, just to be clear… unless I missed something What Device OS API call do I need to use for that subscription? Where do I get an ID of the transaction back?

Of must all be handled by the return code of the function called by the device.

If this is arranged we can use our device in production mode :wink:

[Particle.subscribe()]
When you want to subscribe to any hook-response you’d write this in setup()

  Particle.subscribe("hook-response/", responseHandler, PRIVATE);

and in order to deal with the data you will be sent you’d write your own handler along this line

void responseHandler(const char* event, const char* data) {
  // look at "event" to know which kind of response that was  
  // look at the "data" your server sent back and what you told the webhook to pass on to you
}

One way to insure delivery all of the way to your data store is to use the SSE event stream and an explicit end-to-end acknowledgement. This should cover all of the cases where the data could be lost by keeping the data on-device until it receives a message that it has been safely stored in the cloud.

It’s the 13 - Cloud Datastore guaranteed delivery example in the repository below, but the technique can be used with any data store mechanism, it doesn’t have to be Google cloud datastore.

@ScruffR ok, going to look into this. First need a closer look and test it. But I get what you are telling.

@rickkas7 thank you for the reply. I am going to check this out and test it