This is a follow up to my earlier question in regards to Particle.publish(eventName, data, WITH_ACK). With this topic I hope to get some insides into the design concepts and implementation details of Particle.publish(eventName, data, WITH_ACK).
In the answer from @rickkas7 and his notes on Particle.publish I learned that oneParticle.publish(eventName, data, WITH_ACK) can result in multiple triggers of the configured Webhook when some of the CAOP DATA or ACK messages are lost. This then leads to duplicated data on my server configured as target of the Webhook. I’m now working on implementing the de-duping when the data enters my server so that for the later stages of the server the data policy is exactly once and not at least once.
For this I would like to more deeply understand design concepts and implementation details of Particle.publish(eventName, data, WITH_ACK), so here come my questions (and some assumptions) I hope to get some insides on:
How many times does the Particle.publish(eventName, data, WITH_ACK) repeat the sending of the data before it gives up and returns false? I think I somewhere read 3 times can somebody confirm this? And does it depend on the device (electron, photon, …)? How long does it wait for the ACK before retrying?
Does Particle.publish(eventName, data, WITH_ACK), even if it returns false after the retries, still buffer the publish and trying to resend it later? For how long will it be stored/buffered? How many publishes will be stored/buffered? At what time are they resent and for how many times before the publish is dropped completely?
I apologize for the long/complicated questions, but some details would really help me implementing my server part.
@bittailor I am not really answering your questions - I am curious about designing a solution for an Electron having only used Photons to date.
Do you store event messages on your device until they have sent with a retry mechanism? I assume your retry mechanism is rate limiting.
Do your event messages become obsolete - if not why would you ever drop/delete them?
Apologies if I haven’t seen all the details of your server side solution, but why not just subscribe to the Particle Cloud event stream for the device(s). The Particle Cloud will forward all the events it receives and the device will continue trying until it knows the Particle Cloud has received the message?
The retry I was asking about is inside the Particle.publish(eventName, data, WITH_ACK) but I actually store the messages because my electron is mainly offline and just sends them out once every hour. I do not repeat the send if the Particle.publish(eventName, data, WITH_ACK) returns true.
They do not become obsolete and I do not want to drop them I just want to remove the duplicated messages.
Maybe server was a bit overdone its a hosting that supports PHP so I can consume webhooks but I have no possibility to spawn a long/ever running process that subscribes to the Particle Cloud.
Things are similar but not exactly the same for the Photon. The reason is that the Photon (and P1) use CoAP over TCP. Lost packets are retransmitted at the TCP layer, and if there is sufficient lost the connection is broken and reconnected. So it will behave a little differently. The notes below are for the Electron only.
How many times does the Particle.publish(eventName, data, WITH_ACK) repeat the sending of the data before it gives up and returns false?
Yes, 3 times, then it returns false. It takes about 20 seconds.
Though it actually tries at least once more after returning false.
Does Particle.publish(eventName, data, WITH_ACK), even if it returns false after the retries, still buffer the publish and trying to resend it later?
Yes. If a connection attempt is being made. The publish returns false, but at least one more attempt will be made to send. I've seen three of these outstanding buffered publishes, but you may see more or less. If you have more than one outstanding buffered publish, it may go out out-of-order as well, so beware of that.
You can solve the most common cases by always checking Particle.connected() before doing Particle.publish() though this won't solve all problem entirely. It will solve the out-of-order problem.
Would adding a data counter variable to the Electron help you ?
Then you would use NO_ACK, and send the counter # in the Publish.
Each publish has it’s own unique and consecutive data # in the Publish message.
Your Sever can use this to identify duplicates (likely wont happen with NO_ACK), and can quickly request a “re-publish” from the Electron when data is missing (Server sees a skipped number).
An added benefit is the data savings using NO_ACK Publishes.
My understanding is adding an unsigned long to every publish uses less data than ACK.
@Rftop interesting suggestion. Means I build my own “Reliability” or “Quality of Service” at the application level on top of Particle.publish(eventName, data, NO_ACK) instead of using the WITH_ACK. Will give it a try and see where I end up.