PublishQueuePosixRK with confirm from end service

I want to use PublishQueuePosixRK, but where PublishQueuePosixRK delete on acknowledge, I have a requirement to only delete uploads on confirm back from the end service.

It seems I need to modify these lines starting at line 83: BackgroundPublishRK/BackgroundPublishRK.cpp at main · rickkas7/BackgroundPublishRK · GitHub

        auto ok = Particle.publish(event_name, event_data, event_flags);

        // then wait for publish to complete
        while(!ok.isDone() && state != BACKGROUND_PUBLISH_STOP)
        {
            // yield to rest of system while we wait
            delay(1);
        }

To only OK, when there is a confirm back from a subscribe to the end service reply like:

	Particle.subscribe(String("hook-response/upLoad_" + System.deviceID()), uploadResponseHandler, MY_DEVICES);
    Particle.subscribe(String("hook-error/upLoad_" + System.deviceID()), uploadErrorHandler, MY_DEVICES);

As BackgroundPublishRK runs in a separate thread, can I Particle.subscribe in the main app thread and there modify a variable to be checked in the BackgroundPublishRK thread above? Does that variable need to be declared/protected in a special way?

You’ll probably want to make sure you have a unique identifier in your event payload. Because events can be out-of-order, delayed, or duplicated, you need to make sure the ACK you get back is for the event you think it is. An integer serial number that increases on publish is probably sufficient.

I don’t think you’ll have any special threading issues because you’re probably only passing an integer or boolean between threads. Those are basically atomic in this scenario.

Thanks, good point. Just started testing this in it’s original form. Looks neat!

I will probably try to change the 30 sec retry to 5 minutes, now that data operations count a lot, and a single outage can set off a storm of updates.

1 Like

It seem to be running really well so far, and I am wondering about two things.

  1. Do PublishQueuePosix::clearQueues() clear the actual files?

  2. Maximum number of files. According to AN035 File System | Datasheets | Particle the 2MB filesystem should be used with max 200 files, each with 1 sector for meta data + 1 sector content, and each sector is 4096 Bytes. So as long as each publish is less than 4KB data, I can also get away with 200 files?

1 Like

On point 1. I found it here GitHub - rickkas7/SequentialFileRK: Library for managing sequentially numbered files on the flash file system on Particle Gen 3 devices and the answer is yes the whole directory with files is deleted.

On point 2. I am still not sure I understand AN035 File System | Datasheets | Particle regarding file size. Is it correctly understood that one file is minimum 8kB (two 4K sectors), even if the file only carries one byte of actual data?

Please help me understand this. Thanks.

1 Like

I guess the original question is answered, so I will put the remaining issue in a new topic and set up a support case.

@thrmttnw,

I am also using the PiblishSyncPOSIX and I have observed something I wanted to run by you.

Some of my devices, especially the ones with less than stellar cellular coverage, will end up with a missed data report. I publish my web hooks before I connect to cellular and then connect if it makes sense to. For this reason, all my web hooks are published one the Boron connects and PublishQueuePOSIX clears its cache of messages.

Occasionally, it seems that these messages do not get through, and in looking at my logs, I see three scenarios:

  1. The Boron publishes, but it does not make it to Particle. I can see this happen occasionally in my logs where I can see the device move from the report state to the response wait state without seeing the webhook in the console log. Perhaps the webhook was sent before the data connection was fully established.

  2. The Web hook is in the console log but the device resets because there is no response from the back-end service. When I look at the back-end service (Ubidots) I do not see the “dot” for that report.

  3. The Web hook is in the console and a “dot” was created by Ubidots but the response code (HTTP code 201) was not received by the Ubidots handler in my code so the device counts this as an error.

By breaking it down like this, I thought perhaps some approach could be developed for each case. Here is what I was thinking - please let me know if this makes sense:

For the first case: Perhaps I should turn on WITH_ACK. I looked at the documentation and it is not clear what this will do for sending cached publish events. Do you have any sense of this? Do you use this flag?

For the second case, it seems like your idea of a serialized response would make the most sense. In this approach, if a response was not received in a 6 second window (I believe this is Particle’s window for responses) the webhook could be resent.

For the third case, I thought it might be possible for the Boron to send a webhook to the Ubidots API to get the “last activity” for the device which would do the same thing as the 201 code to validate receipt.

lastActivity only for Get All Variables by Device Endpoint

Please note: The query parameter lastActivity is only available for the endpoint
Get All Variables by Device
https://industrial.api.ubidots.com/api/v2.0/devices/<device_key>/variables/

Am I thinking about this right? If you are doing any of these today, any advice would be appreciated.

Thanks,

Chip