Multi-part webhook responses to wrong handler

[Bellatrix]

I have a webhook that returns four nws forecast chunks, the first three being 512 bytes and the fourth being something less. The first two chunks are being handled by the desired function. The last two are being handled by a function that has nothing to do with it. The two hooks have no prefixes in common.

This webhook I trigger every fifteen minutes:

{
    "name": "NWS-Forecast-RLW",
    "event": "NWS-Forecast-RLW",
    "url": "https://api.weather.gov/gridpoints/FSD/96,64/forecast?units=us",
    "requestType": "GET",
    "noDefaults": true,
    "rejectUnauthorized": true,
    "responseTemplate": "QUT{{properties.updateTime}}QQ{{properties.periods.0.name}}: {{properties.periods.0.detailedForecast}}QQ{{properties.periods.1.name}}: {{properties.periods.1.detailedForecast}}QQ{{properties.periods.2.name}}: {{properties.periods.2.detailedForecast}}QQ{{properties.periods.3.name}}: {{properties.periods.3.detailedForecast}}QQ{{properties.periods.4.name}}: {{properties.periods.4.detailedForecast}}QQ{{properties.periods.5.name}}: {{properties.periods.5.detailedForecast}}QQ{{properties.periods.6.name}}: {{properties.periods.6.detailedForecast}}QQ{{properties.periods.7.name}}: {{properties.periods.7.detailedForecast}}QQ{{properties.periods.8.name}}: {{properties.periods.8.detailedForecast}}QQ{{properties.periods.9.name}}: {{properties.periods.9.detailedForecast}}QQ{{properties.periods.10.name}}: {{properties.periods.10.detailedForecast}}QQ{{properties.periods.11.name}}: {{properties.periods.11.detailedForecast}}QQ{{properties.periods.12.name}}: {{properties.periods.12.detailedForecast}}QQ{{properties.periods.13.name}}: {{properties.periods.13.detailedForecast}}"
}

The second I receive whenever my argon publishes the completion of a HVAC cycle (~ 1 to 5 hours) and looks like this:

{
    "name": "ArgonRLW4_HVAC_EVENT",
    "event": "ArgonRLW4_HVAC_EVENT",
    "url": "https://api.thingspeak.com/update",
    "requestType": "POST",
    "noDefaults": true,
    "rejectUnauthorized": true,
    "responseTemplate": "{{PARTICLE_EVENT_VALUE}}",
    "form": {
        "field1": "{{EVT}}",
        "field2": "{{RT}}",
        "field3": "{{PD_AVG}}",
        "field4": "{{TITOD_AVG}}",
        "field5": "{{TITOD_MAX}}",
        "field6": "{{AT_MAX_MIN}}",
        "field7": "{{AH_MAX_MIN}}",
        "key": "{{key}}"
    }
}

If I only subscribe to the NWS-Forecast-RLW hook I get all four chunks of NWS forecast data cleanly and quite reliably. If I subscribe to the ArgonRLW4_HVAC_EVENT event at the same time, the first two NWS chunks are handled by the NWS-Forecast-RLW handler and the last two are sent to the ArgonRLW4_HVAC_EVENT handler. Note: the NWS-Forecast-RLW handler has quite a bit of parsing to make the forecast text as compact as possible. Has anyone seen this happen and if so were they able to fix it? Might it be that somehow the third and fourth chunks are received while the handler is still processing the prior chunks? Is it possible in such a situation the chunks get passed to another handler??
Thank You,
Bob

[rickkas7]

This isn't a known problem, however you do need to be careful when doing things from a subscription handler. Ideally you should make a copy of the data and return as quickly as possible.

There is a shared buffer used by subscribe and publish, so make sure you never publish from a subscribe handler.

When you publish your HVAC event, is that done from the loop thread, software timer, or worker thread? If it's anything other than the loop thread, it could be happening concurrently with your subscribe handler, which may possibly cause strange behavior, though I'm not positive about that.

[Bellatrix]

Hello Rick,
Thank you for your insight. The HVAC event comes from ArgonRLW4 that publishes HVAC run cycles. It also publishes periodic quiescent parameters every four minutes. PhotonRLW8 has a dot matrix display and displays info from the following events:

1. Outdoor Weather = PhotonRLW5 (TCP)
2. Indoor Climate = PhotonRLW2 (subscription)
3. Indoor Climate Alerts = PhotonRLW2 (subscription)
4. HVAC Run Events = ArgonRLW4 (subscription)
5. HVAC RT Array = ArgonRLW4 (triggered subscription)
6. NWS Forecast = NWS (triggered subscription)

Events 1 to 3 need a constant subscription. 4 and 5 are triggered in the main loop.

All of my publishes are done in loop() except for Particle.publish("particle/device/name") which is done once in setup(). Since I do have five subscriptions I change my subscriptions via a function that unsubscribes and then subscribes to the ones I need. Sort of goofy but there is no way of unsubscribing to just one event AFAIK. I thought I read somewhere that subscriptions should only be done in setup()? Yikes, don't know how I would get around that. I will try to tighten up the NWS handler function although I just append the chunks to a String until the last chunk is received and only then parse the String.
Thank You,
Bob

[rickkas7]

Subscribing and unsubscribing like that will probably cause issues. What is the reason you need to unsubscribe?

[Bellatrix]

I have five events. Three of them need to be constant. The other two subscriptons are triggered and I need to alternate between tjem to stay under the limit of four.

[rickkas7]

The limit isn't exactly 4. You should be able to subscribe to 5 on a Gen 3 or later device.

Also because the subscription is a prefix, a good technique is to have a single prefix for all of your events, and differentiate which from the rest of the event name from the subscription handler. This allows an essentially unlimited number of different events from a single subscription.

[Bellatrix]

Hello Rick,
Both solutions should work. In the short term I have eliminated the Indoor Climate Alert subscription and now have four fulltime subscriptions and everything works smoothly. Since I am running a regular Photon which has a limit of four subscriptions I will soon look into replacing it with a Photon 2 to get five. In future projects I want to look at the single handler solution.
A bit off topic but the conversion of my Outdoor Weather subscription ( four minute interval) to TCP has been working pretty reliably for a couple of weeks and will save 10,800 ops per month (I am currently stuck with the free 100K op limit and was exceeding it routinely). The jump in cost from free to the 720K ops per month plan at $299 is not feasible for me. By converting my Indoor Climate subscription to TCP I could save another 10,800 per month. I have read that subscriptions are more reliable than TCP so I am hesitant to expand until I have a few months of reliability data.
Thank you for your help and the good options you have given me.
Bob

[rickkas7]

Using a non-event system like TCP isn't recommended for commercial products, because of the difficulty of securing the connection due to a lack of TLS/SSL. It's fine for home use, however.

I use UDP multicast with an AES pre-shared key for transmitting sensor data in my house from Photon 1 and Argon devices over Wi-Fi. Most transmit every 30 seconds, though electrical usage updates every 10 seconds and instantaneous wind speed can be updated more than once per second if the speed or direction changes. It would be an insane number of events if transmitted that way.