Particle internal error on publish/subscribe

I have several units out in the field that have reported failures when using the access control system designed around the Particle infrastructure. This began after the cloud outage a couple of weeks ago and happens randomly.

All of the units appear online, appear to publish their data correctly through the console, but fails somewhere internally with Particle as is shown here. Every one that fails, when a publish is sent, this internal Particle message appears too.

The only way I have found to make sure my customer is reliably up and running is to replace the photon or boron on site, this is a big expense to my company in travel to each site, lost time that my techs could be doing other things, and in replacement parts that shouldn’t need to be replaced.

Any idea what may be causing this?

@rickkas7 any ideas here?

The 404 in the first line of the event log indicates that the server the webhook is sending to is returning a 404 (not found) error.

This is causing the second problem, which is the excessive number of 404 errors is causing the webhook server to skip sending requests, because of an excessive number of errors (last line, sleeping) to avoid overloading the server.

So you should check the server logs for your server or service the webhooks are going to to see if there’s any indication of why. Usually 404s are because of a bad URL path.

That brings in a whole slew of new issues. I’m not using any external webhooks, strictly the publish and subscribe functions.

Here is one sides firmware, it has the wiegand device and sends the wiegand data out:
https://go.particle.io/shared_apps/5f22f042e83d02001645aee9

This is the other side writing to the controller and verifying the data:
https://go.particle.io/shared_apps/5e46e809022da40007668988

Are you sure you don’t have a webhook configured that is picking up that event? Remember the event name is a prefix so any event whose name begins with that string will trigger the webhook.

I’ve only got one webhook registered at all, I’ll double check but I’m pretty sure it doesn’t use that event name just for this reason.

I checked, I don’t have a webhook with this event name or even close.