Webhook rate limiting per hostname... across entire cloud?

Cloud
1

Hi folks!

I’m trying to help some folks debug a webhook on their account. They’re using a webhook to forward things to Thingsboard Cloud.

I set up a new webhook, nothing fancy, to a particular domain. I used the particle cli to subscribe to hook-sent, hook-response, hook-error for my devices, and to all the events for my devices. I published a single event from my device, and got

{"data":"Sleeping, too many errors, please wait and try again","ttl":60,"published_at":"2025-11-09T18:10:17.321Z","coreid":"particle-internal","name":"hook-error/HubData/0"}

Huh. I tried again a bit later, and now, looking at the integration in the dashboard, I see 1 success today, 0 errors, and 4 sleeps.

Screenshot 2025-11-09 at 12.19.31 PM

What’s going on here?

I read through the Webhook reference and the tutorial and the troubleshooting guide, and my best guess is this, from Limits:

When a server receiving webhook requests fails many times in rapid successions, the Particle Device Cloud will start to skip sending some events to lighten the load on the receiving server. Specifically, Particle uses an adaptive algorithm to skip webhook attempts when more 4xx or 5xx HTTP status codes than 2xx HTTP status codes are returned by the receiving server.

The server rating algorithm is per server hostname, and does not depend on the webhook that generated it, the account, or the port number being requested.

After a cooldown period of a few seconds without errors, requests will be allowed to be made once again. Events that were skipped will retried after 30 seconds and 1 minute before being dropped.

Is there someone else using the Cloud API who has an error-ridden integration to the same hostname that I’m trying to use? The times I see success are when it’s timed magically right to hit a gap when no one else has errors?

Are there options here? Can support check our webhook and isolate ours from other folks using the same domain? Are there details on exactly what goes into this key? Is it the full hostname, or just the base domain?

Outside of support, the first thing I can think of is to get a unique domain, keep it a secret, and use it to proxy to and from Thingsboard Cloud.

2

What is the hostname that you are sending to? We may be able to add that to the list of hosts or domains that do not aggregate failures across all users for determining whether to webhook sleep or not.

3

thingsboard.cloud

If you could, that would be great!

4

Hi, I’m interested to know if you managed to resolve this issue / what the solution was if you did. We are facing the same propblem. Less than 50% of the events sent from our instruments are successfully passed through. Sometimes the event triggers and success is on the first attempt, other times it takes all 3 attempts and many times totally fails. The uplink converter in ThingsBoard Cloud is pretty lightweight so no real long delays or liklihood of processing errors. I can see the data sent from our instruments is also looking perfectly good every time. Thanks in advance.

5

Yeah!

It did not seem like it was ever added to any allowlist, so I wrote a little tool on val.town to forward messages to Thingsboard. As soon as I did that, our traffic went back to looking perfect.

The tool isn’t quite “general purpose”, so let me know if you’d like some assistance doing something similar.

6

Hi, thanks for replying. I guess your solution was to use an intermediate ‘server’ to cache up the traffic, then meter it out to ThingsBoard with some type of scheduled re-tries until successful. I’ve not come across val.town - looks interesting. I’ll give it a try and hopefully see some improvements

7

That is exactly right. There is nothing special about val.town here. It doesn't even need to batch the traffic, it just needs to be a different hostname than thingsboard has and then relay the traffic.

8

The initial request on this slipped through the cracks, sorry about that. As of last Thursday, thingsboard.cloud should be whitelisted to be excluded from the hostname level protections we have for webhook targets.

I’m tossing a task into the queue to do some evaluation for some other platform/SAAS websites that don’t use unique hostnames per account. If folks stumble on this thread in the future when noticing something similar for a different hostname, please create a new thread and our folks will get make sure us devs can fix it up.

Let me know in this thread if you aren’t seeing this fixed for thingsboard.cloud.

2 Likes