Webhook rate limiting per hostname... across entire cloud?

[adamwolf]

Hi folks!

I’m trying to help some folks debug a webhook on their account. They’re using a webhook to forward things to Thingsboard Cloud.

I set up a new webhook, nothing fancy, to a particular domain. I used the particle cli to subscribe to hook-sent, hook-response, hook-error for my devices, and to all the events for my devices. I published a single event from my device, and got

{"data":"Sleeping, too many errors, please wait and try again","ttl":60,"published_at":"2025-11-09T18:10:17.321Z","coreid":"particle-internal","name":"hook-error/HubData/0"}

Huh. I tried again a bit later, and now, looking at the integration in the dashboard, I see 1 success today, 0 errors, and 4 sleeps.

What’s going on here?

I read through the Webhook reference and the tutorial and the troubleshooting guide, and my best guess is this, from Limits:

When a server receiving webhook requests fails many times in rapid successions, the Particle Device Cloud will start to skip sending some events to lighten the load on the receiving server. Specifically, Particle uses an adaptive algorithm to skip webhook attempts when more 4xx or 5xx HTTP status codes than 2xx HTTP status codes are returned by the receiving server.

The server rating algorithm is per server hostname, and does not depend on the webhook that generated it, the account, or the port number being requested.

After a cooldown period of a few seconds without errors, requests will be allowed to be made once again. Events that were skipped will retried after 30 seconds and 1 minute before being dropped.

Is there someone else using the Cloud API who has an error-ridden integration to the same hostname that I’m trying to use? The times I see success are when it’s timed magically right to hit a gap when no one else has errors?

Are there options here? Can support check our webhook and isolate ours from other folks using the same domain? Are there details on exactly what goes into this key? Is it the full hostname, or just the base domain?

Outside of support, the first thing I can think of is to get a unique domain, keep it a secret, and use it to proxy to and from Thingsboard Cloud.

[rickkas7]

What is the hostname that you are sending to? We may be able to add that to the list of hosts or domains that do not aggregate failures across all users for determining whether to webhook sleep or not.

[adamwolf]

thingsboard.cloud

If you could, that would be great!

[alanseeckts]

Hi, I’m interested to know if you managed to resolve this issue / what the solution was if you did. We are facing the same propblem. Less than 50% of the events sent from our instruments are successfully passed through. Sometimes the event triggers and success is on the first attempt, other times it takes all 3 attempts and many times totally fails. The uplink converter in ThingsBoard Cloud is pretty lightweight so no real long delays or liklihood of processing errors. I can see the data sent from our instruments is also looking perfectly good every time. Thanks in advance.

[adamwolf]

Yeah!

It did not seem like it was ever added to any allowlist, so I wrote a little tool on val.town to forward messages to Thingsboard. As soon as I did that, our traffic went back to looking perfect.

The tool isn’t quite “general purpose”, so let me know if you’d like some assistance doing something similar.

[alanseeckts]

Hi, thanks for replying. I guess your solution was to use an intermediate ‘server’ to cache up the traffic, then meter it out to ThingsBoard with some type of scheduled re-tries until successful. I’ve not come across val.town - looks interesting. I’ll give it a try and hopefully see some improvements

[adamwolf]

That is exactly right. There is nothing special about val.town here. It doesn't even need to batch the traffic, it just needs to be a different hostname than thingsboard has and then relay the traffic.

[Wraithan]

The initial request on this slipped through the cracks, sorry about that. As of last Thursday, thingsboard.cloud should be whitelisted to be excluded from the hostname level protections we have for webhook targets.

I’m tossing a task into the queue to do some evaluation for some other platform/SAAS websites that don’t use unique hostnames per account. If folks stumble on this thread in the future when noticing something similar for a different hostname, please create a new thread and our folks will get make sure us devs can fix it up.

Let me know in this thread if you aren’t seeing this fixed for thingsboard.cloud.

[Swsvy3]

Hi!

Just replying to say that I still have issues sending to thingsboard.cloud. using val.town as adam suggested works great. I have the issue with boron and photon2. The skipped messages seem to occur at random.

[JeffInCO]

I was noticing today that three hours went by where none of my 6 devices were able to send updates to thingsboard.cloud.

My integration history today for one device shows 92 successes, 216 skipped (zero errors). Other devices are similar.

[JeffInCO]

Are others still seeing this issue? It sounds like it's supposed to be fixed, but more than half of my web hooks to ThingsBoard.cloud are skipped!

My devices nominally send 3 events per hour (one event every 20 minutes), sometimes a bit more frequent (like 2 per minute for 10 minutes) if it looks like there is rapid change in the attribute that I'm measuring. In short, the rate is not high.

Any suggestions to help root-cause this? Thanks!

[rickkas7]

Thingsboard.cloud was allow-listed as a multi-tenant cloud service that does not penalize failures across multiple accounts on January 29, 2026.

If you're still having failures some more investigation will be required as it should no longer be the same issue that was reported earlier. What error is reported in your integration failure?

[JeffInCO]

No errors are reported, only successes and skips. See the screenshots below showing successes and skips for all of my integrations, then the 30-day history for one integration. (There were a couple of errors yesterday on a different integration, caused by my experiments, but no errors for the last 30 days besides those few).

About an hour ago, I created a cname record under my own domain that points to ThingsBoard.cloud, then changed two of my webhooks' URLs to target my cname record instead of thingsboard.cloud I also had to disable SSL for the two webhooks targeting my cname record, since obviously ThingsBoard's certificate validates their domain name, not mine.

So far, the two webhooks that target the cname record are looking better than the others that still target ThingsBoard.cloud directly.

I'll see how things look by tomorrow morning.

[rickkas7]

The skipped events are typically a different issue than was originally reported. Skips are usually caused by your device not successfully sending the event. When the cloud throttles a webhook because of external failures it shows up as Sleep in the integration history.

[JeffInCO]

It seems like there is confusion throughout this thread and in the Particle documentation about the difference (or connection between?) "skipped" and "sleeped". The very first post on the thread by @adamwolf shows a log message: {"data":"Sleeping, too many errors..."} message, but then he references the webhook troubleshooting documentation here that talks about skipped events being the grey portion of the bars in the history histogram. That documentation states:

"You will know events are being skipped if you see a hook-error event in your event stream that reads "Sleeping, too many errors, please wait and try again" in the body. You will also see grey bars in your History graph, identified as "skipped" events, corresponding to each event that has been throttled in this way.

Also, how would the webhook know that a "skipped" event occurred if my device did not successfully send it in the first place?

Regardless, here's my stats so far for today (UTC time as noted in particle's description info box). The two red-circled webhooks that have zero "skipped" (called "skipped" in the description info box) are the two that point to my CNAME DNS record. The rest, where there are many skips, sleeps, or whatever they are, point directly to thingsboard.cloud

[JeffInCO]

One other bit of info...

My investigation in this matter began because there were significant segments of time where my data did not make it to thingsboard. As I noted in my first posting, there was a three hour gap in my data from all devices that was completely missing at thingsboard.

What I'm seeing over the last 12 hours is that data seems to have mostly made it to thingsboard for all webhooks, and 100% made it for the two CNAME webhooks, but the non-CNAME datapoints sometimes are delayed by one minute, so maybe due to the up to two retries mentioned in the info box?

BTW, I have a webhook per device. I know this is not ideal (at least it's a pain from a management perspective), but I don't think it matters for the purpose of this issue. It also makes it clear that the integration was triggered by a publish from a particular device at the expected times.

And finally, thanks @rickkas7 for your help!

[JeffInCO]

Over a week has now passed with two of my integrations pointing to "thingsboard.redacted_domain.com", which is a cname record that redirects to thingsboard.cloud. The remainder of my integrations are still pointing to thingsboard.cloud

I have had zero skips in 8 days on the two integrations pointing to the cname record, and numerous skips on the other integrations (business as usual). The two integrations pointing to the cname record necessarily have "enforce SSL" disabled. One of the other integrations also has "enforce SSL" disabled (the first below that shows recent skips), the remainder have enforce SSL enabled though I'm not sure whether it matters since their URL begins with http:// rather than https://

I don't see how this could be a problem with my devices, nor a problem with thingsboard.cloud. It seems clear to me that there is still some rate limiting going by particle when handling webhooks. I guess I now have a workaround with the cname record, but this is less than ideal.

History for all webhooks are below, the first two are those that have pointed to the CNAME record for the last 8 days, showing zero skips. The remainder point directly to thingsboard.cloud. I also have a screenshot of the dig command for the cname record.

[rickkas7]

That does seem like the change to allow list thingsboard.cloud is no longer working. I'll ask to have checking that added to the engineering investigation list.

[JeffInCO]

Thanks, @rickkas7 !

[Swsvy3]

Sorry to bother, but I want to see if there have been any updates on the allow list. Thanks!