Spark.publish( ) throttling . . . something changed recently

Cloud
1

Hi:

Something changed in the last few days. I have a core that is pushing spark.publish( ) events that sometimes exceed 4 events per second (the so-called burst) and sometimes the events come even more dense. Typically, the overflow events would be ignored (fine), but there were enough data points/events to achieve what I was looking to do (psuedo-near-realtime events).

But, recently, the server (and I have tested this on my local server as well as the spark cloud) ignores many more messages. If there is a burst, the server is unresponsive for 4-5 seconds, and then picks up the .publish( ) events again. It is as if the throttling has been increased (even on my personal server).

The only thing that has changed is that I am at a new location with a new internet connection, but I don’t know how that would change anything. I have the .publish( ) events ‘shadowed’ by a Serial.print( ), so I know that data/activity is being read by the core as it should. Again, it’s as if the throttling has be increased.

Any ideas, suggestions. BTW, yes, would love to use websockets or UDP, but these options are currently buggy.

Jason

2

Let’s tag @dave and @BDub for this one, they seem to be the cloud/API guys.

3

@jgeist, there is an outstanding PR on the docs regarding this. It was posted by @kennethlimcp:

-NOTE: Currently, a device can publish at rate of about 1 event/sec, with bursts of up to 4 allowed in 1 second.
+NOTE: Currently, a device can publish at rate of about 1 event/sec, with bursts of up to 4 allowed in 1 second. Back to back burst of 4 messages will take 4 seconds to recover.

So your long bursts are causing the long delays you are seeing :smiley:

2 Likes
4

Hey All!

We haven’t changed any event limits in the past few days, we did roll out some new infrastructure, but I don’t think that would have caused what you’re seeing.

@peekay123 is right in that the firmware is very aggressive about throttling. If you exceed the limit, the firmware will (hard stop) drop all events until you fall back under the limit. Where the cloud will continue to let you hit 60 events / minute, and then let them through at that rate without a hard stop, etc.

Thanks,
David

1 Like
5

So, and I think the question has been proposed earlier, is it is possible to modify the firmware to allow more events through? Alas: probably not. But it is my server (and my firmwares) that I would like to modify here, so I am paying for the bandwidth.

Thanks for the followup!

Jason

6

Hi @jgeist,

It’s open source, if you’re running a local server, you can definitely change the rate limiting! We can also work out increasing limits for you on the main cloud as well depending on the case. Maybe @nexxy would be willing to help walk you through modifying the local cloud and the firmware to increase your limits?

Thanks,
David

7

I am happy to help if this sounds like the route you’d like to take. Another route to consider is buffering your events and publishing them in “batch” on a fixed timer.

Let me know if you’d like help with the firmware/server :smile:

1 Like
8

Hey nexxy,

I would be very interested in seeing what is possible, where should we begin? Thank you @Dave; thank you @nexxy!

jgeist

9

So, this is in SparkCore.js

onCoreSentEvent: function(msg, isPublic) {
            if (!msg) {
                logger.error("CORE EVENT - msg obj was empty?!");
                return;
            }
    
            //TODO: if the core is publishing messages too fast:
            //this.sendReply("EventSlowdown", msg.getId());

and a bit later, this

try {
            if (!global.publisher) {
                return;
            }

            if (!global.publisher.publish(isPublic, obj.name, obj.userid, obj.data, obj.ttl, obj.published_at, this.getHexCoreID())) {
                //this core is over its limit, and that message was not sent.
                this.sendReply("EventSlowdown", msg.getId());
            }
            else {
                this.sendReply("EventAck", msg.getId());
            }
        }
        catch (ex) {
            logger.error("onCoreSentEvent: failed writing to socket - " + ex);
        }
    },

What, where, when is “EventSlowdown” and can it be modified?

Cheers! :smiley:

I’ve edited your post to properly format the code. Please check out this post, so you know how to do this yourself in the future. Thanks in advance! ~Jordy

1 Like