Some published events are being trimmed

Hi,

we have recently noticed that some events published from our devices are being trimmed. We don’t know if the issue is at the Particle broker level or with the webhook.

The devices are running ParticleOS v1.0.0, which support event payload of 622 bytes, but we noticed that the events that are malformed, get trimmed at 255 bytes.

The payload that we send from our devices is present in the webhook data, the part that gets trimmed are normally the attribute fields.

Is this a known bug? If so, is there a workaround for this (without and end-to-end acknowledged chain)?

Thank you,
Nelson

Can you post a particle serial inspect log from one of these devices?
Can you also provide the code that performs the event publish and the webhook definition?

Hi @ScruffR,

this is how the integration looks like (I removed the logs and the real GCP project name):

{
	"integration": {
		"id": "5c18e501087f140720addd97",
		"event": "infarm/status",
		"created_at": "2018-12-18T12:16:01.611Z",
		"responseTopic": "{{PARTICLE_DEVICE_ID}}/hook-response/{{PARTICLE_EVENT_NAME}}",
		"product_id": 7460,
		"logs": [{}],
		"counters": [{
			"date": "20190502",
			"sleep": "2",
			"success": "7822",
			"error": "97"
		}, {
			"date": "20190503",
			"success": "41216"
		}, {
			"date": "20190504",
			"success": "37941"
		}, {
			"date": "20190505",
			"success": "36990"
		}, {
			"date": "20190506",
			"success": "35184"
		}, {
			"date": "20190507",
			"success": "12819",
			"error": "1"
		}],
		"integration_type": "GoogleCloudPubSub",
		"topic": "projects/<GCP_PROLECT_NAME>/topics/status"
	}
}

and the output of particle serial inspect:

~ » particle serial inspect
Platform: 6 - Photon
Modules
  Bootloader module #0 - version 201, main location, 16384 bytes max size
    Integrity: PASS
    Address Range: PASS
    Platform: PASS
    Dependencies: PASS
  System module #1 - version 1002, main location, 262144 bytes max size
    Integrity: PASS
    Address Range: PASS
    Platform: PASS
    Dependencies: PASS
      System module #2 - version 204
  System module #2 - version 1002, main location, 262144 bytes max size
    Integrity: PASS
    Address Range: PASS
    Platform: PASS
    Dependencies: PASS
      System module #1 - version 1002
      Bootloader module #0 - version 7
  User module #1 - version 3, main location, 131072 bytes max size
    UUID: B3A6DCD529BE70FBE24EE959C7305D58E3BC9A04ACDCE8F093C7A1F759097D09
    Integrity: PASS
    Address Range: PASS
    Platform: PASS
    Dependencies: PASS
      System module #2 - version 6
  empty - factory location, 131072 bytes max size

the code that publishes events (is really just the Particle.publish):

bool fb::controller::send_event(const char *_name, const char *_body, bool _private) {
    return Particle.publish(_name, _body, _private ? PRIVATE : PUBLIC);
}

Thank you.

cheers,
nelson

Where is the actual Particle.publish()? Are you sure that your event char array is large enough?

Apologies - just saw the send_event function with the Particle.publish()

yes, the array that holds the events is big enough.

as I mentioned before, the event payload itself is not affected, mostly the attributes are trimmed from the webhook body.

cheers,
nelson

And the formatting of the data into the char array? Are you calling this function/method every time you send an event directly or is there something else between creating the event and publishing like a buffer?

I think you are missing the point here. There's nothing wrong with the event payload:

The payload that we send from our devices is present in the webhook data, the part that gets trimmed is normally the attribute fields.

cheers,
nelson

I'm not quite sure how to interpret that.
So the outgoing event that you Particle.publish() shows up completely in the console, right?
What I'm not entirely sure is then, are you missing the expected data at your receiving server or is the response from the server which should be going back to your device clipped?

It can be that webhook responses get chopped into multiple parts and need to be "stitched" together in the subscription handler.

yes, the payload of the event shows up completely on the console and on the webhook body. only the attributes are clipped.

example of an event body received by our backend event handler (omitting the actual event payload):

{
  "name": "<ORG>/observation",
  "data": "<FULL_EVENT_PAYLOAD>",
  "ttl": 60,
  "published_at": "2019-05-07T10:22:58.329Z",
  "coreid": "

as you can see, the coreid and the remaining attributes were clipped.

sorry, but I’m not sure how can I make this issue more explicit, please let me know if you need any more info to understand this better.

cheers,
nelson

I see now, this doesn’t seem to be an issue with the device OS but rather with the Particle cloud side of things.
@rickkas7 can you chime in as this is something inaccessable to us mere mortals :wink: