InfluxDB/Telegraf Webhook JSON error


#1

I am getting an error from Telegraf when I send information from Particle through the webhook. I publish the following in my Photon’s firmware:

int c=10;
String data=String::format("{ \"tags\" : {\"id\": \"%s\", \"location\": \"%s\"}, \"values\": {\"capacity\": %d}}", "test", "myLoc", c);
Particle.publish("capacity", data, PRIVATE);

This publishes a valid JSON. In the console, I see the photon publishing the following:

{"data":"{ \"tags\" : {\"id\": \"test\", \"location\": \"myLoc\"}, \"values\": {\"capacity\": 10}}","ttl":60,"published_at":"2018-06-21T03:15:15.834Z","coreid":"3c0038000c47363433353735","name":"capacity"}

This is the JSON type I have set up for the webhook:

{
  "event": "{{{PARTICLE_EVENT_NAME}}}",
  "data": "{{{PARTICLE_EVENT_VALUE}}}",
  "coreid": "{{{PARTICLE_DEVICE_ID}}}",
  "published_at": "{{{PARTICLE_PUBLISHED_AT}}}",
  "measurement": "influxdata_sensors"
}

However, when I run ‘systemctl status telegraf’ in the terminal of my AWS instance, I see an error: “json: cannot unmarshal string into Go struct field event.data of type particle.data”

I’m pretty sure I followed all the necessary steps. I’m not sure why telegraf cannot parse the json that is sent to it. I took a look at the telegraf.go code on github. The event struct and data struct look like this:

type event struct {
	Name        string `json:"event"`
	Data        data   `json:"data"`
	TTL         int    `json:"ttl"`
	PublishedAt string `json:"published_at"`
	Database    string `json:"measurement"`
}

type data struct {
	Tags   map[string]string      `json:"tags"`
	Fields map[string]interface{} `json:"values"`
}

I think telegraf can’t parse the data tag in the json from a string (from the photon) to a map in the data struct.

Please let me know what I can do to fix this. Thanks!


#2

You need to escape embedded double quotes in strings (which wasn’t shown before reformatting).
I’ve reformatted your code to make the escaped symbols show up correctly.

See here on how to do that


#3

ScruffR,
I did escape the embedded double quotes in the string I’m publishing. Anyway, it still gives the same error even after I put in the reformatted code you provided.

Which JSON is being sent to Telegraf? Is it the JSON in the form of the string I’m publishing in the firmware:

{"data":"{ \"tags\" : {\"id\": \"test\", \"location\": \"myLoc\"}, \"values\": {\"capacity\": 10}}","ttl":60,"published_at":"2018-06-21T03:15:15.834Z","coreid":"3c0038000c47363433353735","name":"capacity"}

or is it the JSON I configured in the Particle webpage for webhooks:

{
  "event": "{{{PARTICLE_EVENT_NAME}}}",
  "data": "{{{PARTICLE_EVENT_VALUE}}}",
  "coreid": "{{{PARTICLE_DEVICE_ID}}}",
  "published_at": "{{{PARTICLE_PUBLISHED_AT}}}",
  "measurement": "influxdata_sensors"
}

#4

Seeing the same problem here.

@pique did you ever try the test button in the particle console? That is failing for me, so I’ve not actually tried generating data on particle device yet.


#5

I’d not rely on the result of the TEST button since it only sends a “blank” request to the server which would be refused by most hosts.


#6

added some instrumentation to telegraf and this is what is coming in:

{"measurement":"templogger","data":"{\"tags\":{\"id\":\"bec\"},\"values\":{\"temp\":77.54000092,\"hum\":54.60058594}}"}

So data is looking like a string instead of a json object, hence the warning:

json: cannot unmarshal string into Go struct field event.data of type particle.data.

Its not obvious to me how to instruct the particle cloud to send JSON that looks like:

{"measurement":"templogger","data":{"tags":{"id":"bec"},"values":{"temp":77.54000092,"hum":54.60058594}}"}

#7

If I print out what the device is sending, I see:

{"tags":{"id":"bec"},"values":{"temp":77.5039978,"hum":52.69042969}}

So not sure where the escaping of " characters is happening.

It appears that data is always sent as a string (Publish/Subscribe and JSON data in events), so its puzzling to me how the influxdata telegraf webhook can ever work. Did something change, or am I missing something?


#8

cbrake,

Yeah, I still haven’t been able to figure it out.

You’re right–I think it has to do with the data being sent as a String, not a JSON object that can be parsed correctly. Hopefully we can get some advice soon.


#9

There might be a misconception with escaped characters.
When you construct a string (in C/C++, JAVA, …) that contains double quotes you just write "...\"..." to instruct the compiler not to treat the nested double quote as string delimiter, but inject the following character as-is. However, the backslash (\) never becomes part of the created string.
Consequently the receiving end also does not see that backslash in the string, but depending on implementation it might display the string with the escape character (/) in place or not, that’s up to the developer’s intent (display raw string vs. display as it would be written in code).

What may pose a problem tho’ is when you are sending or seeing mixed strings like the one you posted above

Some double quotes are escaped, others aren’t - that’s inconsistent and hence is supposed to fail.

To pinpoint the origin of the problem you may need to show the code that constructs and publishes the string and also how your webhook is declared. Only the combination of that may reveal where the problem gets introduced.
One thing to make sure with a JSON payload is that your webhook should most likely be set to
Request Format: JSON and Advanced Settings -> JSON DATA: Custom in order to properly unpack the event data and forward to the server


#10

ScruffR,

It seems like I am making the same mistake as cbrake. The actual JSON that is being published has a mix of escaped quotes and regular quotes. As you said, some double quotes are escaped and others aren’t, causing the JSON to not be unmarshaled:

{"data":"{ \"tags\" : {\"id\": \"test\", \"location\": \"myLoc\"}, \"values\": {\"capacity\": 10}}","ttl":60,"published_at":"2018-06-26T22:36:24.431Z","coreid":"3c0038000c47363433353735","name":"capacity"}

This is how my current Custom webhook configuration is set up:

{
  "event": "{{{PARTICLE_EVENT_NAME}}}",
  "data": "{{{PARTICLE_EVENT_VALUE}}}",
  "coreid": "{{{PARTICLE_DEVICE_ID}}}",
  "published_at": "{{{PARTICLE_PUBLISHED_AT}}}",
  "measurement": "influxdata_sensors"
}

How should I change this to unpack the event data correctly?


#11

In a custom JSON you should use the nested fields of {{{PARTICLE_EVENT_VALUE}}} to create your own custom JSON.
Currently there seems to be an issue with webhook management, so I can’t setup a test webhook to manufacture a showcase.