InfluxDB Webhooks Hell

I cannot get the webhooks to work after weeks of trying and reading forums and tutorials…

Using influx 1.8 on a local server with ports forwarded and updated conf file.

I have confirmed that the port forwarding of 1619 is working, I have repeatedly checked that the telegraph conf is right, and I have followed both varieties of tutorial (one seems to include additional curly brackets, the other misses quotation marks). Tutorial here: tutorial

I have tried as http https, yes/no on SSL, and setting a password. None work. Getting either a 400 or 401 error (depending on the http(s) use.

Other forum responses have stated to include influx_db, but nothing is working.

At the moment I am using the below as my JSON:

          
{
    "event": "{{{PARTICLE_EVENT_NAME}}}",
    "data": {{{PARTICLE_EVENT_VALUE}}},
    "coreid": "{{{PARTICLE_DEVICE_ID}}}",
    "published_at": "{{{PARTICLE_PUBLISHED_AT}}}",
    "userid": "{{{PRODUCT_USER_ID}}}",
    "fw_version": "{{{PRODUCT_VERSION}}}",
    "public": "{{{PARTICLE_EVENT_PUBLIC}}}",
    "measurement": "particleData",
    "influx_db": "test"
}

note: ‘test’ is a valid influx db I have created and successfully sent data to using UDP.

screenshots below:

I’ve tried all I can, I’m stuck!

1 Like

@ScruffR

Sorry to be a pain, but I am going around in circles. I am happy to share some details of the server via PM if that helps… I already have it working via UDP (also remotely via 3G through this method) so I know my port forwarding works (though UDP is a different port to 1619 recommended in webhooks, i have opened both).

Hi,
Here what works for me (to be input in “custom template”):

{
    "event": "your_event_name",
    "url": "http://influxdb.server:1619/particle",
    "requestType": "POST",
    "noDefaults": true,
    "rejectUnauthorized": false,
    "json": "{\n  \"event\": \"{{{PARTICLE_EVENT_NAME}}}\",\n  \"data\": {\"values\":{{{PARTICLE_EVENT_VALUE}}}},\n  \"coreid\": \"{{{PARTICLE_DEVICE_ID}}}\",\n  \"published_at\": \"{{{PARTICLE_PUBLISHED_AT}}}\",\n  \"influx_db\" : \"telegraf\"\n}"
}

Be sure your event message is formatted as follows:

{
"value_1":123,
"value_2":345,
...
}
1 Like

Thanks so much for this, but still getting the same error... hmm...

Tried also with ssl and a user/password. Still getting the 400 error code when I hit "test"

Was there anything else you tried on this? or is perhaps the "Test" button sending invalid data?

edit: Yes, I changed the influx_db and my URL the right DB and url.

If your influxdb server does not require SSL connexion then you should leave all fields below “JSON data” blank (except HTTP headers, see screenshot attached), and “Enforce SSL” set to “No”.

But I believe there is no issue about connection as 400 error indicates an incorrect request. If you have access to your server via command line, you can shut down Telegraf service and listen to port 1619, to see what it is exactly receiving when Webhooks sends the HTTP request, by using the following command (at least on AWS EC2 which I’m using):

nc -l -p 1619

Should help understand what is incorrect in the request.

2 Likes

Example of what I’m getting with this method:

1 Like

@irwige - would you mind creating a support request for this (support.particle.io)? I’m sure @mstanley (my team’s resident Cloud expert) will be able to help you out!

Thanks for the heads up on this. There was traffic coming through. I did it two ways.

  1. by manually starting the telegraf service and watching the log. The output when I hit “Test” in the webhook page gave me:
2020-07-22T09:56:26Z E! [inputs.webhooks] Error in plugin: invalid character 'e' in literal true (expecting 'r')
  1. I tried nc -l -p 1619 and got:
POST /particle HTTP/1.1
Content-Type: application/json
Accept: application/json
User-Agent: ParticleBot/1.1 (https://docs.particle.io/webhooks)
host: <my url>:1619
content-length: 147
Connection: keep-alive

{
  "event": "testName",
  "data": {"values":test-event},
  "coreid": "api",
  "published_at": "2020-07-22T09:58:52.060Z",
  "influx_db" : "test"
}

I submitted a ticket about this a while ago… no solution yet.

Ticket #93702

I responded to your ticket, it seems your JSON was not correctly formatted. Please test again. The test timed out for me, so I suspect the DB is down.

Thanks @no1089 but I have also tried this.

oK, UPDATE!

I got rid of the error (which was formatting in the JSON), btu now it is not going to my database called “test” but instead the values are appearing in “system_test”???

Any ideas? I have never created this database called “system_test”, it just seems to be a bucket for influx defaults?

Good news you have succeeded to push metrics!

I’m afraid I won’t be able to help on this issue, but maybe you should look at Telegraf config.

1 Like

Thanks again to all who have helped. I found this in my conf file… I have commented it out, but it looks to still be defaulting there. At least I have the particle side sorted. Looks like I need to move to the InfluxDB forums now.

Thanks guys, really appreciate it, especially @gazo, @no1089, and @marekparticle

Would it be helpful if i wrote a little tutorial for complete noobs to understand so this isnt an issue going forward?

The main issue initially was that I was relying on the “Test” button, but this sends a test to InfluxDB that isnt readable. From there it was a rabbit hole of thinking it was in the JSON, but it was only when i started publishing actual data that it smoothly went through.

2 Likes

Ok, full solution below, this has taken me a while to get through and to the devs… THE PARTICLE TO INFLUX TUTORIAL NEEDS TO BE UPDATED.

Step 1) Update your telegraf.conf

sudo nano /etc/telegraf/telegraf.conf

you need to change a few things here, they are:
Update the influxDB credentials. These weren’t current in my .conf.
(note 1, where <> brackets are used, you need to delete the and enter your own data)
(note 2, you need to un-comment the stuff you want to take effect by deleting the ‘#’)

Then, update the Particle Webhooks section (way down the bottom, or use ^W to search for “particle”:
Capture

Now, save ^X, then ‘Y’, then save as ‘telegraf.conf’

then restart the telegraf server:

sudo systemctl restart telegraf.service

Step 2) create the webhook:

copypasta:

{
    "event": "{{{PARTICLE_EVENT_NAME}}}",
    "data": {{{PARTICLE_EVENT_VALUE}}},
    "coreid": "{{{PARTICLE_DEVICE_ID}}}",
    "published_at": "{{{PARTICLE_PUBLISHED_AT}}}",
    "userid": "{{{PRODUCT_USER_ID}}}",
    "fw_version": "{{{PRODUCT VERSION}}}",
    "public": "{{{PARTICLE_EVENT_PUBLIC}}}"
}
  • no username and password needed
  • no SSL

Now, SAVE.
DO NOT use the “Test” it’s a waste of time and will throw an error, even if everything is fine. Also, yes, I had to omit the double quotes on data. It works this way, and smarter people than me can explain why.

Note, the “publishName” is the name of the Particle.publish you will need to use in your microcontroller code in the next step.

Note 2: “event” is the name that will be used for the ‘Measurement’ in InfluxDB. In the above it will use the same name as your particle publish function (in this case “publishName”). If you want to override this, replace the “{{{PARTICLE_EVENT_NAME}}}” with “your_Measurement_name”.

Note 3: the term “measurement” appears to be completely redundant.

Note 4: Dont put a comma , after the last item in the JSON or it will fail without any reasoning (I lost half a day on this!)

Step 3) Particle code (almost there!)

This bit took the life out of me, but I now have it working. You’ll want some tags, add as many or as little as you like, just keep following the format below (you can move the variables outside the loop() if you like).

void setup()
{
}

void loop()
{
  int value2Integer = 10;
  float value3Float = 1.234;

  String data = String::format("{ \"tags\" : {\"influx_db\": \"<databaseName>\", \"<tag1Name>\": \"<tag1Value>\", \"<tag2Name>\": \"<tag2Value>\"}, \"values\": {\"value1Name\": \"%s\", \"value2Name\": %d, \"value3Name\": %f}}", "value1String", value2Integer, value3Float);
  Particle.publish("publishName", data, PRIVATE);
  delay(1000);
}

note: you should include one of the tags as ‘influx_db’ as this is where Telegraf learns the name of the alternative InfluxDB to use. Otherwise it will use the default ‘telegraf’.

Also, you could make the ‘data’ payload a little neater to read in code by adding each section to a new line and appending it with ‘+=’ also… I’ll do that later and see how it goes.

edit: I have this working in the below code:

/////// CONSTANTS ///////
String IDB_Name = "my_database_name";
String IDB_tag1Name = "tag1Name";
String IDB_tag2Name = "tag2Name";
String IDB_tag3Name = "tag3Name";
String IDB_tag1Value = "tag1Value";
String IDB_tag2Value = "tag1Value";
String IDB_tag3Value = "tag1Value";

/////// VARIABLES ///////
String value1Value = String(10.5); // There is probably an easier way to do this?
String value2Value = String(11.5); // In your loop you can set these to your various data.
String value3Value = String(12.5);

void setup()
{
}

void loop()
{
  String payload = "{ \"tags\" : {\"influx_db\": \"";
  payload += IDB_Name;
  payload += "\", \"";
  payload += IDB_tag1Name;
  payload += "\": \"";
  payload += IDB_tag1Value;
  payload += "\", \"";
  payload += IDB_tag2Name;
  payload += "\": \"";
  payload += IDB_tag2Value;
  payload += "\", \"";
  payload += IDB_tag3Name;
  payload += "\": \"";
  payload += IDB_tag3Value;
  payload += "\"}, \"values\": {\"";
  payload += "value1Name\": ";
  payload += value1Value;
  payload += ", \"value2Name\": ";
  payload += value2Value;
  payload += ", \"value3Name\": ";
  payload += value3Value;
  payload += "}}";

  Particle.publish("publishName", payload, PRIVATE);
  delay(1000);
}

DONE!

2 Likes

This is a known issue with all Webhooks for which I have filed an issue ages ago

The Integrations Test-Button triggers a stupid generic request which will in 99.9% of cases not help testing the actual integration at all.

@marekparticle, back then @jvanier had commented on that issue (and a few months later on a related issue) but nothing happened since. I'd consider this a "semi-trivial" enhancement which could be integrated in less than a day but would be beneficial to lots of users and prevent staff from needing to address issues that could be found by the users themselves with it :wink:
A few hours spent well may save days of accumulated efforts to track down multiple issues "caused" by the same shortcoming.

2 Likes

Thanks @ScruffR. I’ve created an internal report about this; we will revisit this issue during the prioritization process for this coming quarter.

1 Like

I recently created a Webhook for InfluxData v2.0 and published the details to my website at:

http://www.savvysolutions.info/savvymicrocontrollersolutions/particle.php?article=particle-webhook-influxdata

1 Like