Webhook to Elastic?

Cloud Integrations
1

Does anyone have experience - or a resource I could look at - to send a JSON payload to Elastic? I know this is an open-ended question but I am still too early in my learning cycle to ask it better.

Here is what I know:

  1. I use Webhooks to send data to Ubidots today.
  2. I would like to send a copy of that same JSON payload to another database
  3. That person is using Elastic and the ELK stack and has given me an account.
  4. I told them what I wanted to do but they simply gave me the URL for their server and added it “with the normal api access for elasticsearch”.

I went to this site for inspiration but, still do not know where to start.
https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started.html
Thanks,

Chip

2

Hey Chip!

I do not have experience with the Elastic Stack API, but I would start here if I were you.
Hope things are ok on your side.
Take care,
Gustavo.

1 Like
3

@gusgonnet,

Thank you for that pointer. I was able to get this working (though still a lot to learn) using the following steps:

  1. Get an Elastic account and the necessary permissions (these were provided to me). Here is the Elastic API reference which you will need to send data: https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html

  2. Define the JSON payload you intend to send but don’t send it yet).

  3. Add these two fields to the payload in the Particle Webhook Integration (saves you from sending this data from the Particle device over cellular):

  "timestamp": "{{PARTICLE_PUBLISHED_AT}}",
  "device": "{{PARTICLE_DEVICE_ID}}",

Refer to the Elastic API for mapping: https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html

  1. Take that payload and use the Elastic mapping API to define the data types for each data element. Mine looked like this. I used Postman to send this command and got a 201 response.
{
  "mappings": {
    "properties": {
	  "timestamp": {"type":"date"},
	  "device": {"type": "keyword"},
	  "Temperature": {"type": "float"},
	  "Humidity": {"type": "float"},
	  "LightLevel": {"type": "integer"},
	  "Soilmoisture1": {"type": "float"},
	  "Soilmoisture2": {"type": "float"},
	  "waterPressure": {"type": "integer"},
	  "Solenoid": {"type": "integer"},
	  "Battery": {"type": "float"},
	  "Resets": {"type": "integer"},
	  "Alerts": {"type": "integer"}
	}
  }
}
  1. Finish the Particle Webhook Integration and send your first data set. My JSON payload looked like this:
          
{
  "timestamp": "{{PARTICLE_PUBLISHED_AT}}",
  "device": "{{PARTICLE_DEVICE_ID}}",
  "Temperature": "{{Temperature}}",
  "Humidity": "{{Humidity}}",
  "LightLevel": "{{LightLevel}}",
  "Soilmoisture1": "{{Soilmoisture1}}",
  "Soilmoisture2": "{{Soilmoisture2}}",
  "waterPressure": "{{waterPressure}}",
  "Solenoid": "{{Solenoid}}",
  "Battery": "{{Battery}}",
  "Resets": "{{Resets}}",
  "Alerts": "{{alerts}}"
}

This creates an “index” in Elastic.

  1. In order to see your data you log into Kabana (the dashboard / visualization engine that is typically installed with Elastic) and create an index pattern. This will take the data you just sent via webhook and map it using the mapping API call you made. Be sure to identify your timestamp in this process. Once this is done, it is hard to change.

Here is the reference for Kibana: https://www.elastic.co/guide/en/kibana/7.6/index.html

The end result will be an “index” of your data which can be searched, visualized and made into dashboards. I am still working on the dashboard part but will share once done.

Still to do:

  • Figure out how to map the Particle Device ID to a more “human friendly” name - perhaps with some lookup table?

  • Figure out how to form a “response template” so I can validate that the data was received by elastic in my sketch.

  • Figure out how to “reindex” data if I add, modify or change the mappings in the future.

Any tips and tricks would be helpful and I hope this helps folks get started.

Thanks, Chip

4

Hey Chip, happy to hear you got this.

that was a smart move.

for the device id, is there a way you can run a query to the particle api at the time the message is received in the Elastic stack? This would be something like a Particle API call that gives you back the device name... I have done this in Firebase with a Cloud function, for instance.

Another alternative would be to fire an event from the particle device and receive then parse the device name (that the particle cloud knows but the device itself doesn't) maybe at boot time and store this info in the device itself and then send it via the publish event that triggers your current webhook.

Cheers,
Gustavo.

1 Like
5

Gustavo,

Thank you for that excellent suggestion on calling the Particle API for the device name. Will give this a try.

Thanks, Chip

6

this is the call I used:

GET https://api.particle.io/v1/products/PARTICLE_PRODUCT_ID/devices?deviceId=DEVICE_ID

You add to that request a header like:
'Authorization': Bearer PARTICLE_TOKEN

And you get a json back and from it I extracted devices[0].name

Gustavo.

1 Like