Bridge from Particle Cloud to MQTT (likely using webhooks)

General
1

I am a big fan of using MQTT as an agnostic data gathering platform. I also believe that the topic hierarchy architecture is crucial to large projects with thousands of deployed units. In the past, I accomplished this with Particle by directly using the MQTT lib on a firmware level. However, with the Gen 3 products, this is no longer possible in a simple way, as mesh connected devices do not have the ability to access TCP.

While we could do a workaround of first sending all data to the gateway, and then forwarding from there to the MQTT broker (c.f. Conceptual Difficulties), this has several disadvantages: code complexity, multiple codebases, dedicated gateway hardware, etc…

Particle.publish(), on the other hand, works perfectly in Gen 3. The devices do not know or care if they are mesh’ed or gateway’ed units. So we can get all the data out of the mesh and up to Particle Cloud with trivial effort, perfect!

Once it’s there, we now have to move it on to the MQTT broker. I have done this in a simple way with webhooks via shitr.io (working code given below), but I now finding myself needing to have a bit of granularity.

Firmware:

// setup() runs once, when the device is first turned on.
void setup()
{
  // Put initialization like pinMode and begin functions here.
}

// loop() runs over and over again, as quickly as it can execute.
void loop()
{
  // Get some data
  String owner = "foo/bar/";
  String topic = owner + "kitchen/fridge";
  float temperature = 20.0f + ((rand() % 100) - 50)/10.0f;
  float co2_ppm = 400 + ((rand() % 10) - 50)/10.0f;
  String data = String::format(
      "{ \"_t\": \"%s\", \"temperature\": %f , \"co2\": %f }",
      topic.c_str(), temperature, co2_ppm);

  // Trigger the integration
  Particle.publish("/", data, PRIVATE);

  // Wait 5 seconds
  delay(5000);
}

Example output published to Particle Cloud:

{
"_t":"foo/bar/kitchen/fridge"
"temperature":18.4
"co2":413.3
}

webhook:

{
    "event": "/",
    "url": "https://try:try@broker.shiftr.io/{{{_t}}}",
    "requestType": "POST",
    "noDefaults": false,
    "rejectUnauthorized": true,
    "body": "{{{co2}}}"
}

Note that we look for the very simple and low bandwidth character / to trigger the hook. Then we use the _t JSON field as the entire base topic hierarchy.

Now, I would like to publish to two final topic hierarchies:

  • foo/bar/kitchen/fridge/temperature
  • foo/bar/kitchen/fridge/co2

But I only have one webhook. In the above example it publishes only co2. The temperature field gets ignored.

It would be madness to make a separate webhook for each topic, as I will have dozens of distinct topics, so what I really need to do is have a teensy bit of intelligence in the webhook. What I would like is that for each JSON field in the output, the webhook triggers an HTTP POST call with the base topic + individual topic (as well as the data in that topic’s field).

I’m coming up blanks for how to do this in the Particle Ecosystem, and all I can think to do is add in a Heroku layer which receives the data from the Particle Cloud and then properly formats it for the MQTT bridge-- which is ugly, costly, and increases complexity by another node.

Anyone have insight into this?

2

I think there is some inconsistency.
The code uses "temp" but you state you see "temperature" in console - I doubt that.
Hence when you expect "temperature" as part of your topic I’d not see how that would get there.

You won’t need multiple webhooks but may need multiple events to target different URLs.

e.g. like these

{
  "_t":"foo/bar/kitchen/fridge"
  "item" :"temperature",
  "value": 18.4
}

and

{
  "_t":"foo/bar/kitchen/fridge"
  "item" :"co2",
  "value": 413.3
}

which then hit a generalised webhook like

{
    "event": "/",
    "url": "https://try:try@broker.shiftr.io/{{{_t}}}/{{{item}}}",
    "requestType": "POST",
    "noDefaults": false,
    "rejectUnauthorized": true,
    "body": "{{{value}}}"
}

AFAIK there is no way one event would trigger multiple requests.

3

@ScruffR, ah, you know that when transferring working code to a minimal example there will be tiny inconsistencies.

I think the forest has might be being missed for the trees. Putting aside the tiny conversion error, what I want to do doesn’t seem to have a way to be done without adding some if conditionals. This makes sense, there’s no way for a simple webhook to go into granular detail for a bespoke hierarchy. The question is whether there’s a way to build a complex webhook.

4

How would you define a “complex webhook”?
As said, webhooks only trigger one request. The payload of that request’s payload can become quite complex but the logic to unpack that payload needs to be on your server side.
Webhooks are not much more than a relay station with some translation capabilities but no means for traversing/looping over arrays, conditional execution or such.

5

I’m hoping to avoid introducing another layer of complexity via a service whose sole function is to translate data from Particle to an MQTT hierarchy. If I were the project manager I would put the kibosh on adding a Heroku app just to add in a layer of abstraction, and so it’s not really something I can recommend to the upstream client.

A shame, really. For our purposes, the Particle Cloud is a somewhat insufficient version of MQTT. It would be nice if the webhook had a little more conditional handling than it already has. It’s almost there with the current regexp conditional and mustache variable substitution, it’s really just missing a simple foreach to loop over a well-defined data set.

@rickkas7 is this something you guys have considered in the past? I’m happy to write up a feature request, just want to make sure I’m not just rehashing past decisions.