Mustache Notation to unpack data in nested lists

Cloud Integrations
1

I'm using a webhook to store data to emoncms.org, but logging multiple feeds from multiple devices at 10sec rate quickly exceeds my allowable data operations. I want to do bulk uploads of my data.

This URL stores 2 minutes of data from 3 different feeds at 10 sec sample rate:

https://emoncms.org/input/bulk?data=[
   [10,"nodeXYZ",{"feedABC":1500},{"feedDEF":1600},{"feedGHI":1700}],
   [20,"nodeXYZ",{"feedABC":1501},{"feedDEF":1601},{"feedGHI":1701}],
   [30,"nodeXYZ",{"feedABC":1502},{"feedDEF":1602},{"feedGHI":1702}],
   [40,"nodeXYZ",{"feedABC":1503},{"feedDEF":1603},{"feedGHI":1703}],
   [50,"nodeXYZ",{"feedABC":1504},{"feedDEF":1604},{"feedGHI":1704}],
   [60,"nodeXYZ",{"feedABC":1505},{"feedDEF":1605},{"feedGHI":1705}],
   [70,"nodeXYZ",{"feedABC":1506},{"feedDEF":1606},{"feedGHI":1706}],
   [80,"nodeXYZ",{"feedABC":1507},{"feedDEF":1607},{"feedGHI":1707}],
   [90,"nodeXYZ",{"feedABC":1508},{"feedDEF":1608},{"feedGHI":1708}],
   [100,"nodeXYZ",{"feedABC":1509},{"feedDEF":1609},{"feedGHI":1709}],
   [110,"nodeXYZ",{"feedABC":1510},{"feedDEF":1610},{"feedGHI":1710}],
   [120,"nodeXYZ",{"feedABC":1511},{"feedDEF":1611},{"feedGHI":1711}]
]

image
image1141×676 32.2 KB

I am trying to adapt it to a situation where the node and feed labels are quite long - about 30 characters each and I will run out of space in my publish string.

Can I send something like this as my event data string:

`{
"t":10,
"n":"veryverylongnamednodeXYZ",
"f":["veryverylongfeedABC","veryverylongfeedDEF","veryverylongfeedGHI"],
"d":[
[1500,1501,1502,1503,1504,1505,1506,1507,1508,1509,1510,1511]
[1600,1601,1602,1603,1604,1605,1606,1607,1608,1609,1610,1611],
[1700,1701,1702,1703,1704,1705,1706,1707,1708,1709,1710,1711]
]
}`

And unpack the elements using Mustache notation to recreate URL like the one above?

The data sampling rate "t" is fixed.
I could fix the number of feeds and data elements but ideally they would both be variable.

2

It would be extremely difficult, bordering on impossible, to do this with mustache templates. Almost all of the data would need to be fixed length for it to even be close to possible.

However, this is exactly what Logic is useful for. It can be triggered by your event and generate an event payload from Javascript. The difference is that the outgoing payload is not limited to 1024 bytes, and you are only charged for the one data operation to upload the bulk data, not the outgoing data.

This is what the code to transform the data would look like (approximately):

function transformData(dataIn) {
    const dataOut = [];

    let rowIndex = 0;

    for(let time = dataIn.t; rowIndex < dataIn.d[0].length; time += dataIn.t, rowIndex++) {
        const row = [];
        row.push(time);
        row.push(dataIn.n);
        for(let colIndex = 0; colIndex < dataIn.f.length; colIndex++) {
            let obj = {};
            const key = dataIn.f[colIndex];
            const value = dataIn.d[colIndex][rowIndex];
            obj[key] = value;
            row.push(obj);    
        }
        dataOut.push(row);
    }

    return JSON.stringify(dataOut);
}

Given the sample data above, the result would be:

[[10,"veryverylongnamednodeXYZ",{"veryverylongfeedABC":1500},{"veryverylongfeedDEF":1600},{"veryverylongfeedGHI":1700}],[20,"veryverylongnamednodeXYZ",{"veryverylongfeedABC":1501},{"veryverylongfeedDEF":1601},{"veryverylongfeedGHI":1701}],[30,"veryverylongnamednodeXYZ",{"veryverylongfeedABC":1502},{"veryverylongfeedDEF":1602},{"veryverylongfeedGHI":1702}],[40,"veryverylongnamednodeXYZ",{"veryverylongfeedABC":1503},{"veryverylongfeedDEF":1603},{"veryverylongfeedGHI":1703}],[50,"veryverylongnamednodeXYZ",{"veryverylongfeedABC":1504},{"veryverylongfeedDEF":1604},{"veryverylongfeedGHI":1704}],[60,"veryverylongnamednodeXYZ",{"veryverylongfeedABC":1505},{"veryverylongfeedDEF":1605},{"veryverylongfeedGHI":1705}],[70,"veryverylongnamednodeXYZ",{"veryverylongfeedABC":1506},{"veryverylongfeedDEF":1606},{"veryverylongfeedGHI":1706}],[80,"veryverylongnamednodeXYZ",{"veryverylongfeedABC":1507},{"veryverylongfeedDEF":1607},{"veryverylongfeedGHI":1707}],[90,"veryverylongnamednodeXYZ",{"veryverylongfeedABC":1508},{"veryverylongfeedDEF":1608},{"veryverylongfeedGHI":1708}],[100,"veryverylongnamednodeXYZ",{"veryverylongfeedABC":1509},{"veryverylongfeedDEF":1609},{"veryverylongfeedGHI":1709}],[110,"veryverylongnamednodeXYZ",{"veryverylongfeedABC":1510},{"veryverylongfeedDEF":1610},{"veryverylongfeedGHI":1710}],[120,"veryverylongnamednodeXYZ",{"veryverylongfeedABC":1511},{"veryverylongfeedDEF":1611},{"veryverylongfeedGHI":1711}]]
1 Like
3

This looks perfect; thanks very much!

4

I created the logic function and deployed it. It's being triggered by my publish event. But now BOTH my integration and my logic function are being triggered by the same event and the integration just sends the event data - not the logic-reformatted data.
How do I set my integration to be triggered by the logic function?

5

Create a separate event name for the integration, not the original event name.

Integrations are triggered with a prefix match so you can't just add something to the end of the event name, but you can insert something before it, so, for example, if the event was "MyEvent" you could have "LogicMyEvent" to make easier to distinguish which events are which.

6

Sorry - It's SO close but my integration is not triggering.
To review:
My device (which is in product 9093) is publishing event "emonBulk" with my raw data payload.
I have a logic function called "reformat_bulk_emoncms" which is triggered by event "emonBulk", takes the raw data and reformats it to match the emoncms bulk upload URL and publishes a new event called "ebulkdoget". This seems to be working well.

image
image944×568 34.7 KB

I want my webhook integration to be triggered by the logic function so I set the integrations's trigger to "ebulkdoget".

image
image857×649 43.2 KB

But, although I can see ebulkdoget getting published, it's not triggering the webhook.

I tried re-creating the webhook - still not getting triggered.

7

What does your logic block Javascript look like? The event name also needs to be changed in the script's publish call to match the webhook event name.

8

here's the integration:

{
    "name": "ebulk webhook ",
    "event": "ebulkdoget",
    "disabled": false,
    "template": "webhook",
    "url": "https://emoncms.org/input/bulk?",
    "requestType": "GET",
    "noDefaults": false,
    "rejectUnauthorized": true,
    "headers": {
        "Authorization": "Bearer zzzzzz8baf1a56ae7d4dd2e3bf2"
    },
    "query": {
        "data": "{{{PARTICLE_EVENT_VALUE}}}"
    }
}

here's the logic function - I did match the event name in the publish call
Particle.publish("ebulkdoget", reformatted, { productId: event.productId });

import Particle from 'particle:core';

export default function reformat({ event }) {
  let data;
  try {
	  data = JSON.parse(event.eventData);
  } catch (err) {
    console.error("Invalid JSON", event.eventData);
    throw err;
  }
  const reformatted = transformData(data);
  console.log(reformatted);
  Particle.publish("ebulkdoget", reformatted, { productId: event.productId });
}

 function transformData(dataIn) {
    const dataOut = [];

    let rowIndex = 0;

    for(let time = dataIn.t; rowIndex < dataIn.d[0].length; time += dataIn.t, rowIndex++) {
        const row = [];
        row.push(time);
        row.push(dataIn.n);
        for(let colIndex = 0; colIndex < dataIn.f.length; colIndex++) {
            let obj = {};
            const key = dataIn.f[colIndex];
            const value = dataIn.d[colIndex][rowIndex];
            obj[key] = value;
            row.push(obj);    
        }
        dataOut.push(row);
    }

    return JSON.stringify(dataOut);
}
9

Within the webhook builder - I selected "Any" for "which of your devices will trigger the webhook."
Is this important? - the event is coming from the logic function not a device at all.
image

10

I think the problem is that your logic-triggered webhook must be in the product that the logic block is created in. It looks like you have the webhook in your developer sandbox, not in the product.

The sandbox webhook triggers directly from the device because it's presumably both claimed to your acccount and part of the product. The logic block triggers because it's part of a product.

The problem is that the logic-generated webhook is only scoped to the product, because it's generated by the product logic block, not the owned device. So only a product integration can be triggered, not a sandbox integration.

11

Yes! I re-created the integration within the product and the whole flow is working beautifully now.
My devices send data packets in "shorthand" that include up to 8 channels of 10 historical samples at 1 min sampling rate. These are parsed and reformatted by the Logic function into a longer format (that greatly exceeds the character limit of the particle.publish() string) that is ingestible by the bulk upload API call at emoncms.org. I have cut my data operation rate down by a factor of >20.
Thanks so much!
Tim

1 Like
12

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.