Webhook to Ubidots with multiple timestamped values ARRAYS

In order to reduce the number of Particle.publish events in a product I have been looking at storing a small cache of readings and then uploading them with timestamps as supported by the Ubidots API.
the validated JSON for this is as so: curl -X POST -H "Content-Type: application/json" -d '{"Temp": "63","Floogit1":[{"value":"28","timestamp": "1505913534000"},{"value": "23","timestamp": "1505913554000"},{"value": "22","timestamp": "1505913574000"},{"value":"50","timestamp":"1505913513000"}],"Floogit2":[{"value": "28","timestamp": "1505913534000"},{"value": "23","timestamp":"1505913554000"},{"value":"22","timestamp":"1505913574000" },{"value": "50","timestamp": "1505913799000"}] }' http://things.ubidots.com/api/v1.6/devices/mymagicaldevice?token=thatisonesexytoken

However Iā€™m struggling to work out how to create the mustache/string that will process this webhook correctly. Iā€™m sure Iā€™m missing something pretty elementary but I just donā€™t see it. This is my JSON definition in the webhook:

{
    "Temp": "{{T}}",
    "Floogit1": [
        {
            "value": "{{v}}",
            "timestamp": "{{t}}"
        },
        {
            "value": "{{v}}",
            "timestamp": "{{t}}"
        },
        {
            "value": "{{v}}",
            "timestamp": "{{t}}"
        },
        {
            "value": "{{v}}",
            "timestamp": "{{t}}"
        }
    ],
    "Floogit2": [
        {
            "value": "{{v}}",
            "timestamp": "{{t}}"
        },
        {
            "value": "{{v}}",
            "timestamp": "{{t}}"
        },
        {
            "value": "{{v}}",
            "timestamp": "{{t}}"
        },
        {
            "value": "{{v}}",
            "timestamp": "{{t}}"
        }
    ]
}

And this is my string / publish

  Status=String::format( "{\"T\": \"%i\"\", \"v\":\"%i\",\"t\": \"%u000\"},{\"v\": \"%i\",\"t\": \"%u000\"},{\"v\": \"%i\",\"t\": \"%u000\"},{\"v\":\"%i\",\"t\":\"%u000\"},{\"v\": \"%i\",\"t\": \"%u000\"},{\"v\": \"%i\",\"t\":\"%u000\"},{\"v\":\"%i\",\"t\":\"%u000\" },{\"v\": \"%i\",\"t\": \"%u000\"}} ", 31,v1,t1,v2,t2,v3,t3,v4,t4,a1,t1,a2,t2,a3,t3,a4,t4,);

Particle.publish("myEvent",Status,PRIVATE);

Iā€™ve been happy up to now with simple flat mustache definitions but this doesnā€™t seem to be quite so simple. When testing this against request.bin I can see a convincing output except all the values are empty:

{"Floogit2":[{"timestamp":"","value":""},{"timestamp":"","value":""},{"timestamp":"","value":""},{"timestamp":"","value":""}],"Floogit1":[{"timestamp":"","value":""},{"timestamp":"","value":""},{"timestamp":"","value":""},{"timestamp":"","value":""}],"Temp":""}

thanks for the edit :wink:

Why the double double quote right after the first %i?

Dodgy search and replace when I was trying the value with and without double quotes sadly not the issue.

Your key/value pairs need to use unique placeholders - reusing {{v}} & {{t}} is ambiguous.
And for the webhook youā€™d just use { .... } without the nested curly braces, otherwise youā€™d need to qualify the placeholders with dot-notation.

OK Iā€™ve given all the tā€™s and vā€™s numbers for the publish (a1, t1 etc) but youā€™ve lost me on the nested curly braces bitā€¦

You use "{ { ... }, { ... }, ... }" but with uniquely named placeholders you can just forget all the inner curly braces and have to outgoing event just as on flat JSON "{ ... }".
But if you want to introduce some hierarchy in your outgoing event string, you should name each nested ā€œbranchā€ and qualify the desired field via its full ā€œpathā€.

e.g.

{ "b1": { "v" : "%i", "t": "%i"}, "b2": { "v": "%i", "t": "%i" }, ... }

and in the webhook definition youā€™d use this to fully qualify the individual fields

...
        {
            "value": "{{b1.v}}",
            "timestamp": "{{b1.t}}"
        },
        {
            "value": "{{b2.v}}",
            "timestamp": "{{b2.t}}"
        },
...
1 Like

Hmmm I like the look of this dot notation and yet on implementing it I seem to get the exact same output.

{
  "Temp": "{{T}}",
  "Floogit1": [
    {
      "value": "{{P1.v}}",
      "timestamp": "{{P1.t}}"
    },
    {
      "value": "{{P2.v}}",
      "timestamp": "{{P2.t}}"
    },
    {
      "value": "{{P3.v}}",
      "timestamp": "{{P3.t}}"
    },
    {
      "value": "{{P4.v}}",
      "timestamp": "{{P4.t}}"
    }
  ],
  "Floogit2": [
    {
      "value": "{{R1.v}}",
      "timestamp": "{{R1.t}}"
    },
    {
      "value": "{{R2.v}}",
      "timestamp": "{{R2.t}}"
    },
    {
      "value": "{{R3.v}}",
      "timestamp": "{{R3.t}}"
    },
    {
      "value": "{{R4.v}}",
      "timestamp": "{{R4.t}}"
    }
  ]
}

Status=String::format( "{\"T\": %i,\"P1\": {\"v\":%i,\"t\": %u},\"P2\":{\"v\": %i,\"t\": %u},\"P3\":{\"v\": %i,\"t\": %u},\"P4\":{\"v\":%i,\"t\": %u},\"R1\":{\"v\": %i,\"t\": %u},\"R2\":{\"v\": %i,\"t\":%u},\"R3\":{\"v\":%i,\"t\":%u },\"R4\":{\"v\": %i,\"t\": %u}} ", 31,etc,etc etc); Serial.print(Status); Particle.publish("event",Status,PRIVATE);

Thanks for the feedback

I think the bit I struggle the most with using webhooks/mustache is getting my head round exactly what it is doing, in particular when is it substituting in a value and when is it expanding a keyname and how much extra content can I insert in the webhook definition to avoid having to send it in the first place.
For instance using @rickkas7 mustache tester I can come up with this as a template

{
"Temp":"{{Temp}}"
"Floogit1":[
{{#Floogit1}}
{
"value":{{value}},
"timestamp":{{timestamp}}
}
{{/Floogit1}}
]
"Floogit2":[
{{#Floogit2}}
{
"value":{{value}},
"timestamp":{{timestamp}}
}
{{/Floogit2}}
]
}

And that would appear to output what I am after but that leaves me confused as to what to publish to the template particularly if I want to keep the byte count down and use some variable expansion in there too.

Would this template need to be part of a fully custom template? As it stands if you try and add that in the console it tells you its invalid ā€œBad Stringā€

Hi there @Viscacha, my two cents here, you have an error in the example endpoint that you are using, you mismatched an slash, ā€˜/ā€™, before of your device label, your endpoint should be something like this:

http://things.ubidots.com/api/v1.6/devices/mymagicaldevice?token=thatisonesexytoken

Hope it helps you.

Regards

Yes sorry I did see that, careless deletion removing my token etc. Final result Iā€™m trying to get to works, parsing it though all the mustache stuff doesnā€™t. Iā€™m starting to wonder if it needs to be done with a JSON file and cannot be done on the console. If that is the case it cannot be done at all as this is a product and you cannot access webhooks for products from the CLI. Personally I think this mustache business is far more comprehensible when extracting values from a JSON string than it it when used to create one.

Well thereā€™s a way to feel stupid, turns out if you set out to do this without a splitting headache &| at a sensible time of the day it is but the work of moments! @ScruffR is indeed correct a completely flat csv of value in the publish and do the 100% of the heavy lifting in the Template. Use the @rickkas7 mustache tester to create the template and in the JSON box start point use the publish you feel like sending .

Status=String::format({\"T\": %i,\"v1\":%i,\"t1\": %u,\"v2\": %i,\"t2\": %u,\"v3\": %i,\"t3\": %u,\"v4\":%i,\"t4\":%u,\"va\": %i,\"ta\": %u,\"vb\": %i,\"tb\":%u,\"vc\":%i,\"tc\":%u ,\"vd\": %i,\"td\": %u} ", all your values here);

To keep things as tight as possible donā€™t bother wrapping all your values in escaped double quotes they are not needed.
Now for the template:

{
  "Temp": "{{T}}",
  "Floogit1": [
    {
      "value": "{{v1}}",
      "timestamp": "{{t2}}000"
    },
    {
      "value": "{{v2}}",
      "timestamp": "{{t2}}000"
    },
    {
      "value": "{{v3}}",
      "timestamp": "{{t3}}000"
    },
    {
      "value": "{{v4}}",
      "timestamp": "{{t4}}000"
    }
  ],
  "Floogit2": [
    {
      "value": "{{va}}",
      "timestamp": "{{ta}}000"
    },
    {
      "value": "{{vb}}",
      "timestamp": "{{tb}}000"
    },
    {
      "value": "{{vc}}",
      "timestamp": "{{tc}}000"
    },
    {
      "value": "{{vd}}",
      "timestamp": "{{td}}000"
    }
  ]
}
          

You will notice Iā€™m using the template to add 000 to each timestamp value, Ubitdots expects time in POSIX milliseconds, Time.now() gives seconds, less bytes this way.

Where had I gone wrong? Well to start with I was for some reason working on the assumption that the variable substitution was done in order rather than by matching variable names. Secondly I was trying to do some of the formatting before transmission and that makes the template harder to create. I think there may still be a way to simplify the template further as it basically has two arrays that are pretty much the same but frankly it works so its good enough. One improvement is pretty obvious t1 and ta etc are the same value each time.

On the plus side messing about with templates did mean I now have a pretty good idea how to use them to extract values from complex JSON strings, something @rickkas7ā€™s tutorial does very well indeed.

Again many thanks to @ScruffR and @jotathebest for their suggestions and corrections and of course @rickkas7 for his webhook pages on github.

3 Likes