Custom Webhook Help Needed

Cloud
1

@bjagpal @jeiden @ScruffR
I’m looking for some help creating a webhook with Custom JSON data that will send data to Microsoft’s Power BI Streaming DataSet so it can be used in custom dashboards which also databases 200,000 data points in a FIFO style.

Microsoft Power BI has a REST API for pushing data into Streaming Datasets that update in real time and that is what I need help with getting set up right.

So in Microsoft Power BI’s web interface you can create Streaming Dataset’s quickly using the window shown in the screen shots below:

Pasted image530×504 9.88 KB

The link to the API Info you see in the screenshot above takes you here if you’re interested: https://go.microsoft.com/fwlink/?linkid=822594

Once you do that you end up with this page:

image.png544×704 12.8 KB

Here is the push URL if you want to try this out:

https://api.powerbi.com/beta/36d257ec-bec7-4daf-9c2b-dc77d8b6da0f/datasets/ce1d5cff-61ef-458f-b486-d5a5571981f7/rows?key=PSyMJSlHZ%2F%2B8zQlOz2UxNpE2jgE2DUVwftPDEJBBfc62vfZ3ZvhdbdCbK438PPlRCcBm3jtgWOS0WNCrm4%2Fi4Q%3D%3D

Now I created a webhook that looks like this and I think it’s correct.

image.png1004×792 51.3 KB


Now when I hit the Test Button I get Error 500 as you can see in the screenshot below:

image.png1200×760 53 KB

The Particle Console shows this when I hit the Test Webhook button:

image.png1318×378 24.8 KB

After saying all that what I really want help with is writing the Particle.Publish function code so the Voltage and SOC data is properly injected and sent over to the Power BI Streaming Dataset.

I tried this code which I edited from the Particle Sparkfun Weathersheild code example.

I do not think the publish code is structured right and I need help with this:

void loop() {

delay(5000);
 
 // Measure Relative Humidity from the HTU21D or Si7021
 float v = 5.0;

 // Measure Temperature from the HTU21D or Si7021
 float soc = 100;
 // Temperature is measured every time RH is requested.
 // It is faster, therefore, to read it from previous RH
 // measurement with getTemp() instead with readTemp()

 char payload[255]; 

 snprintf(payload, sizeof(payload), "{ \"v\":\"v\", \"s\":\"s\", }");

 Particle.publish("SDT", payload, PRIVATE);
}
2

I just quickly skimmed over you publish function, so maybe I’m missing something, by doesn’t your “snprintf” line need to contain “%f” instead of “v”, and “s” ? Then at the end of that line you need to reference your variable names (“v” and “soc”).

Refer to the weather shield code again - I believe the correct format is shown there.

1 Like
3

Thanks for the reply @bjagpal

I switched it to this after more research:

And I think that it’s working now:

image.png1328×134 9.76 KB

I’m still getting a 500 Error message from Microsoft Power BI so I’ll have to wait and see what they have to say.

image.png1314×116 10.5 KB

4

Did you try PostMan to test?

I used your link and got 200 posting your JSON.

5

@BulldogLowell I installed Postman & Posted the Power BI Post Link and also received the 200 Success Message.

image.png1090×612 30.6 KB

Nice to have a new tool to check API’s with :thumbsup:

So it’s good to know it works but now I have to figure out how to properly format the webhook so it sends successfully :smiley:

I guess I still need to work on figuring out how to get the data sent by Particle’s Webhook to look just like this data that provides a 200 Success reply :wink:

@Dave I feel I’m so close to having this working :smile: Can you provide a little guidance on how to craft the Particle Publish & Webhook so it sends successfully.

I need to get the custom JSON to send to their REST API so it looks just like this:

Any help is Greatly Appreciated!

@bjagpal I now have the sample dashboard setup and the Voltage and SOC data is updating in real time every time I post to the API address using Postman. It’s nice to see this working correctly using Postman!

image.png1312×712 197 KB

1 Like
6

Helo,
The problem is that inside JSON contain the characters \.
The question is how to remove?
@Dave

1 Like
7

I actually think that the problem revolves around the double quotes - or rather the lack of them - around the number literal.
In order to have a webhook accept a JSON template, the template has to be valid JSON which forces you to provide it as

...
  "Voltage" : "{{v}}",
  "SoC" : "{{s}},
...

without the double quotes around the value field it wouldn’t be valid as the “mustaches” are seen as strings at that stage.
But that also means that after substitution of the “mustache” terms the double quotes will still be present in the response.

AFAIK there is no way around this yet.
But @Dave might have some updates about that, I’m not aware of :blush:

2 Likes
8

@ScruffR So to you does it look like I’ve formatted the Webhook correctly?

It’s just the double quotes shown around the numbers in the screenshot below that are probably causing the 500 Server error message?

9

Are you supposed to include square brackets [{... . ...}] in your data:

Screen Shot 2016-12-09 at 19.21.08.png880×200 40.3 KB

10

@BulldogLowell Yes, I am based on what Microsoft’s feedback.

I get Error 400 when I do not include them which they said is expected if the [ and ] are not included. Once I included the brackets the Error 400 went away and then I get Error 500 now.

image.png834×374 47.4 KB

11

hmmm… I had a similar type of problem against a different API and could never track down the reason; I just gave up!

I hope you hunt this down and get it working, I’m interested in this solution.

12

Yea I’m hoping @Dave has a solution for us :wink:

13

Hi All,

Sorry about the delayed response, I’ve been flying around a bunch, and finally made it back and am catching up. :slight_smile:

So my guess is that the receiving service wants to get literal numbers, and not strings of numbers. The problem here is in how the custom JSON for the webhook template is defined. Since it’s defined as a JSON object, and not a string of an object, the properties themselves must be parsable JSON. Since literally {{foo}} isn’t JSON, the workaround we added was to make it a string "{{foo}}".

I had started developing a feature to provide a workaround for this, but I don’t think it was ever released. I think we’ve since had another idea that will let us fix this, but it’s something we need to implement and test, so it won’t be available for a while.

In the meantime you can publish the values as literals in your string of a json object:

 snprintf(payload, sizeof(payload), "{ \"v\":%f, \"s\":%f, }", v, soc);

The webhook will still package them in a string, but you’ll be one step closer. :confused:

I’ll ping the team to see if this is something we can get on their radar.

Thanks!
David

2 Likes
14

@Dave Thanks for taking a look at this. I forwarded your reply to Microsoft Power BI also to see if they have any ideas.

1 Like
15

@Dave
I tried to figure out the latest status of the json implementation -> Azure. Same issue I guess with my elektrons:
Asset tracker should send

{ time,
  location:{lat,long}
  alt,
  spd,
  sat,
  dop}

so basically the NMEA vis the publish function to an iot hub. There a stream analysis shall sort them in to colomns:

SELECT
    data.ts as time,
    data.location.lat as latitude,
    data.location.lng as longitude,
    data.alt as altitude,
    data.spd as speed,
    data.sat as stellites,
    data.dop as hdop,
    published_at as ptime,
    coreid as trackerid

and PowerBi sahll be able to access the value separately. But I cannot get it to work.

So far I package all in one statement and it works fine when using:

String json = String::format("{\"ts\":\"%u\",\"lat\":\"%f\",\"lng\":\"%f\",\"alt\":\"%f\",\"spd\":\"%f\",\"sat\":\"%u\",\"dop\":\"%i\"}",
    				    gps.time.value(),
    				    gps.location.lat(),
    				    gps.location.lng(),
    				    gps.altitude.meters(),
    				    gps.speed.kmph(),
    				    gps.satellites.value(),
    				    gps.hdop.value()
                    );
    				Particle.publish("sendpos", json,PRIVATE);

and select * as stream analysis.
But as you might know the PowerBI sees then simply one object that is called ‘data’.

Any update or doc how to send the json correctly? O will I need to setup SEND CUSTOM JSON at the particle integration. ? Is this CUSTOM JSON as described here by @svelde the only solution ?

Thanks for any hint!

16

@copterview

What your trying to do is possible but you will need to use the Auzre integration + custom JSON template to get the data sent over into separate data fields as discussed in the thread you have already found here

1 Like
17

@RWB Thanks for affirming my ideas. I copied a lot of parts together from different sources and mixed up some problems. At the end I simply dropped some blanks - it didn’t cause a transmission error but AZURE didn’t recognize the JSON. Added the blanks and everything works fine…

2 Likes