I am trying to create a webhook using CLI to call Google Firebase realtime database REST apis. Firebase allows to insert server time in data using {".sv":“timestamp”}. Now if I try to include this JSON object in my webhook JSON section, CLI throws an error
webhook creation failed: key .sv must not contain ‘.’
Here is my JSON block from the webhook definition file
Funny thing is sometimes after creation attempt fails and I just publish an event, the webhook that was never created actually triggers and I get proper server timestamp in EPOCH milliseconds inserted in my Firebase database. How is that possible?? This invisible webhook stops triggering after sometime and never appears in Console.
I have two questions
1- First why ‘.’ is not allowed in json keys
2- Is the invisible webhook a bug?
Because the dot (.) is used by JSON to seperate hierachical levels of a structured path.
Just as a forward slash (/) (or backslash \ for Windows) is not allowed in a filename since it's the path seperator.
@ScruffR AFAIK, dot (.) is used in JavaScript to separate hierarchical levels like other languages but within JSON format dot (.) does not have any special meaning. I think it is not even in the JSON operator list. If that were the case then Google would never have used it for accessing server values in firebase like so {".sv":“timestamp”}.
Good point, it seems it’s not the case for vanilla JSON but obviously is enabled for webhooks due to the use of mustache nested objects in custom templates and hence the point stands.
Using unescaped dots would interfere with that feature and hence the “easy” workaround was to just disallow it.
Thanks for the ping! This smells like a bug to me on our end, and I’m chatting with the team about how we can fix it. Due to the nature of what I suspect the bug is, it would be possible for a hook to be briefly created for a few minutes, but as a ‘phantom’ before being removed. Looks like this is maybe an old bug, here’s our public ticket for reference.
I’m not sure when the team will have time to address this, but they’re aware of it again now.
Just wondering what is the planned resolution for this issue? Would we be able to create webhooks with JSON allowing use of dot (.) symbol in key names? OR just ensuing no phantom webhooks get created?
I am still not clear whats the official stance on using dot (.) symbols in key names in JSON section of webhooks.
I have a project where this bug is not letting me insert data directly into firebase realtime database. I have found a way around it but that adds few hundred milliseconds to the whole operation. I have to go from Particle cloud -> Google Pub/Sub -> Google cloud function -> Firebase DB instead of simply going from Particle cloud -> Firebase DB.
I’d need to check with the team to be sure, but I imagine that any valid JSON should be allowed in the webhook JSON template. If they agree with that, then “.notation” properties should be perfectly valid.
Hi, this appears to still not be fixed. I tried many months ago, and again today. If you post an escaped version it’ll send the string, if you post the actual valid JSON then the webhook system actually hangs for me and doesn’t even create it. Awesome.
@noumanh, you will literally go crazy dealing with all the idiosyncrasies (ie bugs) of the webhook system. Even today I was getting tripped up by the old webhook name issue where publishing to ‘myhook2’ will trigger ‘myhook’ and ‘myhook2’. Don’t start me on the handling of >200 status codes.
My current approach to create a timestamp that is equivalent to FB server timestamp is to construct a millis version of epoch on the particle device just before the publish:
[in global declarations]
char charMillis[13];
[in loop]
double millisForFB = (double) Time.now() * 1000; // epoch in seconds changed to millis for firebase
sprintf(charMillis,"%13.0f", millisForFB); // this is the easiest way I knew of stripping the decimal
s.concat(",\"t\":"); // my abbreviated 'time' field in the json post, i change that in the webhook template to a longer variable name to save space in the length constrained publish string
s.concat(charMillis); // make it a string for concat
Compared to your chaining of services (which from experience I don’t recommend due to the HTTP status code handling and constrained 5 second timeout of particle webhook processing), this should be more reliable and cheaper. You could also patch the results via a firebase cloud function but this approach gets it right first time and doesn’t incur (minimal) cloud function costs.
Welcome to suggest improvements, it works but i’m sure there is a nicer way of skinning the cat. I’m not a C guy, my background is distributed web platforms and data centre networks. We just need to roll with the punches until either we can publish via SSL MQTT to our own API’s or someone takes a long hard look at webhooks and dusts out the cobwebs.
I am afraid this is not same as getting the server timestamp. Obviously you can create a similar value locally but the whole idea behind getting the server timestamp is to guarantee the records from multiple sources end up chronologically ordered on the database server (as arrived on the server, not as published from the client). In Firebase case, all clients mention the '.sv' key and when data arrives at the backend, server inserts the timestamp ensuring there is a single reliable source for timestamps regardless of the client type.
If there are multiple client platforms, like electrons, photons, android phones, web clients connecting to firebase, their clock skews will further disturb the ordering at server end. If you really want to use the client clock for timestamp then Firebase recommends that you at least first adjust your clock skew. Below is the excerpt from their website that explains how you might do that
Clock Skew
While firebase.database.ServerValue.TIMESTAMP is much more accurate, and preferable for most read/write operations, it can occasionally be useful to estimate the client's clock skew with respect to the Firebase Realtime Database's servers. You can attach a callback to the location /.info/serverTimeOffset to obtain the value, in milliseconds, that Firebase Realtime Database clients add to the local reported time (epoch time in milliseconds) to estimate the server time. Note that this offset's accuracy can be affected by networking latency, and so is useful primarily for discovering large (> 1 second) discrepancies in clock time.
Unless someone hasn’t explained NTP to the particle staff they will be using the likes of pool.ntp.org and down to the nanosecond from true UTC, same as what Firebase @ Google will be doing. There is a particle function to sync clock, I’d suggest that every 24 hours.
Based on experience I cannot trust that particle webhooks are sent to the destination in order (!!)
What seems to be generally correct is the simple ‘published_at’ stamp which appears to be the received at Particle timestamp (it’d be nice to have this confirmed).
As I was already performing basic Firebase cloud function processing of new records I’ve ended up adding a simple additional update:
const doSimpleDateUpdate = event.data.adminRef.update({fbdate: moment(data.published_at).valueOf()}).then(function(){ console.log('doSimpleDateUpdate complete for ' + coreID); });
// Wrap that in a returned promise
return Promise.all([doSimpleDateUpdate, myMoreInterestingFunctionToProcessResults]).then(results => {
....
});
There is no workaround. The “.” is used in the mustache template expansion and there’s no way to escape it so the whole thing is used as a single JSON field name, instead of a hierarchy of objects or array indexes.
Thanks for the quick reply. I was able to successfully create a webhook by using a custom body instead of choosing the JSON format. I just posted the following into the custom body. Seems to be working.
