As an extension to our anomaly detection example posted last week, we went ahead published the event data from a Photon along with glowfi.sh anomaly detection results and are feeding all in real-time to ubidots via webhook here:
The chart displays the probability of a normal event with the green “scaled anomaly probability” variable and anything below the red “threshold” line is considered an anomalous event by glowfi.sh. The raw sensor variables are in magenta (temperature), purple (wifi strength), and blue (light). These variables have been scaled using ubidot’s “derived variables” for better display.
Help me understand, how come the green line doesn’t cross/approach the threshold every time the temperature reading is bogus ? I see it lining up some times and not others.
Great observation! There are a few reasons why this can happen. The two most relevant ones are webhook request failure, and statistical considerations.
webhooks:
In order to display glowfi.sh predictions along with raw sensor data, we send two POSTs to Ubidots. The first POST is data from the initial particle webhook request to glowfi.sh (the raw sensor data), and the second is a POST from the particle hook-response from glowfi.sh (anomaly predictions). If the second request fails, Ubidots linearly interpolates the line graphs (even though data is missing). In the snapshot below, you can see an instance where the temp sensor goes negative (time ~8:56:40) but the anomaly probability stays above the threshold. The reason for this is that the hook-response with the glowfi.sh prediction for that time was not sent to Ubidots (missing data) and Ubidots interpolates the points across the previous and next points…showing no anomaly.
Statistical Considerations:
Even though the spikes in the temp sensor are obvious and could be removed quite easily with thresholding, as I mention in my post on despiking, anomaly amplitude and duration generally are not known in advance. Additionally, we are determining anomalies in a multi-variate sense across all sensors. In order to be very general in detecting anomalies, our algorithm uses transformations and normalization (e.g., to handle dynamic range variations across sensors), and multi-variate statistical models to estimate normal behavior across all sensor time-series. If the relationships between the sensor series are ill-behaved, then the anomaly predictions will be less robust. I show a snapshot of the sensor space (3-axes of sensor data for our example) below. The statistical relationship between sensors is multi-modal and this complicates the estimation of anomalies. While this can explain some of the behavior you have observed, we believe it is mostly due to the webhooks issue above.
This being said, I’ll post our success/failure rate for our predictions to this post so everyone can see how well we are doing on average with predicting events.
Makes sense.
Thanks for the explanation.
So the webhooks failure you reference is not a particle webhooks issue but one between your servers and ubidots, and if these errors were eliminated and no data was lost, we would expect to see more excursions of the green line below threshold (but still not necessarily 100% of the time because your algorithm is looking at a far broader picture.)
The glowfi.sh - particle webhooks integration is a somewhat complex one. For example, to get device, particle cloud, and glowfi.sh predictions to post to ubidots, the following sequence has to happen:
Raw data publish from device, webhook post to ubidots, webhook post to glowfi.sh API, glowfi.sh server response, hook-response generated in particle cloud (our predictions), device subscribes to that hook response event, device publishes probability event, and webhook post to ubidots.
We have seen some issues with webhooks missing randomly, and are not sure why. We have also run into some burst limitations due to queuing webhooks (we think) but we only post every 5-10s. We are currently working on a simplified version of our integration that posts directly from the device to glowfi.sh via http client and then we post directly to ubidots via our own customizable webhooks.
We should have our alternative integration out for the community in a week or so.
To clarify, the webhooks we are using are particle webhooks not glowfi.sh webhooks. We will update the community when we implement our custom webhooks as an alternative.
