Originally published at: https://blog.particle.io/2019/11/26/machine-learning-102/
In my last post, I showed how you can get started with Machine Learning on Particle devices with our new support for TensorFlow Lite for Microcontrollers. As a part of that post, we looked at an end-to-end example of building an MCU-friendly TensorFlow Lite model, and performing real-time inferencing (aka prediction) against that model. I did so with a toy example in order to keep the focus on the process of using TensorFlow Lite in your Particle apps.
For this post, Iāll bring our ML exploration firmly into the realm of IoT and share how to use a pre-trained ML model to recognize āgesturesā performed in the air by running inference against data streaming from a 3-axis accelerometer. The original source for this project is part of the official TensorFlow repository, though I made a number of modifications to support a different accelerometer breakout, and to add a bit of Particle RGB LED flair at the end. You may not actually be a magician by the end of this post, but youāll certainly feel like one!
Parts and tools
If you want to build one of these wands for yourself, youāll need the following components.
- A Particle Argon, Boron, or Xenon.
- A 3-axis accelerometer, like the LSM9DS1 from Adafruit. If you choose another, youāll need to customize portions of this example.
The Problem Weāre Trying to Solve
One of the reasons that ML on MCUs is such a hot topic these days is because of the amount of data for some solutions can be a deterrent to performing inferencing in the cloud. If youāre working with a high-precision sensor and you need to make a decision based on raw data, you often cannot shoulder the cost of sending all that data to the cloud. And even if you could, the round trip latency runs counter to the frequent need for real-time decision-making.
By performing prediction on a microcontroller, you can process all that raw data on the edge, quickly, without sending a single byte to the cloud.
For this post, Iāll illustrate this with an example that works with a common IoT device: a 3-axis accelerometer, which is connected to a Particle Xenon. The Xenon will use streaming accelerometer data and TensorFlow Lite for Microcontrollers to determine if the person holding the device is making one of 3 gestures:
- A clockwise circle, or āringā gesture;
- A capital W, or āwingā gesture;
- A right to left āslopeā gesture.
Hereās a GIF sample of yours truly performing these gestures.
If one of the three gestures is detected, the onboard RGB LED will light up green, blue, or red, depending on the gesture performed.
Using an Accelerometer Gesture Detection Model
For this post, Iāll be using a pre-trained model thatās included in the example projects for the TensorFlow Lite library. The model, which was created by the TensorFlow team, is a 20 KB convolutional neural network (or CNN) trained on gesture data from 10 people performing four gestures fifteen times each (ring, wing, slope, and an unknown gesture). For detection (inferencing), the model accepts raw accelerometer data in the form of 128 X, Y, and Z values and outputs probability scores for our three gestures and an unknown. The four scores sum to 1, and weāll consider a probability of 0.8 or greater as a confident prediction of a given gesture.
The source for this post can be found in the examples folder of the TensorFlow Lite library source. The model itself is contained in the file magic_wand_model_data.cpp, which, as I discussed in my last post, is a C array representation of the TFLite flatbuffer model itself. For MCUs, we use models in this form because we donāt have a filesystem to store or load models.
Configuring the Magic Wand Application
The entry-point of my application is in the magic_wand.cpp, which contains the setup and loop functions weāre used to. Before I get there however, Iāll load some TensorFlow dependencies and configure a few variables for our application. One of note is an object to set aside some memory for TensorFlow to store input, output and intermediate arrays in.
constexpr int kTensorArenaSize = 60 * 1024; uint8_t tensor_arena[kTensorArenaSize];
The kTensorArenaSize variable represents the number of bytes to set aside in memory for my model to use at runtime. This will vary from one model to the next, and may need to be adjusted based on the device youāre using, as well. I ran this example on a Particle Xenon with the above values, but noticed that the same value on an Argon or Boron tended to cause my application to run out of memory quickly. If you see red SOS errors from your onboard RGB LED when running a TensorFlow project, try adjusting this value first.
Once Iāve configured the memory space for my app, Iāll load my model, and configure my operations resolver. The latter step is notable because itās a bit different than the hello_world example I used in my last post.
static tflite::MicroMutableOpResolver micro_mutable_op_resolver;
micro_mutable_op_resolver.AddBuiltin(
tflite::BuiltinOperator_DEPTHWISE_CONV_2D,
tflite::ops::micro::Register_DEPTHWISE_CONV_2D());
micro_mutable_op_resolver.AddBuiltin(
tflite::BuiltinOperator_MAX_POOL_2D,
tflite::ops::micro::Register_MAX_POOL_2D());
micro_mutable_op_resolver.AddBuiltin(
tflite::BuiltinOperator_CONV_2D,
tflite::ops::micro::Register_CONV_2D());
micro_mutable_op_resolver.AddBuiltin(
tflite::BuiltinOperator_FULLY_CONNECTED,
tflite::ops::micro::Register_FULLY_CONNECTED());
micro_mutable_op_resolver.AddBuiltin(
tflite::BuiltinOperator_SOFTMAX,
tflite::ops::micro::Register_SOFTMAX());
In the hello_world example, we used the tflite::ops::micro::AllOpsResolver, which just loads all available operations into the application. For large models, free memory will be at a premium, so youāll want to load only the operations that your model needs for inferencing. This requires an understanding of the model architecture. For the magic_wand example, I added the operations above (DEPTHWISE_CONV_2D, MAX_POOL_2D, etc.) because those reflect the layers of the CNN model Iām using for this application.
Once everything is configured on the TensorFlow side, itās time to set-up an accelerometer and start collecting real data.
TfLiteStatus setup_status = SetupAccelerometer(error_reporter);
This function lives in the particle_accelerometer_handler.cpp file. True to its name, it configures the accelerometer Iām using for this application. When creating this example, I used the Adafruit LSM9DS1, which contains a 3-axis accelerometer, a gyroscope, and a magnetometer.
To interface with the accelerometer, Iāll use a library. There are a couple of LSM9DS1 libraries available, but this particular application requires the use of the FIFO buffer on the LSM9DS1 for storing accelerometer readings, so for this example I am using the LSM9DS1_FIFO library.
#include <LSM9DS1_FIFO.h> #include <Adafruit_Sensor.h>LSM9DS1_FIFO accel = LSM9DS1_FIFO();
TfLiteStatus SetupAccelerometer(tflite::ErrorReporter *error_reporter)
{
while (!Serial) {}if (!accel.begin())
{
error_reporter->Report(āFailed to initialize accelerometer. Please resetā);
return kTfLiteError;
}accel.setupAccel(accel.LSM9DS1_ACCELRANGE_2G);
accel.setupMag(accel.LSM9DS1_MAGGAIN_4GAUSS);
accel.setupGyro(accel.LSM9DS1_GYROSCALE_245DPS);error_reporter->Report(āMagic starts!ā);
return kTfLiteOk;
}
Once Iāve installed the library, configuration is as simple as calling accel.begin(). With that done, Iām ready to start taking readings.
Reading from the accelerometer
Each time through the loop, Iāll grab data from the accelerometer by calling the ReadAccelerometer function.
bool got_data = ReadAccelerometer(error_reporter, model_input->data.f,
input_length, should_clear_buffer);
This function, which can also be found in the particle_accelerometer_handler.cpp file, reads from the FIFO buffer on the accelerometer one sample at a time. It then downsamples data from 119 Hz, the rate that my accelerometer captures data, to about 25 Hz, the rate that the model was trained on. Next, Iāll assign the x, y, and z values to an array.
while (accel.accelerationAvailable())
{
accel.read();
sensors_event_t accelData, magData, gyroData, temp;
// Read each sample, removing it from the deviceās FIFO buffer
if (!accel.getEvent(&accelData, &magData, &gyroData, &temp))
{
error_reporter->Report(āFailed to read dataā);
break;
}
// Throw away this sample unless itās the nth
if (sample_skip_counter != sample_every_n)
{
sample_skip_counter += 1;
continue;
}
save_data[begin_index++] = accelData.acceleration.x * 1000;
save_data[begin_index++] = accelData.acceleration.y * 1000;
save_data[begin_index++] = accelData.acceleration.z * 1000;
sample_skip_counter = 1;
// If we reached the end of the circle buffer, reset
if (begin_index >= 600)
{
begin_index = 0;
}
new_data = true;
break;
}
This function will run each time through the loop, accumulating readings until we have enough data to perform a prediction, which is around 200 x, y, and z values. After Iāve collected those, I set all the values on the input tensor of my model and Iām ready to make a prediction!
for (int i = 0; i < length; ++i)
{
int ring_array_index = begin_index + i - length;
if (ring_array_index < 0)
{
ring_array_index += 600;
}
input[i] = save_data[ring_array_index];
}
Real-time inferencing with accelerometer data
With a set of accelerometer values in my model input, I can make a prediction by invoking the TFLite interpreter. Notice that the snippet below is identical to the hello world example from my last post. Setting inputs and reading outputs will differ from one TFLite application to the next, but the invocation process follows a consistent API.
TfLiteStatus invoke_status = interpreter->Invoke();if (invoke_status != kTfLiteOk)
{
error_reporter->Report(āInvoke failed on index: %d\nā, begin_index);
return;
}
Once Iāve made a prediction, I work with the output and determine if a gesture was found.
int gesture_index = PredictGesture(interpreter->output(0)->data.f);
The PredictGesture function takes the values from my output tensor and determines if any value is over the 80% confidence threshold Iāve set and, if so, sets that value as the current prediction. Otherwise, an unknown value is set.
int this_predict = -1;
for (int i = 0; i < 3; i++) { if (output[i] > 0.8) this_predict = i;
}
// No gesture was detected above the threshold
if (this_predict == -1) {
continuous_count = 0;
last_predict = 3;
return 3;
}
In addition to interpreting the current prediction, weāre also tracking some state across predictions. Since accelerometer data is highly variable, we should make sure that our model predicts the same gesture multiple times in a row before we formally decide that weāre happy with the prediction. As such, the final portion of PreductGesture tracks the last and current prediction and, if the prediction has occurred a given number of times in a row, weāll report it. Otherwise, weāll report that the gesture was unknown. The kConsecutiveInferenceThresholds is an array of integers that correspond to the number of consecutive predictions we want to see for each gesture before considering a prediction to be valid. This may differ for your project and accelerometer. The values I chose can be found in the particle_constants.cpp file.
if (last_predict == this_predict) {
continuous_count += 1;
} else {
continuous_count = 0;
}
last_predict = this_predict;
if (continuous_count < kConsecutiveInferenceThresholds[this_predict]) {
return 3;
}
continuous_count = 0;
last_predict = -1;
return this_predict;
Now that we have a result, we can do something with it.
Displaying the result
At the end of my loop, with a predicted gesture in hand, I call a function called `HandleOutput` to display something to the user. This function is defined in the particle_output_handler.cpp file and does three things:
- Toggles the
D7LED each time an inference is performed; - Outputs the predicted gesture to the Serial console;
- Sets the onboard RGB Led green, blue, or red, depending on the predicted gesture.
The first step is pretty straightforward, so I wonāt cover it here. For the second, weāll print a big of ASCII art to the console to represent each gesture.
if (kind == 0)
{
error_reporter->Report(
"WING:\n\r* * *\n\r * * * "
"*\n\r * * * *\n\r * * * *\n\r * * "
"* *\n\r * *\n\r");
}
else if (kind == 1)
{
error_reporter->Report(
"RING:\n\r *\n\r * *\n\r * *\n\r "
" * *\n\r * *\n\r * *\n\r "
" *\n\r");
}
else if (kind == 2)
{
error_reporter->Report(
"SLOPE:\n\r *\n\r *\n\r *\n\r *\n\r "
"*\n\r *\n\r *\n\r * * * * * * * *\n\r");
}
It doesnāt look like much here, but if you run the example and perform a gesture, youāll see the art in all its glory.
For the final step, weāll do something unique to Particle devices: take control of the onboard RGB LED. The onboard LED is used by Particle to communicate device status, connectivity, and the like, but itās also possible to take control of the LED and use it in your own applications. You simply call RBG.control(true), and then RGB.color to set an RGB value. For this example, I modified the ASCII art snippet above to set my LED, and release control with RGB.control(false) if a gesture isnāt detected.
if (kind == 0)
{
RGB.control(true);
RGB.color(0, 255, 0); // Green
error_reporter->Report(
āWING:\n\r* * \n\r * * * "
"\n\r * * * *\n\r * * * \n\r * * "
" *\n\r * *\n\rā);
}
else if (kind == 1)
{
RGB.control(true);
RGB.color(0, 0, 255); // Blue
error_reporter->Report(
āRING:\n\r *\n\r * *\n\r * *\n\r "
" * *\n\r * *\n\r * *\n\r "
" *\n\rā);
}
else if (kind == 2)
{
RGB.control(true);
RGB.color(255, 0, 0); // Red
error_reporter->Report(
āSLOPE:\n\r *\n\r *\n\r *\n\r \n\r "
"\n\r *\n\r *\n\r * * * * * * * *\n\rā);
}
else
{
RGB.control(false);
}
Ā
Now, in addition to fancy ASCII art, your device will change the RGB LED when a gesture is detected.
Taking your exploration further
In this post, we took our exploration of TensorFlow Lite into the realm of IoT with an accelerometer demo. Now itās your turn. If you havenāt already, check out the TensorFlowLite Particle library and explore some of the other demos. And if youāre building your own ML on MCUs project with TensorFlow, share it in the comments below!
