Use Tensor Flow Lite and a Particle Xenon to build the ML-gesture wand of your dreams

Originally published at: https://blog.particle.io/2019/11/26/machine-learning-102/

In my last post, I showed how you can get started with Machine Learning on Particle devices with our new support for TensorFlow Lite for Microcontrollers. As a part of that post, we looked at an end-to-end example of building an MCU-friendly TensorFlow Lite model, and performing real-time inferencing (aka prediction) against that model. I did so with a toy example in order to keep the focus on the process of using TensorFlow Lite in your Particle apps.

For this post, Iā€™ll bring our ML exploration firmly into the realm of IoT and share how to use a pre-trained ML model to recognize ā€œgesturesā€ performed in the air by running inference against data streaming from a 3-axis accelerometer. The original source for this project is part of the official TensorFlow repository, though I made a number of modifications to support a different accelerometer breakout, and to add a bit of Particle RGB LED flair at the end. You may not actually be a magician by the end of this post, but youā€™ll certainly feel like one!

Parts and tools

If you want to build one of these wands for yourself, youā€™ll need the following components.

The Problem Weā€™re Trying to Solve

One of the reasons that ML on MCUs is such a hot topic these days is because of the amount of data for some solutions can be a deterrent to performing inferencing in the cloud. If youā€™re working with a high-precision sensor and you need to make a decision based on raw data, you often cannot shoulder the cost of sending all that data to the cloud. And even if you could, the round trip latency runs counter to the frequent need for real-time decision-making.

By performing prediction on a microcontroller, you can process all that raw data on the edge, quickly, without sending a single byte to the cloud.

For this post, Iā€™ll illustrate this with an example that works with a common IoT device: a 3-axis accelerometer, which is connected to a Particle Xenon. The Xenon will use streaming accelerometer data and TensorFlow Lite for Microcontrollers to determine if the person holding the device is making one of 3 gestures:

  • A clockwise circle, or ā€œringā€ gesture;
  • A capital W, or ā€œwingā€ gesture;
  • A right to left ā€œslopeā€ gesture.

Hereā€™s a GIF sample of yours truly performing these gestures.

If one of the three gestures is detected, the onboard RGB LED will light up green, blue, or red, depending on the gesture performed.

Using an Accelerometer Gesture Detection Model

For this post, Iā€™ll be using a pre-trained model thatā€™s included in the example projects for the TensorFlow Lite library. The model, which was created by the TensorFlow team, is a 20 KB convolutional neural network (or CNN) trained on gesture data from 10 people performing four gestures fifteen times each (ring, wing, slope, and an unknown gesture). For detection (inferencing), the model accepts raw accelerometer data in the form of 128 X, Y, and Z values and outputs probability scores for our three gestures and an unknown. The four scores sum to 1, and weā€™ll consider a probability of 0.8 or greater as a confident prediction of a given gesture.

The source for this post can be found in the examples folder of the TensorFlow Lite library source. The model itself is contained in the file magic_wand_model_data.cpp, which, as I discussed in my last post, is a C array representation of the TFLite flatbuffer model itself. For MCUs, we use models in this form because we donā€™t have a filesystem to store or load models.

Configuring the Magic Wand Application

The entry-point of my application is in the magic_wand.cpp, which contains the setup and loop functions weā€™re used to. Before I get there however, Iā€™ll load some TensorFlow dependencies and configure a few variables for our application. One of note is an object to set aside some memory for TensorFlow to store input, output and intermediate arrays in.

constexpr int kTensorArenaSize = 60 * 1024;
uint8_t tensor_arena[kTensorArenaSize];

The kTensorArenaSize variable represents the number of bytes to set aside in memory for my model to use at runtime. This will vary from one model to the next, and may need to be adjusted based on the device youā€™re using, as well. I ran this example on a Particle Xenon with the above values, but noticed that the same value on an Argon or Boron tended to cause my application to run out of memory quickly. If you see red SOS errors from your onboard RGB LED when running a TensorFlow project, try adjusting this value first.

Once Iā€™ve configured the memory space for my app, Iā€™ll load my model, and configure my operations resolver. The latter step is notable because itā€™s a bit different than the hello_world example I used in my last post.

static tflite::MicroMutableOpResolver micro_mutable_op_resolver;
micro_mutable_op_resolver.AddBuiltin(
    tflite::BuiltinOperator_DEPTHWISE_CONV_2D,
    tflite::ops::micro::Register_DEPTHWISE_CONV_2D());
micro_mutable_op_resolver.AddBuiltin(
    tflite::BuiltinOperator_MAX_POOL_2D,
    tflite::ops::micro::Register_MAX_POOL_2D());
micro_mutable_op_resolver.AddBuiltin(
    tflite::BuiltinOperator_CONV_2D,
    tflite::ops::micro::Register_CONV_2D());
micro_mutable_op_resolver.AddBuiltin(
    tflite::BuiltinOperator_FULLY_CONNECTED,
    tflite::ops::micro::Register_FULLY_CONNECTED());
micro_mutable_op_resolver.AddBuiltin(
    tflite::BuiltinOperator_SOFTMAX,
    tflite::ops::micro::Register_SOFTMAX());

In the hello_world example, we used the tflite::ops::micro::AllOpsResolver, which just loads all available operations into the application. For large models, free memory will be at a premium, so youā€™ll want to load only the operations that your model needs for inferencing. This requires an understanding of the model architecture. For the magic_wand example, I added the operations above (DEPTHWISE_CONV_2D, MAX_POOL_2D, etc.) because those reflect the layers of the CNN model Iā€™m using for this application.

Once everything is configured on the TensorFlow side, itā€™s time to set-up an accelerometer and start collecting real data.

TfLiteStatus setup_status = SetupAccelerometer(error_reporter);

This function lives in the particle_accelerometer_handler.cpp file. True to its name, it configures the accelerometer Iā€™m using for this application. When creating this example, I used the Adafruit LSM9DS1, which contains a 3-axis accelerometer, a gyroscope, and a magnetometer.

To interface with the accelerometer, Iā€™ll use a library. There are a couple of LSM9DS1 libraries available, but this particular application requires the use of the FIFO buffer on the LSM9DS1 for storing accelerometer readings, so for this example I am using the LSM9DS1_FIFO library.

#include <LSM9DS1_FIFO.h>
#include <Adafruit_Sensor.h>

LSM9DS1_FIFO accel = LSM9DS1_FIFO();

TfLiteStatus SetupAccelerometer(tflite::ErrorReporter *error_reporter)
{
while (!Serial) {}

if (!accel.begin())
{
error_reporter->Report(ā€œFailed to initialize accelerometer. Please resetā€);
return kTfLiteError;
}

accel.setupAccel(accel.LSM9DS1_ACCELRANGE_2G);
accel.setupMag(accel.LSM9DS1_MAGGAIN_4GAUSS);
accel.setupGyro(accel.LSM9DS1_GYROSCALE_245DPS);

error_reporter->Report(ā€œMagic starts!ā€);

return kTfLiteOk;
}

Once Iā€™ve installed the library, configuration is as simple as calling accel.begin(). With that done, Iā€™m ready to start taking readings.

Reading from the accelerometer

Each time through the loop, Iā€™ll grab data from the accelerometer by calling the ReadAccelerometer function.

bool got_data = ReadAccelerometer(error_reporter, model_input->data.f,
                                  input_length, should_clear_buffer);

This function, which can also be found in the particle_accelerometer_handler.cpp file, reads from the FIFO buffer on the accelerometer one sample at a time. It then downsamples data from 119 Hz, the rate that my accelerometer captures data, to about 25 Hz, the rate that the model was trained on. Next, Iā€™ll assign the x, y, and z values to an array.

while (accel.accelerationAvailable())
{
  accel.read();

sensors_event_t accelData, magData, gyroData, temp;
// Read each sample, removing it from the deviceā€™s FIFO buffer
if (!accel.getEvent(&accelData, &magData, &gyroData, &temp))
{
error_reporter->Report(ā€œFailed to read dataā€);
break;
}

// Throw away this sample unless itā€™s the nth
if (sample_skip_counter != sample_every_n)
{
sample_skip_counter += 1;
continue;
}

save_data[begin_index++] = accelData.acceleration.x * 1000;
save_data[begin_index++] = accelData.acceleration.y * 1000;
save_data[begin_index++] = accelData.acceleration.z * 1000;

sample_skip_counter = 1;
// If we reached the end of the circle buffer, reset
if (begin_index >= 600)
{
begin_index = 0;
}
new_data = true;

break;
}

This function will run each time through the loop, accumulating readings until we have enough data to perform a prediction, which is around 200 x, y, and z values. After Iā€™ve collected those, I set all the values on the input tensor of my model and Iā€™m ready to make a prediction!

for (int i = 0; i < length; ++i)
{
  int ring_array_index = begin_index + i - length;
  if (ring_array_index < 0)
  {
    ring_array_index += 600;
  }
  input[i] = save_data[ring_array_index];
}

Real-time inferencing with accelerometer data

With a set of accelerometer values in my model input, I can make a prediction by invoking the TFLite interpreter. Notice that the snippet below is identical to the hello world example from my last post. Setting inputs and reading outputs will differ from one TFLite application to the next, but the invocation process follows a consistent API.

TfLiteStatus invoke_status = interpreter->Invoke();

if (invoke_status != kTfLiteOk)
{
error_reporter->Report(ā€œInvoke failed on index: %d\nā€, begin_index);
return;
}

Once Iā€™ve made a prediction, I work with the output and determine if a gesture was found.

int gesture_index = PredictGesture(interpreter->output(0)->data.f);

The PredictGesture function takes the values from my output tensor and determines if any value is over the 80% confidence threshold Iā€™ve set and, if so, sets that value as the current prediction. Otherwise, an unknown value is set.

int this_predict = -1;
  for (int i = 0; i < 3; i++) { if (output[i] > 0.8) this_predict = i;
}

// No gesture was detected above the threshold
if (this_predict == -1) {
continuous_count = 0;
last_predict = 3;
return 3;
}

In addition to interpreting the current prediction, weā€™re also tracking some state across predictions. Since accelerometer data is highly variable, we should make sure that our model predicts the same gesture multiple times in a row before we formally decide that weā€™re happy with the prediction. As such, the final portion of PreductGesture tracks the last and current prediction and, if the prediction has occurred a given number of times in a row, weā€™ll report it. Otherwise, weā€™ll report that the gesture was unknown. The kConsecutiveInferenceThresholds is an array of integers that correspond to the number of consecutive predictions we want to see for each gesture before considering a prediction to be valid. This may differ for your project and accelerometer. The values I chose can be found in the particle_constants.cpp file.

if (last_predict == this_predict) {
  continuous_count += 1;
} else {
  continuous_count = 0;
}

last_predict = this_predict;

if (continuous_count < kConsecutiveInferenceThresholds[this_predict]) {
return 3;
}

continuous_count = 0;
last_predict = -1;
return this_predict;

Now that we have a result, we can do something with it.

Displaying the result

At the end of my loop, with a predicted gesture in hand, I call a function called `HandleOutput` to display something to the user. This function is defined in the particle_output_handler.cpp file and does three things:

  • Toggles the D7 LED each time an inference is performed;
  • Outputs the predicted gesture to the Serial console;
  • Sets the onboard RGB Led green, blue, or red, depending on the predicted gesture.

The first step is pretty straightforward, so I wonā€™t cover it here. For the second, weā€™ll print a big of ASCII art to the console to represent each gesture.

if (kind == 0)
{
  error_reporter->Report(
      "WING:\n\r*         *         *\n\r *       * *       "
      "*\n\r  *     *   *     *\n\r   *   *     *   *\n\r    * *       "
      "* *\n\r     *         *\n\r");
}
else if (kind == 1)
{
  error_reporter->Report(
      "RING:\n\r          *\n\r       *     *\n\r     *         *\n\r "
      "   *           *\n\r     *         *\n\r       *     *\n\r      "
      "    *\n\r");
}
else if (kind == 2)
{
  error_reporter->Report(
      "SLOPE:\n\r        *\n\r       *\n\r      *\n\r     *\n\r    "
      "*\n\r   *\n\r  *\n\r * * * * * * * *\n\r");
}

It doesnā€™t look like much here, but if you run the example and perform a gesture, youā€™ll see the art in all its glory.

For the final step, weā€™ll do something unique to Particle devices: take control of the onboard RGB LED. The onboard LED is used by Particle to communicate device status, connectivity, and the like, but itā€™s also possible to take control of the LED and use it in your own applications. You simply call RBG.control(true), and then RGB.color to set an RGB value. For this example, I modified the ASCII art snippet above to set my LED, and release control with RGB.control(false) if a gesture isnā€™t detected.

if (kind == 0)
{
  RGB.control(true);
  RGB.color(0, 255, 0); // Green

error_reporter-&gt;Report(
ā€œWING:\n\r* * \n\r * * * "
"
\n\r * * * *\n\r * * * \n\r * * "
"
*\n\r * *\n\rā€);
}
else if (kind == 1)
{
RGB.control(true);
RGB.color(0, 0, 255); // Blue

error_reporter-&gt;Report(
ā€œRING:\n\r *\n\r * *\n\r * *\n\r "
" * *\n\r * *\n\r * *\n\r "
" *\n\rā€);
}
else if (kind == 2)
{
RGB.control(true);
RGB.color(255, 0, 0); // Red

error_reporter-&gt;Report(
ā€œSLOPE:\n\r *\n\r *\n\r *\n\r \n\r "
"
\n\r *\n\r *\n\r * * * * * * * *\n\rā€);
}
else
{
RGB.control(false);
}

Ā 

Now, in addition to fancy ASCII art, your device will change the RGB LED when a gesture is detected.

Taking your exploration further

In this post, we took our exploration of TensorFlow Lite into the realm of IoT with an accelerometer demo. Now itā€™s your turn. If you havenā€™t already, check out the TensorFlowLite Particle library and explore some of the other demos. And if youā€™re building your own ML on MCUs project with TensorFlow, share it in the comments below!

3 Likes

Brandon - great tutorial, this one I really want to try out - a bit of wizarding magic!

1 Like

Brandon, I just tried to compile the source code against a Xenon with 1.4.2 and I got a lot of warnings (42) and 2 errors.
The first error is Insufficient room for heap. - Is this related to kTensorArenaSize?
The second error is: undefined reference to `kConsecutiveInferenceThresholdsā€™ which weirdly when I go to the definition - finds it in particle_constants.h which if I click on it shows that there are 2 definitions?

Many thanks if you can help me understand these 2 errors.

Errata - just realised that I had accidentally set device OS as 1.4.3 so now donā€™t get error 1 but still see error 2 above,

If both these arrays share the same scope they cannot be called the same and particularly not initialised differently.
Iā€™d say one of these definitions should be extern without initialisation.
Secondly when the first declaration fails any reference to this undefined variable will also be undefined.

Having slept on this I realised that the project.properties was pulling in all the TensorFlowLite/examples which includes the magic_wand files that Brandon has modified from Arduino to Particle. Having removed these and re compile I am still getting an error

In function `PredictGesture(float*)':
/Users/x/Documents/x/VSC_Projects/magic_wand/magic_wand//src/gesture_predictor.cpp:52: undefined reference to `kConsecutiveInferenceThresholds'

This function includes this #include "constants.h" before if (continuous_count < kConsecutiveInferenceThresholds[this_predict]) {

If I look in constants.h it has this extern const int kConsecutiveInferenceThresholds[3];

And in particle_constants.h this

#include "constants.h"

// The number of expected consecutive inferences for each gesture type.
// Established with the Particle Xenon.
// 0 = Wing
// 1 = Ring
// 2 = Slope
const int kConsecutiveInferenceThresholds[3] = {5, 12, 6};

Is it possible that when this was called arduino_constants.h and came before the constants.h file in the /src directory that it processed these the correct way? PS. I donā€™t understand how the same variable can be re-declared - hence request to @bsatrom - because this doesnā€™t seem to work as is.

Update: I have now compiled the magic_wand.cpp. To do this I needed to insert a #include for the particle_constants.h as shown below:

#include "main_functions.h"

#include "particle_constants.h"

Hope this is of any interest/help to anyone else trying to experiment with this.

The D7 LED is flashing to show inferencing is being performed but thus far I have managed only one gesture recognition with output. Getting thereā€¦

I have added an output for when no gesture is recognised because I was wondering why nothing was happening. I am struggling to get any gestures recognised, havenā€™t been able to get an O ring recognised and W wing once plus several Z slopes - see below

Did not recognise that gesture 

SLOPE:
        *
       *
      *
     *
    *
   *
  *
 * * * * * * * *

Did not recognise that gesture
 Did not recognise that gesture 

WING:
*         *         *
 *       * *       *
  *     *   *     *
   *   *     *   *
    * *       * *
     *         *

I guess this points to needing to train my specific wand - or does the wand select its owner?

1 Like

Thanks, @bsatrom

I love seeing these ML examples.

Itā€™s pretty cool seeing what is possible on these cheap connected microcontrollers today.

1 Like

Thanks @RWB!

@armor, glad to see you taking it for a spin! The error you saw is odd, but Iā€™ll dig deeper in the morning. Definitely not something Iā€™m seeing on my end, but perhaps I omitted something from source. Iā€™ll let you know!

I did try turning the LSM9DS1 around on the breadboard to see if that worked - it didnā€™t - I then got no gestures recognised.

Maybe I have misunderstood the last_predict and this_predict - kConsecutiveInferenceThresholds[] is essentially meaning that I need to repeat the gesture a number of times to be certain that it is the gesture in question? These repetitions seem to be very high and much higher than the numbers in the arduino example - was there a reason why you used these numbers?

I set those to those levels because I was seeing false positives if the numbers where any lower. Feel free to tune them to something that works for you. Youā€™ll also want to make sure to perform large gestures over the course of about one second, and make sure the dot on the LSM9DS1 is at the top left when you perform the gesture.

The base model here was trained on a pretty small sample size. I have asked the Google folks for access to the training scripts and will share them if you want to add your own training sample to the set to improve the performance of the model

Youā€™ll also want to make sure to perform large gestures over the course of about one second

I see from the animated giff that you appear to be concentrating pretty - is this why?

make sure the dot on the LSM9DS1 is at the top left when you perform the gesture.

Is this how you mean?

It really needs a lot more training so if you could share the training model that would be great. I will endeavour to post back what I can do to improve the operation. I really would like to build the xenon and IMU plus a battery into the handle of a wand or maybe a light sabre.

It is early days of tinyML, after all. :smiley:

I will certainly share the training scripts once I get ahold of them. In the meantime, if you notice anything amiss in the demo that needs fixing, Iā€™m happy to take PRs on the library!

Hey @armor, good news! I just published an update (v0.1.1) of the TensorFlowLite library that should result in better predictions. Try it out and once loaded, hold the LSM9DS1 breakout away from you with the Xenon USB port towards you to perform the gestures. I am seeing the Wing (X) pretty reliably, as well as the slope (/_), with the ring (O) being the least reliable, which does jive with what the Google team told me was true of the training model.

Speaking of which, the TensorFlow team just released the scripts for performing your own model training yesterday and you can find them here: https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/experimental/micro/examples/magic_wand/train. I am getting runtime errors when using the model (which is 2x smaller!) trained using this approach, and have an email out to the team asking for pointers.

As for collecting your own data for training new or fine-tuning gestures, Iā€™ll be working on some samples for doing that with the Xenon next week. Iā€™ll share my progress here.

Let me know how things work for you once you can try this out again!

Thanks. Perhaps you need to clarify that the original 2 libraries are no longer needed:

dependencies.LSM9DS1_FIFO=1.0.0
dependencies.Adafruit_Sensor=1.0.2

And that this library is needed

dependencies.Arduino_LSM9DS1=1.0.0

Once I had changed all these then it compiled without error or warning.

In terms of results I am getting mostly green LED = Wing but often when I would say it should be a slope. I have noticed that immediately after a Green when holding it still I can get a Blue or a Red. I think I managed a Ring properly once. Isnā€™t part of the ā€˜trickā€™ to know when to start treating the signal stream from the IMU as a gesture movement and the same with the end? I am trying to hold it still after the LED changes from a colour back to White.

I also noticed that I need to hold the breadboard with the boards parallel to me rather than pointing away for it to work at all and it works better if the IMU is on the right of the Xenon. I guess this was how it was trained?

Last observation - is the inferencing working faster? D7 LED appears to me to be flashing a bit faster.

Thanks again - feels like it is getting there.

Looking at the setup used to train this model it was with a sparkfun edge development board - this has a different accelerometer (LIS2DH12 3-axis accelerometer) than the one you selected here for this - could this have a bearing on the accuracy of prediction?

I might have a LIS2DH12 so I am thinking it is worth trying out.

No, that shouldnā€™t impact accuracy. The differences between those specific accelerometers have already been accounted for in firmware.

@bsatrom Hello, looking at the github page it shows that the library is compatible with the P1/photon. However this example does not mention that, is this example compatible with P1/Photon?

Hey @muneebr1, the library is compatible, yes, but Iā€™ve not explicitly tested this example with the Photon/P1, so I canā€™t say. The major constraint will be model size. If the model fits, it should work.

Hi @bsatrom ā€¦ iā€™ve created my own three models for different movements, but when i run magic_wand after updating magic_wand_model_data.cpp file, the following errors are reported:

Didn't find op for builtin opcode 'CONV_2D' version '2'

Invoke failed on index: 0

Didn't find op for builtin opcode 'CONV_2D' version '2'

Invoke failed on index: 3

Didn't find op for builtin opcode 'CONV_2D' version '2'

Invoke failed on index: 6

Didn't find op for builtin opcode 'CONV_2D' version '2'

Invoke failed on index: 9

Didn't find op for builtin opcode 'CONV_2D' version '2'

Invoke failed on index: 12

Didn't find op for builtin opcode 'CONV_2D' version '2'

Invoke failed on index: 15

Didn't find op for builtin opcode 'CONV_2D' version '2'

Invoke failed on index: 18

Didn't find op for builtin opcode 'CONV_2D' version '2'

Invoke failed on index: 21

Didn't find op for builtin opcode 'CONV_2D' version '2'

Invoke failed on index: 24

ā€¦ and so on. Does anyone have a clue about whatā€™s the problem?. any help will be appreciated. Thanks.