Deep Learning with Raspberry Pi and MATLAB

Thank you, everybody, for joining this webinar. My name's Meeshawn Marathe, product manager for Raspberry Pi and Arduino hardware support packages at MathWorks. And today I'm going to talk about deep learning with Raspberry Pi and MATLAB.

In continuation with the webinar series on deep learning with Raspberry Pi from last year, this time I'm going to introduce you to a new improvised workflow to deploy your deep learning applications on Raspberry Pi hardware which will make the deployment process a lot easier.

And to demonstrate this, I'll take the help of an object classification example and deploy it on my Raspberry Pi 4 hardware. If you have a Raspberry Pi, at the end of this webinar, you should be able to get started with deep learning right away. All right, so let's get started.

So here's the agenda for today. I'll begin with a quick demonstration of the deep learning application we intend to deploy on the Raspberry Pi. Next, we'll take a short tour on the deep learning capabilities within MATLAB and also take a look at a typical workflow that you would employ for deploying deep learning applications on the Pi.

From there, we'll move on to the new capabilities available for deploying deep learning applications on the Pi, starting in release 2020a. This will be followed by a detailed walkthrough of our object classification example in MATLAB. And then, finally we'll deploy the application on the Pi and see it in action. With that, let's straightaway jump to our demo.

So this video captures the deep learning object classification example that is running on the Raspberry Pi. In the background, the application is continuously acquiring images with the help of a USB webcam that is connected to the Pi, passing it through a trained deep learning algorithm configured in MATLAB, and then displaying the object predictions over the images in the Pi itself.

You might have noticed that the accuracy was pretty impressive for some of the objects like the bell pepper, loud speaker, and the mouse. The speed might not appear that great since we are processing the algorithm at one frames per second. And we need to understand here that we are not dealing with any GPUs. It's just the CPU of the Raspberry Pi. And it's able to acquire the image, classify, and then display the result, all in under one You could also use other trained neural networks, which are much faster when deployed on embedded devices.

So this is the shipping demo that we will be referring to in this webinar. You can refer to this example using the link given below. This example also comes integrated along with the MATLAB Support Package for Raspberry Pi. It uses the ResNet-50 neural network, which is a pretrained neural network supported by MathWorks to identify the objects captured by the webcam that is connected to the Pi.

So now, let's see how MATLAB enabled us to deploy the deep learning algorithm on the Raspberry Pi. We're going to talk about a deep learning workflow, a simplified view results in four main stages-- data preparation, algorithms and modeling, simulation and testing, and ultimately, deployment into the field. We'll go through all four parts of the workflow, talk about some of the challenges within each stage, and point out how MATLAB can make it all come together efficiently and effectively to deliver value to you.

Data preparation and management turns out to be one of the most critical ingredients to success. And, it's really, really hard. Today, data comes from multiple sensors and databases, data structured or unstructured, the span in time with different time levels, they are quite noisy, they come from different domains, and so forth. The data needs to be prepared for the next stage, that is the algorithms and modeling stage, and it's time consuming.

A great proofpoint here. Andrej Karpathy talked about this in 2018. While doing his work at Stanford, he did not need to spend much time on the data. He was doing research and focused almost exclusively on creating new AI algorithms.

But as director of AI at Tesla, he now spends three-quarters of his time on the data because now he is building a car, not a better performing predictor. You may find that you may end up in the same situation, spending an enormous amount of time inside the data. MATLAB and Simulink can make that time more effective and more productive.

And it's beyond just data manipulation. Data labeling is required for any kind of supervised learning algorithm. And this can be time-consuming and error prone. For deep learning, MATLAB provides data labeling apps like the Image Labeler app to help you use techniques such as semantic segmentation.

Within it, regions of an image are classified. And then, that classification is automatically applied correctly through every frame of a video. We have other labeling apps too all of which can generate code for reuse.

Data preparation also includes preprocessing the input data before passing into the deep learning network. For the Raspberry Pi hardware, additional preprocessing functions are available to assist with this data preparation stage statically on the hardware.

We use these functions in our code to acquire, resize, and display images directly on the Pi. We'll take a closer look at this later in this presentation. Let's move on to the next step in the workflow, the one that perhaps gets the most attention. That's AI-based algorithms and models.

Within the MATLAB environment, you have direct access to common algorithms used for classification and prediction, from regression to deep learning, to clustering. And within MATLAB, you can easily use prebuilt models developed by the broader community, such as ResNet-50 for classification or YOLO for object recognition.

While algorithms and prebuilt models are a good start, it's not enough. Examples are the way engineers learn how to use algorithms and find the best approach for their specific problem. We provide a host of examples for building deep learning models in a wide range of domains and we have over 200 of them in MATLAB.

I mentioned apps for data preparation. We also provide apps for modeling. For deep learning, we ship prebuilt apps that help automate the training step and provide visualizations for understanding and editing deep networks.

So here's a short video on deep network design where you can access pretrained models, visualize an imported model, create a model from scratch, analyze the model, import the data, set the right training options, and then, ultimately, train the network. Finally, you can export and generate code.

In the latest release, there is an app now to manage experiments where you can explore hyperparameters, monitor training progress, define experiments with MATLAB code, keep track of your work, and finally, export the model results.

Deep learning training can take a long time. With our cloud support and third-party integrations, you can now move the training to the appropriate computer platform, be it a GPU, public cloud, private cloud, and so forth. The point is that a single model in MATLAB can be moved to the appropriate computer platform, saving an enormous amount of development time, and allowing you to leverage the right computer technology to accelerate training.

MathWorks is dedicated to improving the performance of deep learning in MATLAB. We have a process in place to measure and track the performance. This enables us to focus our development efforts in the right areas. The following highlights some of the results of these efforts in the past few releases.

For our demo, we'll be running the pretrained ResNet-50 neural network on the Raspberry Pi, which provides decent accuracy at around one frame per second. Being a low-cost portably hobbyist device, the Raspberry Pi might be limited in the type of deep networks that you can practically run on this hardware.

You will need to experiment with different AI models and the right hardware specs to meet your performance requirements. We also know that the broader deep learning community is incredibly active and prolific. And new models are coming out all the time. Because we support the ONNX standard, you have access to those new models and can work with them within the MATLAB environment.

To summarize the modeling step within the MATLAB environment, you can have access to all of the algorithms and methods used to develop models, can work with them at the code level or through an app, and can accelerate training to the appropriate computer platform and can engage with the AI community through ONNX.

Let's move on to step three, simulation and testing. Deep learning models have to be incorporated into a larger system to be useful. Consider automated driving. There's a lot of algorithm work around perception systems, designing perception algorithms that can collect data from multiple sensors, and fuse those data into an understanding of what surrounds the vehicle.

But for those algorithms to be effective, they have to integrate with algorithms for localization, path planning, agented environment management, and controls, all working together to make sure the vehicle accelerates, decelerates, brakes, and steers perfectly in any situation. These can all be accomplished with an overall system simulation that handles not just the perception algorithm, but the entire vehicle itself, other vehicles, and actors, such as pedestrians, across multiple driving scenarios, in multiple weather and environmental conditions, all handled through simulation based in MATLAB and Simulink.

With MATLAB release 2020b, there is now added support for deep learning in Simulink. For deployment, we've got a unique code generation framework that allows models developed in MATLAB to be deployed, anywhere, without having to rewrite the original model. This gives you the ability to test and deploy the entire system and not just the model itself.

MATLAB can generate native code optimized for multiple frameworks. It gives you the flexibility to deploy to lightweight, low-powered embedded devices, such as those used in a car, low-cost rapid prototyping boards, such as Raspberry Pi, edge-based IoT applications such as a sensor and controller on a machine in a factory, and so on and so forth.

MATLAB can also deploy to desktop or server environments, which can allow you to scale from desktop executables, cloud-based enterprise systems on AWS or Azure. Automatic code generation eliminates coding errors and is an enormous value driver for any organization adopting it.

The power and flexibility of our code generation and deployment frameworks is unmatched. To help you understand how we got to where we are today, let's take a look at the MathWorks evolution of deep learning support. Does anyone know when MATLAB first had the ability to train neural networks? The answer is in the mid-'90s, with the initial release of the Neural Network Toolbox.

When did deep learning first came to MATLAB? A popular framework, MatConvNet from Oxford Visual Geometry Group came out in 2014. But MathWorks first officially supported it in 2016. Since then, we've been releasing a stream of major updates. You may notice that some of these updates relate to the broader area of deep learning. Others, like our apps, focus on productivity and some on deployment.

Many ask what's coming next in MATLAB. The answer is, the field is constantly evolving, and in general, this part is difficult to predict. However, you may see that each year, the number of updates in MATLAB have increased over the year prior. MathWorks is committed to tackling this growing and ever-evolving field through our increased development efforts.

One item that I'd like to especially point out is that we now have over 200 examples across multiple domains that can help serve as a great starting point for users looking to apply deep learning in their area of expertise. So as we saw, MATLAB supports deployment of deep neural networks to embedded devices, GPUs, and CPUs.

However, in this webinar, we will be focusing on deployment to Raspberry Pi hardware, as it's powerful, easily accessible, yet inexpensive hardware to get started with prototyping your deep learning algorithms. Additionally, there's a huge community out there which is extensively using Raspberry Pi to prototype a lot of different ideas and applications.

To begin with, I'll be importing and configuring a pretrained neural network in MATLAB, which is the ResNet-50 neural network. In the next step, I'll be creating an application that will talk to the neural network to provide input and obtain and process the classification results. And as you will see later on, the MATLAB Support Package for Raspberry Pi Hardware assists you in designing this stage and prototype it as well for the Raspberry Pi hardware. Finally, I'll show you how to deploy the application on the Pi.

All right. Next, let's take a look at some of the new capabilities available in MATLAB for deep learning using Raspberry Pi starting from MATLAB release 2020a. A new support has been added for Raspberry Pi 4 Model B board. It's more powerful since the onboard RAM can go all up to 8 GB. And this is definitely a big boost for running deep learning applications on the Raspberry Pi.

The next update, and the highlight of today's presentation, is the new C++ code gen capability for all the Raspberry Pi functions in MATLAB. On the right, you can see a list of Raspberry Pi functions that are supported for deployment on the Pi. C++ code can now be directly generated for these functions, which is an important requirement for running deep learning applications on the embedded devices.

Earlier, one had to write custom C++ code to talk to the deep neural networks running on the Raspberry Pi. These were required in order to acquire images from the camera and overwrite them with the production values. But now, the Raspberry Pi functions in MATLAB responsible for this can be directly converted into C++ code, and can be used on the Pi. This is an important step since it saves a lot of time and effort that goes into writing custom codes.

With this, we are now ready to explore the working details of the demo. But before that, let's quickly take a look at the prerequisites. In terms of MathWorks product requirements, you will need Deep Learning Toolbox, MATLAB Coder, and Computer Version Toolbox.

You will also require some additional support packages for Raspberry Pi and deep learning purposes. Let's take a look at them. For working with Raspberry Pi, you will need the MATLAB Support Package for Raspberry Pi. You can download and install it from the Add-Ons menu in MATLAB. And once installed, you would need to configure it for deep learning applications.

And as you can see here, within the Add-Ons menu, you can click on the gear icon against the Raspberry Pi support package. After selecting your Raspberry Pi model, next, you will need to download the MathWorks Raspbian Image customized for deep learning. It's customized to run deep learning applications on the Pi.

The major advantage of this step is that it installs the ARM Compute Library, which is an important requirement for deep learning code gen and deployment on the Pi. Earlier, one had to manually compile and install the library and then set the environment variables for it. All of this is now automated by just downloading and installing the customized image for deep learning.

Moving on, the other support packages that you will need are the MATLAB Coder Interface for Deep Learning Libraries and the Deep Learning Toolbox Model for ResNet-50, since we are using this neural network in the demo.

All right, so now we are all set to move on to the implementation details of our demo. Here's the hardware setup that I'll be using. I'm using a Raspberry Pi 4 Model B board. You can use Raspberry Pi 3 models as well.

However, it's recommended that you use RaspPi4 for deep learning applications due to its faster processor and bigger onboard RAM. You can connect the Pi to your computer using an ethernet cable or you could also connect it to a wireless network, the same network that your computer is connected to.

And then, I have a Microsoft USB webcam, which is connected to the Pi. And of course, you need a computer running MATLAB. As mentioned earlier, this example uses ResNet-50, which is a pretrained neural network supported by MathWorks.

It's trained to over a million images from the image database. And it's got 50 layers capable of classifying images into 1,000 different object categories. You can take a look at this documentation for more details.

Now, you could either use this network as it is or apply transfer learning on a new data subset for your own custom labels. Take a look at this documentation for transfer learning. It helps you get started with retraining an existing convolutional neural network to classify a new set of images.

On a high level, the workflow looks something like this. Initially, you load the new images and the pretrained neural network. Next, you will replace the final layers of the network with new layers adapted with a new data set. You can optionally freeze the rates of the earlier layers in the network by setting the learning rates in those layers to be zero. This can speed up network training since these parameters need not be computed. Finally, train the network with a new data set.

All right. So now let's dig into the demo workflow. Over here, is a step-by-step workflow to deploy any MATLAB function on Raspberry Pi. You can access this space using the link below. Now, for deploying deep learning applications, there is an additional step of configuring a Deep Learning Object and assigning it to the Hardware Configuration Object for Raspberry Pi. Let's dive into these steps one by one.

Here's the documentation for the demo. Initially, you will need to create a successful connection to the Pi from MATLAB. Let's check this out in MATLAB.

Pass the arguments for the IP address, username, and password, and then execute this command. Your IP address might be different from the one used here. The username and password are default values. Great. So the connection was successful. You can now clear this object from the workspace.

Next, we'll open the object classification example in MATLAB. In order to do this, you will need to execute this command. This opens up the MATLAB function.

As you can see here, we are initially creating a Raspberry Pi connection from MATLAB and a camera object, which represents the camera connected to the Pi. For my demo, I replaced the argument here with 2, and saved it as a separate MATLAB function. This is because the argument 1 corresponds to the default Raspberry Pi camera board option, whereas I'm using a Microsoft webcam connected to the Pi's USB hub. You can verify this by executing these commands.

As you can see here, the option 2 corresponds to Microsoft webcam connected to the USB. All right, so next, we'll be importing the trained ResNet network in MATLAB. And we'll be setting the size of the input images. Finally, we'll be running an infinite while loop.

Here, we'll be continuously taking snapshots using the webcam, resize the image according to the neural network's input size requirements, classify the image, and then assign a label, which secures the maximum prediction score. The label, along with the prediction score, is overwritten on the image. All of this happens in a span of one second.

The function Insert Text is a part of the Computer Version Toolbox and Display Image belongs to the Raspberry Pi Support Package. As we can see here, the preprocessing step, which is acquiring and resizing the input images, and the post-processing step of displaying the image and overriding it with classification results.

Both these operations can be done here in MATLAB with the help of the code gen capable functions from MATLAB Support Package for Raspberry Pi and Computer Vision Toolbox. Hence, no additional handwritten C++ code is required for these steps. Next, we can quickly run this algorithm in MATLAB to verify the performance before deploying it on the Pi. Oops. Let's clear the display table.

As you can see here, the algorithm is able to classify different objects with good accuracy.

Now, this code already has the compilation directive added here, which lets MATLAB know that we intend to deploy the MATLAB function so that it can check for any code gen violations or potential errors in the script. Next, we'll use two important functions for deploying the algorithm on the Pi.

First, we'll create a hardware configuration object for the Raspberry Pi. Next, we'll set the language of the generated code to C++. Now comes the additional step for deployment of deep learning applications.

We will need to create an arm-compute deep learning configuration object, specify the architecture of the Raspberry Pi, and the version of the library. Now, we will attach this to the hardware configuration object using this command. All right. So now we are ready to deploy.

That's it. So the function will now be deployed on the Raspberry Pi hardware for standalone execution. A coding report will be generated to present the code generation success results. Since this usually takes a few minutes, we'll skip this part and directly jump to the code gen results. All right. Code generation is successful.

Now comes the interesting part. We will now run the deployed executable that we just downloaded on the Raspberry Pi hardware. We will do so by remotely connecting to the Raspberry Pi hardware. You could also connect a display device to the Pi, and then run the executable file. Note that you can set the location of the generated executable using the same hardware configuration object.

The default location is home/pi/MATLAB_ws. Another interesting point to note here is that MATLAB will generate the same folder structure within this directly that mimics the action working directly in MATLAB. So here's the executable file, generated and saved using MATLAB. We will now open Terminal, navigate to this directory, and then run the executable file.

So now you are seeing a live feed from the webcam connected to the Pi along with the prediction scores. The classification is happening at one frame per second, with a pretty good accuracy for some of the objects.

All right, so now it's time to wrap up this seminar. Today, we looked at a typical deep learning workflow in MATLAB. From there, we went on to see some of the newer capabilities in MATLAB for deploying deep learning applications on the Pi.

The three major highlights were the customized Raspbian image for deep learning, the new C++ code gen capability, and the newly added support for Raspberry Pi 4 Model B board. All of which contributed to an improved and simpler workflow to deploy deep learning applications on the Pi. And then, we demonstrated this new improved workflow with the help of an object classification example employing a trained ResNet-50 neural network. And thus, we saw how easy it is to deploy deep learning applications on the Raspberry Pi using MATLAB.

Here are some resources that you might find useful. This is the official website of MathWorks. You can click on the deep learning button to find a bunch of different resources to help you get started with designing, training, evaluating, and deploying your deep learning applications using MATLAB.

This is a documentation link for deep learning with MATLAB Coder. You can find a bunch of different deep learning examples to experiment with and get started in MATLAB.

This is a documentation link for a bunch of different reference examples that can be deployed on the Raspberry Pi. These examples range from computer version, image processing, to deep learning applications.

Finally, to learn more about deep learning, you can take this Deep Learning Onramp tutorial for free by visiting this link. With the help of these resources and from what we have learned from today's webinar, you are now ready to get started with deploying your deep learning applications on your Raspberry Pi. That's all I have for today's webinar. Thanks, everyone, for your time and attention.