Raspberry Pi, TensorFlow Lite and Qt/QML: object detection example

Helpful? spread the word!

• Author
• Recent Posts

Javier Bonilla

Researcher at CIEMAT - PSA

PhD. in Computer Science / Solar Thermal Energy Researcher

Latest posts by Javier Bonilla (see all)

• Flutter on Raspberry Pi with flutter-pi - 26 November, 2019
• ESP8266 NodeMCU pinout for Arduino IDE - 19 November, 2019
• Cross-compile and deploy Qt 5.12 for Raspberry Pi - 17 November, 2019

In this tutorial, we will see how to integrate TensorFlow Lite with Qt/QML for the development of Raspberry Pi apps. Qt/QML allows us to create rich graphical user interfaces whereas TensorFlow Lite enables on-device machine learning. An open-source example app for object detection is also presented. Have a look to the video below to see this example app in action.

Outline

• Hardware used in this tutorial
• Qt: download, cross-compile and install on Raspberry Pi
• TensorFlow Lite: download and cross-compile for Raspberry Pi
• Raspberry Pi object detection app
• Summary

Hardware used in this tutorial

• Raspberry Pi 3 Model B+
• Raspberry Pi Camera
• Raspberry Pi 3 – 7″ Touchscreen Display
• 2 x Flex Cable for Raspberry Pi (for camera and display)
• 2 x Raspberry Pi 3 B+ Power Supply – 5V 2.5A (for Raspberry Pi and display)

These are affiliate links. This means if you click on the link and purchase the promoted item, we will receive a small affiliate commission at no extra cost to you, the price of the product is the same. We would really appreciate your support to our work and website if this is fine for you.

We need a Linux distribution in our host computer for this tutorial.

Qt: download, cross-compile and install on Raspberry Pi

Have a look at Cross-compile and deploy Qt 5.12 for Raspberry Pi. It provides all the details to do this step. There, you can also find how to set up Qt Creator to deploy Qt apps to Raspberry Pi.

TensorFlow Lite: download and cross-compile for Raspberry Pi

Make sure that you have added the Raspberry Pi GCC cross compiler to your path. Check the tutorial mentioned in the previous step to know more details about how to download and install this compiler.

export PATH=<YOUR_PATH>/raspi/tools/arm-bcm2708/gcc-linaro-arm-
                        linux-gnueabihf-raspbian-x64/bin/:$PATH

First, we have to clone the Git TensorFlow respository in our computer.

git clone https://github.com/tensorflow/tensorflow.git

Then, we have to download TensorFlow Lite compilation dependencies.

cd tensorflow
./tensorflow/lite/tools/make/download_dependencies.sh

Now, we can cross compile TensorFlow Lite for Raspberry Pi.

./tensorflow/lite/tools/make/build_rpi_lib.sh

We can also compile TensorFlow Lite for our Linux distribution (using make), so we can test our multi-platform app in both, our computer and Raspberry Pi. To do this, we have to execute the following command.

make -f ./tensorflow/lite/tools/make/Makefile

Note: If you get an error like this in the compilation process,

undefined reference to NnApiImplementation()

Disable NNAPI by editing the first appearance of BUILD_WITH_NNAPI=true to BUILD_WITH_NNAPI=false in ./tensorflow/lite/tools/make/Makefile. Clean and compile again.

make -f ./tensorflow/lite/tools/make/Makefile clean

NNAPI is TensorFlow Android Neural Network API acceleration which takes advantage of Graphics Processing Units (GPUs), but currently this is only supported on Android devices.

I also encountered a error during the TensorFlow Lite compilation, but it was for the benckmark model.

undefined reference to `tflite::InitTensorFlow()'

Nevertheless, the TensorFlow Lite library (libtensorflow-lite.a) was properly compiled. We can find it in ./tensorflow/lite/tools/make/gen/linux_x86_64/lib/ for 64-bit Linux distributions and in ./tensorflow/lite/tools/make/gen/rpi_armv7l/lib for Raspberry Pi.

Raspberry Pi object detection app

We are ready to test a Qt and TensorFlow Lite app on our Raspberry Pi. The source code of this example app is open source and it is hosted in our GitHub account.

This example uses the TensorFlow starter model for object detection: COCO SSD Quantized MobileNet V1 neural network model.

MobileNets are open-source Convolutional Neural Network (CNN) models for efficient on-device vision. Single Shot Multibox Detector (SSD) is the object detector used by this neural network. This neural network is pre-trained with images from Microsoft Common Objects in Context (COCO) dataset.

The Graphical User Interface of this example app is programmed in QML. In the main screen, the app shows a live camera view. There is a drawer button on the top left side with only one entry: settings. If we go to this option, we will see three tabs: Neural Network & Camera, Screen info and Hardware info & close app.

In the Neural Network & Camera tab, we can configure two neural network options: the minimum confidence level for the detections overlaid on the video frames and the number of cores assigned to TensorFlow Lite. We can also change the camera resolution in this tab.

The Screen info tab allows us to show the inference time (the time taken by the neural network to process one frame) on the main screen and select the objects that we want to detect. The inference time of our neural network running on Raspberry Pi 3 Model B+ is about 1 to 1.5 seconds.

The hardware info & close app tab shows information about our network devices and includes a button to close the app.

If you are interested in more details about the app design or source code, have a look at this tutorial: Tensorflow Lite integration with Qt and Felgo for multi-platform machine learning apps on iOS and Android, most of the source code is shared with this app (except the GUI which is different). You can also inspect the source code of this example app on its GitHub repository.

Some details about the integration between TensorFlow Lite and Qt

The app assumes that the TensorFlow folder is at the same level that our Qt app folder.

The INCLUDEPATH directive in the Qt project file determines where to look for headers in C++ files. The $$PWD variable points to the current folder, i.e. the folder where our Qt project file is located.

INCLUDEPATH += $$PWD/cpp \
               $$PWD/../tensorflow \
               $$PWD/../tensorflow/tensorflow/lite/tools/
                        make/downloads/flatbuffers/include

In the Qt project file, we also have to link to the TensorFlow Lite library.

LIBS += -ltensorflow-lite -ldl

But, we have to specified the path of the TensorFlow Lite library for each platform. We are considering in this example two platforms: Raspbian and the remaining Linux distributions.

Files in the assets folder are distributed together with our application. There are two files: detect.tflite (TensorFlow Lite neural network model for object detection) and labelmap.txt (objects’ labels). The app looks for these files in a nested folder, with respect to the app executable file, called assets.

By default the needed assets are deployed to the /home/pi/qt_apps/TFLite_Qt_Pi/bin/assets folder in our Raspberry Pi. The $${TARGET} variable is substituted by the name of the app (binary filename) we are building, it is the name of the Qt project by default.

Note: we have to create the /home/pi/qt_apps folder in our Raspberry Pi to deploy the example app from Qt Creator.

To test the example app in your development computer (running Linux), set yourself the assets.path in the Qt project file (else block), see below.

# We consider Linux and distinguish between Raspbian (for Raspberry Pi) and other Linux distributions
linux{
    contains(QMAKE_CXX, .*raspbian.*arm.*):
    {
        # TensorFlow Lite lib path
        LIBS += -L$$PWD/../tensorflow/tensorflow/lite/
                           tools/make/gen/rpi_armv7l/lib

        # Assets to be deployed: path and files
        assets.path = /home/pi/qt_apps/$${TARGET}/bin/assets
        assets.files = assets/*
    }
    else {
        # TensorFlow Lite lib path
        LIBS += -L$$PWD/../tensorflow/tensorflow/lite/
                           tools/make/gen/linux_x86_64/lib

        # Assets to be deployed: path and files
        # WARNING: Define yourself the path!
        # assets.path = /home/user/app 
        assets.files = assets/*
    }
}

Note: If the assets are not properly deployed to your Raspberry Pi, just manually copy the assets folder to the /home/pi/qt_apps/TFLite_Qt_Pi/bin/ folder on your Raspberry Pi.

Also make sure that the working directory points to /home/pi/qt_apps/TFLite_Qt_Pi/bin/. This can be set in Qt Creator, left-hand side Projects button, now select the Raspberry Pi kit and choose the Run section. 

Tip: To hide the mouse cursor while executing the example app, we just have to edit the Raspberry Pi user, by default pi, .profile file in our Raspberry Pi.

nano /home/pi/.profile

And add the following lines at the end of this file.

# Hide mouse cursor
export QT_QPA_EGLFS_HIDECURSOR=1

Summary

We have learnt how to integrate TensorFlow Lite with Qt for Raspberry Pi apps in this tutorial. We also applied this to an example app for object detection on device using: a Raspberry Pi camera, a touchscreen display and a pre-trained TensorFlow neural network model for object detection.

I hope you liked the tutorial, please consider to rate this tutorial with the starts you can find below, this gives us feedback about our tutorials. If you have any doubt, proposal, comment or issue, just write below, we will try to help :-).