Smart Camera, AI and Robotics Software Resources for QUALCOMM System on Chip Platforms
Figure 1. QUALCOMM SoC overview, a product matrix of available SoC’s from QUALCOMM can be found here.
The devices in QUALCOMM’s SOC roadmap all follow the same hardware-based architecture as shown in diagram 1, this enables developers to expand software applications and use cases that can be reused across SoCs with lower effort and risk. In order to investigate the software resources available to developers further let us focus on one of the QUALCOMM devices from the premium tier of the roadmap, the QRB5165 SOC better known as the RB5 platform for Robotics applications. The RB5 platform is provided with a complete Ubuntu linux based software release in addition to a separate Linux architecture based on Yocto embedded Linux. Let’s take a closer look at the Ubuntu linux software release and architecture.
Figure 2. Ubuntu linux software architecture overview for the QUALCOMM RB5 platform.
The latest release for the RB5 platform is based on Ubuntu version 20.04, running version 5.4 of the Linux kernel. As you can see in the diagram in addition to the Ubuntu components and operating system, QUALCOMM has combined their proprietary subsystem software (blue), tools and other resources together with middleware to drive GUI applications (Wayland / Weston), integrate audio support (ALSA) to capture video (V4L2) as well as to manage device security (AppArmor). Additionally, to support development on the RB5 device QUALCOMM also provides a suite of tools such as compilers (LLVM and GCC), version control (Git) and debugging (GDB) as well as support for programming languages Phyton and Perl. The apt-get tool allows the developer to enhance the RB5 software architecture with other Debian libraries, tools and resources. This Ubuntu package is maintained by QUALCOMM for a period of two years, this includes critical bug fixes and security patch updates for the components authored by QUALCOMM in the proprietary and open-source sections of the code.
For developers that want to leverage the SoC’s integrated dual camera ISP QUALCOMM provides a dedicated computer vision SDK for the RB5 and other SoCs. This SDK incorporates libraries of the most frequently used computer vision functions including object and face detection, optical flow, depth estimation and geometry correction. These functions run in a logical block in the SoC known as the Engine for Visual Analytics (EVA) which makes the libraries API’s available across the SoC’s, CPU and DSP cores. The computer vision libraries are optimised for ARM processors; however, they have been tuned by QUALCOMM to further reduce power consumption and latency by running on dedicated HW acceleration blocks integrated in QUALCOMM RB5 and other SoCs. These computer vision APIs are also available for use in the SOC’s Hexagon Tensor Processor, a 6-thread scalar DSP with dedicated vector processing capability for pixel processing applications. The computer vision SDK is available for download from QUALCOMM’s developer portal.
Figure 3. Computer vision SDK functions available on QUALCOMM SoC’s such as the RB5 and QCS8250.
In addition to the use of cameras and computer vision many applications today require the use of AI and machine learning to recognise people, objects and events of interest. The QUALCOMM RB5 and other SOCs offer a powerful and dedicated DSP optimised for running AI models along with a dedicated SDK to develop and optimise AI applications. The QUALCOMM Neural Processing SDK is a software accelerated inference-only runtime engine for the execution of deep neural networks. This SDK abstracts the underlying details on the model execution and provides flexibility for users to design and develop machine learning application pipelines with ease. It incorporates the tools to allow developers to take AI models from frameworks such as TensorFlow, Caffe, Caffe2 and ONNX and build an executable file, known as a deep learning container, which can take advantage of the dedicated AI hardware in the SoC. The deep learning container file can be quantized to execute exclusively on the DSP leveraging 8-bit fixed point or it can be configured to execute across the DSP, ARM based multi-core CPU or GPU. Once the DLC file is created, tools can be used to optimise the model with compression and quantization techniques. The QUALCOMM Neural Processing SDK can also be downloaded from QUALCOMM’s Developers Portal.
Figure 4. Workflow to execute AI models on dedicated AI processing hardware in QUALCOMM SOC’s such as the RB5 and QCS8250 platforms.
QUALCOMM’s SoC roadmap contains a number of SoCs which have been optimised for Robotics applications, here is a high-level summary of those devices and the robot applications they are targeting.
Figure 5. QUALCOMM SoC’s for Robotics applications
To enable robotic applications QUALCOMM SoCs such as the RB5 platform are offered with a suite of Robotics software and SDKs which incorporates ROS (Robotics Operating System) and ROS 2 support which are open source (under BSD license) tools and libraries for building Robotics applications. In addition, QUALCOMM offers AWS Robomaker as well as a dedicated Visual SLAM SDK to leverage the on-device computer vision hardware capabilities and multiple cameras to enable the Robot to detect and avoid objects to perform mapping, path planning and other essential robotic functions. Detailed instructions on how to deploy AWS Robomaker and ROS on QUALCOMM’s RB5 Robotics development platform and a comprehensive set of documentation and software resources to enable Robotics developers can be found on QUALCOMM’s developers’ portal.
Figure 6. Software support for QUALCOMM RB5 and other Robotics SoCs.
QUALCOMM SoCs are not only powerful and flexible hardware platforms they are accompanied by a rich suite of software tools, development kits and sample projects to get your project started quickly. For each of the SOC platforms QUALCOMM partners like THUNDERCOMM, their joint venture company, complement QUALCOMM provided software by adding their own IP (AI models, Camera and Sensor integration) which can help a customer to reduce their project risk and accelerate time to market.
For further questions:
Thomas Carmody
+43 1 86305 362
About CODICO
CODICO stands for the design-in-distribution of high quality electronic components. The widespread product portfolio includes active and passive elements as well as products from the field of connection technology. The company operates from the Austrian headquarters in Perchtoldsdorf on the southern outskirts of Vienna as an independent privately owned company, and maintains a large number of sales offices in Germany, Denmark, Italy, France, the Czech Republic, Slovenia, Sweden, the UK, and has other partner companies in Central and Eastern Europe on its team. Backed by a high level of technical expertise, CODICO puts its focus on the Design-In sector. What makes CODICO particularly special is the providing of technical support from the development phase through to the end product, as well as the marketing and sale of exclusively top quality products.