Ethos-U NPU update 2: Executorch support

Over the last few months I have been working on Torx, an ExecuTorch backend in Mesa3D that delegates to the NPU drivers Mesa already contains. Initial Torx support has been merged already, with YOLOX support being currently under review.

Torx sits next to Teflon, the existing TensorFlow Lite delegate, and both use the same Gallium ML API. Today Torx works with the ethosu driver.

Torx implements the backend API that ExecuTorch provides for running portions of a model's graph on specialized hardware such as NPUs or GPUs. It then uses the Gallium ML API to compile and execute those portions on the available hardware.

Unlike TensorFlow Lite, ExecuTorch compiles models in a step separate from execution. The model can therefore be compiled on a different machine from the target, where it is practical to apply more expensive optimizations than edge-class hardware could run.

I worked on this alongside Rob Herring from Arm, who supported the effort on the kernel side.

Below is a demo of object detection with YOLOX-Nano at 416x416, quantized to 8 bits integers, on an i.MX93 board with an Arm Ethos-U65 NPU. Inference takes 17 milliseconds per frame.

The convolutional layers of YOLOX run entirely on the U65. The final decoding steps need floating point operations, so they run on the CPU.

Once the other NPU drivers in Mesa support the ahead-of-time compilation extension to the Gallium ML API, they will be usable from ExecuTorch as well.

To try it yourself, you can follow the instructions in the documentation. If you prefer not to build ExecuTorch from sources, you can use this merge request from Meta's Anthony Shoumikhin. 

Etnaviv NPU update 22: YOLOX support

We have expanded our open-source NPU support for object detection: YOLOX is now running on the Etnaviv driver.

This brings high-performance, open-source AI acceleration to the Vivante VIP line of NPUs, including those found in the NXP i.MX 8M Plus and the Amlogic A311D. Adding YOLOX gives users more options when balancing detection accuracy against available hardware resources.

YOLOX is an open-source object detection model developed by Megvii (Apache 2.0). It is particularly well-suited for edge NPUs, but it is significantly more complex than SSDLite MobileDet (our previously supported model). Getting it running required implementing several new operations in the driver.

As part of this work, we landed support for:

• FullyConnected: A new operation that runs directly on the NN (convolution) cores.
• Reshape, Split, and Concatenate: Handled via metadata changes; these do not execute on the hardware, saving cycles.
• Fused ReLU: Enables activation function hardware on the output.
• Absolute and Logistic: Implemented as lookup table operations on the TP (tensor processing) cores.
• Subtract: Lowered to a convolution, similar to our approach for Add.
• Transpose: Either fused into the next operation or handled as a TP operation.

Additionally, we added support for feature maps in signed 8-bit integers. For certain models, this provides increased accuracy at the exact same computational cost.

This work was performed in partnership with Ideas On Board.

YOLOX is an open-source project developed by Megvii and licensed under the Apache License 2.0.

Ethos-U NPU update 1: The journey to a production-ready mainline driver

Background 

Among Arm's portfolio of IP blocks is the Ethos-U product line, offering acceleration of neural networks at low power and low latency. Arm calls them micro NPUs and though they indeed contain lower numbers of compute units, in my testing they can match the performance of NPUs with up to six times more compute units. In my opinion this is due to a very clean architecture in which the different sub-blocks fit very well with each other, and an excellent software stack that allows for very decent occupancy rates.

Of the SoCs supported by the mainline Linux kernel and containing a Ethos-U NPU, the most popular is NXP's i.MX93, which contains an Ethos U-65 with an advertised 0.5 TOP/s and 640 KB of SRAM.

The author's desk got messier by one board

 

Arm's software stack to go with their IP includes on the low-level side a firmware running on a companion Cortex-M core, an out-of-tree kernel driver and a compiler called Vela that lowers a hardware-independent machine learning graph to the commands that the hardware understands. Their software stack also includes complete and high-quality hardware documentation that made the implementation of the mainline driver much more efficient.

In my conversations with SoC vendors, the mediation of the Cortex-M meant that the ML models couldn't co-execute efficiently on the CPU and NPU, and was cumbersome for their customers to deploy new models. 

The other concern that I heard from SoC vendors and some of their customers was that even though Arm's software stack for the Ethos-U was open source, it wasn't pre-integrated in the Linux distributions that they were using, and there wasn't a standard path for contributing changes back as there is in established, multi-vendor open-source upstream projects. 

Mainline

To solve this deployment friction and eliminate the need for firmware mediation, the community turned to the mainline Linux kernel. 

Rob Herring, widely known in the Linux community from his work on the Device Tree infrastructure, started a completely new kernel driver a bit over a year ago in the DRM/Accel subsystem that would drive the Ethos-U directly from the main CPU, bypassing the Cortex-M and thus the need for firmware. The driver uses the established DRM practices for UAPI and reuses code from the DRM subsystem such as GEM and gpu-drm-scheduler. This driver was merged into the 6.19 version of the Linux kernel, supporting right from the start the i.MX93 SoC from NXP.

In parallel, I started work on the Mesa side, where the userspace portion of DRM drivers are developed. Mesa already contained at that point support for the NPUs from VeriSilicon/Vivante and those found in Rockchip SoCs, so it was quite straightforward to add a third one. The userspace portion of the driver is in charge of translating the machine learning model into the programming of the accelerator hardware, typically divided into hardware configuration and data payloads such as operator coefficients which can be highly compressed.

The hardware drivers in Mesa share code in the form of backends for machine learning frameworks and the related testing and profiling infrastructure. Right now Mesa contains a TensorFlow Lite delegate and a PyTorch/ExecuTorch backend is in development.

The Ethos-U architecture is very clean, as mentioned above, and allows for easy implementation of new operations as used in ML models. Below are the operations currently implemented, as needed by the models that are currently supported and under regression testing.

NPU opcode	TFLite operations
NPU_OP_CONV / NPU_OP_DEPTHWISE	Convolution, Fully Connected
NPU_OP_POOL	Max and Average Pooling
NPU_OP_POOL (NOP)	Concatenation, Strided Slice, Resize
NPU_OP_ELEMENTWISE	Add, Mul, Maximum, Minimum
NPU_OP_ELEMENTWISE (LUT)	Logistic, Tanh, HSwish, Leaky Relu
(fused / passthrough)	Pad, Quantize, Reshape

Among the features that the mainline driver stack already supports are:

• different quantization schemes such as affine per-tensor and per-channel quantization, and signed and unsigned 8-bit integers,
• different schemes of coefficient compression as supported by the hardware decoder, and
• several performance counters for analyzing execution and maximizing occupancy.

When using TensorFlow Lite for inference, the models below are supported: 

Model	Operations	Operation types
MobileNet V1	31	Conv2D, DWConv, Reshape, Softmax
MobileNet V2	65	AvgPool, Conv2D, DWConv, Reshape
SSDLite MobileDet	125	Concat, Conv2D, Custom(Detect), DWConv, Logistic, Reshape
SSD MobileNet V2	111	Concat, Custom(Detect), DWConv, Dequant, Logistic, Quantize, Reshape
YOLOX Nano	154	Concat, Conv2D, Logistic, MaxPool, Mul, Reshape, ResizeNN, ScatterNd, Slice, Sub, Transpose
EfficientDet Lite0	267	Concat, Conv2D, DWConv, Dequant, GatherND, Logistic, MaxPool, Pad, Reshape
Inception V1	83	AvgPool, Concat, MaxPool, Reshape, Softmax
MicroNet Large	14	AvgPool, DWConv, Reshape
MoveNet Lightning	157	ArgMax, Concat, Conv2D, DWConv, Dequant, Div, FloorDiv, GatherND, Logistic, Mul, Pack, Quantize, Reshape, ResizeB, Sqrt, Sub, Unpack
MoveNet Thunder	157	ArgMax, Concat, Conv2D, DWConv, Dequant, Div, FloorDiv, GatherND, Logistic, Mul, Pack, Quantize, Reshape, ResizeB, Sqrt, Sub, Unpack

More recently, support for the Ethos-U85 generation was added, as included in Arm's hardware simulator and in SoCs such as Alif's Ensemble E8. This version of the NPU IP contains improvements intended for more efficient support for transformer-based models, such as VLMs and LLMs. With the upcoming support for ExecuTorch in Mesa, users will be able to use PyTorch to optimize models for this hardware and ExecuTorch to deploy them to their edge/IoT hardware.

Important upcoming work includes performance improvements that will target reduced total memory usage and bandwidth and maximization of SRAM usage.

Rockchip NPU update 6: We are in mainline!

The kernel portion of the Linux driver for the Rockchip NPUs has been merged into the maintainer tree, and will be sent in the next pull request to Linus. The userspace portion of the driver has just been merged as well, in the main Mesa repository.

This means that in the next few weeks the two components of the Rocket driver will be in official releases of the Linux and Mesa projects, and Linux distributions will start to pick them up and package. Once that happens, we will have seamless accelerated inference on one more category of hardware.

It has been a bit over a year since I started working on the driver, though the actual feature implementation took just over two months of that. The rest of the time was spent waiting for reviews and reacting to excellent feedback from many contributors to the Linux kernel. The driver is now much better because of that frank feedback.

What I see in the near future for this driver is support for other Rockchip SoCs and some performance work, to match that of the proprietary driver. But of course, with it being open source, contributors can just start hacking on it and sending patches over for review and merging.

I'm now working on further improvements to the Etnaviv driver for the Vivante NPUs, and have started work with Arm engineers on a new driver for their Ethos line of NPUs. 

So stay tuned for more news on accelerated inference on the edge in mainline Linux!

Rockchip NPU update 5: Progress on the kernel driver

 It has been almost a year since my last update on the Rockchip NPU, and though I'm a bit sad that I haven't had more time to work on it, I'm happy that I found some time earlier this year for this.

Quoting from my last update on the Rockchip NPU driver:

The kernel driver is able to fully use the three cores in the NPU, giving us the possibility of running 4 simultaneous object detection inferences such as the one below on a stream, at almost 30 frames per second.

 

All feedback has been incorporated in a new revision of the kernel driver and it was submitted to the Linux kernel mailing list.

Though I'm very happy with the direction the kernel driver is taking, I would have liked to make faster progress on it. I have spent the time since the first revision on making the Etnaviv NPU driver ready to be deployed in production (will be blogging about this soon), and also had to take some non-upstream work to pay my bills.

Next I plan to cleanup the userspace driver so it's ready for review, and then I will go for a third revision of the kernel driver.

Etnaviv NPU update 21: Support for the NPU in the NXP i.MX 8M Plus SoC is upstream!

Several months have passed since the last update. This has been in part due to the summer holidays and a gig doing some non-upstream work, but I have also had the opportunity to continue my work on the NPU driver for the VeriSilicon NPU in the NXP i.MX 8M Plus SoC, thanks to my friends at Ideas on Board.

CC BY-NC 4.0 Henrik Boye

 I'm very happy with what has been accomplished so far, with the first concrete result being the merge in Mesa of the support for NXP's SoC. Thanks to Philipp Zabel and Christian Gmeiner for helping with their ideas and code reviews.

With this, as of yesterday, one can accelerate models such as SSDLite MobileDet on that SoC with only open source software, with the support being provided directly from projects that are already ubiquitous in today's products, such as the Linux kernel and Mesa3D. We can expect this functionality to reach distributions such as Debian in due time, for seamless installation and integration in products.

With this milestone reached, I will be working on expanding support for more models, with a first goal of enabling YOLO-like models, starting with YOLOX. I will be working as well on performance, as currently we are not fully using the capabilities of this hardware.

Etnaviv NPU update 20: Fast object detection on the NXP i.MX 8M Plus SoC

I'm happy to announce that my first project regarding support for the NPU in NXP's i.MX 8M Plus SoC has reached the feature complete stage.

CC BY-NC 4.0 Henrik Boye

For the last several weeks I have been working full-time on adding support for the NPU to the existing Etnaviv driver. Most of the existing code that supports the NPU in the Amlogic A311D was reused, but NXP used a much more recent version of the NPU IP so some advancements required new code, and this in turn required reverse engineering.

This work has been kindly sponsored by the Open Source consultancy Ideas On Board, for which I am very grateful. I hope this will be useful to those companies that need full mainline support in their products, even if it is just the start.

This company is unique in working on both NPU and camera drivers in Linux mainline, so they have the best experience for products that require long term support and vision processing.

Since the last update I have fixed the last bugs in the compression of the weights tensor and implemented support for a new hardware-assisted way of executing depthwise convolutions. Some improvements on how the tensor addition operation is lowered to convolutions was needed as well.

Performance is pretty good already, allowing for detecting objects in video streams at 30 frames per second, so at a similar performance level as the NPU in the Amlogic A311D. Some performance features are left to be implemented, so I think there is still substantial room for improvement.

At current the code is at a very much proof-of-concept state. The next step is cleaning it all up and submitting for review to Mesa3D. In the meantime, you can find the draft code at https://gitlab.freedesktop.org/tomeu/mesa/-/tree/etnaviv-imx8mp.

A big thanks to Philipp Zabel who reverse engineered the bitstream format of the weight encoding and added some patches to the kernel that were required for the NPU to work reliably.