Ethos-U NPU update 1: The journey to a production-ready mainline driver

Background 

Among Arm's portfolio of IP blocks is the Ethos-U product line, offering acceleration of neural networks at low power and low latency. Arm calls them micro NPUs and though they indeed contain lower numbers of compute units, in my testing they can match the performance of NPUs with up to six times more compute units. In my opinion this is due to a very clean architecture in which the different sub-blocks fit very well with each other, and an excellent software stack that allows for very decent occupancy rates.

Of the SoCs supported by the mainline Linux kernel and containing a Ethos-U NPU, the most popular is NXP's i.MX93, which contains an Ethos U-65 with an advertised 0.5 TOP/s and 640 KB of SRAM.

The author's desk got messier by one board

 

Arm's software stack to go with their IP includes on the low-level side a firmware running on a companion Cortex-M core, an out-of-tree kernel driver and a compiler called Vela that lowers a hardware-independent machine learning graph to the commands that the hardware understands. Their software stack also includes complete and high-quality hardware documentation that made the implementation of the mainline driver much more efficient.

In my conversations with SoC vendors, the mediation of the Cortex-M meant that the ML models couldn't co-execute efficiently on the CPU and NPU, and was cumbersome for their customers to deploy new models. 

The other concern that I heard from SoC vendors and some of their customers was that even though Arm's software stack for the Ethos-U was open source, it wasn't pre-integrated in the Linux distributions that they were using, and there wasn't a standard path for contributing changes back as there is in established, multi-vendor open-source upstream projects. 

Mainline

To solve this deployment friction and eliminate the need for firmware mediation, the community turned to the mainline Linux kernel. 

Rob Herring, widely known in the Linux community from his work on the Device Tree infrastructure, started a completely new kernel driver a bit over a year ago in the DRM/Accel subsystem that would drive the Ethos-U directly from the main CPU, bypassing the Cortex-M and thus the need for firmware. The driver uses the established DRM practices for UAPI and reuses code from the DRM subsystem such as GEM and gpu-drm-scheduler. This driver was merged into the 6.19 version of the Linux kernel, supporting right from the start the i.MX93 SoC from NXP.

In parallel, I started work on the Mesa side, where the userspace portion of DRM drivers are developed. Mesa already contained at that point support for the NPUs from VeriSilicon/Vivante and those found in Rockchip SoCs, so it was quite straightforward to add a third one. The userspace portion of the driver is in charge of translating the machine learning model into the programming of the accelerator hardware, typically divided into hardware configuration and data payloads such as operator coefficients which can be highly compressed.

The hardware drivers in Mesa share code in the form of backends for machine learning frameworks and the related testing and profiling infrastructure. Right now Mesa contains a TensorFlow Lite delegate and a PyTorch/ExecuTorch backend is in development.

The Ethos-U architecture is very clean, as mentioned above, and allows for easy implementation of new operations as used in ML models. Below are the operations currently implemented, as needed by the models that are currently supported and under regression testing.

NPU opcode	TFLite operations
NPU_OP_CONV / NPU_OP_DEPTHWISE	Convolution, Fully Connected
NPU_OP_POOL	Max and Average Pooling
NPU_OP_POOL (NOP)	Concatenation, Strided Slice, Resize
NPU_OP_ELEMENTWISE	Add, Mul, Maximum, Minimum
NPU_OP_ELEMENTWISE (LUT)	Logistic, Tanh, HSwish, Leaky Relu
(fused / passthrough)	Pad, Quantize, Reshape

Among the features that the mainline driver stack already supports are:

• different quantization schemes such as affine per-tensor and per-channel quantization, and signed and unsigned 8-bit integers,
• different schemes of coefficient compression as supported by the hardware decoder, and
• several performance counters for analyzing execution and maximizing occupancy.

When using TensorFlow Lite for inference, the models below are supported: 

Model	Operations	Operation types
MobileNet V1	31	Conv2D, DWConv, Reshape, Softmax
MobileNet V2	65	AvgPool, Conv2D, DWConv, Reshape
SSDLite MobileDet	125	Concat, Conv2D, Custom(Detect), DWConv, Logistic, Reshape
SSD MobileNet V2	111	Concat, Custom(Detect), DWConv, Dequant, Logistic, Quantize, Reshape
YOLOX Nano	154	Concat, Conv2D, Logistic, MaxPool, Mul, Reshape, ResizeNN, ScatterNd, Slice, Sub, Transpose
EfficientDet Lite0	267	Concat, Conv2D, DWConv, Dequant, GatherND, Logistic, MaxPool, Pad, Reshape
Inception V1	83	AvgPool, Concat, MaxPool, Reshape, Softmax
MicroNet Large	14	AvgPool, DWConv, Reshape
MoveNet Lightning	157	ArgMax, Concat, Conv2D, DWConv, Dequant, Div, FloorDiv, GatherND, Logistic, Mul, Pack, Quantize, Reshape, ResizeB, Sqrt, Sub, Unpack
MoveNet Thunder	157	ArgMax, Concat, Conv2D, DWConv, Dequant, Div, FloorDiv, GatherND, Logistic, Mul, Pack, Quantize, Reshape, ResizeB, Sqrt, Sub, Unpack

More recently, support for the Ethos-U85 generation was added, as included in Arm's hardware simulator and in SoCs such as Alif's Ensemble E8. This version of the NPU IP contains improvements intended for more efficient support for transformer-based models, such as VLMs and LLMs. With the upcoming support for ExecuTorch in Mesa, users will be able to use PyTorch to optimize models for this hardware and ExecuTorch to deploy them to their edge/IoT hardware.

Important upcoming work includes performance improvements that will target reduced total memory usage and bandwidth and maximization of SRAM usage.

Rockchip NPU update 6: We are in mainline!

The kernel portion of the Linux driver for the Rockchip NPUs has been merged into the maintainer tree, and will be sent in the next pull request to Linus. The userspace portion of the driver has just been merged as well, in the main Mesa repository.

This means that in the next few weeks the two components of the Rocket driver will be in official releases of the Linux and Mesa projects, and Linux distributions will start to pick them up and package. Once that happens, we will have seamless accelerated inference on one more category of hardware.

It has been a bit over a year since I started working on the driver, though the actual feature implementation took just over two months of that. The rest of the time was spent waiting for reviews and reacting to excellent feedback from many contributors to the Linux kernel. The driver is now much better because of that frank feedback.

What I see in the near future for this driver is support for other Rockchip SoCs and some performance work, to match that of the proprietary driver. But of course, with it being open source, contributors can just start hacking on it and sending patches over for review and merging.

I'm now working on further improvements to the Etnaviv driver for the Vivante NPUs, and have started work with Arm engineers on a new driver for their Ethos line of NPUs. 

So stay tuned for more news on accelerated inference on the edge in mainline Linux!

Rockchip NPU update 5: Progress on the kernel driver

 It has been almost a year since my last update on the Rockchip NPU, and though I'm a bit sad that I haven't had more time to work on it, I'm happy that I found some time earlier this year for this.

Quoting from my last update on the Rockchip NPU driver:

The kernel driver is able to fully use the three cores in the NPU, giving us the possibility of running 4 simultaneous object detection inferences such as the one below on a stream, at almost 30 frames per second.

 

All feedback has been incorporated in a new revision of the kernel driver and it was submitted to the Linux kernel mailing list.

Though I'm very happy with the direction the kernel driver is taking, I would have liked to make faster progress on it. I have spent the time since the first revision on making the Etnaviv NPU driver ready to be deployed in production (will be blogging about this soon), and also had to take some non-upstream work to pay my bills.

Next I plan to cleanup the userspace driver so it's ready for review, and then I will go for a third revision of the kernel driver.

Etnaviv NPU update 21: Support for the NPU in the NXP i.MX 8M Plus SoC is upstream!

Several months have passed since the last update. This has been in part due to the summer holidays and a gig doing some non-upstream work, but I have also had the opportunity to continue my work on the NPU driver for the VeriSilicon NPU in the NXP i.MX 8M Plus SoC, thanks to my friends at Ideas on Board.

CC BY-NC 4.0 Henrik Boye

 I'm very happy with what has been accomplished so far, with the first concrete result being the merge in Mesa of the support for NXP's SoC. Thanks to Philipp Zabel and Christian Gmeiner for helping with their ideas and code reviews.

With this, as of yesterday, one can accelerate models such as SSDLite MobileDet on that SoC with only open source software, with the support being provided directly from projects that are already ubiquitous in today's products, such as the Linux kernel and Mesa3D. We can expect this functionality to reach distributions such as Debian in due time, for seamless installation and integration in products.

With this milestone reached, I will be working on expanding support for more models, with a first goal of enabling YOLO-like models, starting with YOLOX. I will be working as well on performance, as currently we are not fully using the capabilities of this hardware.

Etnaviv NPU update 20: Fast object detection on the NXP i.MX 8M Plus SoC

I'm happy to announce that my first project regarding support for the NPU in NXP's i.MX 8M Plus SoC has reached the feature complete stage.

CC BY-NC 4.0 Henrik Boye

For the last several weeks I have been working full-time on adding support for the NPU to the existing Etnaviv driver. Most of the existing code that supports the NPU in the Amlogic A311D was reused, but NXP used a much more recent version of the NPU IP so some advancements required new code, and this in turn required reverse engineering.

This work has been kindly sponsored by the Open Source consultancy Ideas On Board, for which I am very grateful. I hope this will be useful to those companies that need full mainline support in their products, even if it is just the start.

This company is unique in working on both NPU and camera drivers in Linux mainline, so they have the best experience for products that require long term support and vision processing.

Since the last update I have fixed the last bugs in the compression of the weights tensor and implemented support for a new hardware-assisted way of executing depthwise convolutions. Some improvements on how the tensor addition operation is lowered to convolutions was needed as well.

Performance is pretty good already, allowing for detecting objects in video streams at 30 frames per second, so at a similar performance level as the NPU in the Amlogic A311D. Some performance features are left to be implemented, so I think there is still substantial room for improvement.

At current the code is at a very much proof-of-concept state. The next step is cleaning it all up and submitting for review to Mesa3D. In the meantime, you can find the draft code at https://gitlab.freedesktop.org/tomeu/mesa/-/tree/etnaviv-imx8mp.

A big thanks to Philipp Zabel who reverse engineered the bitstream format of the weight encoding and added some patches to the kernel that were required for the NPU to work reliably.

Etnaviv NPU update 19: Ideas On Board sponsors support for the NXP i.MX 8M Plus SoC

Last week I started work on adding support to the Etnaviv driver for the NPU inside the NXP i.MX 8M Plus SoC (VeriSilicon's VIPNano-SI+).

This work is sponsored by the open source consultancy Ideas On Boards, and will include the same level of support as for the Amlogic A311D SoC, which means full acceleration for the SSDLite MobileDet object detection model.

Right now all kinds of basic convolutions are supported, and work is well on its way for strided convolutions.

For basic convolutions, most of the work was switching to a totally different way of encoding weights. At the low-level, the weights are encoded with Huffman, and zero run length encoding on top. This low level encoding has been already reverse engineered and implemented by Philipp Zabel of Pengutronix, as mentioned in my previous update on the variant of this NPU shipped inside the Amlogic S905D3.

How weights are laid on top of the encoding is also different, so I had to reverse engineer that and implement it in the Mesa driver. That plus some changes on how tiling is computed got basic convolutions working, then I moved to strided convolutions. Pointwise convolutions got supported at the same time as basic convolutions, as they are not any different on this particular hardware.

Strided convolutions are still not natively supported by the hardware, so I reused the code that lowers them to basic convolutions. But the existing jobs that use the tensor manipulation cores to transform the input tensor for strides contained many assumptions that don't hold valid in this hardware.

So I have been reverse engineering these differences and now I have all kinds of strided convolutions supported up to 32 output channels. I feel that these will be done after addressing a couple of details about how the tensor reshuffle jobs are distributed among the available TP cores.

Afterwards I will look at depthwise convolutions, which may be supported natively by the hardware, while on the A311D these were lowered to basic convolutions.

Then on to tensor addition operations, and that should be all that is needed to get SSDLite MobileDet running, hopefully close to the performance of the closed source driver.

I'm very grateful to Ideas On Board for sponsoring this work, for their trust on me to get it done, and for their vision of a fully featured mainline platform that all companies can base their products on without being held captive by any single vendor.

I'm testing all this on a Verdin iMX8M Plus board that was kindly offered by Daniel Lang at Toradex, thanks!

Rockchip NPU update 4: Kernel driver for the RK3588 NPU submitted to mainline

In the past few weeks I have been working on among other things a kernel driver for the NPU in the Rockchip RK3588 SoC, new from the ground up.

It is now fully working and after a good amount of polishing I sent it yesterday to the kernel mailing lists, for review. Those interested can see the code and follow the review process at this link.

The kernel driver is able to fully use the three cores in the NPU, giving us the possibility of running 4 simultaneous object detection inferences such as the one below on a stream, at almost 30 frames per second.

 

The userspace  driver is in a less polished state, but already fully featured. I will be working on this in the next few days so it can be properly submitted for review.

This is the first accelerator-only driver for an edge NPU submitted to the mainline kernel, and hopefully it can serve as a template for the next ones to come, as the differences among NPUs of different vendors are relatively superficial.