Background
Among Arm's portfolio of IP blocks is the Ethos-U product line, offering acceleration of neural networks at low power and low latency. Arm calls them micro NPUs and though they indeed contain lower numbers of compute units, in my testing they can match the performance of NPUs with up to six times more compute units. In my opinion this is due to a very clean architecture in which the different sub-blocks fit very well with each other, and an excellent software stack that allows for very decent occupancy rates.
Of the SoCs supported by the mainline Linux kernel and containing a Ethos-U NPU, the most popular is NXP's i.MX93, which contains an Ethos U-65 with an advertised 0.5 TOP/s and 640 KB of SRAM.
 |
| The author's desk got messier by one board |
Arm's software stack to go with their IP includes on the low-level side a firmware running on a companion Cortex-M core, an out-of-tree kernel driver and a compiler called Vela that lowers a hardware-independent machine learning graph to the commands that the hardware understands. Their software stack also includes complete and high-quality hardware documentation that made the implementation of the mainline driver much more efficient.
In my conversations with SoC vendors, the mediation of the Cortex-M meant that the ML models couldn't co-execute efficiently on the CPU and NPU, and was cumbersome for their customers to deploy new models.
The other concern that I heard from SoC vendors and some of their customers was that even though Arm's software stack for the Ethos-U was open source, it wasn't pre-integrated in the Linux distributions that they were using, and there wasn't a standard path for contributing changes back as there is in established, multi-vendor open-source upstream projects.
Mainline
To solve this deployment friction and eliminate the need for firmware mediation, the community turned to the mainline Linux kernel.
Rob Herring, widely known in the Linux community from his work on the Device Tree infrastructure, started a completely new kernel driver a bit over a year ago in the DRM/Accel subsystem that would drive the Ethos-U directly from the main CPU, bypassing the Cortex-M and thus the need for firmware. The driver uses the established DRM practices for UAPI and reuses code from the DRM subsystem such as GEM and gpu-drm-scheduler. This driver was merged into the 6.19 version of the Linux kernel, supporting right from the start the i.MX93 SoC from NXP.
In parallel, I started work on the Mesa side, where the userspace portion of DRM drivers are developed. Mesa already contained at that point support for the NPUs from VeriSilicon/Vivante and those found in Rockchip SoCs, so it was quite straightforward to add a third one. The userspace portion of the driver is in charge of translating the machine learning model into the programming of the accelerator hardware, typically divided into hardware configuration and data payloads such as operator coefficients which can be highly compressed.
The hardware drivers in Mesa share code in the form of backends for machine learning frameworks and the related testing and profiling infrastructure. Right now Mesa contains a TensorFlow Lite delegate and a PyTorch/ExecuTorch backend is in development.
The Ethos-U architecture is very clean, as mentioned above, and allows for easy implementation of new operations as used in ML models. Below are the operations currently implemented, as needed by the models that are currently supported and under regression testing.
| NPU opcode | TFLite operations |
|---|---|
| NPU_OP_CONV / NPU_OP_DEPTHWISE | Convolution, Fully Connected |
| NPU_OP_POOL | Max and Average Pooling |
| NPU_OP_POOL (NOP) | Concatenation, Strided Slice, Resize |
| NPU_OP_ELEMENTWISE | Add, Mul, Maximum, Minimum |
| NPU_OP_ELEMENTWISE (LUT) | Logistic, Tanh, HSwish, Leaky Relu |
| (fused / passthrough) | Pad, Quantize, Reshape |
Among the features that the mainline driver stack already supports are:
- different quantization schemes such as affine per-tensor and per-channel quantization, and signed and unsigned 8-bit integers,
- different schemes of coefficient compression as supported by the hardware decoder, and
- several performance counters for analyzing execution and maximizing occupancy.
When using TensorFlow Lite for inference, the models below are supported:
| Model | Operations | Operation types |
|---|---|---|
| MobileNet V1 | 31 | Conv2D, DWConv, Reshape, Softmax |
| MobileNet V2 | 65 | AvgPool, Conv2D, DWConv, Reshape |
| SSDLite MobileDet | 125 | Concat, Conv2D, Custom(Detect), DWConv, Logistic, Reshape |
| SSD MobileNet V2 | 111 | Concat, Custom(Detect), DWConv, Dequant, Logistic, Quantize, Reshape |
| YOLOX Nano | 154 | Concat, Conv2D, Logistic, MaxPool, Mul, Reshape, ResizeNN, ScatterNd, Slice, Sub, Transpose |
| EfficientDet Lite0 | 267 | Concat, Conv2D, DWConv, Dequant, GatherND, Logistic, MaxPool, Pad, Reshape |
| Inception V1 | 83 | AvgPool, Concat, MaxPool, Reshape, Softmax |
| MicroNet Large | 14 | AvgPool, DWConv, Reshape |
| MoveNet Lightning | 157 | ArgMax, Concat, Conv2D, DWConv, Dequant, Div, FloorDiv, GatherND, Logistic, Mul, Pack, Quantize, Reshape, ResizeB, Sqrt, Sub, Unpack |
| MoveNet Thunder | 157 | ArgMax, Concat, Conv2D, DWConv, Dequant, Div, FloorDiv, GatherND, Logistic, Mul, Pack, Quantize, Reshape, ResizeB, Sqrt, Sub, Unpack |
More recently, support for the Ethos-U85 generation was added, as included in Arm's hardware simulator and in SoCs such as Alif's Ensemble E8. This version of the NPU IP contains improvements intended for more efficient support for transformer-based models, such as VLMs and LLMs. With the upcoming support for ExecuTorch in Mesa, users will be able to use PyTorch to optimize models for this hardware and ExecuTorch to deploy them to their edge/IoT hardware.
Important upcoming work includes performance improvements that will target reduced total memory usage and bandwidth and maximization of SRAM usage.
