Imported from xynexus/hipfire (
.agents/skills/npu-kernel-build/SKILL.md). Install upstream withnpx skills add xynexus/hipfire --skill npu-kernel-build. Copyright stays with the author.
NPU kernel build — MLIR-AIE / IRON
AMD NPU naming map
AMD's NPU naming is a tangle of marketing names, microarchitecture names, and toolchain identifiers that don't align. Reference table:
| Marketing name | Silicon examples | NPU gen | Tile ISA | Toolchain target | TOPS |
|---|---|---|---|---|---|
| XDNA (Ryzen AI) | Phoenix (7040), Hawk Point (8040) | NPU1 | AIE2 / AIE-ML | aie2, aie2txn, aie2dpu |
16 |
| XDNA2 (Ryzen AI 300) | Strix Point, Strix Halo | NPU2 | AIE2P / AIE-ML v2 | aie2p |
50 |
| XDNA3 (future) | Kraken | NPU3 | AIE4 | aie4 |
TBD |
Name equivalences:
- AIE = AI Engine (Xilinx/Versal origin)
- AIE2 = second-gen AI Engine = AMD's AIE-ML (NPU1 / XDNA)
- AIE2P = AIE2+ = AMD's AIE-ML v2 (NPU2 / XDNA2); adds hardware tanh, wider accumulators
- XDNA / XDNA2 = AMD's brand for the NPU IP block, not the tile ISA
- NPU1 / NPU2 = MLIR-AIE device identifiers (
AIEDevice.npu1,AIEDevice.npu2)
Critical ISA difference: AIE2 needs LUT-based tanh (getTanhBf16 from <lut_based_ops.h>).
AIE2P has hardware tanh (aie::tanh<bfloat16>(accum.to_vector<float>())). Using the wrong one
fails to compile or produces wrong output. See the tanh section below for exact call patterns.
Check AGENTS.local.md for which NPU generation is on this machine.
AIE tile microarchitecture
Each AIE compute tile contains two separate processors that run in parallel:
1. Scalar VLIW processor (the "chess core")
- 32-bit RISC-like, handles control flow, addressing, scalar arithmetic
- Runs loop counters, pointer increments, branch logic
AIE_PREPARE_FOR_PIPELININGtells the Peano scheduler to overlap scalar and vector ops across iterations (software pipelining) — scalar computes addresses for N+1 while the vector unit executes N
2. Vector SIMD processor (the "vector unit")
- Processes 16 BF16 values per cycle (AIE2: 256-bit; AIE2P: 512-bit wide but same API)
- Target of
aie::mul,aie::add,aie::tanh, etc. aie::begin_restrict_vector<16>feeds the vector unit load/store path
Plus a DMA engine running concurrently with both processors
- Moves data between tile-local SRAM and the NoC / neighboring tiles
- ObjectFIFOs in IRON map to DMA channels — tile N+1 is fetched while N executes
Key gotcha — auto vs explicit vector type: aie::mul(vec, vec) returns
accum<__accfloat, 16>, not vector<bfloat16, 16>. Using auto for intermediates
keeps them as accum; the scalar core can't bypass an accumulator back to the vector
input path, stalling the pipeline and breaking chained ops (no mul(vec, accum) overload).
Always declare intermediates as aie::vector<bfloat16, 16> explicitly.
Tile local memory: 64 KB SRAM per compute tile, shared between both processors and the DMA. At BF16 (2 bytes): 64 KB = 32 K elements. The streaming tile pattern (objectfifo) exists precisely because full tensors don't fit — only the active tile slice lives in SRAM at once.
This is 64 KB on both AIE2 (NPU1) and AIE2P (NPU2) — BaseNPU1TargetModel and
BaseNPU2TargetModel both derive from AIE2TargetModel, whose
getLocalMemorySize() is 0x10000. Do not use 32 KB: that is the AIE1/Versal
value (AIE1TargetModel, 0x8000), which UG1079 documents and which does not
apply to any XDNA NPU. Verify against the installed toolchain rather than a
Versal-era manual:
mlir_aie/include/aie/Dialect/AIE/IR/AIETargetModel.h.
Memory tiles: 512 KB each (getMemTileSize() = 0x80000), one per column —
so 2 MiB total on NPU1 (4 columns), 4 MiB on NPU2 (8 columns). AIE1 has no
memory tiles at all (getMemTileSize() = 0).
Toolchain locations
| Tool | Path |
|---|---|
| Python venv | ~/.venv |
| aiecc | ~/.venv/bin/aiecc |
| Peano (clang++) | ~/.venv/lib/python3.12/site-packages/llvm-aie/bin/clang++ |
| AIE headers | ~/.venv/lib/python3.12/site-packages/mlir_aie/include/ |
| Pre-built AIE2 kernels | ~/.venv/lib/python3.12/site-packages/mlir_aie/include/aie_kernels/aie2/ |
| Pre-built AIE2P kernels | ~/.venv/lib/python3.12/site-packages/mlir_aie/include/aie_kernels/aie2p/ |
| AIE2 runtime lib (LUT tanh) | ~/.venv/lib/python3.12/site-packages/mlir_aie/aie_runtime_lib/AIE2/ |
| AIE2P runtime lib | ~/.venv/lib/python3.12/site-packages/mlir_aie/aie_runtime_lib/AIE2P/ |
| xclbinutil | /opt/xilinx/xrt/bin/xclbinutil — not on PATH by default; prepend manually |
Critical: xclbinutil is at /opt/xilinx/xrt/bin/ but not on PATH. aiecc silently skips
xclbin assembly if it can't find it — no error, just no .xclbin output. Always prepend:
os.environ["PATH"] = "/opt/xilinx/xrt/bin" + os.pathsep + os.environ.get("PATH", "")
IRON API quick reference
from ml_dtypes import bfloat16 # NOT np.bfloat16 — numpy doesn't have it
import numpy as np
from aie.iron import ExternalFunction
from aie.iron.algorithms.transform import transform_parallel_binary, transform_binary
from aie.iron.device import NPU1, NPU2
from aie.utils import set_current_device
from aie.utils.compile import compile_mlir_module, compile_external_kernel
# Prefer detect_npu() over hardcoding; see "Autodetection" section below.
set_current_device(NPU1()) # must be called before any transform_* call
tile_ty = np.ndarray[(tile_size,), np.dtype[bfloat16]]
kernel = ExternalFunction(
name="my_kernel",
source_file="/path/to/kernel.cc",
arg_types=[tile_ty, tile_ty, tile_ty, np.int32], # (in0, in1, out, n)
include_dirs=[str(AIE_INCLUDE), str(AIE_RUNTIME_LIB)], # AIE2: runtime lib has lut_based_ops.h
)
# Single-core (for debugging / small sizes)
mlir_module = transform_binary(kernel, in0_buf, in1_buf, out_buf, tile_size=tile_size)
# All columns in parallel (preferred)
mlir_module = transform_parallel_binary(kernel, in0_buf, in1_buf, out_buf, tile_size=tile_size)
tile_size must be a multiple of 16. hidden_size must be divisible by tile_size * num_cols.
Get num_cols from get_current_device().cols after set_current_device.
Autodetection
Detect the installed NPU generation at build time via pyxrt rather than hardcoding:
_NAME_TO_NPU = {
"npu1": "npu1", "Phoenix": "npu1",
"npu4": "npu2", "npu5": "npu2", "npu6": "npu2", "Strix": "npu2", "Krackan": "npu2",
}
def detect_npu() -> str:
import ctypes, sys
ctypes.CDLL("/opt/xilinx/xrt/lib/libxrt_coreutil.so.2", mode=ctypes.RTLD_GLOBAL)
sys.path.insert(0, "/opt/xilinx/xrt/python")
import pyxrt
name = pyxrt.device(0).get_info(pyxrt.xrt_info_device.name)
for substr, key in _NAME_TO_NPU.items():
if substr in name:
return key
raise RuntimeError(f"Unknown device: {name!r}")
pyxrt.xrt_info_device.name returns strings like "RyzenAI-npu1" or "RyzenAI-npu4".
See tools/npu/build_qwen35_swiglu.py for the full implementation. The Cargo
build.rs passes --npu auto by default; override with HIPFIRE_NPU_TARGETS=npu1,npu2.
Compilation
import tempfile, shutil
from pathlib import Path
# target_arch: "aie2" for NPU1, "aie2p" for NPU2
with tempfile.TemporaryDirectory(prefix="hipfire_npu_build_") as tmp:
tmp = Path(tmp)
compile_external_kernel(kernel, tmp, target_arch="aie2")
compile_mlir_module(
mlir_module=mlir_module,
insts_path=tmp / "insts.bin",
xclbin_path=tmp / "final.xclbin",
work_dir=tmp,
)
shutil.copy2(tmp / "final.xclbin", out_dir / xclbin_name)
shutil.copy2(tmp / "insts.bin", out_dir / instr_name)
Produces:
final.xclbin— path1 forxdna1_bf16_*_create; DPU kernel id0x901, kernel nameMLIR_AIEinsts.bin— path2 forxdna1_bf16_*_create; raw NPU instruction binary
Kernel C++ conventions (AIE2 + AIE2P portable)
The toolchain is identical across NPU generations — same Peano compiler, same
mlir_aie package. Only the target_arch flag ("aie2" / "aie2p") and the
tanh implementation differ. Gate on __AIE_ARCH__ (20=AIE2, 21=AIE2P).
#include "aie_kernels/aie_kernel_utils.h"
#include <aie_api/aie.hpp>
// AIE2: include BOTH lut_based_ops files — .h has getTanhBf16 (inline) + extern
// table decls; .cpp has the actual table data. The .cpp does NOT include .h.
// AIE2P: no LUT needed; aie::tanh is hardware but takes float input (see below).
#if __AIE_ARCH__ == 20
# include "lut_based_ops.h"
# include "lut_based_ops.cpp"
#endif
#include <stdint.h>
using namespace aie;
extern "C" {
void my_kernel(bfloat16 *restrict in0, bfloat16 *restrict in1,
bfloat16 *restrict out, const int32_t n) {
auto it0 = aie::begin_restrict_vector<16>(in0);
auto it1 = aie::begin_restrict_vector<16>(in1);
auto it_o = aie::begin_restrict_vector<16>(out);
AIE_PREPARE_FOR_PIPELINING
AIE_LOOP_MIN_ITERATION_COUNT(1)
for (int i = 0; i < n; i += 16) {
aie::vector<bfloat16, 16> a = *it0++;
aie::vector<bfloat16, 16> b = *it1++;
// Use explicit vector<bfloat16,16>, not auto, for all intermediates
aie::vector<bfloat16, 16> result = aie::mul(a, b);
*it_o++ = result;
}
}
}
aie::mul return type and the auto trap
aie::mul(vec, vec) returns accum<__accfloat, N>, not vector<bfloat16, N>.
-
AIE2: The chess scalar core cannot bypass an accumulator back into the vector input path. Using
autofor any intermediate that feeds anothermuloraddstalls the pipeline because there's no vec×accum overload. Always use explicitaie::vector<bfloat16, 16>for those intermediates — the implicitaccum → vector<bfloat16>conversion fires on assignment. -
AIE2P:
autois generally fine for non-tanh paths, but any variable that will be passed as an argument to anotheraie::mulmust be explicitly typed asaie::vector<bfloat16, 16>. There is nomul(vector<bfloat16>, accum)overload on either arch.
tanh: arch-specific call patterns
AIE2 (NPU1) — LUT-based:
// x must be vector<bfloat16,16>
aie::vector<bfloat16, 16> t = getTanhBf16(x);
AIE2P (NPU2) — hardware, but input must be vector<float>:
// half_x is the accum from aie::mul(x, 0.5f)
auto half_x_acc = aie::mul(x, half);
auto t = aie::tanh<bfloat16>(half_x_acc.to_vector<float>());
aie::tanhonly has avector<float, N>overload — passingvector<bfloat16>directly gives "no matching function"..to_vector<float>()is a method onaccum, not onvector<bfloat16>. Ifhalf_xhas already been stored asvector<bfloat16>, there is no.to_vectoron it — keep the accum alive withautountil the tanh call.- The
<bfloat16>template parameter onaie::tanhselects the return type.
Tile grid and column_width
xrt-smi "6×5=30" = 6 columns × 5 rows = 30 total tiles (all types). Breakdown:
- 6 physical columns: 1 shim-only column + 5 data columns
- 5 rows: 1 shim row + 4 core rows
NPU1().cols = 4= compute columns that accept kernel PDIs- Firmware
total_col = 5= max column-width a hw_context may request
Driver validation: num_col = xclbin.column_width * core_rows / core_rows = xclbin.column_width.
If column_width > 5 → DRM_AMDXDNA_CREATE_HWCTX returns EINVAL.
transform_parallel_binary with set_current_device(NPU1()) correctly generates column_width=4.
A stale xclbin built when the device was set to NPU2 will have column_width=8 and silently fail
at runtime. Always rebuild xclbins after switching --npu target or changing the build script.
Host runtime
aie.xrt.XCLBin is broken under XRT 2.25 (hardcoded assertion "RyzenAI-Phoenix",
but XRT 2.25 returns "RyzenAI-npu1"). Use XRTHostRuntime instead:
import ctypes, os, sys
# Must be loaded BEFORE pyxrt to supply a weak vtable symbol
ctypes.CDLL("/opt/xilinx/xrt/lib/libxrt_coreutil.so.2", mode=ctypes.RTLD_GLOBAL)
sys.path.insert(0, "/opt/xilinx/xrt/python")
from aie.utils.hostruntime.xrtruntime.hostruntime import XRTHostRuntime
from aie.utils.hostruntime.xrtruntime.tensor import XRTTensor
from aie.utils.npukernel import NPUKernel
import numpy as np
from ml_dtypes import bfloat16
# Allocate host-accessible device buffers (group_id=0, host_only flags)
t_in0 = XRTTensor(input_array, dtype=bfloat16, device="cpu")
t_out = XRTTensor((hidden_size,), dtype=bfloat16, device="cpu")
npu_kernel = NPUKernel(xclbin_path=xclbin, insts_path=instr, kernel_name="MLIR_AIE")
rt = XRTHostRuntime()
handle = rt.load(npu_kernel)
result = rt.run(handle, [t_in0, t_out])
t_out.to("cpu")
output = t_out.numpy()
The internal kernel call is kernel(3, insts_bo, insts_nbytes, *data_bos).
Instruction BO uses flags=pyxrt.bo.cacheable, group_id=kernel.group_id(1);
data BOs use flags=pyxrt.bo.host_only, group_id=0.
libhipfire_xdna1.so FFI contract
Artifacts feed into xdna1_bf16_swiglu_create(xclbin_path, instr_path, hidden_size, result_out).
See crates/hipfire-arch-qwen35/src/xdna1_ffi.rs for the full Rust FFI wrapper.
Env vars to activate:
export HIPFIRE_QWEN35_FFN_BF16=xdna1
export HIPFIRE_QWEN35_XDNA1_XCLBIN=/path/to/qwen35-swiglu-8960.xclbin
export HIPFIRE_QWEN35_XDNA1_INSTR=/path/to/qwen35-swiglu-8960-instr.bin
export HIPFIRE_XDNA1_LIB=target/npu/libhipfire_xdna1.so # default
export HIPFIRE_QWEN35_FFN_BF16_TRACE=1 # per-layer timing
Build script template
See tools/npu/build_qwen35_swiglu.py and tools/npu/silu_mul_bf16.cc.
Pre-built kernel sources
~/.venv/lib/python3.12/site-packages/mlir_aie/include/aie_kernels/
Both aie2/ and aie2p/ subdirectories exist. Available kernels:
| File | Computes |
|---|---|
swiglu.cc |
silu(x·w2) · (x·w1) — fused weight-mul (use our silu_mul_bf16.cc when projections are pre-done) |
silu.cc |
silu(x) elementwise |
rms_norm.cc |
RMSNorm (reduction + scale) |
rope.cc |
RoPE rotary embeddings |
mm.cc |
Matrix multiply |
gelu.cc |
GeLU activation |
softmax.cc |
Softmax |
Under-utilized hardware features (ranked)
Source of truth: docs/npu/npu-kernel-design-guide.md:779 (op-surface survey vs. actual
hipfire usage). Value is framed against the documented tax: real W4A8 prefill is pinned at
~5 TOPS of ~40–55 capacity by per-tile C load/store + objectfifo lock ping-pong, and M512
batching is blocked by memtile/shim channel saturation.
Most current kernels are SPMD (one program replicated across tiles) fed through
shim→memtile→core objectfifos. That is one point in the design space, not the hardware's
model — each tile has its own program memory, so cores may run entirely different
programs (benchmarks/npu_gemm_tuning/r71/r71_gen.py places 17 distinct kernel functions
by (col,row); even columns get output-projection accumulators, all columns get attention
buffers). Several features below exist specifically to break the walls that the
SPMD+memtile pattern runs into.
| # | feature (MLIR op) | status | lever |
|---|---|---|---|
| 1 | accumulator cascade (aie.cascade_flow, put_mcd/get_scd) |
unused (r5 proto only) | split K down a column, C stays in the flowing accumulator, stored once → kills the per-tile C reload. ~3–8× prefill. |
| 2 | runtime params (aiex.npu.rtp_write) |
unused | one parametric xclbin across tile-count/batch/scale — fixes program-store overflow + M256→M512 without a new image. |
| 3 | packet-switched routing (aie.packet_flow, amsel, dma_bd_packet) |
unused | ID-tag multiple streams over one channel → attacks the shim-stream ceiling + saturated memtiles. |
| 4 | DMA n-D transform + pad (dma_bd bd_dim_layout/bd_pad_layout) |
barely used | free transpose/reshape/K-tail-pad in the memtile → frees vector cycles spent on aie::transpose. |
| 5 | counting-semaphore / lock-free (AcquireGreaterEqual, disable_synchronization) |
unused | batch-acquire N objects, skip lock gen where static-safe → cuts the per-tile acquire/release tax. |
| 6 | wider resident-C tiling (deeper aie::mmul expansion) |
partial (2×2) | keep more C sub-tiles in the register file across K → fewer C load/stores (lower-risk partial of #1). |
| 7 | core direct stream (aie.put_stream/get_stream) |
unused | low-latency small-operand path (bias, per-group scales, RMSNorm stats) without a DMA/lock round trip. |
| 8 | trace unit (aie.trace, trace.event) |
unused | per-tile event/PC/cycle trace to host — localizes which stage costs the tax (stalls are otherwise only inferred). |
| 9 | parametric BD chains (aie.bd_chain, dma_start_bd_chain) |
unused | define ping-pong once, concretize at runtime → shrinks runtime_sequence bloat (dispatch-floor lever). |
| 10 | fuller weight-replay (iter_count/repeat_count) |
partial, BLOCKED | one weight object feeds many activation macros; gated on #3 to free the channel it needs. |
The rule: a feature is "under-utilized" only until measured to help — log the ones that
don't. #1 (cascade) is measured at 4–10× over the memtile GEMM on feed-free compute
(npu-kernel-design-guide.md:726); the cascade stream sits unused in every shipped kernel,
AMD's included. #2/#3/#4 most directly attack the program-store and channel-saturation
bottlenecks.
Constraints that are real (per-tile, not per-design)
- L1 64 KB per core — an objectfifo's buffer × depth must fit (depth 2 double-buffering halves your working tile).
- Program store 16 KB per core — binds each core's own text, not the sum of a heterogeneous design. Balancing work between cores is how overflow gets fixed.
- ~2 MM2S channels per column shim — feeding >1 core/column directly fails aiecc with "number of output DMA channel exceeded"; requires memtile routing (or #3).
- npu1 DMA BD dimension cap = 1023 (10-bit). Contiguous stripes larger than that must be
split into two dims;
benchmarks/npu_gemm_tuning/r14/r14_gen.py::_splitis the reference implementation. npu2 does not have this cap, so npu2-authored generators often omit it. - DDR ceiling ~13–16 GB/s on npu1, saturating at only ~4–8 cores — more cores cannot
raise aggregate bandwidth (
benchmarks/npu_gemm_tuning/r12/README.md).
Do not confuse the aggregate DDR ceiling with the per-kernel weight path. A W4A8 multi-core GEMM at DFlash shapes (M=16) measures ~9.3 GB/s on the weight stream — below the aggregate ceiling, and that gap is real headroom, not measurement noise. The distinction was established by a control: cutting 1 MB of redundant activation traffic left the time unmoved (899.3 → 905.5 µs), so weight bytes bind while A and C ride concurrently on other channels and are effectively free. Computing aggregate-bytes ÷ time and calling it a ceiling will silently overstate how saturated you are — measure the binding stream, and prove which one binds by varying the others.
