Skip to main content

The Neural Processor — Reference Notes

Condensed notes for the slides. Everything here is either measured on duos-frog-9f7f (SG2000, SECOND_CPU=arduino-tpu) or quoted from a vendor document; the two are marked.

Numbers marked measured came from a command on the board. Numbers marked spec are from a datasheet or manual and were not reproduced.

Part I — The Machine

The part

Vendor name Cvitek TPU
Internal name bm1822 (useless as a search term — it collides with materials science and finance)
Claimed peak 0.5 TOPS INT8 (spec)
Clock fpll/3 = 500 MHz, independent of the CPU
Registers tiu @ 0xc101000, tdma @ 0xc100000
Local memory 32 KB, 8 banks × 4 KB @ 0x0C000000
Device node /dev/cvi-tpu0
Driver soph_tpu.ko
Runtime libcviruntime.so, libcvikernel.so, libcvimath.so
Symbols in libcvikernel 313 (measured)
CPU next to it C906, RV64GCV, 1050 MHz measured with rdcycle

The C906 is RVV 0.7.1, not 1.0. Debian gcc-14 with -march=rv64gcv emits RVV 1.0 and gives you SIGILL. Use the Xuantie toolchain with -march=rv64imafdcv0p7 -mabi=lp64d. Intrinsics are compiler builtins, not library functions; GCC 10 does not auto-vectorise here.

The operation list

This is the machine. There is no thirteenth thing.

convolution            depthwise_convolution    matrix_multiplication
max_pooling            min_pooling              avg_pooling
element_wise_add       element_wise_mul         element_wise_ge
lut / lookup_interp_table                       shuffle
tensor_copy (g2l, l2g, l2l)                     tensor_copy_stride

The 313 exported symbols are variants of these: per-dtype, per-layout, per-shape entry points.

What is missing

Missing Consequence
any scalar unit no scalar code
branch / control flow no data-dependent control flow
divide none, anywhere
exp, log, sqrt, tanh nonlinearities are interpolated table lookups

softmax is therefore not a primitive. It decomposes into max, subtract, LUT-exp, sum, and a reciprocal — and the reciprocal is a LUT and a multiply, because there is no divide.

Memory

32 KB of software-managed scratchpad. No cache.

There is no tag array, no replacement policy, and no cache miss — because there is no cache. The compiler emits g2l (global to local) and l2g copies around each operation, and the memory schedule is part of the compiled artifact.

The comparison that matters: the C906 next door has a 32 KB L1 and a 64 KB L2. The accelerator’s entire memory is the size of one CPU cache. Every model of any size is therefore a streaming problem — tile the computation so each tile’s working set fits in 32 KB, and DMA tiles in and out.

This is the GPU deck’s “expose or hide” decision taken to its conclusion, with two orders of magnitude less memory than a compute unit’s LDS.

CV181X_HW_NPU_SHIFT = 3 and EU_SHIFT = 4 give 8 × 16 = 128 lanes. That does not obviously produce 0.5 TOPS at 500 MHz and the discrepancy is unresolved; do not quote a reconciliation.

The three-way comparison

CPU GPU NPU
Area goes to not stalling something else to run the one operation
Programmable yes yes no
Programmer supplies an algorithm parallelism a graph
Breaks when cache too small work diverges op not on the list
Degrades gracefully catastrophically absolutely

The last row is the one that matters. A slow CPU program is slow. A badly shaped GPU kernel is 16× slow. An NPU given an operation it does not have does not run at all — no emulation, no fallback, no “it will just be slower”.

Part II — Precision

The quantisation enum

typedef enum { NONE, BF16, INT8_SYM, INT8_ASYM } QuantType;

There is no FP32. --quantize F32 fails at lowering:

ASSERT INFO: Not Implemented
  at .../TopLowering.h:366

Note that it is a lowering failure, not a parse failure: the graph type-checks and then cannot be emitted.

So quantisation is not an optimisation applied at the end. It is the interface.

bf16 is real compute

Operand bf16 — 1 sign, 8 exponent, 7 mantissa
Accumulator fp32
Output rounded to bf16

Evidence. Multiply matrices of small integers in bf16 on the NPU and compare against bf16_round(exact_int64) computed on the host: 0 of 65,536 elements differ (measured). Max absolute error against the exact answer is 4.0 near a value of 1473, which is exactly half a bf16 ULP at that magnitude.

The test is the argument. If the accumulator is fp32 and rounding happens once, the output must equal bf16_round(exact). If it accumulated in bf16, or rounded per partial sum, it would drift. It matches exactly.

An earlier version of these notes said bf16 was storage-only. That was wrong.

What you pay

  • No FP32. Every model is a quantisation decision.
  • Static shapes. One compiled model per shape.
  • 32 KB of scratchpad, no cache.
  • No standard. Compiler, runtime and driver are one vendor’s.

The fixed-function bet buys efficiency by giving up generality; the bill arrives as a compiler you did not write and cannot fix.

Part III — Archaeology

Not examinable. Three dates, one argument.

Year The claim
Kung & Leiserson, Systolic Arrays for VLSI 1978 Put the dataflow in the wires; values meet in the middle
DianNao (Chen et al., ASPLOS) 2014 A fixed-function accelerator beats a general core on energy per op
TPU v1 (Jouppi et al., ISCA) 2017 Ship it: 256 × 256 MAC array, int8 only, no caches

Kung & Leiserson is the origin: values flow between processing elements on a fixed rhythm, so each element does a multiply-add per beat and never touches memory. The name is a physiology metaphor.

DianNao is the paper that made the argument in energy terms rather than FLOPs, and its central observation — that for a neural accelerator the memory system dominates energy by an order of magnitude, so the design work is buffering and reuse, not arithmetic — is exactly the 32 KB slide.

TPU v1 is the same design point as bm1822, four years earlier and ~400× bigger.

The idea waited fifty years for a workload regular enough to schedule at compile time. That is a statement about workloads, not architectures — the same reason Itanium’s static scheduling failed and Groq’s works.

Part IV — The Stack

Five layers

your model          ONNX
      │
TPU-MLIR            the compiler             ← open source
      │
.cvimodel           a schedule, not code
      │
CVI_NN / CVI_RT     model API / context API
      │
cvikernel           the 313 operations
      │
/dev/cvi-tpu0       the driver
      │
bm1822              tiu @ 0xc101000, tdma @ 0xc100000

Above the line you own. Below it you do not. The line is in an unusual place: the compiler is open and everything under it is a blob.

On a GPU the ISA is public and the driver is closed. Here the compiler is the hackable piece, so debugging means reading the emitted schedule and inferring what the hardware did.

The compiler

model_transform --model_name dec --model_def decode.onnx \
    --input_shapes [[1,128,224],[1,128,224],[1,1,224],[1,128,224],[1]] \
    --mlir dec.mlir

model_deploy --mlir dec.mlir --chip cv181x --quantize BF16 \
    --model dec_bf16.cvimodel
  • Input: ONNX or Caffe. Not TFLite. PyTorch goes via jit.trace.
  • Quantisation: INT8 (symmetric or --asymmetric), BF16, F16. No FP32.
  • Shapes: static only. --merge_weight plus model_tool --combine shares weights across several shapes.
  • Chips: cv180x, cv181x (ours), cv182x, cv183x, cv184x, cv186x.

model_tool --info reports what you need before you run anything:

Cvimodel Version: 1.4.0
CviModel Need ION Memory Size: 8.5 MB

Version 1.4.0 is emitted by TPU-MLIR 1.30.2 and accepted by the board’s 2024.12 runtime — the format has been stable across a two-year gap. CVI_NN_GetModelVersion on the board tells you what the runtime accepts.

A .cvimodel is Model → Program → Routine → Tensor plus a command buffer: a pre-computed list of tensor operations with their DMA schedule. It is not a program. There is no branch, no loop, no call.

Is there a standard?

No.

Attempt Outcome
NNEF (Khronos) an exchange format with no runtime. Effectively dead
ONNX Runtime has execution providers — none for cv181x
OpenCL / Vulkan compute not supported by this runtime
Caffe supported
TFLite not supported, despite what the marketing material says

Every vendor has the same shape: RKNN (Rockchip), QNN (Qualcomm), Edge TPU (Google), Ethos-U (Arm). A compiler, a blob format, a runtime, and lock-in.

ONNX is the only standardised piece, and it is standardised at the input to a compiler that belongs to one vendor.

The samples are all camera-shaped

About 60 binaries in /mnt/system/usr/bin/ai. Every one goes through the TDL SDK, which builds a VPSS frame before loading a model. Our image does not load the camera stack, so none of them can run.

The only entry point is the bare API:

CVI_NN_RegisterModel(path, &model);
CVI_NN_GetInputOutputTensors(model, &in, &n_in, &out, &n_out);
memcpy(in[0].sys_mem, data, in[0].mem_size);
CVI_NN_Forward(model, in, n_in, out, n_out);
CVI_NN_CleanupModel(model);

Five calls. examples/npu/run/run_model.py is exactly this in ~150 lines of ctypes.

There is no compiler on the board — no gcc. Probes are written in Python against the runtime library with hand-declared structs. That is a feature: a ctypes probe is a text file you can scp and edit.

Part V — Present and Unusable

The symptom

Everything a working platform looks like:

$ ls /dev/cvi-tpu0              crw------- root root 245, 0
$ lsmod | grep soph_tpu         soph_tpu  28822  0
$ ls /mnt/system/lib/           libcviruntime.so  libcvikernel.so  libcvimath.so
$ nm -D --defined-only libcvikernel.so | wc -l
313

And it dies on the smallest allocation the API can express:

CVI_RT_Init -> rc=0
CVI_RT_MemAlloc(4096) ...
ion ioctl fail:: Cannot allocate memory
python3: cvi_device_mem.cpp:451: mem_alloc_raw: Assertion `0' failed.
Aborted (core dumped)          # exit 134

Not weights. Not a tensor. Not a command buffer. 4 KiB.

The library asserts

// cviruntime/src/soc/common/cvi_device_mem.cpp:451
assert(0);          // ← reached on ENOMEM

A recoverable ENOMEM becomes SIGABRT. Therefore:

  • CVI_NN_RegisterModel cannot report out-of-memory — it will not return.
  • Every line of error handling above it is unreachable.
  • You cannot write a program that gracefully degrades when the NPU is unavailable, because asking kills you. You must know out of band.

This is why the fix had to be a build-time decision rather than a runtime check.

ION and the carveout

A fixed-function accelerator addresses memory by physical address. There is no IOMMU, so the memory must be physically contiguous, and physically contiguous memory cannot be conjured at runtime.

reserved-memory {
    ion { compatible = "ion-region"; size = <0x00 0x4000000>; };  /* 64 MiB */
};

The carveout is taken before the kernel’s page allocator exists. reserved-memory is the only API: no module parameter, no sysfs knob, no runtime override. cvitek_ion.c reads the size at probe time.

And “a heap that allocates” is not “a heap that works”:

heap mmap + write paddr
ion_system_heap ok 0x0
unnamed ok 0x0
unnamed ENOMEM —

Two of three heaps hand back memory a DMA engine cannot use — mappable, writable, and not physically contiguous.

Cvitek’s ION is not the upstream ION uapi. ion_allocation_data has no align field and carries fd/paddr/name; ION_IOC_HEAP_QUERY does not implement the “ask for zero to learn the count” convention. Guess the struct and you get ENOTTY.

One knob, two meanings

# configs/duos/settings.mk
SECOND_CPU ?= camera
ifeq ($(SECOND_CPU),arduino)
ION_SIZE=0

The SG2000 has one C906L that can either run the vendor camera stack or be handed to Arduino via remoteproc. S00kmod decided which by asking how big the carveout was:

if [ $ion_size != 0 ]; then insmod soph_vi.ko; insmod soph_vpss.ko; ...; fi
insmod soph_tpu.ko        # unconditional
ION_SIZE > 0  ⇒  carveout exists  ⇒  camera stack loads
                                  ⇒  vendor RTOS takes the C906L
                                  ⇒  Arduino personality is dead

The NPU needs the carveout. Arduino needs the C906L. Nothing about those two facts should have been connected.

The soph_tpu.ko line is the tell: the TPU module loads unconditionally, so the system looks like it has a TPU regardless of the carveout.

The fix

Four changes to the image build, not the board:

file change
configs/duos/settings.mk new SECOND_CPU=arduino-tpu, ION_SIZE=64
configs/chip/sg200x/chip.mk inject the rproc fragment for the new personality
scripts/addons/load-systemko/S00kmod gate the camera stack on /boot/arduino
scripts/addons/preconfigure/addon.mk write /boot/arduino for both flavours

The condition became “is this an Arduino build” — what it always meant — instead of “is the carveout non-zero”, which was a proxy that happened to correlate.

A new SECOND_CPU value rather than ION_SIZE=64 on the command line, because SECOND_CPU is part of the docker build-volume key. Overriding ION_SIZE alone reuses the same build tree, and $(BUILDDIR)/$(BOARD)-$(VARIANT)/memmap.py has no prerequisites — so make does not regenerate it and you get a stale memory map compiled into a fresh-looking image.

Caution

The first attempt edited the ion size in the .dtb on the running board and rebooted it. The board never came back and had to be reflashed.

reserved-memory is consumed before the kernel’s allocator exists, so a wrong size does not fail gracefully. There is no recovery path that does not involve removing the card. Never patch a device tree on a board without a console, and never reboot one.

Part VI — The Microbenchmark

Method

Operands are integers in [-8, 7], so:

  • bf16 represents them exactly (7-bit mantissa),
  • int8 represents them exactly,
  • the reference is computable in int64, exactly.

Only compare like with like:

NPU CPU
bf16 bfv (RVV) / bfs (scalar)
int8 i8v (RVV) / i8s (scalar)

An earlier version of this work had an “NPU bf16 vs CPU int8” column. It was wrong and its conclusions were withdrawn. bf16 and int8 compute different functions.

Results

All times in ms, N × N multiply (measured).

N NPU bf16 NPU int8 CPU i8v CPU i8s CPU bfv CPU bfs
32 0.25 0.34 0.05 0.15 0.18 0.23
64 0.28 0.56 0.38 1.22 1.49 1.96
128 0.39 1.36 3.93 9.91 11.87 14.40
256 0.95 4.65 26.43 94.24 114.41 133.53
512 4.49 18.25 220.76 916.86 2295.38 1976.07

bf16, NPU vs CPU bfv (bit-identical output)

N NPU CPU who wins
32 0.25 0.18 CPU 1.4×
64 0.28 1.49 NPU 5.3×
128 0.39 11.87 NPU 30×
256 0.95 114.41 NPU 120×
512 4.49 2295.38 NPU 511×

int8, NPU vs CPU i8v

N NPU CPU who wins
32 0.34 0.05 CPU 6.8×
64 0.56 0.38 CPU 1.5×
128 1.36 3.93 NPU 2.9×
256 4.65 26.43 NPU 5.7×
512 18.25 220.76 NPU 12×

Crossover: N ≈ 64 (bf16), N ≈ 128 (int8). Set by a fixed per-inference cost of ~0.25–0.35 ms — the same number as the 898-parameter convnet’s 0.28 ms.

The ratio that grows

N³ predicts 8× for a doubling of N. Measured 256 → 512:

growth
NPU bf16 4.6× (better than N³ — amortising the fixed cost)
CPU bfv 20× (worse than N³ — fell out of cache)

At N=512 each matrix is 1 MB, three matrices is 3 MB, and the C906 has 32 KB L1 and 64 KB L2. The naive kernel walks a column with a 2 KB stride. So part of the widening ratio is the CPU’s problem, not the NPU’s triumph.

Correctness

pairing reference result
CPU i8s / i8v exact int64 A@B exact, max err 0
CPU bfs / bfv bf16_round(exact) exact, max err 0
NPU bf16 bf16_round(exact) exact, max err 0
NPU int8 exact int64 A@B max err 11.0

All five implementations produce bit-identical output at N=256 and N=512 (int8 exact; bf16 bf16_round(exact)).

NPU int8 is not exact: max error 11.0 against an output quantisation step of 11.49 — about one step. Input and weight thresholds were forced to 127 so the int8 operands would be exact and the accumulator scale 1.0, but a second int8 stage (AC_MatMul_si8 → C_MatMul, qscale 11.49) sits between the matmul and the fp32 output.

The anomaly

int8 is slower than bf16 on this NPU.

N bf16 int8 penalty
32 0.25 0.34 1.4×
64 0.28 0.56 2.0×
128 0.39 1.36 3.5×
256 0.95 4.65 4.9×
512 4.49 18.25 4.1×

Reproducible across all five sizes and over repeated runs. This inverts the marketing: INT8 is the headline “0.5 TOPS” mode and it is the slow path here. The consequence is large — the NPU’s advantage over the CPU is 511× in bf16 and 12× in int8, because the C906’s int8 path (16 lanes, widening MAC) is genuinely good.

Cause not identified. Hypotheses:

  1. An extra requantise routine — the tensor map shows two routines for int8 (matmul → int8 intermediate → requantise to fp32) and one for bf16. But the extra stage is 65,536 elements and should cost microseconds, not 3.7 ms, so this explanation does not fit the magnitude.
  2. Different tiling in the int8 matmul.
  3. A different accumulator path.

Distinguishing these requires reading the emitted TPU dialect, not the ONNX graph. That is the next experiment.

Caveats

The CPU kernels are hand-written and unoptimised: blocked int8 scalar with 4×4 register blocking, blocked int8 RVV, blocked fp32 scalar, blocked fp32 RVV. They reach 1.27 GOPS (int8 RVV) and 0.29 GFLOP/s (fp32 RVV), far from what the hardware should manage. Several attempts to improve them made things worse.

So: the crossover points are solid (set by NPU fixed cost, not CPU kernel quality); the large-N ratios are an upper bound.

Part VII — The Demo

The pipeline

text → espeak-ng (IPA) → token ids → encoder (CPU) → vocoder (NPU) → WAV

Steady state, measured through the web API:

stage ms
phonemise (espeak-ng) ~140
encoder (duration.onnx, onnxruntime on C906) ~230
vocoder (NPU, bucketed, resident) ~384
total ~760

For 1.82 s of audio — 2.4× real time, with the compute-heavy stage on the accelerator.

The split is forced by shapes

  • The encoder’s output length is sum(predicted durations) — data-dependent, so no static graph can express it. TPU-MLIR gives UNREACHABLE at .../Interfaces/Range.cpp:75.
  • The vocoder needs static shapes (bucketing supplies them) and is the expensive part.

python3-onnxruntime is packaged for riscv64 in Debian (1.21.0), and the encoder is 3.6 MB. So: shape-dynamic part on the CPU, expensive static part on the NPU.

Board dependencies:

sudo apt install -y python3-numpy python3-onnxruntime espeak-ng

phonemizer is not packaged for Debian, but it is a thin wrapper over espeak-ng, so espeak-ng -q --ipa does the same job.

The model

owensong/Inflect-Nano-v2-ONNX, Apache-2.0:

  • duration.onnx 3.6 MB — encoder + duration predictor
  • decode.onnx 12.6 MB — flow + HiFi-GAN vocoder
  • Both graphs are RNG-free. VITS’s stochastic sampling is externalised as a zp_noise input, so feeding zeros is deterministic. (This is what killed Piper: RandomNormalLike → UNREACHABLE in RandnLike.cpp.)
  • It ships the text frontend, which the vendor’s own TTS sample omits.

Shapes: text_len = 2·n_phonemes + 1, wav_len = 256·mel_len, 24 kHz.

Bucketing

mel_len ms audio real time model
128 192.6 1.37 s 7.1× 7.6 MB
256 379.3 2.73 s 7.2× 8.5 MB
512 800.2 5.46 s 6.8× 11.3 MB

Linear in mel_len — the vocoder is compute-bound and the fixed cost is amortised by 128 frames.

Zero-pad the mel along the time axis but keep y_mask at 1.0 over the padding, so the vocoder sees no hard edge and does not click. Trim the output to mel_len × 256 samples.

Boundary cost is real: a 258-frame sentence lands in the 512 bucket — 2× the work for 1.01× the audio. Measured: mel_len 258, vocoder 807 ms instead of the ~390 ms a 256-frame model would take.

Keep it resident

ms
per-request (shell out, register, run, exit) 3127
first call with resident handle 519
second call, resident 385

CVI_NN_RegisterModel allocates the weights and the command buffer in ION on every call. An 8× difference from a purely structural change.

The Gather bug

The compiler rejected four channel flips:

TgReverseKernel.cpp:18: Assertion `(axis == 0 || axis == 3)` failed

They came from /flow/flows.{1,3,5,7}/Slice — starts=[-1], ends=[INT64_MIN], axes=[1], steps=[-1]. The steps tensor comes from a Constant node, not an initializer, so scanning initializers misses it.

The mistake. Replace each flip with Gather(axis=1, reversed_indices). It compiles, runs at 7× real time, and is completely wrong:

expected:  channel 0 = 127 ... channel 127 = 0     (reversed)
NPU:       channel 0 =   0 ... channel 127 = 127   (identity!)

TPU-MLIR’s Gather on cv181x silently ignores the index vector. It is a pass-through. It fails open.

The diagnostic that separated precision from structure: feed constant inputs (0.5, 0.1, 1.0, 0.0 — all exact in bf16). Precision cannot change a constant-input result. The NPU produced rms 0.0294 against the reference’s 0.00415, so the fault was structural.

The fix — exploit what the backend does support (Reverse on axis 0):

Transpose(1,0,2) → Slice(axis 0, step −1) → Transpose(1,0,2)

Verified in isolation first (ch0=127, ch127=0), then applied.

test wave corr spectrum corr
constant inputs +0.9983 0.9993
random inputs +0.9311 0.9819
full text pipeline +0.9468 0.9774

rms 0.0604 (reference) vs 0.0625 (NPU). What remains is bf16 rounding noise.

The rule: any substitution made to work around a backend limitation needs its own isolated test before it is trusted. A compiler that accepts an operation and silently ignores part of it is worse than one that rejects it.

The A/B on the slide

Same sentence, same encoder output, same padding and trimming:

fp32 on the C906 (onnxruntime) 116,248 ms
bf16 on the TPU 807 ms
wave correlation 0.9252
spectrum correlation 0.9675

Both files are 66,048 samples at 24 kHz (2.75 s), aligned at lag 0.

The 116 s is a real measurement and not a fair one. onnxruntime on riscv64 has no tuned kernels for this core, so the generic CPU path is doing something much worse than 116 s worth of arithmetic. A hand-written fp32 kernel would be far faster and has not been written. What the number establishes is narrower: fp32 inference of a 12.6 MB model on this board, through the packaged runtime, is not a deployment option. Do not quote 144× as a speedup.

Part VIII — Synthesis

Seven machines

Machine Accelerator Attach Precision Programmable
sg2000 bm1822 · 0.5 TOPS on-die int8, bf16 no
celedyr Polaris 10 · 2016 PCIe slot fp32 yes
kraken MI50 · 2018 PCIe slot fp32 yes
typhon 8 × Gaudi2 · 2022 PCIe + switch bf16 yes
unicorn RX 7600 · 2023 PCIe slot fp32 yes
vampire Radeon 8060S · 2025 on-die, PCIe endpoint fp32 yes
pegasus Arc Battlemage · 2025 PCIe slot fp32 yes

One machine in this table cannot be programmed, and it is the one on the desk.

The sg2000 counterfactual

To put a programmable accelerator on this die you would need:

  1. A bus. The sg2000’s best off-chip link is USB 2.0 at 60 MB/s. One PCIe 4.0 ×16 slot is 31,500 MB/s — 525× more. There is nothing to attach a GPU to.
  2. Address space. A 4 GB physical map cannot hold a 16 GB BAR. You would get the 256 MiB paging window that celedyr has.
  3. Coherency. The C906’s caches do not snoop a DMA engine, and PCIe adds a non-coherent link with an IOMMU in between.
  4. Power and pins. A PCIe PHY and SerDes on a ~3 W part.
  5. Amdahl. A 1 GHz dual-issue C906 could not feed a GPU anyway.

Each is solvable. Together they describe a different chip.

The seven claims

  1. Fixed-function means no ISA. The op list is the machine; an operation that is not on it does not run — not slowly, not at all.
  2. Precision is the interface. No FP32 path. bf16 is real compute; int8 is a different function.
  3. The compiler is the open piece and the bugs live there. Gather fails open.
  4. A device can be present and unusable. Presence is not usability.
  5. Coupling is the bug. Two requirements expressed as one test costs days and leaves no trace in any log.
  6. Fixed cost sets the crossover. ~0.3 ms per inference; the accelerator loses below N≈64.
  7. Measure like with like, or you measure nothing.

The interesting engineering is not in the accelerator. It is in the two interfaces on either side of it: the compiler above, and the memory below.

Reproducing

# host: toolchain (gitignored, ~6 GB, Python 3.10 via uv)
./examples/npu/setup-tools.sh
./examples/npu/smoke/smoke.sh            # tiny ONNX -> cvimodel, no board

# board: is the NPU there?
cat /sys/kernel/debug/ion/cvi_carveout_heap_dump/total_mem   # 67108864
lsmod | grep -E 'soph_tpu|soph_vi'                           # tpu only
sudo python3 examples/npu/probe/probe.py                     # MemAlloc + paddr

# board: run a model
sudo python3 examples/npu/run/run_model.py model.cvimodel 20

# board: the matmul tables
sudo python3 examples/npu/mmul/run_mm.py model.cvimodel A.bin C.bin 30

# board: text to speech
python3 examples/npu/tts/pipeline.py "hello from the neural processor"

Known open items

  • int8 slower than bf16 on the NPU, 4.9× at N=256. Unexplained; needs the emitted TPU dialect.
  • The CPU matmul kernels are naive. RVV fp32 reaches only 1.3× over scalar where 3–4× is expected.
  • CV181X_HW_NPU_SHIFT/EU_SHIFT arithmetic does not reconcile with 0.5 TOPS. Unresolved.
  • The A/B audio has not been judged by ear. Correlation 0.9252 could still be audibly wrong.