The Neural Processor

An integer machine bolted to a floating-point one

CS 4250 — Computer Architecture

2026-11-04

Roadmap

One lecture. Five parts, then a lab on Friday that assumes you were here.

1. The machine — what it is, and what it cannot do
2. The stack — five layers, and why there is no standard
3. The bring-up — a device that was present and unusable
4. The numbers — one matmul, measured on both machines
5. The demo — text to speech, split across two processors

Parts 4 and 5 are the point. Parts 1–3 are what you have to know to believe them.

Say the shape out loud on the first slide. Students plan around it, and a one-lecture deck that turns into a survey has already lost.

The ordering is deliberate and slightly unusual: most NPU talks go architecture → toolchain → benchmarks, and the benchmarks are always a vendor bar chart. We are going to do the reverse. The bring-up comes before the numbers because the numbers are only interesting if you know what it cost to get them, and because “the device was present and unusable” is the single most transferable thing in this deck.

If you are running short, the two slides you can drop are The Samples Are All Camera-Shaped and Keep It Resident. Nothing later depends on either.

What We Will Not Do Today

• No neural networks. No layers, no backprop, no training. This is a computer architecture lecture that happens to be about a chip used for inference.
• No CUDA-equivalent tutorial. There is nothing to write. That is the subject, not an omission.
• No camera pipeline. The vendor’s samples all want one; we deliberately do not have one.
• No toolchain install. The compiler runs on your laptop and it is a six-hour afternoon. Friday’s lab uses models that are already compiled.

Two questions we will answer:

1. What does a fixed-function accelerator actually compute?
2. What does it cost you to use one?

Scope-setting matters more than usual because “NPU” invites both digression and hype. The neural-network material is a different course; the interesting architectural claim here is that this thing has no instruction set worth the name, and every consequence follows from that.

The toolchain point is worth being firm about. TPU-MLIR installs into a Python 3.10 venv, pins a yanked OpenCV, needs setuptools<80 because it imports pkg_resources, needs torch even for ONNX input, and ships a bundled glibc 2.35 that shadows a newer host libc and kills every subprocess. I have the script; it is not a lab exercise.

If someone asks “so how do I get a model on it” — the answer is in Part 2, and the honest answer is “you compile it on a different machine, and you debug it by faith”.

Where We Are

We have spent the term on the CPUs. The die has more than CPUs:

Block	What it is
C906 × 1	1 GHz RV64GCV, runs Linux 1050 MHz measured
C906L × 1	the little core, handed to remoteproc for Arduino
8051	always-on domain, no firmware shipped
TPU	Cvitek bm1822, 0.5 TOPS INT8
VPU	H.264/H.265 encode and decode
ISP, VPSS	camera front end

Half a gigabyte of DRAM, ~3 W, and an accelerator with a clock of its own: fpll/3 = 500 MHz, independent of the CPU’s 1050.

This is a recap of the Beyond the CPU deck’s engine zoo, with one block promoted to the front.

The clock detail is worth a sentence because students assume accelerators run at the CPU clock. The TPU is on its own divider from the video PLL — fpll/3. That is why the peak arithmetic and the measured arithmetic disagree by a factor of five later on, and it is a nice concrete instance of the clock-tree material from the power deck.

Do not oversell 0.5 TOPS. It is a marketing number computed at peak, and Part 4 is about the gap between it and reality.

Part I — The Machine

Two Ways to Spend a Transistor, Again

The GPU deck had two columns. This machine is the third point of the triangle.

	CPU	GPU	NPU
Area goes to	not stalling	having something else to run	the one operation
Programmable	yes	yes	no
Programmer supplies	an algorithm	parallelism	a graph
Breaks when	working set outgrows cache	work diverges	the op is not on the list
Degrades	gracefully	catastrophically	absolutely

A CPU hides latency. A GPU exposes it and makes you fix it. An NPU deletes the question — there is no latency to hide because there is no instruction stream.

Put the three columns on the board and leave them there.

The “degrades” row is the one to land. A slow CPU program is slow. A badly shaped GPU kernel is 16× slow. An NPU given an operation it does not have does not run at all — there is no fallback, no emulation, no “it will just be slower”. That is a qualitatively different failure mode and it is why the compiler matters so much more here than anywhere else in the course.

If someone objects that GPUs also refuse to run code they cannot compile: they do not. A GPU has an ISA. You can put anything in it. This machine has no ISA you can write, which is the next slide.

The Op List Is the Machine

libcvikernel.so exports 313 symbols. They are not instructions. They are operations, and the list is short enough to read:

convolution            depthwise_convolution    matrix_multiplication
max_pooling            min_pooling              avg_pooling
element_wise_add       element_wise_mul         element_wise_ge
lut / lookup_interp    shuffle                  tensor_copy (g2l, l2g, l2l)

That is the machine. There is no thirteenth thing.

Compare with the C906 next to it: RV64GCV, and the “G” alone is dozens of instructions before you get to the vector unit. This is a machine whose entire vocabulary fits on one slide.

The symbol count is real — nm -D --defined-only libcvikernel.so | wc -l on the board gives 313. Most of those are variants: per-dtype, per-layout, per-shape entry points for the same handful of operations. The distinct operations are what is on the slide.

The point to make: this is not a reduced CPU. It is a completely different kind of object. You cannot write a loop, because there is nothing to loop with. You describe a dataflow graph and the compiler schedules it onto these operations.

Good question to plant: what happens when your model needs an operation that is not on this list? Answer: it either gets decomposed into these, or it gets pushed off the chip to the CPU. Part 5 shows that happening for real.

No Divide, No Branch

What is missing from that list is more interesting than what is on it.

Missing	Consequence
any scalar unit	no scalar code at all
branch / control flow	no data-dependent control flow
divide	no division, anywhere
exp, log, sqrt, tanh	nonlinearities are table lookups

lookup_interp_table is how you get exp. You precompute a table on the host, the hardware interpolates, and that is your sigmoid.

So softmax is not a primitive — it is a chain of max, subtract, LUT-exp, sum, divide. And divide does not exist, so it is a reciprocal LUT and a multiply.

This is the slide that makes “what is an NPU” concrete. It is not a fast floating-point unit. It is a machine with no arithmetic beyond multiply and add, plus a table lookup, and everything else is built out of those.

The precision cost of the LUT is real and worth naming: an interpolated table over a bounded input range is not exp to full precision. That is fine for a sigmoid in the middle of a network and not fine for anything that needs to be accurate in the last bit. This is the same argument as the bf16 material in Part II, arriving early.

The practical consequence students will meet: any model with a divide, a sqrt, or an unusual activation either gets rewritten into LUT form by the compiler or gets pushed back to the CPU. The compiler will tell you which — and if it does not, Part 5’s Gather story is what happens when it lies.

32 KB

The entire on-chip memory is 32 KB, in 8 banks of 4 KB at 0x0C000000.

	Size
TPU local memory (lmem)	32 KB
C906 L1 data cache	32 KB
C906 L2	64 KB
The vocoder’s smallest bucket	7.6 MB

The accelerator’s entire memory is the size of one CPU cache. And unlike a cache, it is software-managed: it is a scratchpad, there is no cache controller, and if you want something in it you copy it there.

That copy is tdma at 0xc100000 — a DMA engine with its own registers. The memory schedule is part of the program. On a GPU the hardware hides this; here you write it.

This is the most architecturally interesting number in the deck and it deserves a minute.

32 KB is not a cache. There is no tag array, no LRU, no replacement policy, and no cache miss — because there is no cache. There is a scratchpad and a DMA engine, and the compiler emits g2l (global to local) and l2g copies around each operation. That is why the vendor’s “tensor map” output looks like a schedule and not like code.

The comparison to the CPU next door is the point: the C906’s L1 is the same size, and the C906 also has an L2. The accelerator has 32 KB and nothing else, which is why every model of any size is a streaming problem — you tile the computation so that each tile’s working set fits in 32 KB, and you DMA tiles in and out.

The GPU deck’s “expose or hide” slide made the same argument at 64 KB of LDS per compute unit. This is that decision taken to its conclusion, with two orders of magnitude less memory.

The EU_SHIFT arithmetic (CV181X_HW_NPU_SHIFT=3, EU_SHIFT=4, so 8 × 16 = 128 lanes) is worth mentioning only if someone asks about width — it does not obviously produce 0.5 TOPS, and I have not resolved the discrepancy. Say so rather than inventing a reconciliation.

Part II — Precision Is the Interface

int8 and bf16, and Nothing Else

The quantisation enum has four values:

typedef enum { NONE, BF16, INT8_SYM, INT8_ASYM } QuantType;

Ask for FP32 and the compiler does not warn you. It stops:

ASSERT INFO: Not Implemented
  at .../TopLowering.h:366

So quantisation is not an optimisation you apply at the end. It is the interface. There is no path from your float model to this chip that does not pass through a conversion, and the conversion is where the accuracy goes.

The Not Implemented is at the lowering stage, which means the FP32 graph parses, type-checks, and then dies when it tries to emit cv181x operations. That is worth distinguishing from a parse error: the toolchain accepts the model and then cannot lower it. Students who have fought compilers will recognise the feeling.

The vendor’s own manual says the part supports “efficient INT8 and high-dynamic-range BF16”, which is accurate and is also the whole story.

Do not let the room conclude that INT8 is the fast mode. Part 4 measures the opposite, and the explanation is still open.

bf16 Is Not Storage-Only

An easy and wrong conclusion: “bf16 is just a compact weight format, the maths happens in int8.” It is not.

Operand format	bf16 (1 sign, 8 exponent, 7 mantissa)
Accumulator	fp32
Output	rounded to bf16

Test: multiply two matrices of small integers in bf16 on the NPU, and compare against bf16_round(exact_int64_result) computed on the host.

0 of 65,536 elements differ.

Max absolute error against the exact answer is 4.0, near a value of 1473 — which is exactly half a bf16 ULP at that magnitude. That is not a bug. That is what bf16 costs.

I got this wrong early on and said bf16 was storage-only. The bit-exactness test is what settled it, and it is worth describing as a method rather than a result: if you believe the accumulator is fp32, then rounding once at the end must reproduce bf16_round(exact) exactly, and if the accumulator were bf16 you would see accumulated rounding drift. It does, and we do not.

The 4.0 error is the useful number for students. It is not “the NPU is imprecise”. It is “the NPU rounds to bf16, and half an ULP at 1473 is 4.0”. The error is exactly the format’s quantisation step, no more.

Why this matters for Part 5: bf16 being real compute is the reason the vocoder works at all, and bf16’s 8-bit exponent being the same as fp32’s is the reason the waveform does not clip or vanish.

What We Bought / What We Paid

Bought

• 0.5 TOPS INT8 on a ~3 W budget, on-die, no bus.
• Weights compress 4× against fp32, which matters when the whole machine has half a gigabyte of DRAM.
• Deterministic: fixed shapes, fixed schedule, no branch predictor to mispredict.

Paid

• No FP32 at all. Every model is a quantisation decision.
• Static shapes. One compiled model per shape.
• 32 KB of scratchpad and no cache.
• No standard. The compiler, the runtime, and the driver are one vendor’s.

The fixed-function bet buys efficiency by giving up generality. The bill arrives as a compiler you did not write and cannot fix.

This is the Part I summary of the whole deck, and it is worth reading slowly because every later slide is an instance of one of the four bullets in the “paid” column.

The “deterministic” bullet is underrated and worth a sentence: there is no branch predictor, so there is no misprediction, so the same input takes the same time every time. For a real-time system that is worth a lot — it is the same property the FreeRTOS deck wanted from the little core.

If someone asks about power: I have not measured it. The “~3 W” is the SoC figure, not the TPU’s share, and I would not quote a TPU-only number.

Part III — Archaeology

Archaeology: The Fixed-Function Idea

Three dates, and the same argument each time.

	Year	The claim
Kung & Leiserson, systolic arrays	1978	Put the dataflow in the wires; let values meet in the middle
DianNao	2014	A fixed-function neural accelerator beats a general core on energy per op
Google TPU v1	2017	Ship it: 256 × 256 MAC array, int8 only, no caches

The idea is fifty years old. What changed is not the idea — it is that the workload finally became regular enough to schedule at compile time.

Archaeology slide — not examinable. Flag that out loud.

Kung and Leiserson (1978, “Systolic Arrays for VLSI”) is the origin: instead of storing intermediate values, you let them flow between processing elements on a fixed rhythm, so each element does a multiply-add per beat and never touches memory. The name is a physiology metaphor — the heart pumps, the data pulses. Every NPU dataflow is a descendant.

DianNao (Chen et al., ASPLOS 2014) is the paper that made the case in energy terms rather than FLOP terms, and it is the one to recommend. Its central observation is that for a fixed-function neural accelerator, the memory system dominates energy by an order of magnitude, so the design work is about buffering and reuse, not arithmetic. That is exactly the 32 KB slide.

TPU v1 (Jouppi et al., ISCA 2017) is the one that shipped, and the detail worth quoting is that it is int8 only with no caches — a 256×256 MAC array fed from a software-managed buffer. The same design point as the bm1822 we have, four years earlier and 400× bigger.

The reason it waited is the interesting part, and it is the same reason Itanium’s static scheduling failed in the GPU deck: you can only schedule statically if the program is predictable. General-purpose code is not. A neural network is. That is a statement about workloads, not about architectures.

Part IV — The Stack, and the Absence of a Standard

Five Layers

  your model            ONNX
        │
  TPU-MLIR              the compiler        ← open source
        │
  .cvimodel             a schedule, not code
        │
  CVI_NN  /  CVI_RT     model API / context API
        │
  cvikernel             the 313 operations
        │
  /dev/cvi-tpu0         the driver
        │
  bm1822                tiu @ 0xc101000, tdma @ 0xc100000

Above the line you own. Below the line you do not. The line sits in an unusual place: the compiler is open and everything under it is a blob.

Draw this on the board and keep it up for the rest of Part IV.

The layering is the same shape as any accelerator stack — a compiler, a model format, a runtime, a kernel, a driver, hardware. What is different is the openness boundary. On a GPU, the ISA is public and the driver is a black box. Here the compiler is the open, hackable piece and everything below it is opaque, including the thing you would normally call “the instruction set”.

That inversion has a practical consequence: when something is wrong, you can read the compiler and you cannot read the runtime. So you debug by looking at the emitted schedule and inferring what the hardware did. Part 5’s Gather bug is exactly that, and it took a day.

The two register blocks are worth naming because they appear in the vendor’s crash output: tiu is the tensor instruction unit, tdma is the tensor DMA.

ONNX In, cvimodel Out

model_transform --model_name dec --model_def decode.onnx \
    --input_shapes [[1,128,224],[1,128,224],[1,1,224],[1,128,224],[1]] \
    --mlir dec.mlir

model_deploy --mlir dec.mlir --chip cv181x --quantize BF16 \
    --model dec_bf16.cvimodel

A .cvimodel is not a program. It is a Model → Program → Routine → Tensor graph plus a command buffer: a pre-computed list of tensor operations with their DMA schedule. model_tool --info reports what it will need:

Cvimodel Version: 1.4.0
CviModel Need ION Memory Size: 8.5 MB

That memory line is the number to read before you run anything. It is the working set the board has to find, and Part V is about what happens when it cannot.

The two commands are the whole toolchain, and the second one is where all the time goes — quantisation, tiling, and op selection all happen in model_deploy.

The five input shapes are the vocoder’s: m_p_exp, logs_p_exp, y_mask, zp_noise, noise_scale. That is a real example, not a toy, and it shows why static shapes hurt: the 224 is the mel length, and it has to be a literal.

The version line matters more than it looks. This compiler emits 1.4.0 and the board’s 2024.12 runtime accepts it, so the format has been stable across a two-year gap. That is worth knowing before you plan around it — and worth checking with CVI_NN_GetModelVersion on the board if you ever see a model that will not load.

The “not a program” point is the conceptual one. There is no branch, no loop, no call. It is a dataflow graph with a schedule attached, which is why the compiler has to know every shape at compile time.

Is There a Standard?

No. And it is worth being precise about what that means.

Attempt	What happened
NNEF (Khronos)	a standard exchange format, no runtime. Effectively dead
ONNX Runtime	has execution providers — there is none for cv181x
OpenCL / Vulkan compute	not supported by this runtime at all
Caffe, TFLite	Caffe yes, TFLite no — check model_deploy --help

Every vendor has the same shape: RKNN (Rockchip), QNN (Qualcomm), Edge TPU (Google), Ethos-U (Arm). A compiler, a blob format, a runtime, a lock-in.

ONNX is the only standardised piece, and it is standardised at the input to a compiler that belongs to one vendor.

This slide answers the question students always ask — “is there an ONNX Runtime backend for this?” — and the answer is no, and the reason is structural rather than lazy.

Be careful with the TFLite row. The vendor’s marketing material claims TFLite support, and I have read claims that the SG2000 accepts “Caffe, PyTorch, ONNX, MXNet, and TensorFlow Lite”. Our own model_deploy --help lists ONNX and Caffe only. Trust the binary over the brochure, and say that out loud — it is a better lesson than the fact itself.

The RKNN/QNN/Edge TPU comparison is the generalisation: this is not a Cvitek problem, it is the industry’s answer to the same economics. Nobody standardises the layer that is their moat.

NNEF is worth one sentence because it is the cautionary tale: Khronos did try to standardise the exchange format, and a format without a runtime is a document, not an ecosystem.

The Samples Are All Camera-Shaped

The board ships about 60 sample binaries in /mnt/system/usr/bin/ai. None of them can run on our image.

$ sudo sample_img_object_detection YOLOV5_DET_COCO80 . bus.jpg
[base_image.cpp:163] [E] allocate memory failed

Every one of them goes through the TDL SDK, which builds a VPSS frame before it loads a model. Our image deliberately does not load the camera stack, so there is no VPSS, so there is no entry point.

The only way in is the bare API, and it is five calls:

CVI_NN_RegisterModel(path, &model);
CVI_NN_GetInputOutputTensors(model, &in, &n_in, &out, &n_out);
memcpy(in[0].sys_mem, data, in[0].mem_size);
CVI_NN_Forward(model, in, n_in, out, n_out);
CVI_NN_CleanupModel(model);

This is the practical wall that costs a week, and it is worth putting on a slide because it is invisible from outside. The SDK has samples, the samples look like documentation, and not one of them applies.

The five calls are the whole API you need, and the fact that they are five calls is the good news: once you find them, everything works. run_model.py in the examples directory is exactly this, in 150 lines of ctypes.

The ctypes detail is worth a slide of its own in the notes: there is no compiler on the board. No gcc. So the probes were written in Python against the runtime library, declaring the structs by hand from cviruntime.h. That turned out to be a feature — a ctypes probe is a text file you can scp and edit, where a cross-compiled binary is a build-system problem. If someone asks how to write C for this board, the answer is “you cross-compile, and you will wish you had not”.

Part V — Present and Unusable

The Device That Was Present and Unusable

Everything a working platform looks like:

$ ls /dev/cvi-tpu0              crw------- root root 245, 0
$ lsmod | grep soph_tpu         soph_tpu  28822  0
$ ls /mnt/system/lib/           libcviruntime.so  libcvikernel.so  libcvimath.so
$ nm -D --defined-only libcvikernel.so | wc -l
313

And the first thing it does is die on 4 KiB:

CVI_RT_Init -> rc=0
CVI_RT_MemAlloc(4096) ...
ion ioctl fail:: Cannot allocate memory
python3: cvi_device_mem.cpp:451: mem_alloc_raw: Assertion `0' failed.
Aborted (core dumped)          # exit 134

Not weights. Not a tensor. Not a command buffer. The smallest allocation the API can express.

Three things are true at once here and they are worth separating, because the temptation is to conclude “the driver is broken”.

The device node exists. The module is loaded. The runtime library initialises and returns success. The symbol table has 313 entries. Every test that says “is the hardware there” passes.

And nothing runs. That is the slide.

The 4 KiB is the detail that makes it concrete: this is not a capacity problem where you could run a smaller model. It is not “your model is too big”. It is the very first byte of memory the library asks for.

If someone asks how you find this: the vendor sample fails with a generic “allocate memory failed”, and you get the real story by calling the runtime directly from Python, which is examples/npu/probe/probe.py.

assert(0)

// cviruntime/src/soc/common/cvi_device_mem.cpp:451
void* CviDeviceMem::mem_alloc_raw(bmctx_t ctx, size_t size) {
    ...
    assert(0);          // ← reached on ENOMEM
}

A recoverable ENOMEM is turned into SIGABRT. Which means:

• CVI_NN_RegisterModel cannot report out-of-memory to its caller.
• It cannot return an error code, because it will not return.
• Every line of error handling above it is unreachable code.

A library that asserts on a recoverable condition is not a library you can write defensively against. It is a library you can only avoid provoking.

This is the sharpest single lesson in the deck and it generalises far past this chip.

The contract most C libraries offer is “return non-zero and set errno”. That contract lets the caller decide what to do: retry, fall back to a CPU path, report to the user, exit cleanly. An assert deletes the caller’s agency. The process is gone; there is no recovery, and there was never a decision to make.

The practical consequence on this board: you cannot write a program that gracefully degrades when the NPU is unavailable, because asking the NPU whether it is available kills you. You have to know out of band. That is why the image fix in the next two slides had to be a build-time decision and not a runtime check.

If a student argues that assert is fine in a debug build — yes, and this is the release build, and the assert is still there. Say that.

This is also why the probe had to be a separate process: the probe’s job is to die informatively, and a probe that shares a process with the application takes the application with it.

ION, Carveout, and No IOMMU

A fixed-function accelerator addresses memory by physical address. There is no IOMMU on this part, so the memory it touches must be physically contiguous — and physically contiguous memory cannot be conjured at runtime.

reserved-memory {
    ion { compatible = "ion-region"; size = <0x00 0x4000000>; };  /* 64 MiB */
};

So the carveout is taken before the kernel’s page allocator exists, and reserved-memory is the only API. There is no module parameter, no sysfs knob, no runtime override. cvitek_ion.c reads the size at probe time and that is that.

And “a heap that allocates” is not “a heap that works”:

heap	mmap + write	paddr
ion_system_heap	ok	0x0
unnamed	ok	0x0
unnamed	ENOMEM	—

Two of three heaps hand back memory a DMA engine cannot use.

This slide is the mechanism behind the previous one, and the three-heap table is the part students find surprising.

The table comes from a probe that goes around the vendor library and talks to /dev/ion directly. Three heaps are registered. Two of them will allocate and give you a pointer you can mmap and write to. Both return paddr = 0, meaning the memory is not physically contiguous — it is ordinary page-allocator memory that happens to be mappable. That is fine for a CPU and useless for a bus master with no IOMMU.

The third heap is the carveout, and it is the only one that can feed the DMA engine — and it is the one that is empty.

Worth a detour if the room is interested: Cvitek’s ION is not upstream ION. ion_allocation_data has no align field and carries fd/paddr/name instead, and ION_IOC_HEAP_QUERY does not implement the “ask for zero to learn the count” convention. Guess the struct and you get ENOTTY. That cost an hour.

The general lesson: reserved-memory is a boot-time decision, and this is why the fix had to be an image rebuild rather than a patch.

One Knob, Two Meanings

The zero-sized carveout was not an accident. It was a build-time decision:

# configs/duos/settings.mk
SECOND_CPU ?= camera
ifeq ($(SECOND_CPU),arduino)
ION_SIZE=0

The SG2000 has one little C906L that can either run the vendor camera stack or be handed to Arduino via remoteproc. S00kmod decided which by asking how big the carveout was:

if [ $ion_size != 0 ]; then insmod soph_vi.ko; insmod soph_vpss.ko; ...; fi
insmod soph_tpu.ko        # unconditional

ION_SIZE > 0  ⇒  carveout exists  ⇒  camera stack loads
                                  ⇒  vendor RTOS takes the C906L
                                  ⇒  Arduino personality is dead

The NPU needs the carveout. Arduino needs the C906L. Nothing about those two facts should have been connected.

This is the whole bug, and it is a design bug rather than an implementation bug, which is why it is worth a slide.

Two independent requirements — “the TPU needs physically contiguous memory” and “the Arduino personality needs the little core” — were expressed as one test on one integer. The test was correct for the case it was written for. It became wrong the moment someone wanted the third combination.

The soph_tpu.ko line is the tell: the TPU module loads unconditionally, so the system looks like it has a TPU regardless of the carveout. That is why the failure is so confusing — the module is there, the node is there, the symbols are there, and the memory is not.

If someone asks why the fix was a new SECOND_CPU value rather than ION_SIZE=64 on the command line: SECOND_CPU is part of the docker build volume key. Overriding ION_SIZE alone reuses the same build tree, and $(BUILDDIR)/$(BOARD)-$(VARIANT)/memmap.py has no prerequisites — so make does not regenerate it and you get a stale memory map in a fresh-looking image. That is a genuinely nasty class of bug and a good story about build systems.

Coupling Is the Bug

The fix was four small changes to the image build, not to the board:

file	change
configs/duos/settings.mk	new SECOND_CPU=arduino-tpu, ION_SIZE=64
configs/chip/sg200x/chip.mk	inject the rproc fragment for the new personality
scripts/addons/load-systemko/S00kmod	gate the camera stack on /boot/arduino
scripts/addons/preconfigure/addon.mk	write /boot/arduino for both flavours

The condition became “is this an Arduino build”, which is what it always meant, instead of “is the carveout non-zero”, which was a proxy that happened to correlate.

Caution. The first attempt edited the ion size in the .dtb on the running board and rebooted it. The board never came back and had to be reflashed. A boot-time reservation is not something to patch on a machine you cannot put a console on.

Read the table as a lesson in what “fixing it properly” costs: four files, in a build system, none of them on the device.

The reframing is the intellectual content. ion_size != 0 was a proxy for “is the camera stack wanted”. Proxies correlate until they do not, and then they are indistinguishable from bugs. The fix replaces the proxy with the thing it was standing in for.

The caution is not decoration. I did exactly this, on a board with no serial console, and it cost a reflash. Say it plainly and say why: reserved-memory is consumed before the kernel’s allocator exists, so a wrong size does not fail gracefully — the kernel maps memory that is not there, or does not map memory that is, and the result is a board that does not boot. There is no recovery path that does not involve removing the card.

That is also the rule for Friday’s lab: nobody edits a device tree, and nobody reboots a board. If a lab exercise seems to need a reboot, the exercise is wrong.

Part VI — The Microbenchmark

How to Measure an Accelerator

You cannot get a clean answer from float data. So we chose data where both representations are exact:

• Operands are integers in [-8, 7].
• bf16 represents them exactly (7-bit mantissa, small integers).
• int8 represents them exactly.
• The reference is computable in int64, exactly, on the CPU.

And only compare like with like:

NPU	CPU
bf16	bfv (RVV) / bfs (scalar)
int8	i8v (RVV) / i8s (scalar)

An earlier version of this work had an “NPU bf16 vs CPU int8” column. It was wrong, and its conclusions were withdrawn. bf16 and int8 compute different functions — one rounds to bf16 on output, the other quantises to int8.

Put the retraction on the slide. It is the most useful thing on it.

The temptation with an accelerator benchmark is to find the biggest number, and the biggest number here was available by comparing the NPU’s bf16 path against the CPU’s int8 path — which is comparing two different functions and calling the difference a speedup. It is not a speedup; it is a category error. The column existed for a day and its conclusions are withdrawn.

The [-8, 7] choice is the methodological fix and it is worth dwelling on. Both formats are exact over that range, so: - a difference in output can only come from the arithmetic, not the conversion; - the reference is an exact integer matrix multiply, computable in int64 with no floating point anywhere.

That is what lets Part 5’s “bit-exact” claims mean something. If the operands were arbitrary floats, “the NPU agrees with the CPU” would be a statement about tolerances, not about arithmetic.

The one asymmetry that remains: the NPU int8 path is not exact — max error 11.0 against a quantisation step of 11.49, i.e. about one step. That is a real difference and it is on the next-but-one slide.

The Crossover

N × N matrix multiply, milliseconds measured:

N	NPU bf16	CPU bfv		NPU int8	CPU i8v
32	0.25	0.18	CPU 1.4×	0.34	0.05	CPU 6.8×
64	0.28	1.49	NPU 5.3×	0.56	0.38	CPU 1.5×
128	0.39	11.87	NPU 30×	1.36	3.93	NPU 2.9×
256	0.95	114.41	NPU 120×	4.65	26.43	NPU 5.7×
512	4.49	2295.38	NPU 511×	18.25	220.76	NPU 12×

It goes both ways. The accelerator loses below N ≈ 64 (bf16) and N ≈ 128 (int8), because it pays a fixed cost before it computes anything.

This is the slide to spend time on. Read the bold cells and let the room see the direction change.

The shape: at N=32 the CPU wins both columns, and it wins int8 by 6.8× — the C906’s vector int8 path is genuinely good. By N=64 bf16 has flipped and int8 has not. By N=128 both have flipped and the NPU never loses again.

The fixed cost is the explanation and it is measurable independently: a 900-parameter convnet takes 0.28 ms on this NPU. So ~0.25–0.35 ms is what you pay for one inference of anything, and any matmul the CPU can do in less than that is a CPU job.

Two caveats to state honestly: - The CPU kernels are hand-written and unoptimised — blocked int8 with 4×4 register blocking, blocked RVV, and so on. They reach 1.27 GOPS int8 and 0.29 GFLOP/s fp32, which is far from what the hardware should manage. Several attempts to improve them made things worse. So the ratios at large N are an upper bound. - The crossover points are not affected by that, because they are set by the NPU’s fixed cost, not by how good the CPU kernel is.

That second point is the one to emphasise. The interesting result survives the weakness of the baseline.

The Ratio That Grows, and Why You Shouldn’t Believe It

The bf16 column ends at 511×. That number is real and it is also partly a lie.

N	NPU bf16	CPU bfv	ratio
256	0.95	114.41	120×
512	4.49	2295.38	511×
growth 256→512	4.6×	20×

N³ predicts 8× for both. The NPU did 4.6×. The CPU did 20×.

The CPU did not get slower because the problem got harder — it fell out of its cache. Part of the widening ratio is the CPU’s problem, not the NPU’s triumph.

This is the slide that separates a benchmark from a measurement.

Both implementations do the same N³ work. Going from 256 to 512 is 8× the multiply-adds. The NPU took 4.6× the time, which is better than N³ — it is amortising its fixed cost over more work. The CPU took 20×, which is 2.5× worse than N³.

Why? Because at N=512 each matrix is 1 MB, three matrices is 3 MB, and the C906’s L1 is 32 KB with a 64 KB L2. The naive kernel walks a column of bb with a 2 KB stride. At N=256 that pattern still mostly hit cache; at N=512 it does not. The CPU fell off a cliff that has nothing to do with the accelerator.

So the honest statement of the 511× is: “at N=512, with a naive CPU kernel, on a machine with 96 KB of cache, the NPU is 511× faster.” Every clause in that sentence is load-bearing.

This is the same lesson as the memory-hierarchy deck, arriving uninvited: a benchmark that changes two things at once measures neither. If you want the NPU’s speedup you have to hold the CPU implementation quality fixed, and our CPU implementation is not good enough to hold it fixed at large N.

Same Answer, Different Speed

The reason we could use integer data: the outputs are checkable.

Implementation	Reference	Result
CPU i8s / i8v	exact int64 A@B	exact, max err 0
CPU bfs / bfv	bf16_round(exact)	exact, max err 0
NPU bf16	bf16_round(exact)	exact, max err 0
NPU int8	exact int64 A@B	max err 11.0

All five implementations produce bit-identical output at N=256 and N=512 (for int8, exact; for bf16, bf16_round(exact)).

The NPU is not less accurate. It computes the same function, 120× faster.

The exception is int8: max error 11.0 against an output quantisation step of 11.49 — about one step. A second int8 stage (AC_MatMul_si8 → C_MatMul, qscale 11.49) sits between the matmul and the fp32 output.

The bit-exactness is the strongest claim in the deck and it is worth explaining how it was established rather than just asserting it.

The inputs were chosen so that no rounding happens on the way in. Then: - if the NPU’s bf16 path accumulates in fp32 and rounds once, its output must equal bf16_round(exact_int64); - if it accumulated in bf16, or rounded per-partial-sum, it would drift.

It matches exactly, on all 65,536 elements, at two sizes. That is a much stronger statement than “the error is small”, and it is the kind of statement you can only make because the operands were exact.

The int8 exception is worth naming precisely because it is the honest one. We forced the input and weight thresholds to 127 so the int8 operands would be exact and the accumulator scale would be 1.0. Even so, there is a second quantisation stage between the matmul and the output, with a step of 11.49. The observed max error of 11.0 is one step. So the int8 path is a different function from the exact matmul, by about one quantisation step, and that is the real cost of int8 — not speed.

The Measurement That Surprised Us

int8 is slower than bf16 on this NPU.

N	bf16	int8	penalty
32	0.25	0.34	1.4×
64	0.28	0.56	2.0×
128	0.39	1.36	3.5×
256	0.95	4.65	4.9×
512	4.49	18.25	4.1×

This inverts the marketing. INT8 is the headline “0.5 TOPS” mode and here it is the slow path. The consequence is large: the NPU’s advantage over the CPU is 511× in bf16 and only 12× in int8.

Hypotheses, none confirmed: an extra requantise routine (the tensor map shows two routines for int8, one for bf16); a different tiling; a different accumulator path. The requantise stage is over 65k elements and should be cheap, so I do not believe that explanation. Cause not identified.

This is the slide to be most careful with, because it is the one where the honest answer is “I do not know”.

What is solid: the measurement, reproducible across five sizes and repeated runs, on one board, with the model compiled by the same compiler version. int8 is 1.4× to 4.9× slower than bf16, peaking at N=256.

What is not solid: any explanation. The tensor map shows that the int8 model needs two routines — TPU matmul producing an int8 intermediate, then a requantise/dequantise to fp32 — where the bf16 model needs one. That is a real structural difference and it is the obvious suspect. But the extra stage is 65,536 elements of elementwise work on a machine that does hundreds of GOPS, so it should cost microseconds, not 3.7 ms. The number does not fit the story.

The remaining candidates are about how the int8 matmul is tiled, or which accumulator path it selects, and both require reading the emitted TPU dialect rather than the ONNX graph. That is the next experiment.

Why put an unresolved question on a slide: because it is a real one, it is reachable by a student with a laptop and the toolchain, and pretending otherwise would make the rest of the deck less trustworthy. The deck’s other numbers are believable partly because this one is admitted.

If someone proposes “maybe int8 is just badly supported on this part” — that is consistent with the data and would be a fine conclusion, but we cannot distinguish it from a tiling artefact yet.

Part VII — The Demo

Text to Speech, Split by Shape

text → espeak-ng (IPA) → token ids → encoder (CPU) → vocoder (NPU) → WAV

Stage	Where	ms
phonemise (espeak-ng)	CPU	~140
encoder (duration.onnx)	CPU, onnxruntime	~230
vocoder (bucketed, resident)	NPU	~384
total		~760

For 1.82 s of audio. That is 2.4× faster than real time, and the compute-heavy stage is the one on the accelerator.

The split is not a design choice. It is forced by the shapes:

• the encoder’s output length is sum(predicted durations) — data-dependent, so no static graph can express it. Range.cpp:75, UNREACHABLE.
• the vocoder needs static shapes, which bucketing supplies, and it is the expensive part.

This is the payoff slide and the pipeline diagram is worth drawing rather than reading.

The shape argument is the architectural content, and it is a direct consequence of Part IV’s “static shapes only”. The encoder produces a tensor whose length is the sum of the predicted phoneme durations. That number is not known until runtime, so it cannot appear in --input_shapes, so the encoder cannot be a cvimodel. Full stop.

But the encoder does not have to be on the NPU. python3-onnxruntime is packaged for riscv64 in Debian (1.21.0), and the encoder is only 3.6 MB and takes 230 ms on the C906. So the answer is: put the shape-dynamic part on the CPU, put the expensive static part on the NPU. That is a genuinely good division of labour and it fell out of the constraint rather than being designed.

The 140 ms for phonemisation is worth flagging as a surprise: text front-end work is 18% of the total, on a machine with two vector units. Amdahl does not care that it is not the interesting part.

The table is one sentence, and it is not the one in the demo. These are the numbers from the shortest measured run. The encoder and vocoder both scale with input length, so the demo sentence — 49 symbols, 258 mel frames — measures phonemise 149 ms, encoder 378 ms, vocoder 807 ms, total 1335 ms for 2.75 s of audio. Same pipeline, different lengths, and the longer one pays the bucket boundary cost described two slides from here. Do not mix the two sets of numbers in one sentence.

Board dependencies are python3-numpy python3-onnxruntime espeak-ng. Note that phonemizer is not packaged for Debian, but it is a thin wrapper over espeak-ng, so espeak-ng -q --ipa does the same job in one subprocess.

Bucketing

One model per shape is useless when the length depends on the text. The answer is a small set of compiled shapes, and pick the smallest that fits.

mel_len	ms	audio	real time	model
128	192.6	1.37 s	7.1×	7.6 MB
256	379.3	2.73 s	7.2×	8.5 MB
512	800.2	5.46 s	6.8×	11.3 MB

Linear in mel_len, so the vocoder is compute-bound and the fixed cost is amortised by 128 frames.

The cost is at the boundary. A 258-frame sentence lands in the 512 bucket: 2× the work for 1.01× the audio. Measure it yourself — the same sentence through the live server shows mel_len 258, vocoder 807 ms.

The three-row table is the good news: near-perfect linear scaling, ~7× real time, and the models are small enough to keep all three resident.

The boundary effect is the bad news, and it is freshly measured. The sentence I use for the demo phonemises to 258 mel frames. The bucket ladder is 128, 256, 512, so 258 rounds up to 512 — and the vocoder runs the 512-frame model, which takes 807 ms instead of the ~390 ms a 256-frame model would. Two frames over a boundary cost 400 ms.

The engineering answer is more buckets, or a bucket at 320. The pedagogical answer is that this is exactly the tiling/rounding waste from the GPU deck’s block-size material, in a different costume: you pay for the granularity of your hardware’s shape constraint.

The padding trick is worth a sentence: zero-pad the mel along the time axis, but keep y_mask at 1.0 over the padding so the vocoder does not see a hard edge and produce a click. Then trim the output to mel_len × 256 samples. Verified: a 200-frame input selects the 256 bucket and returns exactly 51,200 samples.

Keep It Resident

The first version shelled out to a runner per request. Same model, same board:

	ms
per-request (shell out, register, run, exit)	3127
first call with a resident handle	519
second call, resident	385

CVI_NN_RegisterModel allocates the weights and the command buffer in ION, every time you call it. So a per-request registration pays for the load on every request.

An 8× difference from a purely structural change — no arithmetic changed, no model changed. Per-request registration makes the demo feel broken.

This is the cheapest lesson in the deck and the one most likely to matter to students in their own projects.

Nothing here is an optimisation. The model is the same, the computation is the same, the hardware is the same. The only change is when RegisterModel is called: once at startup, or once per request. That is an 8× difference.

The mechanism is ION. Registration allocates the weights and the command buffer in the carveout — the 64 MiB from Part V — and the allocation and copy are the expensive part. Doing it per request means doing it a thousand times.

The generalisation: on an accelerator, the cost of getting ready often dominates the cost of the work. The GPU deck made the same point with the 4096-versus-6144 cutoff, where the PCIe transfers flipped the answer between two cards. Here it is model registration.

Also worth saying: 385 ms is the resident figure and it is 5.5× real time, not 7×, because the web server path adds its own overhead. Both numbers are true; they measure different things.

The Bug That Failed Open

The compiler rejected four channel flips:

TgReverseKernel.cpp:18: Assertion `(axis == 0 || axis == 3)` failed

They came from /flow/flows.{1,3,5,7}/Slice — steps=[-1] on axis 1. The steps tensor comes from a Constant node, not an initializer, so scanning initializers misses it.

The mistake that cost a day. Replace each flip with Gather(axis=1, reversed_indices). It compiles. It runs at 7× real time.

expected:  channel 0 = 127 ... channel 127 = 0     (reversed)
NPU:       channel 0 =   0 ... channel 127 = 127   (identity!)

TPU-MLIR’s Gather on cv181x silently ignores the index vector. It is a pass-through. It fails open.

This is the slide that justifies the whole deck’s attitude to the toolchain, so tell it as a story rather than as a fact.

The symptom was a vocoder output that was uncorrelated with the reference (0.025) and had lost its high band. That looks exactly like precision loss, and the natural conclusion is “bf16 is not good enough for a vocoder” — which would have been a plausible, publishable, and completely wrong conclusion.

The diagnostic that separated the two is the one to teach. Feed constant inputs. If every input is 0.5 or 1.0, all of which are exact in bf16, then no amount of rounding can change the answer. The NPU produced rms 0.0294 where the reference produced 0.00415. Precision cannot do that. So the fault was structural, not numerical, and the search moved from “is bf16 good enough” to “which operator is wrong”.

The isolated test is the other half: build a two-op graph that just flips a channel, run it, and look. Expected ch0=127, ch127=0; got ch0=0, ch127=127. Four missing flips, each of which scrambled the flow.

Why “fails open” is the phrase to use: a compiler that rejects an operator tells you immediately. A compiler that accepts it and ignores part of it produces a model that runs, at speed, with plausible-looking output. It is strictly worse than not supporting the operator at all.

The Bug That Failed Open (cont.)

The fix exploits what the backend does support — Reverse on axis 0:

Transpose(1,0,2) → Slice(axis 0, step −1) → Transpose(1,0,2)

Verified in isolation first (ch0=127, ch127=0), then applied to the vocoder.

test	wave corr	spectrum corr
constant inputs	+0.9983	0.9993
random inputs	+0.9311	0.9819
full text pipeline	+0.9468	0.9774

rms 0.0604 (reference) vs 0.0625 (NPU). What remains is bf16 rounding noise — which is what bf16 should cost.

The fix is deliberately ugly, and the ugliness is the point: it expresses the operation using the subset of operations the backend actually implements, rather than the operation you meant.

The order of operations matters and is worth stating as a rule: verify the substitution in isolation before trusting it in the model. The isolated test is five lines and it takes a minute. The alternative is another day of “why is my waveform wrong”.

The three correlation numbers are the evidence that bf16 is fine. Constant inputs at 0.9983 says the structure is right. Random inputs at 0.9311 says the structure is right for non-trivial data. The full pipeline at 0.9468 says the whole chain — phonemiser, encoder, tokeniser, vocoder — agrees.

The gap between 0.9311 and 0.9468 is worth a sentence: the full pipeline scores higher than random inputs, which is mildly surprising and is explained by the reference and the NPU both being driven by the same encoder output. Do not over-read either number; they are correlations with a single deterministic reference, not a perceptual metric.

And the honest limit: I cannot hear it. Correlation 0.95 could still be audibly wrong in a way that a spectrogram hides. That is what the demo slide is for.

The Demo

Same sentence, same encoder output, same padding, two vocoders. The only difference is bf16.

fp32 on the C906 (onnxruntime)	116,248 ms
bf16 on the TPU	807 ms
wave correlation	0.9252
spectrum correlation	0.9675

Reference — fp32 on the CPU

NPU — bf16 vocoder

“This is what a bf16 accelerator costs you, audibly.” If you cannot tell them apart, that is the answer — and it is a better answer than the correlation number.

Play both. Do not narrate over them, and do not tell the room which is which before they hear it if you can avoid it — ask for a show of hands on whether anyone can tell, and then reveal.

The two files are the same sentence through the same encoder, with the vocoder run twice: once as fp32 on the C906, once as bf16 on the TPU. The padding and trimming are identical, so the only variable is precision. Both are 66,048 samples, 2.75 s, and the best cross-correlation is at lag zero — so they are aligned and the comparison is not confounded by a delay.

Be careful with the 116 seconds. It is a real measurement and it is not a fair one. onnxruntime on riscv64 has no tuned kernels for this core; the generic CPU path is doing something much worse than 116 s worth of arithmetic. A hand-written fp32 kernel would be far faster, and I have not written one, so I do not know how much faster. What the number does establish is narrower and still useful: fp32 inference of a 12.6 MB model on this board, through the packaged runtime, is not a deployment option. The NPU is not 144× faster at the arithmetic; it is the only one of the two paths that finishes in a time a user would accept.

Say that distinction out loud rather than quoting 144× as a speedup. It is the same discipline as the retracted column in Part VI.

This is also the question tts/RESULTS.md raises and never answers — “very likely muffled speech rather than noise, but that is a guess until someone listens”. Now the room listens.

If the audio is unavailable, the live server is at http://duos-frog-9f7f:8080/ and takes a sentence and returns a WAV. Fall back to that, but the A/B is better because it is a comparison rather than a demonstration.

Honest framing: I still cannot hear it. If the room says the NPU version is muffled, that is a real finding and it changes the conclusion of Part II — the precision section would need to say “bf16 is exact for matmul and audibly lossy for a waveform”. Say that rather than defending the number.

Part VIII — Synthesis

Seven Machines, Seven Answers

Machine	Accelerator	Attach	Precision	Programmable
sg2000	bm1822 TPU · 0.5 TOPS	on-die	int8, bf16	no
celedyr	Polaris 10 · 2016	PCIe slot	fp32	yes
kraken	MI50 · 2018	PCIe slot	fp32	yes
typhon	8 × Gaudi2 · 2022	PCIe + switch	bf16	yes
unicorn	RX 7600 · 2023	PCIe slot	fp32	yes
vampire	Radeon 8060S · 2025	on-die, still a PCIe endpoint	fp32	yes
pegasus	Arc Battlemage · 2025	PCIe slot	fp32	yes

Read the last column down. One machine in this table cannot be programmed, and it is the one on your desk.

The sg2000 counterfactual, in one line: to put a programmable accelerator on this die you would need a bus (it has none above USB 2.0), address space (a 4 GB map cannot hold a 16 GB BAR), and coherency (it has neither). It would be a different chip.

This is the memory deck’s “Four Cores, Four Answers” and the GPU deck’s “Seven Machines, Seven Answers”, continued. It is the slide to photograph.

The column that matters is the last one. Every other accelerator in this table has an ISA and a programming model. The sg2000’s does not, and that is not a deficiency — it is the design point. It is a ~3 W part with no bus and 512 MB of DRAM; a programmable accelerator would be starved by the CPU that feeds it and would not fit the power budget.

The counterfactual is compressed to one line here because the GPU deck spent five consequences on it. If the room has not seen that deck, expand: discovery (a GPU would need no device-tree node), address space, coherency, power and pins, and Amdahl. Item five is the one students miss — even with a free GPU on a free bus, a 1 GHz dual-issue C906 could not keep it fed.

The honest summary: the sg2000 is the integrate everything design point, and it is self-consistent. The TPU is on the die because there is nowhere else to put it.

What We Actually Learned

1. Fixed-function means no ISA. The op list is the machine, and an operation that is not on it does not run — not slowly, not at all.
2. Precision is the interface. There is no FP32 path. bf16 is real compute (fp32 accumulate, bf16 round, bit-exact), and int8 is a different function.
3. The compiler is the open piece and the bugs live there. Gather fails open, and a pass-through that runs at 7× real time is worse than a rejection.
4. A device can be present and unusable. Driver loaded, node present, 313 symbols exported, and the first 4 KiB aborts the process. Presence is not usability.
5. Coupling is the bug. Two independent requirements expressed as one test is the failure mode that costs days, and it is not visible in any log.
6. Fixed cost sets the crossover. ~0.3 ms per inference means the accelerator loses below N≈64, and the number is a property of the launch, not the arithmetic.
7. Measure like with like, or you measure nothing. The retracted column in Part VI is on the slide because the mistake is instructive.

The interesting engineering is not in the accelerator. It is in the two interfaces on either side of it: the compiler above, and the memory below.

Read these as the examinable summary. Each is a sentence a student should be able to expand into a paragraph.

Claim 6 is the one to put on the exam, because it is the one that transfers: every accelerator in this course has a fixed cost per invocation, and the question “when does the accelerator win” is always “when the work exceeds the fixed cost”, never “when the peak is higher”.

Claim 7 is unusual to put in a summary and it belongs there. The room has watched me admit an unexplained result (int8 slower than bf16) and a retracted comparison. Those admissions are part of the content, not a failure of it.

Claim 4 and claim 5 are the two that generalise past this chip. They are the ones worth remembering in ten years, when this part is long obsolete and the student is bringing up something else.

Friday

There is a lab on Friday. It assumes Parts I–V of this deck.

• The boards are already on the image that works. Do not flash, do not edit a device tree, do not reboot.
• You will run models that are already compiled. The compiler stays on my laptop — the install is a six-hour afternoon and it is not the exercise.
• Bring a way to listen to a WAV file.

The lab is not “make the NPU go”. It goes. The lab is “find the fixed cost, find the crossover, and explain both” — the measurements in Parts VI and VII, reproduced by you, on hardware you can touch.

Keep this short. It is a pointer, not a preview, and the lab handout is the authority.

The safety line is not boilerplate — the caution slide in Part V is a story about a board that did not come back. Say it plainly and mean it.

Why the compiler is not in the lab: the install needs Python 3.10 (the system is 3.12), a yanked OpenCV, setuptools<80 for pkg_resources, torch even for ONNX input, and a bundled glibc 2.45 that shadows the host’s and kills subprocesses. Every one of those is a real bug in someone else’s package. None of them teaches computer architecture.

The “bring headphones” note is practical: the demo is the payoff and a room full of laptop speakers will not do it justice.

Discussion Prompts

• int8 is the advertised mode and it is 4.9× slower than bf16 here. Design an experiment that distinguishes “extra requantise stage” from “different tiling” from “measurement artefact”.
• The NPU has 32 KB of scratchpad and no cache. Why not a cache? Argue both sides, then say what a 4 KB cache would cost you in area and in predictability.
• A 258-frame sentence pays for 512 frames. What would you change — more buckets, a different bucket ladder, or a dynamic-shape mechanism — and what does each cost?
• The vendor library asserts on ENOMEM. Design the API you would have written instead, and say what the caller can now do that it could not.
• Gather fails open. How would you test a compiler to find that class of bug before it reaches a customer?
• The encoder is on the CPU because its output length is data-dependent. Is there a model architecture that would let it move to the NPU?

Every one is answerable from this deck. None has a single right answer.

These are designed to be argued in class or written as homework.

The first is the best one, because it is a real open problem with a real answer somewhere in the emitted TPU dialect. A student who wants to chase it has everything they need: the toolchain, the models, the board, and a discrepancy of 3.7 ms that no current explanation accounts for.

The fifth is the one that generalises furthest. “How would you test a compiler to find operators that silently do the wrong thing” is a question about verification, and the answer — differential testing against a reference implementation, on constant inputs first — is exactly how this bug was found, by accident, a day too late.

Do not answer them. Let the room do it.

Refs

Primary sources

• CVITEK TPU SDK Development Guide — the operator list, QuantType, the runtime API. Distributed by SFTP, not on the web; the copy in docs/ is the one to read.
• SG200x Software Developer’s Manual — cviruntime, cvikernel, cvibuilder, cvimath. 75,894 lines; the TPU sections are the ones that matter.
• TPU-MLIR — the compiler, and its CV18xx appendix (Appx.02_cv18xx_guide.rst) which documents this exact chip family.
• cvimodel.fbs — the format. Version 1.4.0, and the schema is the clearest documentation of what a cvimodel actually is.
• Kung & Leiserson (1978), Systolic Arrays for VLSI; Chen et al. (2014), DianNao; Jouppi et al. (2017), In-Datacenter Performance Analysis of a TPU.

Measured on our hardware

• probe.py and ion_probe.py — allocation, heaps, paddr
• run_model.py — the five-call API, latency, CVI_NN_GetModelVersion
• mm_cpu.c, mm_peak.c, run_mm.py — the Part VI tables
• pipeline.py, vocoder.py, server.py — the Part VII numbers
• rdcycle for the 1050 MHz C906 clock

Companion

• Reference Notes · ../gpu-slides/ · ../memory-slides/ · ../power-slides/ · ../hints/tpu.md

Point at the measured list specifically: every number in this deck that is marked green came from a command on a board the students can log into. The hints/tpu.md file is the long-form version of Parts I–V, with the full bring-up narrative and the reproduction steps.

The three papers are worth recommending individually: Kung & Leiserson for the idea, DianNao for the energy argument, and Jouppi for the shipped design point. A student who reads all three has the whole intellectual history of this class of machine.

The cvimodel.fbs recommendation is the sleeper. It is a FlatBuffers schema, it is short, and reading it tells you more about what the compiler produces than the manual does.