The TPU

The SG2000 has a second accelerator that nobody has used yet: a 0.5 TOPS INT8 neural-network engine. Unlike the GPU story (which is about a machine we don’t have), this one is on the die, has a driver loaded, has a full runtime on the Debian image, and is still almost unusable — for reasons that are instructive.

This page is what the hardware is, what it can actually do, and what “TPU development” means on this class of chip.

What is in the chip

From the TRM (§2.3.2) and the live device tree:

CVITEK TPU, ~0.5 TOPS INT8. Self-described as a “TPU scheduling engine” feeding “tensor processor cores”.
Clocked from fpll / 3 → clk_tpu = 500 MHz by default (TRM table 8.4), with clk_tpu_fab alongside it.
Two register blocks and two interrupts, and the split is the whole design:

block address interrupt job

tiu 0xc101000 tiu_irq the arithmetic

tdma 0xc100000 tdma_irq the data movement
The IP is internally 1822 (the kernel library exports cvk1822_* and bmk1822_*, plus cvikernel_get_chip_info_cv181x). The same core is in the CV1800B (plain Duo) and the CV181x family; the SG2000 is a CV181x.

block	address	interrupt	job
`tiu`	`0xc101000`	`tiu_irq`	the arithmetic
`tdma`	`0xc100000`	`tdma_irq`	the data movement

0.5 TOPS at 500 MHz is 1000 INT8 ops per cycle, i.e. roughly 512 INT8 MACs per cycle if the peak is quoted at the default clock. Treat that as an inference from the two published numbers, not a datasheet line.

What it can actually do

The TPU is not a small CPU. It is a fixed-function pipeline with a specific, short list of things it can do — this is the list from libcvikernel.so, i.e. the actual op set the compiler emits:

TIU (arithmetic) - convolution, depthwise_convolution, convolution_qdm (quantised depth multiplier), matrix_multiplication, matrix_multiplication_qdm - max_pooling, min_pooling, average_pooling - element_wise_{add,sub,mul,mac,max,min,ge,copy,arith_shift} - element_wise_{and,or,xor}_{int8,int16}, element_wise_mul_qdm - lookup_table (how activations like sigmoid/exp get done) - bf16 variants of a few (bf16_element_wise_ge, bf16_min_pooling)

TDMA (data movement) - tensor_copy in every direction: g2l (global→local), l2g, l2l, tg2l, tg2tg - transposing copies (nc_transposed, cw_transposed, chw_rotated, row_col_transposed), decompressed/compressed copies, tensor_fill_constant, tensor_lrn_shift

Memory model - gmem = DRAM, lmem = on-chip scratchpad, tgmem = tensor-group memory. lmem is allocated by the compiler (cvikernel_lmem_alloc_tensor), not by the hardware at run time. There is no cache — this is a scratchpad machine, the same “inverted memory hierarchy” idea as the GPU lectures, done with an explicit two-level address space instead of a hierarchy. - ps32 matrices (int32 partial sums) exist as a first-class type, which is what makes INT8 convolution with 32-bit accumulators expressible.

Everything else a real framework needs — softmax, NMS, resize, argmax, the YOLO decode — is not on the TPU. libcviruntime.so carries a set of CPU fallbacks for exactly this (softmax.cpp, yolo_detection.cpp, roi_pooling.cpp, interp, quant, …). A model runs partly on the array and partly on the C906s.

The numbers

Cvitek’s own benchmark table for the CV181X (the SG2000’s family) at 500 MHz DDR3, Rel-v1.5, as republished in the Milk-V docs. Times are per inference, “Utilization” is the vendor’s own figure:

model	size	ION	ms	FPS	util
resnet50	26 M	27.35 MB	41.5	24.1	74.5 %
resnet18	12 M	12.01 MB	18.26	54.8	79.5 %
mobilenet_v2	3.6 M	4.40 MB	7.07	141.4	49.2 %
squeezenet_v1.1	1.3 M	1.90 MB	4.82	207.5	64.7 %
shufflenet_v2	1.5 M	1.62 MB	1.53	653.6	21.7 %
googlenet	7.0 M	8.24 MB	19.62	51.0	65.0 %
inception_v3	24 M	25.49 MB	55.95	17.9	81.9 %
densenet121	8.2 M	10.52 MB	36.77	27.2	62.2 %
mobilenet_ssd	6.1 M	8.58 MB	14.79	67.6	62.5 %
retinaface_mnet25	513 K	1.36 MB	3.47	288.2	56.5 %
yolo_v3_tiny	8.8 M	14.32 MB	26.82	37.3	83.2 %
yolov5s	8.2 M	25.31 MB	99.36	10.1	68.8 %
yolox_s	10 M	21.62 MB	140.5	7.1	76.5 %
arcface_res50	43 M	43.86 MB	63.1	15.9	80.1 %
alphapose	41 M	42.95 MB	62.54	16.0	75.4 %

Do the arithmetic on the well-known FLOP counts and a picture appears: resnet18 (~1.8 GFLOP), resnet50 (~4.1 GFLOP) and shufflenet_v2 (~0.15 GFLOP) all land near ~100 GOPS sustained, and yolov5s lands near ~165 GOPS. Against a 0.5 TOPS peak that is 20–35 % of peak, and the vendor’s own utilization column agrees.

Two lessons:

Peak TOPS is a marketing number. Budget ~100 GOPS on this part.
The ION column is the other half of the story. retinaface_res50 needs 60 MB of working memory on a board with 512 MB total, and yolov2_416 needs 59 MB. Model footprint, not arithmetic, is what decides whether a model fits — same conclusion as the memory-hierarchy lectures, one layer out.

Cvitek’s own perf notes say that when a system gets busy, tdma_exe_ms grows while tiu_exe_ms stays flat: the arithmetic units starve on memory bandwidth. The tiu/tdma split in the register map is exactly this tension, exposed.

The stack: five layers

On the board, bottom to top:

layer	what	on our image
5	TDL SDK — turnkey pipelines (face, YOLO, pose, OCR, ASR)	`libtdl_core/ex/utils.so` + ~60 `sample_*` binaries in `/mnt/system/usr/bin/ai/`
4	CVI_NN — model-level API: `CVI_NN_RegisterModel`, `CVI_NN_GetInputOutputTensors`, `CVI_NN_Forward`	`libcviruntime.so`
3	CVI_RT + cvikernel — `CVI_RT_RunCmdbuf` submits a command buffer built from `cvk1822_tiu_` / `cvk1822_tdma_` calls	`libcvikernel.so`, `libcvimath.so`
2	kernel driver	`soph_tpu.ko` → `/dev/cvi-tpu0`, major 245
1	the array	`0xc100000`/`0xc101000`

Layer 3 is the interesting one for a course: a command buffer is the unit of work, built by software from typed op calls, then handed to the kernel in one go. The compiler’s real job is choosing the tiling that fits lmem and emitting that buffer.

Layers 4 and 5 are what the docs call “TPU development”. Note the docs’ own advice: the tpu-introduction page is for “simple tests” and tells you to use the TDL SDK for anything real.

Is there a standard for this?

No. There is no open standard, no portable API, and no way to program this array the way you program a GPU. What exists is a de-facto pipeline that essentially every edge-NPU vendor has independently converged on, and Cvitek follows it exactly:

ONNX  ──►  vendor compiler  ──►  vendor blob  ──►  vendor runtime  ──►  fixed-function array
      (TPU-MLIR)              (.cvimodel)      (CVI_NN)                (/dev/cvi-tpu0)

What is genuinely standardised is only the left end:

ONNX as the interchange format. (Cvitek also takes Caffe; not TFLite.) PyTorch goes through jit.trace → ONNX. Everything else goes through ONNX.
MLIR as the compiler substrate. TPU-MLIR is an MLIR-based compiler, like almost every modern NPU toolchain (and like IREE, TVM, XLA).

What looks like a standard but is not usable here:

Khronos NNEF — an interchange format that never got traction; Cvitek doesn’t support it.
OpenCL / Vulkan compute / SPIR-V — there is no programmable ISA exposed, so there is nothing to compile to. The array is not a core.
ONNX Runtime execution providers / TVM BYOC — architecturally the right hook, but no Cvitek backend exists; the only public artifact is the vendor’s compiler.
Arm Ethos-U + Vela — same shape of solution (a compiler that tiles for a scratchpad MAC array), different ISA, Arm-only.

So the practical answer to “what’s the standard” is: ONNX in, .cvimodel out, and the vendor’s CVI_NN/CVI_RT C API. Below that the cvikernel op set is a vendor binary with no published encoding.

That is the same position as Rockchip (RKNN), Amlogic, Qualcomm (QNN/SNPE), Google Coral (Edge TPU compiler), and every other edge NPU. The lock-in is not an accident of this vendor; it is what a fixed-function array plus a proprietary scheduler gives you.

Getting a model onto it

The only open piece of the toolchain is TPU-MLIR (https://github.com/sophgo/tpu-mlir), and it is the only way to put your own network on this chip. It runs on x86 in a Docker image, not on the board:

# fp32 ONNX -> fp32 MLIR, with the preprocessing baked in
model_transform --model_name yolov5s --model_def ../yolov5s.onnx \
    --input_shapes [[1,3,640,640]] \
    --mean 0.0,0.0,0.0 --scale 0.0039216,0.0039216,0.0039216 \
    --keep_aspect_ratio --pixel_format rgb \
    --output_names 326,474,622 \
    --test_input ../image/dog.jpg \
    --test_result yolov5s_top_outputs.npz \
    --mlir yolov5s.mlir

# calibration table for INT8 (100-1000 images)
run_calibration yolov5s.mlir --dataset ../COCO2017 --input_num 100 -o yolov5s_cali_table

# MLIR -> .cvimodel  (--processor cv181x for the Duo S)
model_deploy --mlir yolov5s.mlir --quantize INT8 \
    --calibration_table yolov5s_cali_table \
    --processor cv181x \
    --test_input yolov5s_in_f32.npz --test_reference yolov5s_top_outputs.npz \
    --tolerance 0.85,0.45 \
    --model yolov5s_cv181x_int8_sym.cvimodel

Constraints worth knowing before you pick a network:

ONNX or Caffe only. Not TFLite.
INT8 symmetric or BF16 only. No FP32. BF16 is ~3–4× slower than INT8.
No dynamic shapes. One .cvimodel per shape; several can be merged into one file with shared weights (--merge_weight + model_tool --combine).
No model encryption.
model_tool --info x.cvimodel tells you the ION footprint before you deploy.

The state of our board

Installed (cvitek-tpusdk-duos, cvitek-middleware-duos, cvitek-osdrv-duos): driver loaded, /dev/cvi-tpu0 present, soph_tpu in lsmod, the full runtime and TDL SDK in /mnt/system, and ~60 sample binaries.

Not installed: any model. There is not one .cvimodel on the filesystem, so every sample exits with a usage message. Prebuilt ones for this chip family live in https://github.com/milkv-duo/cvitek-tdl-sdk-sg200x (cvimodel/), and that repo also carries the TDL headers, which the Debian package does not ship.

And there is a real bug in this image. The Cvitek stack allocates all of its working memory from an ION carveout, which the device tree declares with size zero:

reserved-memory {
    ion { compatible = "ion-region"; size = <0x0 0x0>; };
};
cvitek-ion {
    heap_carveout@0 { compatible = "cvitek,carveout"; memory-region = <&ion>; };
};

so the first allocation fails and the whole media/AI stack stops:

[0] carveout heap size:0 bytes, used:0 bytes
ion allocated len=0x281400 failed
_sys_ion_alloc failed

The TPU needs physically contiguous memory and the SG2000 has no IOMMU, so this carveout is not optional. In the vendor tree the size is not a literal at all — it is CVIMMAP_ION_SIZE, generated from the board’s memmap.py, and for the Duo S that file says ION_SIZE = 170 * SIZE_1M, i.e. 170 MiB placed just below the FreeRTOS region (0x95400000–0x9fe00000). Our Debian image was built with it set to zero.

The fix is one property in /boot/fdt/linux-image-*/cvitek/cv181x_milkv_duos_sd.dtb:

size = <0x0 0x0>;   →   size = <0x0 0xAA00000>;    /* 170 MiB, or 0x4000000 for 64 */

which is a same-length in-place patch — decompile with dtc -I dtb -O dts, edit, recompile — and needs a reboot. Keep a copy of the original .dtb.

Note that this is the same carveout the camera pipeline (VI/VPSS/VO) and the video codec use, so an SG2000 image with a zero-sized ION heap cannot run the multimedia stack at all, TPU or not. That is worth knowing before blaming your model.

(Status: measured on hardware. CVI_RT_Init succeeds; the first CVI_RT_MemAlloc(4096) fails on the empty carveout and the vendor library asserts rather than returning an error, so the process aborts. The other two ION heaps do hand back mappable memory, but with paddr=0 — not contiguous, so not usable by a bus master with no IOMMU. Full output in examples/npu/probe/RESULTS.md.)

Open items

The ION patch above was applied to duos-toad-9f7f and the board did not come back after the reboot. It needs a power cycle before the patch can be confirmed. The original DTB is saved next to the patched one as cv181x_milkv_duos_sd.dtb.orig; copying it back over cv181x_milkv_duos_sd.dtb reverts the change.
No .cvimodel has been run on this board yet. The first measurement worth taking is resnet18 or mobilenet_v2, where the vendor table above gives a target (18.26 ms / 7.07 ms) to check against — that tells you immediately whether the carveout fix landed.
The TDL headers are not in the Debian package. They are in https://github.com/milkv-duo/cvitek-tdl-sdk-sg200x (include/cvi_tdl/), along with prebuilt scrfd and yolov5s cvimodels for cv181x.