Skip to main content

Probe results — 2026-11-03

Board: Milk-V Duo S, debian@duos-toad-9f7f, scpcom Debian image (duos-e_sd-develop.img), kernel 5.10.260-20260911-7+duos, freshly reflashed so the device tree is stock.

Question: can anything execute on the TPU at all?

Answer: no. The runtime initialises, then aborts on its first allocation. The cause is the zero-sized ION carveout in the device tree, and the vendor library turns that into a SIGABRT rather than an error return.

1. The vendor runtime

$ sudo python3 -u probe.py
libcviruntime: /mnt/system/lib/libcviruntime.so
CVI_RT_Init -> rc=0 handle=0x7b21e0
  CVI_RT_MemAlloc(4096) ...
reopen ion dev
reopen ion dev success
reopen ion dev
reopen ion dev success
reopen ion dev
reopen ion dev success
ion ioctl fail:: Cannot allocate memory
ion ioctl fail:: Cannot allocate memory
ion ioctl fail:: Cannot allocate memory
python3: /build/tpusdk/cviruntime/src/soc/common/cvi_device_mem.cpp:451:
         virtual bm_memory* CviDeviceMem::mem_alloc_raw(bmctx_t, size_t):
         Assertion `0' failed.
Aborted (core dumped)   [exit 134]

Three things worth keeping for the slides:

  • CVI_RT_Init returns 0. The driver opens fine and the runtime is healthy.
  • The failure is on 4 KiB — the smallest thing it can ask for. It never gets as far as weights, tensors, or a command buffer.
  • It retries three times, then asserts. mem_alloc_raw ends in assert(0), so a recoverable ENOMEM becomes a process abort. There is no error path to handle from above; CVI_NN_RegisterModel cannot report “out of memory” because the library will not return.

2. The kernel allocator underneath

Bypassing the vendor code and talking to /dev/ion directly. Note this is not the upstream ION uapi — Cvitek’s ion_allocation_data has no align field and carries fd/paddr/name, and ION_IOC_HEAP_QUERY does not implement the “ask for 0 to learn the count” convention.

$ sudo python3 -u ion_probe.py
3 heap(s) registered:
  id= 2  type=0 (system)  name='ion_system_heap'
  id= 1  type=0 (system)  name=''
  id= 0  type=0 (system)  name=''

4 KiB allocations, one heap at a time:
  id=2 ion_system_heap         -> fstat ok; mmap+write ok; paddr=0x0
  id=1 (unnamed)               -> fstat ok; mmap+write ok; paddr=0x0
  id=0 (unnamed)               -> Cannot allocate memory (errno 12)

Reading:

  • id 0 is the carveout, and it is empty — ENOMEM on 4 KiB. That is the reserved-memory/ion { size = <0x0 0x0> } node, and it is the heap the vendor runtime asks for.
  • id 1 and id 2 work. They hand back real, mappable memory. So ION itself is not broken; one heap is.
  • But paddr=0x0 for both. They are not physically contiguous, so they are useless to the TPU, which addresses memory by physical address and has no IOMMU. A working heap is not the same as a usable one.

Heap ids are ordered as registered; the device tree declares only heap_carveout@0, and Cvitek’s driver registers its own system and contig heaps alongside it, which is why there are three.

What this rules in and out

  • Out: running a model, loading a .cvimodel, or measuring anything on a stock image. Nothing gets past the first 4 KiB.
  • Out: fixing it from userspace, unless the runtime can be steered to a contiguous heap. CVI_RT_Global_SetMemAllocCallback is the documented hook, but its callback must return a CVI_RT_MEM, an opaque bm_memory * whose layout is not published — so a userspace allocator has nothing it can construct.
  • In, but not this phase: a non-zero reserved-memory/ion size, which is a device-tree edit and a reboot. Off the table by instruction, and the last attempt at it cost a card reflash.

Reproducing

./examples/npu/probe/run.sh

Both probes only open the driver and ask for memory. Nothing is submitted to the TPU, and nothing on the system is modified.