Add Axelera Metis NPU detector plugin v.0.19 - #24270
patrykorwat wants to merge 15 commits into
Conversation
patrykorwat
changed the title
 |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
hawkeye217
left a comment
hawkeye217
left a comment
There was a problem hiding this comment.
The backpressure change and the duty cycle reset both look right, and step 2 is fixed.
I'd like to keep kmod and pciutils out of the base image. The runtime only uses them for a preflight check in list_devices() where it greps lspci and lsmod output, and both tools just read files that are already visible in the container, so the plugin can provide two small shims instead. Details inline.
Still open: the colon alias in init-devices/run (see my question about the long bind syntax), the type == "axelera" fold in stats/util.py, the error message that tells users to pass the colon node, the two function level imports, the duplicate close()/shutdown(), the dead model_url split, and _find_model_json picking the first subdirectory. The runtime and onnx factories are injectable, so _decode, the letterbox mapping, and the zip extraction guard can get unit tests without hardware.
| wget \ | ||
| lbzip2 \ | ||
| procps vainfo acl \ | ||
| procps vainfo acl kmod pciutils \ |
There was a problem hiding this comment.
I'd rather not add these to the base image for everyone. list_devices() doesn't need either tool to do real work, it just runs a bare lspci and lsmod through popen and greps the output (the "No target device found in lspci output" and "No AIPU driver found in lsmod output" strings are in libaxruntime.so). Both tools only read /sys/bus/pci/devices and /proc/modules, which are readable in the container, so the plugin can provide its own.
Can you try writing these two scripts to a directory the plugin owns, like /config/model_cache/axelera/bin, and prepending it to os.environ["PATH"] right before _point_runtime_at_installed_firmware()? Only do it when shutil.which() doesn't find the real tool.
#!/bin/sh
echo "Module Size Used by"
while read -r name size used _; do
printf '%-19s %8s %s\n' "$name" "$size" "$used"
done < /proc/modules#!/bin/sh
for dev in /sys/bus/pci/devices/*; do
[ -r "$dev/vendor" ] || continue
read -r vendor < "$dev/vendor"
read -r device < "$dev/device"
printf '%s Device %s:%s\n' "${dev##*/}" "${vendor#0x}" "${device#0x}"
doneThe second prints a Device 1f9d:<id> line per card, which is what the runtime matches. I can't run list_devices() without the card, so let me know if the count check passes.
There was a problem hiding this comment.
applied the change with having /config/model_cache/axelera/bin in PATH
| done < <(find /dev/bus/usb -type d -print0) | ||
| fi | ||
|
|
||
| # The Axelera runtime enumerates Metis cards by the colon-separated node name |
There was a problem hiding this comment.
Before adding an env var, can you try the long bind syntax? --device and short form volumes: split on colons, but the long form doesn't. With the raw node still under devices: for the cgroup grant, this might put the colon node in the container with no script change, and it would work under --user too.
devices:
- /dev/metis-0-1-0
volumes:
- type: bind
source: /dev/metis-0:1:0
target: /dev/metis-0:1:0If that doesn't work, a generic DEVICE_ALIASES variable next to DEVICE_ACL_PATHS that takes comma separated alias=target pairs is the shape I'd want.
There was a problem hiding this comment.
regarding init-devices: Docker rejected both variants during tests (too many colons in the long volume format and the mount format), so the symlink + DEVICE_ALIASES approach remains.
| cycles = [ | ||
| detector.run_duty_cycle.value | ||
| for detector in detectors.values() | ||
| if detector.detector_config.type == "axelera" |
There was a problem hiding this comment.
This is still a per-vendor branch in stats/util.py. Every ObjectDetectProcess has the Value now, so the fold can write any non-negative run_duty_cycle into npu_usages[detector.detector_config.type]["npu"] when that entry exists, and _axelera_run_duty_cycle goes away.
| if not devices: | ||
| raise RuntimeError( | ||
| "axelera: no Metis device enumerated; check that the " | ||
| "/dev/metis-<bus>:<dev>:<fn> node (colon form) is passed " |
There was a problem hiding this comment.
--device rejects colons, so users can't follow this. Point at the raw /dev/metis-<bus>-<dev>-<fn> node and whatever ends up creating the alias.
There was a problem hiding this comment.
regarding init-devices: Docker rejected both variants during tests (too many colons in the long volume format and the mount format), so the symlink + DEVICE_ALIASES approach remains.
| os.makedirs(cache, exist_ok=True) | ||
| logger.info("axelera: downloading model from %s", model_url) | ||
| try: | ||
| urllib.request.urlretrieve(_url_with_scheme(model_url), zip_path) |
There was a problem hiding this comment.
If a model download is interrupted, urlretrieve(..., zip_path) can leave a partial file at the final persistent cache path. Later startups see that file, skip the download, and reject it as a non-ZIP archive without deleting it. This prevents the Axelera detector from initializing until the cached file is manually removed. Download to a temporary file and atomically rename it after successful validation, or remove the destination when downloading or validation fails.
6ca6583 to
dc8d47f
Compare
Runs Frigate inference on the Axelera Metis NPU through the vendor's low-level axelera.runtime API with pip-installed, sha256-pinned wheels from Axelera's public artifactory (self-contained: auditwheel-repaired libs + bundled firmware). - async send_input/receive_output contract (FIFO pairing, overload emits zero rows instead of raising, frames are copied out of camera shm) - generic YOLO-family decoder driven by the compiled model's manifest.json (padded input geometry, dequant params, baked postprocess graph via onnx) - labels come from the compiled model's model_info.json; empty labelmaps fail loudly at init
The card exposes no utilization counter usable next to a live detector, so the poller reports the board-controller temperature read via axcmd (cached 5s; the read must not race detector init).
…me enumeration The Axelera runtime enumerates Metis cards via the colon-separated node name created by the vendor udev rule on the host (e.g. /dev/metis-0:1:0) and probes kernel state with lsmod/lspci. Containers only receive the raw /dev/metis-0-1-0 node, so the detector never found the card. - init-devices: create /dev/metis-BB:DD:F aliases for each Metis node - install_deps: add kmod and pciutils so the runtime's driver check works
… size, per-class NMS) - download model zips through ModelDownloader (atomic .part rename; a truncated download no longer poisons the cache) - stop(): release the Context (releases connection/model/instance); the previous instance.free() call never existed in the runtime API and was silently swallowed - honor input_pixel_format: flip to RGB planes only when Frigate delivers BGR (input_pixel_format: bgr); with rgb the frame passes through as-is - docs/catalog: width/height = compiled 640x640 input (square regions no longer get squashed twice), document input_pixel_format: bgr - NMS via xyxy_to_xywh_for_nms + per-class cv2.dnn.NMSBoxes, matching the other detector plugins instead of a hand-rolled class-agnostic pass
The same board readout was surfaced twice on the System page: once as the detector temperature (get_hardware_temperatures) and once on the NPU card (npu_usages). Keep the NPU card, where the value belongs.
The runtime exposes no device-side usage counter (mvm_utilisation is a configuration limit, not a readout), so the NPU card rendered empty. ModelInstance.run blocks until the AIPU completes: the detector process now times each run and publishes busy/wall over a 2s window through a small file under the tmp cache dir; the stats poller reads it and sets the npu key. A stale file (detector idle/restarting) reports nothing rather than a stale number.
Preprocess reused three fresh arrays per frame (resize output, quantised copy, strided channel flip at write time) and refilled the whole padded input buffer. Now resize writes into a scratch with cv2 dst=, the BGR->RGB swap is a cvtColor into scratch (no strided copy), the int8 quantise runs with out=, and only the border strips the content rect does not cover get memset. Measured 39% faster on the 704x576 -> 640x640 path. Dequantise allocated two fp32 temporaries per head (astype then the dequant expression) before the transpose copy. copyto now casts int8->fp32 while transposing straight into a reused fp32 feed buffer, and the zp/scale math runs in place. Measured 78% faster for the YOLOX-S head set, outputs bit-identical (int8->fp32 cast is lossless). Argmax+gather replaces max+argmax (one class-axis reduction instead of two), and the postprocess session pins intra_op_num_threads=1: one frame is in flight through the decoder, extra ORT threads only add context switches.
Replace the tmp-file duty-cycle handoff (and the FRIGATE_TMP env var nothing else reads) with a third multiprocessing.Value alongside avg_inference_speed and detection_start: ObjectDetectProcess creates it, AsyncDetectorRunner passes it to the detector, and the plugin publishes busy/wall of the blocking ModelInstance.run over a 2s window, mirroring how memryx receives its shared state. No filesystem path, no staleness window, no atomic rename, and stats/hardware.py no longer imports the plugin module to find the file. The producer also ticks the window while the queue is empty, so the duty cycle falls to 0 when the card idles instead of sticking at its last value. Name it what it is: host-side occupancy of the run call, including DMA and driver time, not an AIPU core counter (the runtime exposes no device-side one). The NPU entry now reads temperature from the board controller plus the detector-reported run duty cycle folded in where the detector processes are in reach.
Context.list_devices() keys off the lspci BDF and needs the colon-form node to be present, so surface an empty enumeration at init with the fix-it hint instead of a bare device_connect(None) ConnectionError several frames deeper. Uses the DeviceInfo the runtime enumerated rather than relying on the pick-first implicit path.
… cycle on restart Greptile P1: _push_result dropped a completion when the result queue was full, stranding the runner's send_times entry and the input SHM mapping (1 submission : 1 result contract). Replace the drop with a stop-aware blocking put so an accepted submission always produces exactly one delivered completion. Greptile P2: start_or_restart did not clear run_duty_cycle when starting a replacement process, so stats could show the previous process's duty cycle while the new detector was still initializing. Reset it to the -1.0 sentinel alongside detection_start, inside the async branch. Greptile P2 (offline docs): list axelera among the detector types whose pre-seeded runtime files live under /config/model_cache/runtimes/<type>.
…VICE_ALIASES, generic duty-cycle fold) install_deps.sh: drop kmod/pciutils from the base image. The runtime's list_devices() only greps popen'd lspci/lsmod output over /sys and /proc, so the plugin now installs shell stand-ins in its own /config/model_cache/axelera/bin and prepends it to PATH when the real tools are missing (hardware-verified: the stand-in lspci prints the Device 1f9d:<id> line the runtime matches, the lsmod shape matches /proc/modules). init-devices: replace the Metis-keyed alias block with a generic DEVICE_ALIASES env var (comma separated alias=target pairs, next to DEVICE_ACL_PATHS). Docker rejects colon-named nodes in --device AND in bind mounts (probed: 'too many colons' on both the long volume form and the mount form), so the in-container symlink stays necessary; docs carry the pair users must set. stats/util.py: fold any detector's published run_duty_cycle into the matching npu_usages entry generically; _axelera_run_duty_cycle and the per-vendor branch are gone. plugin: the no-device RuntimeError now names the raw dash-form node (--device rejects colons, users can only pass the raw node) and points at DEVICE_ALIASES for the colon alias.
…n spellings) stats/util.py: several detector processes can share one hardware type (e.g. two axelera cards on one host). Iterating and assigning made the displayed NPU usage depend on dict order, last writer wins. Collect the published windows per type and write the busiest one, matching what the removed per-vendor helper did with max(). plugin: the remaining British spellings (dequantise family) corrected to the American forms the repo requires. plugin: model path resolution follows hailo8l's single-string pattern (URL or local path or preset name in one value), the download uses urllib directly like memryx, and _find_model_json now raises on an ambiguous multi-model directory instead of returning whichever sorts first. plugin: AxeleraDetector exposes shutdown() (the async contract hook base.py calls) instead of close(); __del__ guards a partially initialised instance.
dc8d47f to
2ef33d4
Compare
2 participants
Proposed change
Continuation of #24036. Adds support for the Axelera Metis AI accelerator as a Frigate object detector.
Type of change
For new features
AI disclosure
AI tool(s) used: Hermes with deepseek-ai/DeepSeek-V4-Flash-0731 and Qwen/Qwen3.8-Flash-Next
How AI was used:code generation, testing, debugging, documentation
Extent of AI involvement: implemented the Frgate to Axelera API bridge, generated documentation in the asked placea
Human oversight: did go through the generated code and made sure safe to use, tested the implementation on a stream from 7 IP cameras detecting objects with yolox-s-coco-onnx model.
Checklist
enlocale.ruff format frigate)