NPU-GPU-Pipeline

Ryzen™ AI NPU-GPU Pipeline

Model Pipelining on NPU and iGPU

We showcase an application that has models strategically distributed and off-loaded to the NPU and integrated GPU (iGPU), based on varying intensity of computation requirements and hardware support. The application consists of two Convolutional Neural Network (CNN) based models that run on the NPU, and one generative model that is off-loaded to the iGPU. ONNX Runtime provides support for Vitis AI EP and DirectML EP, used to demonstrate inference on the NPU and iGPU respectively. Some of these models operate concurrently, thus utilizing the accelerator resources to their full potential. Please refer to NPU-GPU article for high level implementation

Prerequisites and Environment Setup

Install Ryzen AI Software using the automatic installer Link. This should create a conda environment that can be used for this example. Check the installation path variables

Below instructions can be executed on Conda command prompt or Miniforge Prompt

1. Create a clone of the Ryzen AI installation conda environment to add required python packages

set RYZEN_AI_CONDA_ENV_NAME=ryzen-ai-<version>
conda create --name npu-gpu-pipeline --clone %RYZEN_AI_CONDA_ENV_NAME%
conda activate npu-gpu-pipeline

2. Set RyzenAI Environment variable

# Location of RyzenAI software installation path or default at "C:\Program Files\RyzenAI\<version>"
set RYZEN_AI_INSTALLATION_PATH=<Path to RyzenAI Installation>

3. Install dependencies for Yolov8 and RCAN:

cd <RyzenAI-SW>\demo\NPU-GPU-Pipeline
python -m pip install -r requirements.txt 

4. Download pre-quantized ONNX models from Huggingface for Yolov8 and RCAN in the same directory.

5. Install dependencies for Stable Diffusion:

python -m pip install -r stable_diffusion\requirements-common.txt 

6. Make sure XLNX_VART_FIRMWARE is set to point to the correct xclbin from the VOE package

echo %XLNX_VART_FIRMWARE%

7. Copy vaip_config.json from the installed VOE package to this directory

copy %RYZEN_AI_INSTALLATION_PATH%\voe-4.0-win_amd64\vaip_config.json .

Running the application

1. Generate optimized stable diffusion model using Olive:

cd stable_diffusion 
python stable_diffusion.py --provider dml --optimize

The optimized FP16 models should be generated in models\optimized-dml.

2. Run the example (the following command off-loads Stable Diffusion to the iGPU and Yolov8+RCAN to the NPU):

cd ..
python pipeline.py -i test/test_img2img.mp4 --npu --provider_config vaip_config.json --igpu