Llama Fast API Server

Installation

FastAPI

Windows

pip install fastapi

macOS

pip3 install fastapi

Llama-cpp-python

Windows + CUDA

pip install llama-cpp-python --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cu121

Windows + CPU

pip install llama-cpp-python --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cpu

macOS + Metal

pip3 install llama-cpp-python --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/metal