How to add cloudflare workers AI via litellm proxy UI?

[tan-yong-sheng]

Since cloudflare has officially supported OpenAI-compatible api for Cloudflare workers AI (see https://developers.cloudflare.com/workers-ai/configuration/open-ai-compatibility/), I'm thinking how we could we add it into llmlite proxy UI?

For example, the sample API_BASE_URL will be "https://api.cloudflare.com/client/v4/accounts/${env.CLOUDFLARE_ACCOUNT_ID}/ai/v1"

Here is the UI that I am tinkering with to add in this Cloudflare Workers AI....

Hope for help, thanks.

[tan-yong-sheng]

I put these on config.yaml instead

model_list:
  - model_name: cloudflare/llama-3-8b-instruct
    litellm_params:
      model: cloudflare/@cf/meta/llama-3-8b-instruct

  - model_name: cloudflare/llama-2-7b-chat-fp16
    litellm_params:
      model: cloudflare/@cf/meta/llama-2-7b-chat-fp16

  - model_name: cloudflare/mistral-7b-instruct-v0.1
    litellm_params:
      model: cloudflare/@cf/mistral/mistral-7b-instruct-v0.1

  - model_name: cloudflare/deepseek-coder-6.7b-base-awq
    litellm_params:
      model: cloudflare/@hf/thebloke/deepseek-coder-6.7b-instruct-awq

[creedjhem]

Did you manage to make it work ?

[tan-yong-sheng]

Hi @creedjhem, ya, I managed to get it work for Cloudflare Workers AI via using LiteLLM's config.yaml file:

Reference: https://docs.litellm.ai/docs/proxy/configs

For your config.yaml file, add these:

# CLOUDFLARE WORKERS AI
  - model_name: cloudflare/*
    litellm_params:
      model: cloudflare/*
      # Replace <YOUR_COUDFLARE_ACCOUNT_ID> with your cloudflare_account_id
      api_base: "https://api.cloudflare.com/client/v4/accounts/<YOUR_COUDFLARE_ACCOUNT_ID>/ai/run/"
      api_key: "<YOUR_CLOUDFLARE_API_KEY>"
      # not sure how to set  tpm, as it offers 10,000 tokens/day
      drop_params: true
      modify_params: true

To find your cloudflare account ID: https://developers.cloudflare.com/fundamentals/setup/find-account-and-zone-ids/
To create cloudflare API key: https://developers.cloudflare.com/fundamentals/api/get-started/create-token/

Below is the config.yaml file I set up to connect various LLM api, like gemini, github models, openai, AWS bedrock. You could take it as reference:

litellm-config.txt

---

Just some self-promo, I am a blogger at https://tanyongsheng.com/blog/ and https://www.tanyongsheng.com/note/ who is keen on open source technology.