Cloud Native LLM runtime proposal

[daixiang0]

For python developers, litellm maybe a good choice. That would be great we can do it in a runtime and any developers can use LLMs in own language way rather than packaging HTTP/gRPC calls by themselves.

Now we have many LLM APIs like OpenAI, Azure AI, Cohere, LLaMA, AWS bedrock, Kserve, OpenVINO and so on, migrate from one to the other still need many code changes.

I propose that we can do it in a Cloud Native LLM runtime, then developers can migrate from one to the other only by config.

[zanetworker]

Related doc here: https://docs.google.com/document/d/1FQN_hGhTNeoTgV5Jj16ialzaSiAxC0ozxH1D9ngCVew/edit

[daixiang0]

The whole proposal is here, and point out difference between gateway and runtime.