Dynamically change model with api_key without os.environ

### Description

We are letting users to choose their own models and providing their own API key, thus we're unable to set `os.environ` as it will affect all users on the service. I see there is a `infer_provider_class` but no `infer_model_class` to pass in a provider to.

### References

_No response_