native fetch has a default timeout of 300s

[narengogi]

What Happened?

This hasn't surfaced so far because of our use of workerd environment,
but native fetch from node js has a deafult 300s timeout nodejs/node#46375
so if the response headers take longer than 5mins to arrive, we get this exception

"html-message": "Message: fetch failed Cause: HeadersTimeoutError: Headers Timeout Error Name: TypeError"

and it's entirely strange that this is not configurable
since portkey has undici as a dependency already, we should use a custom agent with a higher default timeout, since the code is tied in with the native fetch, this might need some wriggling around to respect the user sent custom timeout options

What Should Have Happened?

No response

Relevant Code Snippet

No response

Your Twitter/LinkedIn

No response

[jrokiskyjhuapl]

@narengogi we got bit by this issue due to the longer runtimes of some openai models. If you are planning on developing a solution to this, please make the timeout value configurable

[narengogi]

@jrokiskyjhuapl sorry for teh delay, keeping this open, @VisargD add a fix for this