Feature Request: setting to force range requests on or off

[jcushman]

The logic to detect whether range requests are supported or not is complicated and brittle. (The openFile function in runtime_browser.ts).

• It depends on two settings, reliable_head_requests and allow_full_http_reads, that have effects that are hard to understand even when reading the code.
• It's been been updated multiple times as different edge cases are discovered with how servers report range support.
• It's still not robust. In my current project I was only using servers that support range requests, but I had to spend a bunch of time discovering by trial and error that I needed SET reliable_head_requests = true to get range requests to work in the Vite dev server, but SET reliable_head_requests = false to get them to work with files served on source.coop (which I think is S3 under the hood).
• The ways to make it more robust involve extra requests to the server, which hurt performance.

I think much of this could be mitigated if there was a setting like force_range_requests, on/off/unset, that worked like:

• If on, skip checks and assume range requests work.
• If off, skip checks and assume range requests won't work.
• If unset, fall back to current logic.

This would work well in two situations:

• I'm using duckdb-wasm as an application designer in a controlled setting where I know which server will serve the database files, and I know whether it supports range requests. It's on the application designer to check, and their fault if it doesn't work. (This was the situation that made me post here.)
• I'm using duckdb-wasm in the shell as a user, trying to connect to some slightly non-compliant server, and I don't mind just setting force_range_requests and seeing what happens.

I think both of these situations are well communicated by offering a force_... flag -- if you use it, what happens is on you, but we have it because you might know better than we do.

[jcushman]

Note I don't think this is anyone's fault who implemented various versions of the detection logic. I think it's a situation where the spec is vague and different servers have done different things, so any version of the detection logic that is efficient for some servers is going to be broken for other servers -- hence the detection logic being updated in circles, and a way to opt out of detection logic entirely is desirable.

[jjhbw]

I'm running into the same issue. My server supports range requests, but DuckDB thinks it doesn't because of the HEAD request. Pulling in the whole Parquet file is not an option for me, so it would be great if I could force it to use range requests.