Removing the limitation of 256KB for header size

[gahfy]

This allows users to not be restricted to the limitation of 256KB for the size of the header, in order to solve #8472 :

How to reproduce

Python3 server code

import socket
import random
import string

def generate_random_string(length):
    return ''.join(random.choices(string.ascii_letters + string.digits, k=length))

random_string = ''.join(random.choices(string.ascii_letters + string.digits, k=256 * 1024 - 48))

# Création d'un socket TCP
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as server_socket:
    server_socket.bind(('127.0.0.1', 8080))
    server_socket.listen()

    while True:
        conn, addr = server_socket.accept()
        with conn:
            response = (
                "HTTP/1.1 200 OK\n"
                "Content-Type: text/plain\n"
                f"MyHeader:{random_string}\n"
                "Content-Length: 14\n"
                "\n"
                "Erreur de requête"
            )
            conn.sendall(response.encode('utf-8'))

Kotlin Client code after/before the PR (Comment/Uncomment lines 10/11)

import okhttp3.OkHttpClient;
import okhttp3.Request;
import okhttp3.Response;
import okhttp3.ResponseBody;
import okhttp3.internal.http1.HeadersReader;

public class OkHttpTest {
  private static final String ENDPOINT = "http://127.0.0.1:8080";

  public static void main(String... args) {
    OkHttpClient client = new OkHttpClient.Builder().build();
    //OkHttpClient client = new OkHttpClient.Builder().headerSizeLimit(HeadersReader.HEADER_NO_LIMIT).build();

    try {
      Request request = new Request.Builder()
        .url(ENDPOINT)
        .build();
      try (Response response = client.newCall(request).execute()) {
        ResponseBody body = response.body();
        System.out.println(body.string());
      }
    } catch(Exception e) {
      e.printStackTrace();
    }
  }
}

Before, you will get an EOFException, after, everything works fine

[yschimke]

Seems reasonable. Let me confirm with specs and also my past decisions.

[yschimke]

Why -2 instead of zero? Just avoiding -1?

[gahfy]

Yes, and I'm not sure about the behavior then of source.readUtf8LineStrict(headerLimit) with a negative number in the case of an empty line, which wouldn't break the rule limit. So putting it strictly negative will handle that scenario.

[yschimke]

Given we now have two codepaths here, and it was already confusing. Should we catch and re throw in all cases with context such as remaining header length/"NO_LIMIT"?

[gahfy]

I'm not sure to understant what you mean here ? If there is no limit and we do source.readUtf8LineStrict(headerLimit), it will throw an exception, and the next source.readUtf8LineStrict() will be run on the next line if I understant it well.
So how would you perform the completion of the line in the catch clause ?

[yschimke]

It's probably worth catching and rethrowing and explaining issue so it's clear why

• newline not found within limit
• newline now found by end of stream

[gahfy]

Not sure to really understand what you mean. Can you write 1 or 2 line of pseudo code, to make sure I understand it well (no need to make something correct, it does not need to compile either, it's just in order to get an idea of what you mean)

[yschimke]

I'm mainly concerned about the confusion with this limit and the limit used by Okio.

https://github.com/square/okio/blob/597a65527c987d9f221dd5607c49e49f934df2de/okio/src/jvmTest/kotlin/okio/ReadUtf8LineTest.kt#L66

  throw EOFException(
    "\\n not found: limit=" + minOf(buffer.size, limit) +
      " content=" + data.readByteString().hex() + '…'.toString(),
  )

So if I've set the header length limit to 1024, but I might get 0, 10, or 1024 as the limit in the exception message. I'd be confused if I saw a different limit here.

Maybe something like

    val line = try {
      if (headerLimit == HEADER_NO_LIMIT) source.readUtf8LineStrict()
      else source.readUtf8LineStrict(headerLimit)
    } catch (eofe: EOFException) {
      throw EOFException(buildString {
        append("Could not read a complete header")
        if (headerLimit != HEADER_NO_LIMIT) {
          append(" within $headerLimit bytes")
        }
      }).apply {
        initCause(eofe)
      }
    }
``

But I'm open to suggestions.

[yschimke]

In #3613 we removed a system property lookup. Not sure why we didn't move to a config.

[yschimke]

My one concern about this is that the current default is way above internet infrastructure. So likely to break on proxies or other clients.

[gahfy]

My one concern about this is that the current default is way above internet infrastructure. So likely to break on proxies or other clients.

I agree with that, that is why in the #8472 issue, I suggested also to not change anything, and just update the documentation. Btw, very long headers do not break any RFC 2616 rule. Maybe making sure that we can handle any response which does not break any rule would make the library also usable by people with that weird server response.

[yschimke]

I don't know about the documentation vs config choice.

But if we want to move forward with this, could you add some tests of header limits that show the messages that will be seen.