DISCUSS

mod_h2 - a http/2 modules for apache httpd
==========================================
The mod_h2 Apache httpd module implements the HTTP2 protocol (h2+h2c) on
top of libnghttp2 for httpd 2.4 servers. For a general description, see
the README, installation issues are detailed in INSTALL.

This document is for discussion of the module's internals, current issues
and exploration of ideas.


THE GOALS
---------
The goals of this module can be stated as follows:
- make the full htpp2 standard available in Apache httpd
- provide it as a module on top of httpd 2.4.x
- support deployment on standard unix platforms


HOW IT WORKS
------------
The architecture of this module is heavily inspired by Google's mod_spdy:
The incoming, parallel requests (http2 streams) are dispatched to a thread
pool, responses are collected and multiplexed on the original connection.

The major players in ascii art:

  h2_conn -> h2_session ------> h2_mplx ----> h2_task / h2_worker
  (socket)   (nghttp2_session)            |-> h2_task / h2_worker
                                          |-> h2_task / h2_worker
                                                ...

h2_session: by using nghttp2 API, is doing the http2 frame work, stream
            states, flow control, etc. Sits as connection level filter
            on standard httpd connections. Gets active either by ALPN
            selection or as HTTP/1.1 Upgrade from a request.

h2_mplx:    is a somewhat specialized bucket_brigate. It multiplexes data
            buckets associated with stream IDs in both directions and
            has some specials to reset streams or announce response headers.
            It also performs flow control on the downlink of streams.

h2_task:    having own conn_rec instance, plus in/out filters at connection
            level, is converting http2 streams into http/1.1 requests and
            parses back responses to http2 usable headers and data.


LIFETIMES
---------
For each connection that uses HTTP/2, a new h2_session is created. That lives
as long as all objects it creates: h2_stream and h2_task instances. So, when
closing a h2_session, this waits until all associated h2_streams have
been destroyed. h2_streams will only be destroyed when their h2_task is either
removed from the schedule queue or has terminated.

Insofar, the lifetimes from h2_session/h2_stream have the similar relations as
conn_rec/request_rec with the exception that there can be many simultaneous
h2_streams active per h2_session (and in various worker threads).


THREAD HANDLING
---------------
h2_session is only ever accessed from the thread handling the original
connection. Same for h2_stream instances. The execution of what is necessary
for execution of a h2_stream happens in h2_task. h2_task gets instantiated
in the connection thread, but is the handed to a worker and, apart from
checking its status atomically, not called by any other thread.

The object that shovels the data packages back and forth and is accessed from
multiple threads is h2_mplx. h2_tasks use it to retrieve their input and feed
it their output. h2_mplx blocks h2_tasks when no input is available or
the amount of output queue has reached a certain maximum.

There is a nice thread pool in apr-util which even suports priority scheduling.
It would be good to exchange the h2_worker(s) for this pool, except mod_h2
has the use case that streams can be aborted by the client and the
corresponding task needs to be removed from the schedule without any
blocking wait for a possibly running task to complete. This is not offered
by apr-util.


LOCKS / CONDITIONS
------------------
apr_thread_mutex_t is used for locking. apr_thread_cond_t is used for blocking
and signalling. The number of such objects created grows linear with the 
number of parallel main connections, plus the number of worker threads.

This means it is not influenced by the number of outstanding requests. The 
intention for this is to allow, possibly, many outstanding requests per HTTP/2 
connection without consuming unnecessary server resources.


MEMORY HANDLING
---------------
The session pool is a sub pool of the main connection pool, with the twist
that it has its own allocator (apr_allocator_t). That allocator is protected
with a apr_thread_mutex (one instance per session). All further sub-pools share
this allocator.

Protecting the allocator allows sub pools to live in other threads 
concurrently, Necessary for parallel processing of HTTP/2 streams. (As 
alternative, using root pool for streams was tested, but resulted in poorer
performance).

Everything related to a h2_stream (and even the struct itself) is allocated
from a new subpool. This guarantuees that all memory is recycled when the 
stream is destroyed. Same holds true for h2_tasks, which get their own 
sub pools.

FILE BUCKET HANDLING
--------------------
Requests for static resources result most often in a single file bucket being
send as body of the response. This would ideally be placed into eh h2_mplx,
thus finished the h2_task and freeing the h2_worker for other things.

This is difficult for the following 2 reasons:
1. Finishing the h2_task will free pool and run registered cleanup functions
   that closes the file. The file needs to traverse this boundary. Early
   attempts at doing this nicely have failed.
2. In load scenarios, we quickly run out of open file handles. With 100 max
   parallel stream per connection, the process limits can be reached quite
   easily.

Solving 1) can be done by careful coding and good debugging of what the apache
runtime does here. Solving 2) requires some kind of resource booking scheme
inside the httpd child process, it seems.

For now, mod_h2 reads files when placing the data into the h2_mplx. That means
that files are completely read before the h2_task is finished and the h2_worker
is available again. This limits the number of open files in buckets to the 
number of h2_worker (by order of magnitude - there can be sub requests etc.).

This works well and stable, but does not allow the transfer speeds of httpd's
optimized HTTP/1 implementtion.


DISCUSSION / OPEN QUESTIONS
---------------------------
- HTTP/2 Padding feature is not implemented. As RFC7540, Ch. 10.7 describes, 
  using a fixed length padding is counter-productive, same as simple random
  schemes. The ideal padding is supposed to be chosen by the application. So
  maybe a response note or special header should determine it?
- HTTP/2 Priority handling of streams is implemented in nghttp2, HOWEVER it
  has no effect on h2_task scheduling. If there is a backlog of streams for
  processing by workers, the stream with the highest priority should be
  processed first. This information is currently not available via the nghttp2
  API.