Support multiple input dockerfiles (single frontend)

[fenollp]

Description

As I am generating larger and larger Dockerfiles, I am hitting size limts and some are getting rejected. cc docker/buildx#3637 #6097 (comment)

What I am actually doing is creating and building increasingly larger Dockerfiles, progressively stitching them together (while making use of the build cache). (That's just due to my problem space.)

I am proposing to allow an alternative to --local dockerfile=DIR, taking 2+ --local dockerfiles=FILE, still along with a single frontend or gateway. These many build descriptors would then be topologicaly sorted and pieced together to form one large LLB graph (bypassing the input file limits).
Each descriptor still has to be parseable by the frontend/gateway on their own. (For a Dockerfile this also means the ordering of its multiple stages has to be correct).
Duplicate stages (contents and names) should be allowed when they match exactly.
Also, --opt target=NAME should be required.

To illustrate, this:

# my/first.Dockerfile
FROM ubuntu AS b
RUN echo World! >/b

# second.Dockerfile
FROM alpine AS a
RUN printf %s 'Hello ' >/a

# final.Dockerfile (parses but fails to build on its own)
FROM ubuntu AS ab
RUN \
  --mount=from=a,source=/a,dst=/a \
  --mount=from=b,source=/b,dst=/b \
    cat /a /b >/c
FROM scratch AS final
COPY --from=ab /c /

buildctl build --frontend dockerfile.v0 --local context=. --opt target=final --local dockerfiles=my/first.Dockerfile --local dockerfiles=second.Dockerfile --local dockerfiles=final.Dockerfile --output type=local,dest=.

...would be exactly equivalent to:

# /tmp/assembled.Dockerfile
FROM alpine AS a
RUN printf %s 'Hello ' >/a

FROM ubuntu AS b
RUN echo World! >/b

FROM ubuntu AS ab
RUN \
  --mount=from=a,source=/a,dst=/a \
  --mount=from=b,source=/b,dst=/b \
    cat /a /b >/c
FROM scratch AS final
COPY --from=ab /c /

buildctl build --frontend dockerfile.v0 --local context=. --opt target=final --local dockerfile=/tmp/assembled.Dockerfile --output type=local,dest=.

This is probably doable today with bake and using contexts and the target: syntax. I am looking for a buildctl way.

What I am proposing should not conflict with existing behavior and would allow for LLB graphs no longer limited by frontend file sizes (as of v0.27.1: between 10510099 and 10549763 bytes it seems).

I have not seen this approach mentioned in previous "linking frontends" discussions e.g. here or there.

It is possible this file size limit exists only in buildx side, in which case we should move this proposal there.