Merge pull request #491 from ipfs/trustless-gateway-responses

trustless gateway: document probing path (identity cid)

Author: lidel

src/http-gateways/path-gateway.md

@@ -296,6 +296,8 @@ The request succeeded.
 
 If the HTTP method was `GET`, then data is transmitted in the message body.
 
+If the HTTP method was `HEAD`, then no body should be sent.
+
 ### `206` Partial Content
 
 Partial Content: range request succeeded.
@@ -310,23 +312,35 @@ The new, canonical URL is returned in the [`Location`](#location-response-header
 
 ### `400` Bad Request
 
-A generic client error returned when it is not possible to return a better one
+A generic client error returned when it is not possible to return a better
+one. For example, this can be used when the CID is malformed or its codec is
+unsupported.
 
 ### `404` Not Found
 
-Error to indicate that request was formally correct, but traversal of the
-requested content path was not possible due to a invalid or missing DAG node.
+Error to indicate that request was formally correct but either:
+
+* traversal of the requested content path was not possible due to a invalid or
+missing DAG node, or
+* the requested content is not retrievable from this gateway.
+
+Gateways MUST use 404 to signal that content is not available, particularly
+when the gateway is [non recursive](#recursive-vs-non-recursive-gateways), and only provides access to a known
+dataset, so that it can assess that the requested content is not part of it.
 
 ### `410` Gone
 
 Error to indicate that request was formally correct, but this specific Gateway
-refuses to return requested data.
+refuses to return requested data even though it would have normally provided
+it.
 
-Particularly useful for implementing [deny lists](#denylists), in order to not serve malicious content.
+`410` is particularly useful to implement [deny lists](#denylists), in order to not serve blocked content.
 The name of deny list and unique identifier of blocked entries can be provided in the response body.
 
 See: [Denylists](#denylists)
 
+See also: [`451 Unavailable for Legal Reasons`](#451-unavailable-for-legal-reasons).
+
 ### `412` Precondition Failed
 
 Error to indicate that request was formally correct, but Gateway is unable to
@@ -359,23 +373,45 @@ See: [Denylists](#denylists)
 
 ### `500` Internal Server Error
 
-A generic server error returned when it is not possible to return a better one.
+A generic server error returned when it is not possible to return a better
+one. An internal server error signals the general unavailability of the
+gateway.
 
 ### `502` Bad Gateway
 
-Returned immediately when Gateway was not able to produce response for a known reason.
-For example, when gateway failed to find any providers for requested data.
+Error that indicates that a Gateway was not able to produce response for a
+known reason: for example, in the case of
+[recursive gateways](#recursive-vs-non-recursive-gateways), in the event of
+failure to find any providers for requested data. `502` indicates that the
+request can be retried and is not a permanent failure.
+
+This error response SHOULD include
+[`Retry-After`](#retry-after-response-header) HTTP header to indicate how long
+the client should wait before retrying.
 
-This error response SHOULD include [`Retry-After`](#retry-after-response-header) HTTP header to indicate how long the client should wait before retrying.
+Gateways SHOULD return `404` instead of `502` when the content is known to be
+unretrievable: for example, when the Gateway is
+[non-recursive](#recursive-vs-non-recursive-gateways) and the content is known
+to not be available.
 
 ### `504` Gateway Timeout
 
-Returned when Gateway was not able to produce response under set time limits.
-For example, when gateway failed to retrieve data from a remote provider.
+Error that indicates that the Gateway was not able to produce response under
+set time limits: for example, when gateway failed to retrieve data from a
+remote provider. `504` indicates that the request can be retried and is not a
+permanent failure.
+
+There is no generic timeout, Gateway implementations SHOULD set timeouts based
+on specific use cases.
 
-There is no generic timeout, Gateway implementations SHOULD set timeouts based on specific use cases.
+This error response SHOULD include
+[`Retry-After`](#retry-after-response-header) HTTP header to indicate how long
+the client should wait before retrying.
 
-This error response SHOULD include [`Retry-After`](#retry-after-response-header) HTTP header to indicate how long the client should wait before retrying.
+Gateways SHOULD return `404` instead of `504` when the content is known to be
+unretrievable: for example, when the Gateway is
+[non-recursive](#recursive-vs-non-recursive-gateways) and the content is known
+to not be available.
 
 ## Response Headers
 
@@ -641,9 +677,13 @@ Optional, present in certain response types:
 
 ### `Retry-After` (response header)
 
-Gateway returns this header with error responses such as [`429 Too Many Requests`](#429-too-many-requests) or [`504 Gateway Timeout`](#504-gateway-timeout).
+Gateway SHOULD return this header with error responses such as [`429 Too Many Requests`](#429-too-many-requests), [`504 Gateway Timeout`](#504-gateway-timeout) or `503` (server maintainance).
 
-The "Retry-After" header indicates how long the user agent ought to wait before making a follow-up request.
+The "Retry-After" header indicates how long the user agent ought to wait before making a follow-up request. It uses the following syntax:
+
+```
+Retry-After: <delay-seconds>
+```
 
 See Section 10.2.3 of :cite[rfc9110].
 
@@ -835,6 +875,21 @@ The usual optimizations involve:
   - The downside of this approach is that it will always be slower than
     skipping child block resolution.
 
+## Recursive vs non-recursive gateways
+
+A *recursive Gateway* is a gateway which generally attempts to fetch content
+from a third party it does not control by triggering lookups and retrievals. A
+recursive Gateway may not know in advance whether it can obtain and return a
+piece of content as the availability of it is out of its control. It may also
+suggest that clients retry failed requests later via `502` and `504` responses
+status codes.
+
+A *non-recursive Gateway* is gateway which accesses a known content-set and,
+under normal operation conditions, knows with certainty whether content
+requested can be obtained or not. Non-recursive gateways SHOULD prevent
+unnecessary retries from clients when the content is known to be unavailable
+by returning `404`.
+
 [dag-pb-format]: https://ipld.io/specs/codecs/dag-pb/spec/#logical-format
 [dag-json]: https://ipld.io/specs/codecs/dag-json/spec/
 [dag-cbor]: https://ipld.io/specs/codecs/dag-cbor/spec/

src/http-gateways/trustless-gateway.md

@@ -4,15 +4,21 @@ description: >
   The minimal subset of HTTP Gateway response types facilitates data retrieval
   via CID and ensures integrity verification, all while eliminating the need to
   trust the gateway itself.
-date: 2024-04-17
+date: 2025-03-06
 maturity: reliable
 editors:
   - name: Marcin Rataj
     github: lidel
-    url: https://lidel.org/
+    affiliation:
+      name: Shipyard
+      url: https://ipshipyard.com
   - name: Henrique Dias
     github: hacdias
-    url: https://hacdias.com/
+  - name: Héctor Sanjuán
+    github: hsanjuan
+    affiliation:
+      name: Shipyard
+      url: https://ipshipyard.com
 xref:
   - url
   - path-gateway
@@ -47,10 +53,14 @@ Optional `path` is permitted for requests that specify CAR format (`?format=car`
 
 For block requests (`?format=raw` or `Accept: application/vnd.ipld.raw`), only `GET /ipfs/{cid}[?{params}]` is supported.
 
+Client and Server implementations SHOULD include support for the [`GET` probe path](#dedicated-probe-paths).
+
 ## `HEAD /ipfs/{cid}[/{path}][?{params}]`
 
 Same as GET, but does not return any payload.
 
+Client and Server implementations SHOULD include support for the [`HEAD` probe path](#dedicated-probe-paths).
+
 ## `GET /ipns/{key}[?{params}]`
 
 Downloads data at specified IPNS Key. Verifiable :cite[ipns-record] can be requested via `?format=ipns-record` or `Accept: application/vnd.ipfs.ipns-record`.
@@ -233,7 +243,9 @@ In case both are present in the request, the value from the [`Accept`](#accept-r
 
 # HTTP Response
 
-Below MUST be implemented **in addition** to "HTTP Response" of :cite[path-gateway].
+Below MUST be implemented **in addition** to "HTTP Response" of
+:cite[path-gateway], with special attention to the "Response Status Codes" and
+the "Recursive vs non-recursive gateways" sections.
 
 ## Response Headers
 
@@ -438,3 +450,29 @@ returned as [application/vnd.ipfs.ipns-record](https://www.iana.org/assignments/
 A Client MUST confirm the record signature match `libp2p-key` from the requested IPNS Name.
 
 A Client MUST [perform additional record verification according to the IPNS specification](https://specs.ipfs.tech/ipns/ipns-record/#record-verification).
+
+# Appendix: Notes for implementers
+
+## Dedicated Probe Paths
+
+Trustless gateways SHOULD provide probing endpoints as described below.
+
+### `GET /ipfs/bafkqaaa`
+
+`bafkqaaa` is the identity empty CID. This endpoint can be used to probe that
+that the endpoint corresponds to a trustless gateway.
+
+For block requests (signaled by `?format=raw` and `Accept: application/vnd.ipld.raw`), when supported, it MUST return `200 OK`
+and an empty body.
+
+For CAR requests (signaled by `?format=car` and `Accept: application/vnd.ipld.car`), when supported, it MUST return `200 OK` and a valid CAR file with CAR Header `roots` set to `bafkqaaa`. Identity block MAY be skipped in the CAR Data section.
+
+This specific identity CID is special for probing. Other random
+identity CIDs MAY not be handled.
+
+### `HEAD /ipfs/bafkqaaa`
+
+`bafkqaaa` is the identity empty CID. If this endpoint is enabled, the gateway
+MUST support [`HEAD` requests](#head-ipfs-cid-path-params).
+
+The response is the same as [`GET`](#get-ipfs-bafkqaaa) but without body and all headers are optional.