Progress Update

[Barabazs]

Progress update April 16, 2022

A lot of time has been dedicated to research and writing tests for this project.

It didn't take long to realise that it would not be sufficient to write the implementation while following the IPFS documentation. The docs are interesting for endusers and developers that interact with the protocol on a "shallow" level.

E.g. what multibase encodings are supported for a CIDv1?

Let's check the docs https://docs.ipfs.io/concepts/content-addressing/#identifier-formats :

• CIDv0: appears to be base 58-encoded

• CIDv1:

  • There's a link to the CID specs
  • A link to the decoding algorithm
  • case-insensitive base32 is the default encoding
  • "but use of the shorter base36 is encouraged for IPNS names to ensure same text representation on subdomains."
  • "A CID can be represented as both text and as a stream of bytes."
  • "Sometimes, a hexadecimal (opens new window) representation of raw bytes is prefered for debug purposes."
  • Reading further we can see that the following hex encodings should be supported:
    • base16
    • base32
    • base36

  From the CID specs and the decoding algorithm we can infer that a CIDv0 is always encoded with base58btc and no explicit encoding is mentionned for CIDv1.

With that information in mind, I started working with writing tests for working with these encodings. This later turned out to be insufficient and much more encodings are supported for certain cases. How do we solve this conundrum? A (probably) naive approach would be brute-force converting a CIDv1 to each encoding listed in the multibase docs and testing it against the CID Inspector.

In v0.0.5 we can currently validate CIDv0, CIDv1, URL with IPFS path and URL with an IPFS subdomain.

Currently supported

• CID
  • v0
  • v1
• URL
  • IPFS
  • IPNS
• Subdomain
  • IPFS
  • IPNS
• Path
• ...

---

Latest release at time of writing is available on:

• Github v0.0.5
• PyPi v0.0.5

Cross-post from IPFS/community

[Barabazs]

Progress update April 24, 2022

I was finally able to dedicate some time to the project and implemented support for IPFS paths.

In v0.0.6 we can currently validate CIDv0, CIDv1, URL with IPFS path, URL with an IPFS subdomain and IPFS path.

Currently supported

• CID
  • v0
  • v1
• URL
  • IPFS
  • IPNS
• Subdomain
  • IPFS
  • IPNS
• Path
  • IPFS
• ...

---

Latest release at time of writing is available on:

• Github v0.0.6
• PyPi v0.0.6

[Barabazs]

Progress update May 9, 2022

py-is_ipfs has now reached version number 0.1.0 and can be considered as entering the beta phase.
For this release I carefully selected testing data and converted these to the most used encodings.

py-is_ipfs can now validate both IPFS and IPNS for the most common use cases. see below

As mentionned before, the docs are not always clear and concise, which resulted in me compiling a table of accepted encodings.

encoding	CIDv0	CIDv1	IPFS URL subdomain gateway	IPFS URL path gateway	IPNS URL subdomain gateway	IPNS URL path gateway	IPFS content path	IPNS content path
base2	❌	✔️	❌	✔️	❌	✔️	✔️	✔️
base8	❌	✔️	❌	❌	❌	❌	❌	❌
base10	❌	✔️	❌	❌	❌	❌	❌	❌
base16	❌	✔️	❌	✔️	❌	✔️	✔️	✔️
base32	❌	✔️	✔️	✔️	❌	✔️	✔️	✔️
base32hex	❌	✔️	✔️	✔️	❌	✔️	✔️	✔️
base36	❌	✔️	✔️	✔️	✔️	✔️	✔️	✔️
base36upper	❌	✔️	✔️1	✔️	✔️1	✔️	✔️	✔️
base58flickr	❌	✔️	❌	✔️	❌	✔️	✔️	✔️
base58btc	✔️	✔️	❌	✔️	❌	✔️	✔️	✔️
base64	❌	✔️	❌	❌	❌	❌	❌	❌
base64url	❌	✔️	❌	✔️	❌	✔️	✔️	✔️

While py-is_ipfs can now correctly validate all IPFS/IPNS addressing, it should still be considered as a work in progress. The main logic (Validator(input).is_ipfs()) will not change, but the underlying methods are subject to change.

Currently supported

• CID
  • v0
  • v1
• URL
  • Path Gateway
    • IPFS
    • IPNS
  • Subdomain Gateway
    • IPFS
    • IPNS
• Path
  • IPFS
  • IPNS
• ...

---

Latest release at time of writing is available on:

• Github v0.1.0
• PyPi v0.1.0

Footnotes

1. browsers convert uppercase to lowercase for subdomains, resulting in same result as base36 ↩ ↩²

[Barabazs]

Progress update May 18, 2022

I am currently working on implementing native URLs, as this is often used in the web3-space. Unfortunately I've been swamped with other work and haven't made much progress. Or at least, not enough to share here.

If all goes well, a new update should be released soon ™️.

[Barabazs]

Progress update May 26, 2022

I mentioned a new release in the previous progress update and actually pushed 2 new releases. Unfortunately the first release contained wrong validation logic for native URLs (ipfs:// and ipns://)...

As noted before, the IPFS docs and specs are all over the place and there never seems to be one single source of truth that we can fully trust. I started by following the information from the IPFS docs in the section about "Native URLs".

According to the specifications the native address format is the same as a subdomain gateway HTTP URL, but the protocol is replaced by ipfs or ipns and the gateway host + port are replaced with a CID (or a dnslink-name for IPNS). They even include a few clear examples.

They also mention that we should avoid case-sensitive CID in IPFS://, because some user agents will convert the CID to lowercase for URL-like addresses. They also state that, "Use of CIDv1 in Base32 or Base36 is advised." And further down the page, they also refer to "the best and most up-to-date source of truth about IPFS addressing can be found in the IPFS in-web-browsers repo".

The information is largely the same as in the docs, but it goes a bit deeper on a technical level. Here I missed/misinterpreted the most crucial part where they briefly mention normalization of the cid:

ipfs://{cidv1base32}
ipfs://{cidv0} → redirect → ipfs://{cidv1base32} # CIDv0 is case-sensitive Base58, does not work as Origin authority

ipns://{libp2p-key-in-cidv1base36}
ipns://{libp2p-key-in-base58} → redirect → ipns://{libp2p-key-in-cidv1}  # Base58, does not work as Origin authority

ipns://{dnslink}
ipfs://{dnslink} → redirect → ipns://{dnslink} # just to improve UX :-)

I interpreted this as: use the same validation logic as for the subdomain gateways and just make sure the protocol is accounted for. CIDv0 can't be used because it has to be normalised. So I wrote some code and tests, everything looked correct (based on the documentation) and I pushed a new release.

The next day I tried to validate NFT metadata URLs from OpenSea and noticed they were all formatted as ipfs://CIDv0. I then tried to go to these URLs with Brave (browser) and, to my surprise, the image appeared correctly. What then followed was a deep dive in the IPFS repos, going through issues and commits, from one link to the next until I could finally piece together all the information I needed to implement a correct validation.

TL;DR

Basically a native IPFS/IPNS URL follows the same logic as the path gateway pattern (not subdomain as they mention in the docs). The subdomain pattern is recommended, but that does not invalidate the path gateway pattern.

The latest release (0.3.0 at the time of writing) includes the correct validation logic and a few tweaks. It also includes a missing valid encoding for the IPFS subdomain gateway pattern.

---

Currently supported

• CID
  • v0
  • v1
• URL
  • Path Gateway
    • IPFS
    • IPNS
  • Subdomain Gateway
    • IPFS
    • IPNS
• Path
  • IPFS
  • IPNS
• Native URL
  • IPFS
  • IPNS
• ...

---

Latest release at time of writing is available on:

• Github v0.3.0
• PyPi v0.3.0

[Barabazs]

Progress update June 14, 2022

As most features are already implemented, there is not much progress to report. I added a very basic CLI to the latest release. Although I suspect this library is more useful when integrated as part of a larger framework.

The next step is writing documentation and adding a few examples.

Currently supported

• CID
  • v0
  • v1
• URL
  • Path Gateway
    • IPFS
    • IPNS
  • Subdomain Gateway
    • IPFS
    • IPNS
• Path
  • IPFS
  • IPNS
• Native URL
  • IPFS
  • IPNS
• ...

---

Latest release at time of writing is available on:

• Github v0.4.0
• PyPi v0.4.0

[Barabazs]

Progress update June 24, 2022

py-is_ipfs is now fully supports typing. Until now it implicitly supported typing, but that is obviously not enough to explicitly support typing.
While finishing up with my own package I fell into a rabbit hole and also wrote stubs for the two libraries this package relies on. I plan on submitting a PR to both projects later on.

Currently supported

• CID
  • v0
  • v1
• URL
  • Path Gateway
    • IPFS
    • IPNS
  • Subdomain Gateway
    • IPFS
    • IPNS
• Path
  • IPFS
  • IPNS
• Native URL
  • IPFS
  • IPNS
• ...

---

Latest release at time of writing is available on:

• Github v0.5.1
• PyPi v0.5.1