Number of slashes in the osdf protocol

[aowen-uwmad]

Proposal

Anytime the osdf:///<address> sequence is expected or looked for, Pelican should accept the osdf://<address> case.

Background

Pelican has a dedicated osdf protocol for handling transfers to/from the OSDF.
The osdf protocol is a shorthand for calling the pelican protocol using the OSDF osg-htc.org federation.

For accessing the OSDF using the pelican protocol, the URL is

pelican://osg-htc.org/<namespace>/<objectname>

In theory, the osdf protocol URL would be

osdf://osg-htc.org/<namespace>/<objectname>

but since the osdf plugin is only ever used for the OSDF, the federation URL is superfluous and can be dropped. So in practice, the federation URL is omitted:

osdf:///<namespace>/<objectname>

User perspective

OSPool users (and facilitators) are frequently confused by the three consecutive slashes when using the osdf protocol via the HTCondor file transfer plugin.
This seems to be an expectations issue, because in general people are used to two slashes in HTTP addresses.
And indeed, all other HTCondor file transfer plugins use two slashes (plugin://<address>), so even experts in HTCondor can be confused by this.

That is, most users intuition guides them to do:

osdf://<namespace>/<objectname>

Implementation

Supposedly this has been done for the HTCondor file transfer plugin itself.
But discussion with the team brought up that development of other clients/plugins may not have been conscious of this choice.

Thus:

1. A review of the current places where the osdf:///<address> syntax is expected should be done, to check if it also accepts osdf://<address>.
2. Moving forward, development always pairs osdf:///<address> and osdf://<address>.

[turetske]

I believe this has already been fixed in #1589

[turetske]

Re-opening this because it isn't currently implemented in PelicanFS and I realized this was a discussion for all clients

[aowen-uwmad]

I am no longer interested in pushing for a programmatic solution to this discussion. The reason is that I noticed when I use my browser to open a PDF file, it has file:///path/to/my.pdf in the address bar. This emphasizes a point that @turetske made in a in-person conversation that the 3 slashes is inherited from other web protocols, but I hadn't appreciated it until I saw my browser doing it.

I think the resolution to this discussion moving forward should now be focused on how we communicate to and educate users about the Pelican & OSDF URLs. Having a concrete example that exists outside of Pelican helps provide a conceptual handhold for new users to understand that this isn't just an arbitrary decision by the dev team.

We'll have to work on how exactly we word this framing in the documentation and trainings, and until we make such improvements I anticipate further user confusion about the 2 vs. 3 slashes.

[aowen-uwmad]

To round things out, here's my draft of how to write this up.
Perhaps this can be a sub-page in a "FAQ" section of the manual.

Why are there three slashes in a row in the OSDF URL?

We frequently get the question, "How many slashes should there be in the OSDF URL, two (osdf://address/to/object) or three (osdf:///address/to/object)?"
The answer to the question is "three".

This often confuses users, so it's worth explaining the origin of the OSDF URL.

Short version

A little background

If you open a file in your web browser, you may have noticed that the full address in the address bar looks something like file:///path/to/file.
On the other hand, when you are viewing a webpage from the internet, the full address looks something like https://pelicanplatform.org/about.
When using the file protocol, there's only one computer that your browser ever needs to talk to: yours.
When using an internet based protocol like https, there's many possible computers on the internet that your browser may talk to.

Similarly, when using the osdf protocol to transfer an object using Pelican, there's only one Pelican Federation that Pelican ever needs to talk to: the Open Science Data Federation.
On the other hand, when using the pelican protocol, there's many possible Pelican Federations that Pelican may talk to.

The short explanation

There are three slashes when using the osdf protocol with Pelican for the same reason there are three slashes when using the file protocol with your browser.

Long version

We've answered the original question above, but by introducing a new question: why does the file protocol have three slashes in its address? Again, consider the analogy to the web browser.

When using the internet, you need to provide the browser with the URL of the page you want to view.
The URL has the following components:

1. The communication protocol, e.g., https://,
2. The hostname of the website, e.g., mywebdomain.com,
3. The path of the file on the website, e.g., /examples/my-webpage.html.

Putting all three components together, the address becomes https://mywebdomain.coim/examples/my-webpage.html.

When loading the file from your computer, you still need to provide a URL for the file to view.
Following the syntax described above, the components of this URL would be

1. The communication protocol, file://,
2. The hostname of the "website", localhost,
3. The path of the file on the "website", e.g., /examples/my-webpage.html.

But in this scenario, the hostname (localhost) is redundant because when you use the file:// protocol, the browser will only ever talk to the localhost.
So instead of combining all 3 components into a single address, only parts 1 and 3 are combined, and part 2 is left out.
Therefore the address becomes file:///examples/my-webpage.html.
The three slashes in a row occurs because the hostname localhost was not included in the address.

If you know the path to a file on your computer that can be viewed using a web browser, you can enter the address including the localhost part of the URL, and it should still open!

Pelican and the OSDF operates in a similar fashion. In general, if you want to get an object via the Pelican Client, you need to provide a Pelican URL:

1. The communication protocol, pelican://,
2. The "hostname" of the Pelican Federation, e.g., myfederation.org,
3. The "path" of the object in the Federation, e.g., /my-namespace-prefix/my-object.

Combining the components yields the Pelican URL pelican://myfederation.org/my-namespace-prefix/my-object.

To get an object from the OSDF via the Pelican Client, the components of the URL would be

1. The communication protocol, osdf://,
2. The "hostname" of the OSDF, osg-htc.org,
3. The "path" of the object in the Federation, e.g., /my-osdf-prefix/my-object.

As with the file protocol above, specifying both the osdf communication protocol and the osg-htc.org "hostname" of the federation is redundant.
Therefore only parts 1 and 3 are combined, with part 2 being left out, leading to the address osdf:///my-osdf-prefix/my-object.

Pseudo-code version

Let's define a function that excludes the "hostname" part of the URL if it is not required, in this case, when the URL is using either the file:// or osdf:// protocols:

>> def print_address(protocol, hostname, path):
..     if protocol in ["file://", "osdf://"]:
..         address = protocol + path
..     else:
..         address = protocol + hostname + path
..     
..     print(address)
..

We can now use this function to print the URLs for the examples described above:

>> print_address("https://", "mywebdomain.com", "/examples/my-webpage.html")
https://mywebdomain.com/examples/my-webpage.html

>> print_address("file://", "localhost", "/examples/my-webpage.html")
file:///examples/my-webpage.html

>> print_address("pelican://", "myfederation.org", "/my-namespace-prefix/my-object")
pelican://myfederation.org/my-namespace-prefix/my-object

>> print_address("osdf://", "osg-htc.org", "/my-osdf-prefix/my-object")
osdf:///my-osdf-prefix/my-object

[aowen-uwmad]

A version of the help text is being included in #1817 .