Create "pelican object remove" command

[aowen-uwmad]

Pelican Service:

• Client
• Plugin
• Registry
• Director
• Origin
• Cache
• Other (please give the detail)

Is your feature request related to a problem? Please describe.
An increasing number of researchers only have access to the data storage via the connected Origin.
Since changes to an object require a change to the object name, over time multiple similar copies of objects will accumulate in the storage while only the most recent copy is used.
Removing the old copies of the object is necessary to prevent unbounded growth in the storage.
Currently, the only way to do this is with direct access to the storage, which is not possible for users whose access is only via the Client.

Describe the solution you'd like
Implement a pelican object remove command.

• Requires write permissions & authentication (or at least, similar level)
• Prompts user to confirm removal (can be bypassed using --force flag)
• Includes somewhere a warning about not affecting cached copies, and proscription against reusing object name
• Recursive functionality

Additional context
In principal, a user could use pelican object put <0-byte object> <PelicanURL removal target> to "remove" the object contents from the storage. This is NOT intuitive to users. Furthermore, the storage would accumulate 0-byte objects and for a large enough number of files, the file metadata for said objects would be a non-trivial contribution to the storage usage.

[brianaydemir]

@aowen-uwmad See #1685 and #1816.

[matyasselmeci]

pelican object delete does seem to exist, but does not show up as a subcommand with pelican object --help.

% pelican object --help
...
Available Commands:
  copy        Copy a file to/from a Pelican federation
  get         Get a file from a Pelican federation
  ls          List objects in a namespace from a federation
  put         Send a file to a Pelican federation
  share       Generate a string for sharing access to a namespace.
Note the sharing is based on prefixes; all object names matching the prefix will be accessible
  stat        Stat objects in a namespace from a federation
  sync        Sync a directory to or from a Pelican federation
...

% pelican object delete --help
Delete an object or a collection

Usage:
  pelican object delete {object} [flags]

(v7.15.2)

[aowen-uwmad]

Obscuring functionality does not prevent users from discovering it, nor using it.
If we truly do not want users to invoke this command, then the code path should not exist.
I believe, however, that we do need a command like this for the users, otherwise the object store behind the origin can grow without bound.

Thus, the command (and its help text) should build in warnings and guard rails such that user has no choice but to be aware of the consequences of the action. For example,

$ pelican object delete <PelicanURL>

WARNING! Removing an object from the origin will NOT remove it from any caches.
         Furthermore, uploading an object with the same name can result in corrupted versions 
         at said caches.

Are you sure you want to proceed? (y/n)
> 

Prompting the user forces them to take a second and think about what they are doing. The warning makes sure that they have information to inform their decision.

[brianaydemir]

Furthermore, uploading an object with the same name can result in corrupted versions at said caches.

By this token, object put also needs a warning. And it raises a related question for what the HTCondor file transfer plugin should do.

---

I'd maybe also name this command pelican origin object-remove. The thing we're fundamentally interacting with and modifying is an origin, not the object as it exists in the federation (cf. caches still retaining a copy).

[matyasselmeci]

I wonder if it's possible for the origin to keep track of whether an object has been cached -- or at least whether something claiming to be a cache has downloaded an object. If something's never been cached, it should be safe to delete or overwrite...

[github-actions]

This issue is stale because it has been open for 60 days with no activity.

[aowen-uwmad]

I'd maybe also name this command pelican origin object-remove. The thing we're fundamentally interacting with and modifying is an origin, not the object as it exists in the federation (cf. caches still retaining a copy).

I second this suggestion.