obj: Create an S3 client common utility library

[jgoldsch12]

Move obj_s3_client to a common utility library providing AWS S3 client support for all object based plugins.

What?

This PR moves the s3 client library code to a common utility location.

Why?

Moving to a common utility location enables the creation of additional object plugins which depend on this s3 client library

How?

Moving files, editing references and test code

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[github-actions]

👋 Hi jgoldsch12! Thank you for contributing to ai-dynamo/nixl.

Your PR reviewers will review your contribution then trigger the CI to test your changes.

🚀

[aranadive]

@jgoldsch12 please see PR #1127. Im already doing something similar when introducing CRT support and there are more things that are common between S3/S3CRT clients.

[jgoldsch12]

@jgoldsch12 please see PR #1127. Im already doing something similar when introducing CRT support and there are more things that are common between S3/S3CRT clients.

Hi @aranadive, thanks for bringing that to my attention. I'm working on a new object based plugin which is an alternative to OBJ. I moved the S3 client code around so that there can be some code sharing between plugins. This would also enables changes like yours to introduce new SDKs in a common manner.

[aranadive]

@jgoldsch12 please see PR #1127. Im already doing something similar when introducing CRT support and there are more things that are common between S3/S3CRT clients.

Hi @aranadive, thanks for bringing that to my attention. I'm working on a new object based plugin which is an alternative to OBJ. I moved the S3 client code around so that there can be some code sharing between plugins. This would also enables changes like yours to introduce new SDKs in a common manner.

Can you add more details around what alternative you are trying to add? Im just trying to understand the overall flow and experience from NIXL side so we can organize better.

[jgoldsch12]

@jgoldsch12 please see PR #1127. Im already doing something similar when introducing CRT support and there are more things that are common between S3/S3CRT clients.

Hi @aranadive, thanks for bringing that to my attention. I'm working on a new object based plugin which is an alternative to OBJ. I moved the S3 client code around so that there can be some code sharing between plugins. This would also enables changes like yours to introduce new SDKs in a common manner.

Can you add more details around what alternative you are trying to add? Im just trying to understand the overall flow and experience from NIXL side so we can organize better.

Sure. I'm preparing a plugin which is optimized to work with Dell's ObjectScale object storage using RDMA data transfers for S3 compatible storage. The plugin introduces support for CUDA memory and GPU Direct Storage transfer for objects. It utilizes the S3 client library interface (reorganized by this PR), but provides put/get implementations to utilize RDMA for data transfer, instead of HTTP streams.