GPUDirect Storage prototype tutorial #3317

mikaylagawarecki · 2025-04-08T14:22:48Z

No description provided.

pytorch-bot · 2025-04-08T14:22:51Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/tutorials/3317

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure, 1 Unrelated Failure

As of commit a5f98f1 with merge base 63295e8 ():

NEW FAILURE - The following job has failed:

Build tutorials / pytorch_tutorial_build_worker (6, 15, linux.g5.4xlarge.nvidia.gpu) (gh)
ValueError: expected a tensor object

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

Check spelling / pyspelling (gh) (trunk failure)

This comment was automatically generated by Dr. CI and updates every 15 minutes.

albanD · 2025-04-08T14:31:01Z

.jenkins/validate_tutorials_built.py

@@ -31,6 +31,7 @@
    "prototype_source/vmap_recipe",
    "prototype_source/torchscript_freezing",
    "prototype_source/nestedtensor",
+    "prototype_source/gpu_direct_storage", # requires specific filesystem + GPUDirect Storage to be set up


Doesn't it run in compat mode with a random machine?

You need a specific filesystem

prototype_source/gpu_direct_storage.py

albanD · 2025-04-08T15:06:23Z

prototype_source/gpu_direct_storage.py

+# The loading flow is the inverse, we can ``torch.load`` under the ``torch.serialization.skip_data`` context
+# manager to load everything except the storage bytes. This means that any tensors in the checkpoint will be
+# created but their storages will be empty (i.e. the tensors will be created via ``torch.empty``). If the
+# tensors to be loaded to are persistent, one can use the ``torch.cuda.gds.gds_register_buffer`` API to register


The register API is not used here?

albanD · 2025-04-08T15:06:40Z

prototype_source/gpu_direct_storage.py

+    f.load_storage(v.untyped_storage(), offset)
+    assert torch.equal(v, sd[k])
+
+del f


Similar synchronization question as above

I don't think synchronization is needed after the call as cuFileRead/Write are blocking operations that block until IO is complete https://docs.nvidia.com/gpudirect-storage/api-reference-guide/index.html#cufileread. You might need to synchronize before these ops (rather than after) though

albanD · 2025-04-08T15:07:04Z

prototype_source/gpu_direct_storage.py

+# We also provide ``torch.cuda.gds.gds_register_buffer`` to register the
+# tensor storages as GPUDirect Storage buffers. See `here
+# <https://docs.nvidia.com/gpudirect-storage/best-practices-guide/index.html#cufile-bufregister-fileread-filewrite>`_
+# for when one should do this.


Not sure what this means?
We should either mention when you would want to do that. Or skip this section.

mikaylagawarecki · 2025-04-08T15:27:13Z

prototype_source/gpu_direct_storage.py

+# If you are continuously saving the same state dictionary during training, you
+# would only need to obtain the offsets once and the same offsets can be re-used. Similarly if tensor is going to
+# be loaded to repeatedly one can use the ``torch.cuda.gds.gds_register_buffer`` which wraps
+# ``cuFileBufRegister`` to register the storages as gds buffers.


@albanD is this better?

svekars

Just a few minor nits

svekars · 2025-04-09T16:03:39Z

prototype_source/gpu_direct_storage.py

+GPUDirect Storage enabes a direct data path for direct memeory access transfers
+between GPU memory and storage, avoiding a bounce buffer through the CPU.
+
+In version ``2.7``, we introduced some prototype APIs to ``torch.cuda.gds`` that serve as thin wrappers around


Suggested change

In version ``2.7``, we introduced some prototype APIs to ``torch.cuda.gds`` that serve as thin wrappers around

In version **2.7**, we introduced a new prototype APIs to ``torch.cuda.gds`` that serve as thin wrappers around

svekars · 2025-04-09T16:03:54Z

prototype_source/gpu_direct_storage.py

+(prototype) Using GPUDirect Storage
+====================================
+
+GPUDirect Storage enabes a direct data path for direct memeory access transfers


Suggested change

GPUDirect Storage enabes a direct data path for direct memeory access transfers

GPUDirect Storage enables a direct data path for direct memory access transfers

svekars · 2025-04-09T16:08:39Z

prototype_source/gpu_direct_storage.py

+
+In version ``2.7``, we introduced some prototype APIs to ``torch.cuda.gds`` that serve as thin wrappers around
+the `cuFile APIs <https://docs.nvidia.com/gpudirect-storage/api-reference-guide/index.html#cufile-io-api>`_
+that can be used with ``torch.Tensor``.


wondering if we can continue the phrase to show the value proposition. For example (it's just an example):

Suggested change

that can be used with ``torch.Tensor``.

that can be used with ``torch.Tensor`` to achieve improved I/O performance.

svekars · 2025-04-09T16:09:57Z

prototype_source/gpu_direct_storage.py

+################################################################################
+# Using GPUDirect Storage with ``torch.save`` and ``torch.load``
+# =============================================================
+# GPUDirect Storage requires a storage alignment of 4KB. One can toggle this using


Suggested change

# GPUDirect Storage requires a storage alignment of 4KB. One can toggle this using

# GPUDirect Storage requires a storage alignment of 4KB. You can toggle this by using

svekars · 2025-04-09T16:10:15Z

prototype_source/gpu_direct_storage.py

+# Using GPUDirect Storage with ``torch.save`` and ``torch.load``
+# =============================================================
+# GPUDirect Storage requires a storage alignment of 4KB. One can toggle this using
+# ``torch.utils.serialization.config.save.storage_alignment`` to toggle this


Suggested change

# ``torch.utils.serialization.config.save.storage_alignment`` to toggle this

# ``torch.utils.serialization.config.save.storage_alignment``:

svekars · 2025-04-09T16:12:58Z

prototype_source/gpu_direct_storage.py

+# If you are continuously saving the same state dictionary during training, you
+# would only need to obtain the offsets once and the same offsets can be re-used. Similarly if tensor is going to
+# be saved or loaded to repeatedly one can use the ``torch.cuda.gds.gds_register_buffer`` which wraps
+# ``cuFileBufRegister`` to register the storages as gds buffers.


Suggested change

# ``cuFileBufRegister`` to register the storages as gds buffers.

# ``cuFileBufRegister`` to register the storages as GDS buffers.

svekars · 2025-04-09T16:14:18Z

prototype_source/gpu_direct_storage.py

+    assert torch.equal(v, sd[k])
+
+################################################################################
+# The loading flow is the inverse, we can ``torch.load`` under the ``torch.serialization.skip_data`` context


Suggested change

# The loading flow is the inverse, we can ``torch.load`` under the ``torch.serialization.skip_data`` context

# The loading flow is the inverse: you can use ``torch.load`` with the ``torch.serialization.skip_data`` context

svekars · 2025-04-09T16:14:36Z

prototype_source/gpu_direct_storage.py

+################################################################################
+# The loading flow is the inverse, we can ``torch.load`` under the ``torch.serialization.skip_data`` context
+# manager to load everything except the storage bytes. This means that any tensors in the checkpoint will be
+# created but their storages will be empty (i.e. the tensors will be created via ``torch.empty``).


Suggested change

# created but their storages will be empty (i.e. the tensors will be created via ``torch.empty``).

# created but their storages will be empty (for example, the tensors will be created via ``torch.empty``).

svekars · 2025-04-09T16:15:00Z

prototype_source/gpu_direct_storage.py

+
+del f
+
+# Summary


Suggested change

# Summary

# Conclusion

svekars · 2025-04-09T16:16:21Z

prototype_source/gpu_direct_storage.py

+# in conjunction with ``torch.save`` and ``torch.load`` on local filesystem. Do
+# file in issue in the PyTorch GitHub repo if you have any feedback.


Suggested change

# in conjunction with ``torch.save`` and ``torch.load`` on local filesystem. Do

# file in issue in the PyTorch GitHub repo if you have any feedback.

# in conjunction with ``torch.save`` and ``torch.load`` on local filesystem.

# Please file an issue in the PyTorch GitHub repo if you have any feedback.

malfet · 2025-04-10T18:50:57Z

prototype_source/gpu_direct_storage.py

@@ -0,0 +1,126 @@
+"""
+(prototype) Using GPUDirect Storage


I know next to nothing about SEO, but it feels like one can discover this tutorial only if user knows a very specific technical term. Shouldn't it be something like "Accelerating data loading with GPU Direct Storage"?

GPUDirect Storage prototype tutorial

255da51

facebook-github-bot added the cla signed label Apr 8, 2025

mikaylagawarecki requested a review from albanD April 8, 2025 14:24

mikaylagawarecki marked this pull request as draft April 8, 2025 14:24

albanD reviewed Apr 8, 2025

View reviewed changes

address comments

ba7f4e8

mikaylagawarecki commented Apr 8, 2025

View reviewed changes

one more fix

a5f98f1

mikaylagawarecki added the 2.7 label Apr 9, 2025

svekars reviewed Apr 9, 2025

View reviewed changes

malfet reviewed Apr 10, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GPUDirect Storage prototype tutorial #3317

GPUDirect Storage prototype tutorial #3317

mikaylagawarecki commented Apr 8, 2025

pytorch-bot bot commented Apr 8, 2025 •

edited

Loading

albanD Apr 8, 2025

mikaylagawarecki Apr 8, 2025

albanD Apr 8, 2025

albanD Apr 8, 2025

mikaylagawarecki Apr 8, 2025 •

edited

Loading

albanD Apr 8, 2025

mikaylagawarecki Apr 8, 2025

svekars left a comment •

edited

Loading

svekars Apr 9, 2025

svekars Apr 9, 2025

svekars Apr 9, 2025

svekars Apr 9, 2025

svekars Apr 9, 2025

svekars Apr 9, 2025

svekars Apr 9, 2025

svekars Apr 9, 2025

svekars Apr 9, 2025

svekars Apr 9, 2025

malfet Apr 10, 2025

	In version ``2.7``, we introduced some prototype APIs to ``torch.cuda.gds`` that serve as thin wrappers around
	In version 2.7, we introduced a new prototype APIs to ``torch.cuda.gds`` that serve as thin wrappers around

	GPUDirect Storage enabes a direct data path for direct memeory access transfers
	GPUDirect Storage enables a direct data path for direct memory access transfers

	that can be used with ``torch.Tensor``.
	that can be used with ``torch.Tensor`` to achieve improved I/O performance.

	# GPUDirect Storage requires a storage alignment of 4KB. One can toggle this using
	# GPUDirect Storage requires a storage alignment of 4KB. You can toggle this by using

	# ``torch.utils.serialization.config.save.storage_alignment`` to toggle this
	# ``torch.utils.serialization.config.save.storage_alignment``:

	# ``cuFileBufRegister`` to register the storages as gds buffers.
	# ``cuFileBufRegister`` to register the storages as GDS buffers.

	# The loading flow is the inverse, we can ``torch.load`` under the ``torch.serialization.skip_data`` context
	# The loading flow is the inverse: you can use ``torch.load`` with the ``torch.serialization.skip_data`` context

	# created but their storages will be empty (i.e. the tensors will be created via ``torch.empty``).
	# created but their storages will be empty (for example, the tensors will be created via ``torch.empty``).

		# in conjunction with ``torch.save`` and ``torch.load`` on local filesystem. Do
		# file in issue in the PyTorch GitHub repo if you have any feedback.


		del f

		# Summary

GPUDirect Storage prototype tutorial #3317

Are you sure you want to change the base?

GPUDirect Storage prototype tutorial #3317

Conversation

mikaylagawarecki commented Apr 8, 2025

pytorch-bot bot commented Apr 8, 2025 • edited Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/tutorials/3317

❌ 1 New Failure, 1 Unrelated Failure

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mikaylagawarecki Apr 8, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

svekars left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pytorch-bot bot commented Apr 8, 2025 •

edited

Loading

mikaylagawarecki Apr 8, 2025 •

edited

Loading

svekars left a comment •

edited

Loading