💡 [REQUEST] Add the latest PyTorch features into basics Autograd tutorial

[svekars]

🚀 Describe the improvement or the new tutorial

The Autograd tutorial does a good job of covering the basic graph creation and then running .backward(). However, there are new features from the latest PyTorch that can be highlighted in the tutorial.

cc: @soulitzer

Existing tutorials on this topic

https://pytorch.org/tutorials/beginner/basics/autogradqs_tutorial

Additional context

No response

cc @subramen @albanD @jbschlosser

[soulitzer]

The Autograd tutorial does a good job of covering the basic graph creation and then running .backward().

I agree the overall the tutorial is in a good place as the first things someone reads as they are starting out (e.g. how to do basic backprop on a module with .backward().

There are definitely a lot of new features not covered here, although many also require some more concepts to be introduced (e.g. covering activation memory could lead to us introducing activation checkpointing / saved tensor hooks). I think we can either do that by repositioning this tutorial to also cover intermediate concepts OR having a separate "intermediate autograd" tutorial.

If we continue to keep this as a brief "starting out" only tutorial, we should be careful to have a balance between providing enough information for the user to read more, but also keep it approachable, e.g., briefly mention/link the newer features related to each section in the form of optional notes.

I think some improvements for now we could make are:

• disabling gradient tracking should link to the autograd mechanics doc that goes into disabling gradient tracking more in depth

• for the computational graph section, we can introduce some more tools to help visualize that better:
  TORCH_LOGS="+autograd", torchviz