Enable all tests

[samanklesaria]

Now that deprecated functionality has been removed, we can remove the restrictions given when running the test suite during CI. This PR depends on the functionality in the branch pseudo_main.

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/audio/4061

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 8 New Failures

As of commit e45fe5e with merge base a645da6 ():

NEW FAILURES - The following jobs have failed:

• .github/workflows/bandit.yml (gh)
• .github/workflows/integration-test.yml (gh)
• .github/workflows/lint.yml (gh)
• .github/workflows/pr-labels.yml (gh)
• .github/workflows/unittest-macos-cpu.yml (gh)
• .github/workflows/unittest-windows-cpu.yml (gh)
• .github/workflows/unittest-windows-gpu.yml (gh)
• Build documentation / build / Build doc (gh)
  RuntimeError: Command docker exec -t 10fbf26ac27afe8fe9e52e9f167373427c9ef292f7306b2464f9b6f43410648e /exec failed with exit code 2

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[samanklesaria]

The torchscript consistency tests don't seem to work with manual autograd in python (torch.autograd.Function overloads). Pretty much all the torchscript consistency checks seem to have this issue. The only way forward is to either:
A) Investigate more whether it's possible to get torch.autograd.Function working with torchscript
B) Delete the torchscript consistency checks.
As torchscript is essentially deprecated anyway, I propose option (B)

[pearu]

It is possible to fix torchscript consistency tests by avoiding autograd functions under jit-scripting. For instance, rnnt_loss tests pass when applying the following patch:

diff --git a/src/torchaudio/functional/functional.py b/src/torchaudio/functional/functional.py
index 5d4284d5..a72c6091 100644
--- a/src/torchaudio/functional/functional.py
+++ b/src/torchaudio/functional/functional.py
@@ -1821,7 +1821,13 @@ def _rnnt_loss(
     if blank < 0:  # reinterpret blank index if blank < 0.
         blank = logits.shape[-1] + blank
 
-    costs = RnntLoss.apply(
+    if torch.jit.is_scripting():
+        # TorchScript does not support autograd functions
+        func = torch.ops.torchaudio.rnnt_loss_forward
+    else:
+        func = RnntLoss.apply
+
+    costs = func(
         logits,
         targets,
         logit_lengths,
@@ -1831,6 +1837,9 @@ def _rnnt_loss(
         fused_log_softmax
     )
 
+    if torch.jit.is_scripting():
+        costs = costs[0]
+
     if reduction == "mean":
         return costs.mean()
     elif reduction == "sum":