[E2E] HF/Timm/Torchbench models got "eager_two_runs_differ"

[libohao1201]

🐛 Describe the bug

The following models got "eager_two_runs_differ"

		eager_two_runs_diff
LNL	HF
	Timm
	Torchbench	Super_SloMo (train_eager_fp32/amp_bf16) pytorch_CycleGAN_and_pix2pix (train_eager_fp32/bf16/amp_bf16)
BMG	HF
	Timm
	Torchbench	Super_SloMo (train_eager_fp32) pytorch_CycleGAN_and_pix2pix (train_eager_fp32/bf16)
ARC	HF	DistilBertForMaskedLM(train_fp16_eager)
	Timm	convnext_base (train_eager) jx_nest_base (train_eager) swin_base_patch4_window7_224 (train_eager) twins_pcpvt_base (train_eager) coat_lite_mini (train) convit_base mobilevit_s tnt_s_patch16_224
	Torchbench	hf_Reformer(train_eager) timm_regnet (train_eager_fp32)

Versions

Env:

stock pytorch	https://github.com/pytorch/pytorch/tree/90b7dcf2c5ee13b892701822f2abbc0e64f5584d pip install --pre torch==2.6.0.dev20241202+xpu torchvision==0.20.0.dev20241202+xpu torchaudio==2.5.0.dev20241202+xpu --index-url https://download.pytorch.org/whl/nightly/xpu
torch-xpu-ops	Commit: bf4bab1 Commit: 0f48ac0 (including #1187) - for UT
Driver	32.0.101.6314 32.0.101.6252(bmg)
Conda	python 3.10
transformer	243e186efbf7fb93328dd6b34927a4e8c8f24395

[mengfei25]

Super_SloMo got same fail on PVC for FP32 training

[jianyizh]

For Super_SloMo and pytorch_CycleGAN_and_pix2pix, pytorch did not set deterministic, see
https://github.com/pytorch/pytorch/blob/2a7e997b3f1805b077810d7fef87cabc4411eeea/benchmarks/dynamo/common.py#L3582

pytorch_CycleGAN_and_pix2pix is reasonable to fail on eager-two-run-diff because of atomic ops.

Super_SloMo can get pass if I use torch.use_deterministic_algorithms(True,warn_only=True) and #1370 to avoid atomic ops.