which GPU did you used and how much time and RAM consumed training process?

[gustawdaniel]

I have RTX 3060 Laptop with 6 GB and after step 35 374 / 2 000 000 I see.:

RuntimeError: CUDA out of memory. Tried to allocate 2.00 MiB (GPU 0; 5.70 GiB total capacity; 3.79 GiB already allocated;
2.12 MiB free; 3.90 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  
See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF