Hardware
CPU: Intel i9
RAM: 64 GB
GPU: NVIDIA GeForce RTX 4070 Ti SUPER
VRAM: 16 GB GDDR6X
OS: Windows 11
CUDA: 13
NVIDIA Driver: 616
Problem
I have installed a fresh/new installation of AI Toolkit and created a new training job for Ideogram 4.
I used the default training settings, with only the following changes:
Cache text embeddings: Enabled
Low VRAM: Enabled
The training fails during the transformer loading/dequantization stage.
The error occurs while attempting to dequantize the transformer weights:
dequantizing transformer weights 211 fp8 -> torch.bfloat16
The training does not proceed beyond this point.
Important comparison
The same PC and GPU can run Ideogram 4 image generation in ComfyUI without any problems.
The GPU is therefore capable of running Ideogram 4 inference successfully. The problem appears specifically when AI Toolkit attempts to load/dequantize the transformer for training.
Expected behavior
AI Toolkit should successfully load the Ideogram 4 transformer and begin the training process using the Low VRAM configuration.
Actual behavior
AI Toolkit fails while dequantizing the transformer weights from FP8 to torch.bfloat16:
dequantizing transformer weights 211 fp8 - torch.bfloat16
Training never starts.
Steps to reproduce
Install a fresh/current version of AI Toolkit.
Create a new training job for Ideogram 4.
Leave the training configuration at its default settings.
Enable Cache Text Embeddings.
Enable Low VRAM.
Start the training.
AI Toolkit begins loading the transformer.
Training fails during FP8 → BF16 dequantization.
Additional information
This does not appear to be a general GPU performance or VRAM problem, because the same NVIDIA RTX 4070 Ti SUPER (16 GB VRAM) can successfully generate Ideogram 4 images in ComfyUI.
The failure seems to be specifically related to the way AI Toolkit handles the FP8 transformer weights and their conversion/dequantization to torch.bfloat16 during training.
I would appreciate confirmation whether the RTX 4070 Ti SUPER / 16 GB VRAM is supported for Ideogram 4 training and whether there is a known issue with FP8 transformer dequantization on this GPU.
Hardware
CPU: Intel i9
RAM: 64 GB
GPU: NVIDIA GeForce RTX 4070 Ti SUPER
VRAM: 16 GB GDDR6X
OS: Windows 11
CUDA: 13
NVIDIA Driver: 616
Problem
I have installed a fresh/new installation of AI Toolkit and created a new training job for Ideogram 4.
I used the default training settings, with only the following changes:
Cache text embeddings: Enabled
Low VRAM: Enabled
The training fails during the transformer loading/dequantization stage.
The error occurs while attempting to dequantize the transformer weights:
dequantizing transformer weights 211 fp8 -> torch.bfloat16
The training does not proceed beyond this point.
Important comparison
The same PC and GPU can run Ideogram 4 image generation in ComfyUI without any problems.
The GPU is therefore capable of running Ideogram 4 inference successfully. The problem appears specifically when AI Toolkit attempts to load/dequantize the transformer for training.
Expected behavior
AI Toolkit should successfully load the Ideogram 4 transformer and begin the training process using the Low VRAM configuration.
Actual behavior
AI Toolkit fails while dequantizing the transformer weights from FP8 to torch.bfloat16:
dequantizing transformer weights 211 fp8 - torch.bfloat16
Training never starts.
Steps to reproduce
Install a fresh/current version of AI Toolkit.
Create a new training job for Ideogram 4.
Leave the training configuration at its default settings.
Enable Cache Text Embeddings.
Enable Low VRAM.
Start the training.
AI Toolkit begins loading the transformer.
Training fails during FP8 → BF16 dequantization.
Additional information
This does not appear to be a general GPU performance or VRAM problem, because the same NVIDIA RTX 4070 Ti SUPER (16 GB VRAM) can successfully generate Ideogram 4 images in ComfyUI.
The failure seems to be specifically related to the way AI Toolkit handles the FP8 transformer weights and their conversion/dequantization to torch.bfloat16 during training.
I would appreciate confirmation whether the RTX 4070 Ti SUPER / 16 GB VRAM is supported for Ideogram 4 training and whether there is a known issue with FP8 transformer dequantization on this GPU.