🐞 <Training Patchcore model on GPU takes twice longer than CPU>

[LeMinhNgan]

Describe the bug

I am training a dataset of about 900 images on both CPU and GPU and find it traing very slow on my RTX5090 that is not logical.

• CPU: 1.33s/it
• RTX5090: 2.31s/it
  Please find the attached screenshot for detail

Dataset

N/A

Model

PatchCore

Steps to reproduce the behavior

datamodule = Folder(root= cfg.root, name= cfg.name, normal_dir= cfg.train_normal_dir, abnormal_dir= cfg.val_abnormal_dir, normal_test_dir= cfg.val_normal_dir, 
                        train_batch_size= cfg.batch_size, eval_batch_size= cfg.batch_size, num_workers=0) 
    
evaluator = Evaluator(val_metrics = [AUPR(fields=["pred_score", "gt_label"]), AUROC(fields=["pred_score", "gt_label"]), F1Max(fields=["pred_score", "gt_label"]), F1Score(fields=["pred_score", "gt_label"])])

pre_processor= PreProcessor(transform= Compose([ToImage(), ToDtype(dtype= torch.float32, scale=True), RGB(), Resize(size= cfg.img_h_w, interpolation= InterpolationMode.BICUBIC, antialias=True), Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])]))

model_paras= {'evaluator': evaluator, 'pre_processor': pre_processor}

#MODE1: CPU
engine = Engine(default_root_dir= cfg.root, accelerator= "cpu" ,callbacks= [PrintValidationResults()])  

#MODE2: RTX5090
engine = Engine(default_root_dir= cfg.root, accelerator= "gpu" ,callbacks= [PrintValidationResults()]) 

engine.fit(datamodule=datamodule, model=model)    

--> output:
MODE1: CPU. training speed is 1.33s/it
MODE1: RTX5090. training speed is 2.31s/it

OS information

OS information:

• OS: windows11
• Python version: 3.10
• Anomalib version: 2.4.2
• PyTorch version: 2.7.0
• CUDA/cuDNN version: 12.8
• GPU models and configuration: RTX5090

Expected behavior

Please check the code of trainer on GPU

Screenshots

Pip/GitHub

pip

What version/branch did you use?

2.4.2

Configuration YAML

n/a

Logs

n/a

Code of Conduct

• I agree to follow this project's Code of Conduct

[github-actions]

Thanks for submitting this issue! It has been added to our triage queue. A maintainer will review it shortly.

[rajeshgangireddy]

Hi @LeMinhNgan
It looks like #3651 fixes this. Can you check now if this fixed and the speeds are as expected?