Test-only parameter for model finetuning

[hanshupe007]

I am running the command below to get the baseline before I start fine-tuning the model and was hoping the --test-only parameter is exactly doing that. The result is 0% accurate predictions: Validation loss: 26.6137 (Exact: 0.00% | Partial: 0.00%)

My test samples are just basic words, fine-tuning is slowly increasing the accuracy. So I am wondering what could be a reason for the poor result?

python references/recognition/train_pytorch.py crnn_vgg16_bn --train_path path/to/your/train_set --val_path path/to/your/val_set --epochs 5 --device 0 --pretrained --test-only

[felixdittrich92]

Hi @hanshupe007 👋,

crnn_vgg16_bn was the first model we trained (the vocab differs to all other models - we need to retrain it at some time)

So for crnn_vgg16_bn you need to pass --vocab=legacy_french all other models use the default french

[felixdittrich92]

(doctr-dev) felix@felix-Z790-AORUS-MASTER:~/Desktop/doctr$ USE_TORCH=1 python3 /home/felix/Desktop/doctr/references/recognition/train_pytorch.py parseq --train_path=/home/felix/Desktop/Other/text_recognition_validation --val_path=/home/fel
ix/Desktop/Other/text_recognition_validation --pretrained --test-only
Namespace(arch='parseq', train_path='/home/felix/Desktop/Other/text_recognition_validation', val_path='/home/felix/Desktop/Other/text_recognition_validation', train_samples=1000, val_samples=20, font='FreeMono.ttf,FreeSans.ttf,FreeSerif.ttf', min_chars=1, max_chars=12, name=None, epochs=10, batch_size=64, device=None, input_size=32, lr=0.001, weight_decay=0, workers=None, resume=None, vocab='french', test_only=True, freeze_backbone=False, show_samples=False, wb=False, push_to_hub=False, pretrained=True, sched='cosine', amp=False, find_lr=False, early_stop=False, early_stop_epochs=5, early_stop_delta=0.01)
Validation set loaded in 0.1673s (51494 samples in 805 batches)
Running evaluation
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 805/805 [00:26<00:00, 30.00it/s]
Validation loss: 0.272104 (Exact: 90.00% | Partial: 90.60%)

[felixdittrich92]

(doctr-dev) felix@felix-Z790-AORUS-MASTER:~/Desktop/doctr$ USE_TORCH=1 python3 /home/felix/Desktop/doctr/references/recognition/train_pytorch.py crnn_vgg16_bn --pretrained --test-only --vocab=legacy_french
Namespace(arch='crnn_vgg16_bn', train_path=None, val_path=None, train_samples=1000, val_samples=20, font='FreeMono.ttf,FreeSans.ttf,FreeSerif.ttf', min_chars=1, max_chars=12, name=None, epochs=10, batch_size=64, device=None, input_size=32, lr=0.001, weight_decay=0, workers=None, resume=None, vocab='legacy_french', test_only=True, freeze_backbone=False, show_samples=False, wb=False, push_to_hub=False, pretrained=True, sched='cosine', amp=False, find_lr=False, early_stop=False, early_stop_epochs=5, early_stop_delta=0.01)
Validation set loaded in 0.0706s (2460 samples in 39 batches)
Running evaluation
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 39/39 [00:01<00:00, 20.72it/s]
Validation loss: 2.9171 (Exact: 16.46% | Partial: 27.68%)

[hanshupe007]

Great, thanks!