Finetuning recipes for text recognition

[ajkdrag]

Out of the box pretrained models perform well, but for my usecase, where we have bank check images, the recognition output is something that needs to be improved. Sometimes amazon is parsed as ama2on, other times it becomes totally weird.

An example with crnn_mobilenet_large:

CAEIMIGER/RMAFOR

I tried finetuning the models by this recipe but it performed worse:

• Use Google Cloud Vision (or Azure/Amazon cloud OCR models) to get output for 100 images.
• Finetune on this dataset.
• Tryout different learning rate etc.

Is there a guideline on how am I supposed to do this? Should I add in some synthetic data to the mix or some other datasets like FUNSD?

[felixdittrich92]

Hi @ajkdrag 👋
Have you already tried the other recognition models for example vitstr_small or parseq ?
In general i highly recommend (If not enough real data available) to generate synth data for training (for example with synthtiger) and validate on real samples (also if it's not so much).
If the license doesn't matter you can also include crops from FUNSD / Cord / SROIE sure :)

[ajkdrag]

My rationale behind not using synthtiger was that the pretraining would have already have been done on synthetic data and my dataset is all scanned bank documents which have sometimes mottled ink, scan lines etc. I haven't used synthtiger but i can try it.

[ajkdrag]

Any estimate on how many images are needed for fine-tuning?

[hanshupe007]

I have similar issues as you described, which model and fine-tuning approach did work best for you?