Irregular OCR inference time for the same image in a loop (docTr vs. onnxTr)

[govindbagaria]

@felixdittrich92 Hello,

I’m experiencing irregular inference times when I repeatedly run OCR on the same image in a loop. The memory profile remains pretty constant, but the time taken to process each iteration varies significantly—even though I’m feeding the exact same image each time.

I’ve tried this with both onnxTr and docTr, each set up to disable multiprocessing and set a small primitive cache capacity. Below are my minimal reproducible scripts, plus screenshots of the resulting time profiles and memory usage.

onnxTr Code:
``
import os
os.environ["ONNXTR_MULTIPROCESSING_DISABLE"] = "TRUE"
os.environ["ONEDNN_PRIMITIVE_CACHE_CAPACITY"] = "1"
os.environ["ONNXTR_CACHE_DIR"] = "onxx_models"

import time
import tqdm
import matplotlib.pyplot as plt
from onnxtr.io import DocumentFile
from onnxtr.models import ocr_predictor

model = ocr_predictor(
det_arch="db_mobilenet_v3_large",
reco_arch="crnn_vgg16_bn",
assume_straight_pages=False,
straighten_pages=True
)
times = []

file_path = "img.jpg"

for i in tqdm.tqdm(range(0, 100)):
start_time = time.time()
doc = DocumentFile.from_images(file_path)
result = model(doc)
end_time = time.time()
inference_time = end_time - start_time
times.append(inference_time)

plt.figure(figsize=(8, 5))
plt.plot(range(0, 100), times, marker='o', linestyle='-', color='b', label='Inference Time')
plt.title('OCR Inference Time per Iteration (onnxTr)')
plt.xlabel('Iteration')
plt.ylabel('Time (seconds)')
plt.legend()
plt.grid(True)
plt.savefig('onnxTr_loop.png', dpi=150, bbox_inches='tight')
``

docTr Code:
``
import os
os.environ["DOCTR_MULTIPROCESSING_DISABLE"] = "TRUE"
os.environ["ONEDNN_PRIMITIVE_CACHE_CAPACITY"] = "1"

import time
import tqdm
import matplotlib.pyplot as plt
from doctr.io import DocumentFile
from doctr.models import ocr_predictor

model = ocr_predictor(
det_arch="db_mobilenet_v3_large",
reco_arch="crnn_vgg16_bn",
pretrained=True,
assume_straight_pages=False,
straighten_pages=True
)
times = []

file_path = "img.jpg"

for i in tqdm.tqdm(range(0, 100)):
start_time = time.time()
doc = DocumentFile.from_images(file_path)
result = model(doc)
end_time = time.time()
inference_time = end_time - start_time
times.append(inference_time)

plt.figure(figsize=(8, 5))
plt.plot(range(0, 100), times, marker='o', linestyle='-', color='b', label='Inference Time')
plt.title('OCR Inference Time per Iteration (docTr)')
plt.xlabel('Iteration')
plt.ylabel('Time (seconds)')
plt.legend()
plt.grid(True)
plt.savefig('docTr_loop.png', dpi=150, bbox_inches='tight')
``

Observations

1. Memory Usage: Profiling indicates memory usage remains fairly constant across all 100 iterations, both in docTr and onnxTr.
2. Inference Time:

• Despite using the same exact image, the inference time spikes and dips unpredictably.
• Even with DOCTR_MULTIPROCESSING_DISABLE / ONNXTR_MULTIPROCESSING_DISABLE set to TRUE and small ONEDNN_PRIMITIVE_CACHE_CAPACITY (or ONNXTR_CACHE_DIR), we still see significant variations in latency.
• The first few iterations can sometimes be slower (likely due to warm-up), but we’re seeing fluctuations that continue far beyond the warm-up phase.

Screenshots
docTr Memory Profiling:

docTr Time Profiling:

onnxTr Memory Profiling:

onnxTr Time Profiling:

Questions

• Has anyone else observed these sorts of fluctuations with docTr or onnxTr for repeated inference on a single image or running the same on a loop of images?
• Any insights into internal caching mechanisms, parallelization threads, or oneDNN overhead that might cause these irregular timings?
• Are there recommended environment variables or flags beyond what’s shown above that can help stabilize inference speed?

System Info
OS: macOS
Processor: 2.6 GHz 6-Core Intel Core i7
Graphics: AMD Radeon Pro 5300M 4 GB----Intel UHD Graphics 630 1536 MB
Python Version: 3.10
docTr Version: v0.10.0
onnxTr Version: v0.6.0

[felixdittrich92]

Hi @govindbagaria 👋🏼,

Thanks for reporting I will have a deeper look after my vacation (06.01.) 👍
Can you reproduce the same behaviour also with assume_straight_pages=True and/or straighten_pages=False ? :)

Best,
Felix

[govindbagaria]

The issue might be because of the kernels taking up much resources in my system. Not able to replicate the same on anther device. Marking this as not relevant for now.

[felixdittrich92]

Thanks for the update 👍