TypeError: RetrievalEvaluator.__init__() missing 1 required positional argument: 'retriever'

[XuHuang441]

Hi there,

I encountered an issue while trying to evaluate a retriever on ViDoRE. I followed all the steps provided on the page and ran the following command after installing the package via pip install vidore-benchmark:

vidore-benchmark evaluate-retriever \
    --model-name vidore/colpali-v1.2 \
    --collection-name "vidore/vidore-benchmark-667173f98e70a1c0fa4db00d" \
    --split test

However, after the "Forward pass documents process," I received the following error:

TypeError: RetrievalEvaluator.__init__() missing 1 required positional argument: 'retriever'

Thanks

[pjoziak]

Same here. Also noted that the command in README has an extra CLI kwarg specified --model-class colpali, can this be related that this was dropped from CLI kwargs but is still expected somewhere deeper in the stack?

[p114514-new]

Change line 134 of vidore_benchmark/retrievers/utils/vision_retriever.py(in the pip installed package):
mteb_evaluator = RetrievalEvaluator()
to
mteb_evaluator = RetrievalEvaluator(self.model)
solves the problem.