feat: add colpali retriever#25

Open

paknikolai wants to merge 110 commits intoepam:developmentfrom

paknikolai:f/colpali_retriever

Commits on Jun 30, 2025

reapplied commits
Nikolai Pak
committed
fixed path
Nikolai Pak
committed

Commits on Jul 1, 2025

upgraded dependencies
Nikolai Pak
committed
fixed pathes
Nikolai Pak
committed
added dependencies
Nikolai Pak
committed

Commits on Jul 7, 2025

added test for colpali retriever
Nikolai Pak
committed

Commits on Jul 8, 2025

added code for caching results
Nikolai Pak
committed
fixed the test cahce and added cache itsef
Nikolai Pak
committed
added e2e test code
Nikolai Pak
committed
updated code of test to match the query
Nikolai Pak
committed
updated test and cache
Nikolai Pak
committed
updated test files
Nikolai Pak
committed

Commits on Jul 9, 2025

updated caches for the tests
Nikolai Pak
committed
updated cache
Nikolai Pak
committed
added config yaml for colpali and updated tests
Nikolai Pak
committed
added image size to colpali config
Nikolai Pak
committed

Commits on Jul 10, 2025

added creation of colpaliresource on creation,
Nikolai Pak
committed
moved colpali resource to colpali folder
Nikolai Pak
committed
partlt changed the calculation of scores
Nikolai Pak
committed

Commits on Jul 11, 2025

updated retriever code to use page embeddings and score them and then select k chunks from top pages
Nikolai Pak
committed

Commits on Jul 15, 2025

changed enum+str to strenum, added consistency check for the model name and type
Nikolai Pak
committed
added lock for gpu related operations
Nikolai Pak
committed
added cpu pools usage
Nikolai Pak
committed
changed bloat16 to float16
Nikolai Pak
committed
import fixes
Nikolai Pak
committed
fixed lint issues
Nikolai Pak
committed
fixed cache and refactored cache code
Nikolai Pak
committed
removed ignores
Nikolai Pak
committed
removed unnecessary processor caching
Nikolai Pak
committed

Commits on Jul 16, 2025

switched device choice to existing function
Nikolai Pak
committed
Merge branch 'development' into f/colpali_retriever
Nikolai Pak
committed
removed unnecessary setting of lib version
Nikolai Pak
committed
added comment for the colpali-engine dependency
Nikolai Pak
committed

Commits on Jul 21, 2025

put colpali config outside of the request config
Nikolai Pak
committed

Commits on Jul 23, 2025

added batch that is being collected from different tasks and then being processed
Nikolai Pak
committed

Commits on Jul 24, 2025

added hasg based cache
Nikolai Pak
committed

Commits on Jul 25, 2025

added test for parallel queries and images
Nikolai Pak
committed
fixed moving to cpu embeddings
Nikolai Pak
committed
added additional pools not to block other pools
Nikolai Pak
committed
replaced progress bar
Nikolai Pak
committed

Commits on Jul 28, 2025

caching models in docker on creating image
Nikolai Pak
committed

Commits on Aug 5, 2025

fixed donwload model script
Nikolai Pak
committed
fixed docker file
Nikolai Pak
committed
added file with common models info and pathes, changed docker file to use it
Nikolai Pak
committed
changed config according to fields in app config
Nikolai Pak
committed
added cache in model while saving it
Nikolai Pak
committed
fixed device
Nikolai Pak
committed
fixed model cache save
Nikolai Pak
committed
added copying additional files for docker to download models
Nikolai Pak
committed
fixed format
Nikolai Pak
committed
removed from resource config model type, now it is being calculated based on model name
Nikolai Pak
committed
fixed tests
Nikolai Pak
committed

Commits on Aug 6, 2025

fixed format
Nikolai Pak
committed
Merge branch 'development' into f/colpali_retriever
Nikolai Pak
committed

Commits on Aug 7, 2025

changed doc_id to original index that was in from_doc_records
Nikolai Pak
committed
poetry lock
Nikolai Pak
committed
updated poetry lock with minimal changes comapring to original one
Nikolai Pak
committed
fixed test
Nikolai Pak
committed

Commits on Aug 8, 2025

renamed file to match cache for test
Nikolai Pak
committed
updated cahce method, and replaced cahce files
Nikolai Pak
committed
changed yaml name
Nikolai Pak
committed
added fair queue to process tasks
Nikolai Pak
committed
renamed function
Nikolai Pak
committed
changed variables order
Nikolai Pak
committed
Merge branch 'development' into f/colpali_retriever
Nikolai Pak
committed
changed colpali version to realeased version
Nikolai Pak
committed

Commits on Aug 21, 2025

made separate docker file+changed script to download model
Nikolai Pak
committed
added arg for base image
Nikolai Pak
committed
added stage to donwload model and some logs to verify env variables
Nikolai Pak
committed
fixed model env variable and added perint to log when loading the model
Nikolai Pak
committed
fixed env
Nikolai Pak
committed

Commits on Aug 22, 2025

removed unseccessary variable from docker
Nikolai Pak
committed
Merge branch 'development' into f/colpali_retriever
paknikolai
authored
updated readme
Nikolai Pak
committed

Commits on Sep 8, 2025

Merge branch 'development' into f/colpali_retriever
Nikolai Pak
committed

Commits on Sep 10, 2025

removed image size field because it was unecessary
Nikolai Pak
committed
removed unused import
Nikolai Pak
committed
changed paramerers order to be consistent with embeddings
Nikolai Pak
committed
removed unnecessary print
Nikolai Pak
committed
removed unused dependency
Nikolai Pak
committed

Commits on Sep 11, 2025

moved load model to different function and removed set config from resourse
Nikolai Pak
committed
replaced with map with progress query and image processing
Nikolai Pak
committed
made a str enum and renamed map fro known models
Nikolai Pak
committed
removed batch prcessor
Nikolai Pak
committed
fixed queries
Nikolai Pak
committed
fixed format
Nikolai Pak
committed
fixed lint
Nikolai Pak
committed
fixed queries
Nikolai Pak
committed
added batch size to parameters
Nikolai Pak
committed

Commits on Sep 12, 2025

moved model call inside resource
Nikolai Pak
committed
removed unusd config
Nikolai Pak
committed
fixed docker file since folder with dial rag is already copied
Nikolai Pak
committed
fixed downloading script
Nikolai Pak
committed

Commits on Sep 16, 2025

merged develop branch
Nikolai Pak
committed
fixed format
Nikolai Pak
committed
added some comments
Nikolai Pak
committed

Commits on Sep 17, 2025

Merge branch 'development' into f/colpali_retriever
paknikolai
authored

Commits on Sep 18, 2025

fixed type for images in calculate_images_embeddings
Nikolai Pak
committed
simplified replacing cached resource in tests
Nikolai Pak
committed
created constabt for the test folder
Nikolai Pak
committed

Commits on Sep 19, 2025

removed colpali index from colpali resource
Nikolai Pak
committed
moved common embeddings to separate folder
Nikolai Pak
committed
Merge branch 'development' into f/colpali_retriever
paknikolai
authored
separated download scripts
Nikolai Pak
committed
corrected argument line
Nikolai Pak
committed
replaced env variable for model path with config
Nikolai Pak
committed
removed padding of embeddings
Nikolai Pak
committed

Commits on Sep 25, 2025

changed hf_home to cache models there
Nikolai Pak
committed
updated donwload
Nikolai Pak
committed
Merge branch 'development' into f/colpali_retriever
Nikolai Pak
committed