[RFC] Propose a new GenAIExample - visual search and QA

[llin60]

This RFC proposes a new GenAIExample that integrates a multi-modal search engine with a visual QA assistant, so that the QnA assistant could be a better helper given the search results as visual context. The search engine and VQA assistant can also work independently as well.

This application serves as an excellent use case for industries such as surveillance, smart cities, and other domains requiring efficient analysis of large-scale visual data.

[yinghu5]

Hi @llin60
Thank you a lot for new example, will escalate for further discussion.
Just for reference:
Visual QnA example: https://github.com/opea-project/GenAIExamples/tree/main/VisualQnA
MultimodalQnA: https://github.com/opea-project/GenAIExamples/tree/main/MultimodalQnA, which may comprehend a mix of textual, visual, and audio facts drawn from the document contents.

[Copilot]

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

[Copilot]

The curl example for the embeddings endpoint omits the HTTP method and headers, which may mislead users expecting a complete API request (e.g., missing '-X POST' and 'Content-Type: application/json'). Consider updating the example for consistency with the other endpoints.

Suggested change

	curl http://localhost:6000/v1/embeddings
	-X POST
	-d '{"input":"traffic jam"}'
	-H 'Content-Type: application/json'
	curl -X POST http://localhost:6000/v1/embeddings \
	-H "Content-Type: application/json" \
	-d '{"input":"traffic jam"}'

[Copilot]

The curl command for the retrieval endpoint is missing important flags (such as '-X POST' and required headers) that are needed for a proper API call. Please revise the example to include these details for clarity.

Suggested change

	curl http://localhost:7000/v1/retrieval
	-X POST
	-d "{"embedding":${text_embedding},"search_type":"similarity", "k":4}"
	-H 'Content-Type: application/json'
	```
	curl http://localhost:7000/v1/retrieval \
	-X POST \
	-d '{"embedding":"<text_embedding_placeholder>","search_type":"similarity","k":4}' \
	-H 'Content-Type: application/json'
	# Replace <text_embedding_placeholder> with the actual text embedding value.

[Copilot]

The curl example for the combined search and Q&A endpoint omits the HTTP method and payload details, which might lead to confusion. Consider including the POST method and a sample payload to match the other examples.

Suggested change

	curl http://localhost:8888/v1/dbsearch_qna
	-X POST
	-d '{"text":"traffic jam"}'
	-H 'Content-Type: application/json'
	curl http://localhost:8888/v1/dbsearch_qna \
	-X POST \
	-d '{"text":"traffic jam", "context_images": ["image1.jpg", "image2.jpg"], "k": 5}' \

[llin60]

Hi, thank you for the info. I've studied the existing examples for multi-modal applications. It seems that they process visual data by converting to text. However, in the application we are proposing, we need to store the visual data authentically, as the original images/videos are the targets in interest. Details can be found in the documentation.