Add wrapper tool for FactReasoner

[jbrry]

Summary 📝

This PR adds FactCheckTool which is a wrapper to call the hosted FactReasoner service.

It also adds some helper tools which enable further downstream analysis: a VectorDBTool which is based on a Chroma vector database is added so that search artefacts can be stored for downstream analysis or so that FactReasoner can use it for context retrieval.

Details

VectorDBTool: akd/tools/vector_db_tool.py used for interacting with a vector database. It uses ChromaDB for persistent storage and sentence-transformers for creating embeddings. The .index() method is used for adding documents, and _arun runs the retrieval based on a query.

DeepLitSearchAgent Integration: The new functionality has been added as part of a DeepLitSearchAgent run. This shows how we can use the new tools to work along with the DeepLitSearchAgent.

Usage

Documentation for the backend deployment is available here.

Standalone example:

from akd.tools.fact_check import FactCheckInputSchema, FactCheckOutputSchema, FactCheckToolConfig, FactCheckTool

from pydantic import HttpUrl

fact_check_config = FactCheckToolConfig(base_url=HttpUrl("https://factreasoner-service-app.1yhbkn094k2v.us-south.codeengine.appdomain.cloud"), polling_interval_seconds=10)
fact_check_tool = FactCheckTool(config=fact_check_config)


sample_question = "What is the coriolis effect?"
sample_answer = "The Coriolis effect describes the pattern of deflection taken by objects not firmly connected to the ground as they travel long distances around the Earth."

fact_check_input = FactCheckInputSchema(
    question=sample_question,
    answer=sample_answer,
)

print(f"\n--- Running Fact-Check on sample answer ---\n{sample_answer}\n")
try:
    fact_check_result = await fact_check_tool.arun(params=fact_check_input)

    print("\n--- Fact-Check Complete ---")
    score = fact_check_result.fact_reasoner_score.get("factuality_score", 0)
    num_supported = len(fact_check_result.supported_atoms)
    num_not_supported = len(fact_check_result.not_supported_atoms)

    print(f"Factuality Score: {score:.2%}")
    print(f"Supported Atoms: {num_supported}")
    print(f"Not Supported Atoms: {num_not_supported}")
    print(f"Graph ID: {fact_check_result.graph_id}")
    
    # graph methods
    if fact_check_result.graph_id:
        graph_id = fact_check_result.graph_id

    print(f"\nFetching graph structure for ID: {graph_id}...")
    graph_json_url = f"{fact_check_tool.config.graph_json_endpoint}/{graph_id}?format=full"
    graph_response = await fact_check_tool.api_client.get(graph_json_url)

    if graph_response.status_code == 200:
        graph_data = graph_response.json()
        print(f"Successfully fetched graph JSON. Found {len(graph_data.get('nodes', []))} nodes and {len(graph_data.get('edges', []))} edges.")
        # print(graph_data) # uncomment for full object.
    else:
        print(f"Failed to fetch graph JSON. Status: {graph_response.status_code}")

except Exception as e:
    print(f"\nExample failed. Ensure the fact-checking service is running.")
    print(f"Error: {e}")

It returns the factuality score based on the question and response pair:

--- Fact-Check Complete ---
Factuality Score: 100.00%
Supported Atoms: 3
Not Supported Atoms: 0
Graph ID: 92ed5024-ad09-48f6-b296-357bdc3e38f7

Fetching graph structure for ID: 92ed5024-ad09-48f6-b296-357bdc3e38f7...
Successfully fetched graph JSON. Found 15 nodes and 36 edges.

I am also listing the test files for each of these tools, which cover initialization and running of the tools and can serve as documentation:

• VectorDBTool: test script example
• FactCheckTool: test script example.

[NISH1001]

@jbrry Could you please add usage section in the PR as well.

Some sample PRs:

• Refactor the scraper module to make it robust and extensiblity #78
• Add CombinedCodeSearchTool for unified code search  #77

etc

[NISH1001]

[Initial superficial ocmments]

[movinam]

Thanks for the PR! I think the text splitter tool is not needed - we can just use the langchain implementation/or any existing implementation and pass the documents to the vector db to be indexed. Please see comments.

[NISH1001]

@jbrry There's merge conflict with develop. Could you pull in develop latest changes and resolve?

[github-actions]

❌ Tests failed (exit code: )

📊 Test Results

• Passed: 0
• Failed: 0
• Warnings: 0
• Coverage: 0%

⚠️ Note: Test counts are 0, which may indicate parsing issues or early test failure. Check the workflow logs for details.

Branch: feature/vector_database_tool
PR: #79
Commit: 982ae7e

📋 Full coverage report and logs are available in the workflow run.

[github-actions]

❌ Tests failed (exit code: 1)

📊 Test Results

• Passed: 221
• Failed: 6
• Warnings: 109
• Coverage: 72%

Branch: feature/vector_database_tool
PR: #79
Commit: cc83fbe

📋 Full coverage report and logs are available in the workflow run.

[github-actions]

❌ Tests failed (exit code: )

📊 Test Results

• Passed: 0
• Failed: 0
• Warnings: 0
• Coverage: 0%

⚠️ Note: Test counts are 0, which may indicate parsing issues or early test failure. Check the workflow logs for details.

Branch: feature/vector_database_tool
PR: #79
Commit: 9ed9312

📋 Full coverage report and logs are available in the workflow run.

[github-actions]

❌ Tests failed (exit code: 1)

📊 Test Results

• Passed: 221
• Failed: 6
• Warnings: 109
• Coverage: 72%

Branch: feature/vector_database_tool
PR: #79
Commit: 03cea67

📋 Full coverage report and logs are available in the workflow run.

[github-actions]

❌ Tests failed (exit code: 1)

📊 Test Results

• Passed: 221
• Failed: 6
• Warnings: 109
• Coverage: 72%

Branch: feature/vector_database_tool
PR: #79
Commit: 6a87351

📋 Full coverage report and logs are available in the workflow run.

[github-actions]

❌ Tests failed (exit code: 1)

📊 Test Results

• Passed: 221
• Failed: 6
• Warnings: 108
• Coverage: 72%

Branch: feature/vector_database_tool
PR: #79
Commit: 6a87351

📋 Full coverage report and logs are available in the workflow run.

[NISH1001]

Can we rename the default name to something else? like akd_vdb or something?

[NISH1001]

Nitpick

[github-actions]

❌ Tests failed (exit code: 1)

📊 Test Results

• Passed: 249
• Failed: 5
• Warnings: 113
• Coverage: 73%

Branch: feature/vector_database_tool
PR: #79
Commit: 5f7c9b6

📋 Full coverage report and logs are available in the workflow run.

[github-actions]

❌ Tests failed (exit code: 1)

📊 Test Results

• Passed: 221
• Failed: 6
• Warnings: 108
• Coverage: 72%

Branch: feature/vector_database_tool
PR: #79
Commit: 69533eb

📋 Full coverage report and logs are available in the workflow run.