Skip to content

add optional WFGY RAG failure map evaluation pack#216

Open
sabbellasri wants to merge 1 commit into
AI4Finance-Foundation:masterfrom
sabbellasri:rag-failure-map-feature
Open

add optional WFGY RAG failure map evaluation pack#216
sabbellasri wants to merge 1 commit into
AI4Finance-Foundation:masterfrom
sabbellasri:rag-failure-map-feature

Conversation

@sabbellasri
Copy link
Copy Markdown

Refs #212

@onestardao this PR adds a first docs-only draft of the optional WFGY evaluation pack discussed in the issue.

Summary

This PR adds an optional docs-only evaluation pack for FinGPT RAG workflows that maps common financial QA failure patterns to the WFGY 16-problem ProblemMap.

The goal is to give FinGPT users a lightweight troubleshooting guide for real-world RAG deployments over filings, earnings calls, news, and simple tool-augmented finance workflows, without changing any core model or training code.

Changes

  • Added fingpt/FinGPT_RAG/wfgy_evaluation_pack/README.md
  • Linked the new pack from fingpt/FinGPT_RAG/README.md
  • Linked the optional pack from the top-level README.md

What's Included

  • A compact symptom -> ProblemMap -> "inspect this part of the FinGPT stack first" troubleshooting table
  • A full No.1 to No.16 mapping from the WFGY taxonomy to likely FinGPT pipeline components
  • Representative financial RAG failure cases covering hallucinated risk statements, retrieval drift over filings, prompt drift in multi-step financial QA, and misrouted tool or time-series calls

Notes

  • This is a documentation-only contribution
  • No changes were made to FinGPT model weights, training scripts, or benchmark code
  • The WFGY taxonomy is used as an optional external diagnostic framework for users who want a shared vocabulary for RAG debugging

Reference

Upstream WFGY ProblemMap:
https://github.com/onestardao/WFGY/blob/main/ProblemMap/README.md

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant