Selective Disabling of Loaded Data.

[sbmandava]

Once we Ingest the data into the RAG using tg-load-pdf is there a way to disable the PDF from RAG. i.e Looking for some method in restrict the scope to only the new set of document uploaded and disable the past vector/knowledge graph references

[sbmandava]

Nevermind, found the response in using collections. "-c" option.

tg-load-pdf -c nasa data/sources/Challenger-Report-Vol1.pdf
tg-query-graph-rag -c nasa -q 'Give me 20 facts about the space shuttle Challenger'

[JackColquitt]

We're working on a lot of features specifically for managing data in the graph/vector stores. While the power of knowledge graphs is being able to load huge amounts of disparate data into a single store to unlock previously unknown relationships, there are many use cases where you would want more fine grained control over what is in the stores.

[sbmandava]

Agree with the ideal world of single store, when the quality data is in.

But in real enterprise world lot of data is redundant or obsolete or not reflective of true state and thus we spend 80% of time cleaning up the data to make it ready for Knowledge Graph, else our RDF Tuples has weird relationships which get fed into RAG's and chat responses.

Been in real-world situations that one wrong document ingestion messed up the data quality by 50% and had to revert to prior snapshot. Appreciate you can think of snapshots / export / import capabilities for the ingested system.

[JackColquitt]

Trying to solve this issues is exactly our vision around what we're calling "knowledge cores". I definitely think this problem gets overlooked with RAG using just vector embeddings. If you have low quality embeddings, then you relying on a snapshot system to be able to revert back to previous "versions".

We don't want to have to rely on a snapshot style system, because that requires storing all of that data from this snapshots to even be able to revert back to previous versions. That's why we're making the "knowledge cores" modular, so that they can be removed and added to the system as needed.

How are you cleaning the data for entering into the Knowledge Graph now? Is this a manual process or reviewing the triples and manually editing them?

[cybermaggedon]

@sbmandava you are correct, we envisage the collection and user being used to group documents. Please be aware that the implementation is not 100% complete, it works for Qdrant vector store plus Cassandra graph store. Neo4j implementation is not present yet.