Implement QnA fine-tune

Author: mshumayl

.gitignore

@@ -0,0 +1 @@
+.env

documents/amaterasu.txt

@@ -0,0 +1 @@
+Amaterasu, also known as Amaterasu Omikami or Ohirume no Muchi no Kami, is the goddess of the sun in Japanese mythology. One of the major deities (kami) of Shinto, she is also portrayed in Japan's earliest literary texts, the Kojiki (c. 712 CE) and the Nihon Shoki (720 CE), as the ruler (or one of the rulers) of the heavenly realm Takamagahara and the mythical ancestress of the Imperial House of Japan via her grandson Ninigi. Along with her siblings, the moon deity Tsukuyomi and the impetuous storm god Susanoo, she is considered to be one of the "Three Precious Children" (mihashira no uzu no miko / sankishi), the three most important offspring of the creator god Izanagi.

documents/jiraiya.txt

@@ -0,0 +1 @@
+Jiraiya, originally known as Ogata Shuma Hiroyuki, is the toad-riding protagonist of the Japanese folk tale Katakiuchi Kidan Jiraiya Monogatari ("The Tale of the Gallant Jiraiya"). The tale was originally a Yomihon that was published in 1806–1807, and was adapted into a serialized novel that was written by different authors and published in 43 installments from 1839 to 1868; one of its illustrators was woodblock artist Kunisada. Kawatake Mokuami then wrote a kabuki drama based on the first ten parts of the novel, which premiered in Edo in 1852, starring Ichikawa Danjuro VIII in the leading role. Since then the story has been adapted into, several films, video games, and manga and has also influenced various other works.

documents/susanoo.txt

@@ -0,0 +1 @@
+Susanoo (historical orthography: 'Susanowo') is a kami in Japanese mythology. The younger brother of Amaterasu, goddess of the sun and mythical ancestress of the Japanese imperial line, he is a multifaceted deity with contradictory characteristics (both good and bad), being portrayed in various stories either as a wild, impetuous god associated with the sea and storms, as a heroic figure who killed a monstrous serpent, or as a local deity linked with the harvest and agriculture. Syncretic beliefs that arose after the introduction of Buddhism to Japan also saw Susanoo becoming conflated with deities of pestilence and disease.

qna.py

@@ -0,0 +1,50 @@
+import time
+from dotenv import load_dotenv
+from langchain.embeddings import OpenAIEmbeddings
+from langchain.document_loaders import DirectoryLoader, TextLoader
+from langchain.text_splitter import CharacterTextSplitter
+from langchain.vectorstores import Chroma
+from langchain.chains import RetrievalQA
+from langchain import OpenAI
+
+def process_data():
+    print("Getting data...")
+    loader = DirectoryLoader("documents", glob="**/*.txt")
+    documents = loader.load()
+    print("Documents loaded.")
+    
+    text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)
+    print("Text splitted.")
+    
+    texts = text_splitter.split_documents(documents)
+    
+    print(texts)
+    
+    return texts
+
+def get_embeddings(texts):
+    print("Getting embeddings...")
+    embeddings = OpenAIEmbeddings()
+    print("Embeddings loaded.")
+    
+    print("Creating vector store...")
+    vector_store = Chroma.from_documents(texts, embeddings)
+    print("Vector store created.")
+    
+    qna = RetrievalQA.from_chain_type(
+        llm=OpenAI(),
+        chain_type="stuff",
+        retriever=vector_store.as_retriever()
+    )
+    
+    return qna
+
+def query(prompt, qna_retriever):
+    print(f"User prompt: {prompt}")
+    print(f"Answer: {qna_retriever.run(prompt)}")
+
+if __name__ == '__main__':
+    load_dotenv()
+    texts = process_data()
+    qna_retriever = get_embeddings(texts)
+    query("What did Ichikawa Danjuro star in?", qna_retriever)