One of the biggest concerns for users right now is the quality of responses produced by LLMs. Natural language Generative AI models can produce unexpected or unwanted responses to prompts. This can be caused by any number of factors, including:
- Insufficient information in the training data
- Insufficient context in the prompt
- Lack of capability of the model itself
- Hostile intent by the user providing the prompt ("jailbreaking")
There are several different techniques currently being utilized in order to provide more accurate responses to prompts. We will briefly discuss these different techniques, the pros and cons of each, and scenarios where they might be used.
Prompt Engineering is the process of adding additional context to the prompt to provide "grounding" to the AI model and make it more likely to produce the desired response and less likely to produce undesirable outputs. For example, in a chatbot application, the system would inject additional instructions and data into the prompt before the user's actual input, to provide context to the model.
Fine tuning is the process of further training on a pre-trained model on a task specific dataset, resulting in a new "custom" LLM that has been optimized for the provided examples.
Retrieval Augmented Generation is a pattern that works with pretrained Large Language Models (LLM) and your own data to improve LLM performance. As discussed previously in the demo, data is first converted into vector emebeddings and placed into a vector index. When given a prompt, the prompt is converted into a vector embedding, and then compared to the vector embeddings generated by the data. The vector embeddings with the most similarity to the prompt are pulled from the index, and the user is given an answer to the prompt based on these portions of the document, typically referred to as the "context."
| Strategy | Pros | Cons |
|---|---|---|
| Prompt Engineering | -Easy to perform as it does not require any additional technical skills. -Can be combined with the two other strategies to maximize performance. |
- Requires a lot of trial and error to get the desired results. - Can be very time consuming. -No change to the LLM is occurring so it is still limited to the capabilities of the base model. |
| Fine-Tuning | -The LLM becomes specifically customized to the new data it was fine-tuned on. | - Fine-Tuning is very expensive and time consuming. - The model has to be fine-tuned again if new data needs to be added. |
| RAG | - Able to access a large variety of documents. - Creating and searching through a vector index is economical. |
- Still need to perform prompt engineering for maximum benefits. -Performance of RAG is dependent on the quality of data and documents in the vector index. |
-
The fashion company ChaosShop is constantly adding new items to their store, but wants customers to be able to use their interactive chatbot that gives up to date suggestions and information about their products.
- RAG would be a good solution to use in this scenario, as the vector index can be continuously updated, and the LLM can give up to date responses to customers.
-
The consulting firm ERS has a specific set of financial reports that they produce every single month that has a very rigid format that does not change on a month to month basis.
- Fine-tuning would be a good solution in this scenario as the reports does not change much and it would be worth it for the firm to invest in fine-tuning a model.
-
Miko is a graphic designer trying to use DALL-E to help her come up with some potential ideas for a social media flyer.
- Prompt engineering would be a good solution in this scenario to help Miko select a specific idea for her flyer.
That's all folks! Thanks for attending this demo.
This demo was adapted from the azure-openai-demo by revodavid and Use your own data with Azure Open AI Service from Microsoft Learn.