What is Retrieval Augmented Generation?

What is Retrieval Augmented Generation?

The modern LLMs, such as ChatGPT or Claude, are able to provide answers to questions such as "Who is the current president of US?". This is a great feat because these systems are not built with this information. The dataset used for building these LLMs often have a past cutoff date and they may not carry the latest information. So the question arises, how are they able to do that?

The answer is RAG or Retrieval Augmented Generation. Simply speaking, before the LLM generates an answer, it augments its knowledge by retrieving latest information pertaining to the user's query, in this case the president of the United States. 

A RAG system does the following - 

1. Retrieval: Finding relevant information from a source.
2. Augmentation: Appending that information to the user's query.
3. Generation: Generating the LLM response using user query and augmented information.

How does Retrieval work?

Let us spend a little time understanding how the retrieval step works, because that is the key to this whole process. The problem we are solving for is to find / retrieve the right sources of information for a user's query. Let us look at it with an example.

Let us say we are building a RAG system that answers queries about a restaurant's menu. The core information that the system is grounded in is menu items and their descriptions.

A user comes into the system's chatbot and asks "Does butter chicken contain nuts?". Now, for an effective RAG system, it needs to retrieve the information pertaining to butter chicken from the menu items and place it with the query to the LLM.

There are various ways of doing that:

1. One way would be to do keyword search. Search for menu items that have the words "Butter" and "Chicken" and insert that along with the query to the LLM. This will work but this approach has the issue of capturing other items as well which have the words "butter" and "chicken" in them and adding additional information to the LLM.
2. Another way, and this is the preferred way, would be to retrieve menu information based on understanding the meaning and using that to match whether the menu contains butter chicken. This mechanism is known as Semantic Search. This is the preferred way of performing retrieval in a RAG system and we will explain that more in the next section.

What is Semantic Search?

Plainly speaking, Semantic Search is the mechanism of retrieving relevant information to a query based on the meaning rather than just keyword search. 

A semantic search system has the following components:

1. Database/Source Information/Documents: This is the superset of all the information that semantic search can query on. This can include pdfs, database information etc. Anything relevant for the system you are trying to build.
2. Knowledge Representation / Embedding: Since semantic search is a meaning based search, we do not just want to store the raw information. We also need a way to store the knowledge that the data carries. This is what is called as "Embeddings". Embeddings create a mathematical representation of the information that is carried in raw data. This is achieved through specialized AI models built for this purpose.
3. Vector Database: Once we have embeddings, we need a place to store them and that can be used for searching. This is what a vector database does. Some popular options are Chroma, Pinecone, Weaviate and pgvector (a Postgres extension). For small projects, an in-memory store works just fine. 
4. Similarity Search: When a user asks a question, we convert the question into an embedding using the same model, to capture the meaning of the query. Then we ask the vector database for the stored embeddings that are "closest" to it. This is done via mechanisms such as cosine similarity, which finds the top closest embeddings.

Let us go back to our restaurant example. The question "Does butter chicken contain nuts?" gets converted into an embedding. The vector database compares it against the embeddings of all menu items and returns the butter chicken entry as the closest match. It may also return a couple of other close items, like chicken tikka masala, but these are ranked by how relevant they are. This works even if the user types "Is there any cashew in the murgh makhani?", because the meaning is the same even though the words are different. 

How does Augmentation work?

Augmentation is the simplest step. We take the information returned by retrieval and add it to the prompt we send to the LLM, along with the user's question and some instructions. A typical augmented prompt looks like this:

You are a helpful assistant for a restaurant. Answer the customer's question
using only the menu information below. If the answer is not in the menu
information, say that you don't know and suggest asking the staff.

Menu information:
- Butter Chicken: Tandoori chicken simmered in a creamy tomato gravy with
  butter, cream and a cashew paste. Contains: dairy, nuts.
- Chicken Tikka Masala: ...

Customer question: Does butter chicken contain nuts?

How does Generation work?

The LLM now reads the augmented prompt and generates a response. Since the relevant facts are right there in the prompt, the model does not need to guess. In our example it would answer something like: "Yes, our butter chicken contains cashew paste, so it is not suitable for anyone with a nut allergy."

Important learnings from building a production RAG system

1. Chunking: For long texts, rather than generating an embedding for the entire document, chunking/splitting it into smaller parts and then generating the embeddings is more efficient. A single embedding from a large text can lead to drift and knowledge not being captured accurately.
2. Keyword search still has a place: Semantic search can miss exact terms like dish codes or brand names. Many production systems combine both, which is called hybrid search.
3. Tell the model what to do when it doesn't know: Without that instruction, the LLM will happily fill gaps with plausible guesses. For something like allergens, that can be dangerous.

Conclusion

RAG is a power mechanism to improve the LLM's response quality and context. It provides the LLM with the missing pieces of information which it can utilize to provide a contextually relevant answer.

If you wanna build your own RAG based LLM, I provide dedicated sessions on the same! Feel free to book some sessions and we can deploy your production ready RAG system.