Understanding RAG: Benefits, Limitations, and When to Use It

Kshitij Nehete — Sat, 11 Jul 2026 11:56:52 GMT

Have you ever asked an AI a question and received an answer that sounded confident but was completely wrong? This happens because Large Language Models (LLMs) only know the information they learned during training. They cannot automatically access your company's latest documents, recent policies, or newly published content.

This is where Retrieval-Augmented Generation (RAG) comes in. Instead of answering directly, a RAG system first searches a knowledge base for relevant information. It then provides that information to the LLM, allowing it to generate a response based on the retrieved context instead of relying only on its training data.

A basic RAG pipeline is easy to understand. A user asks a question, the system searches for the most relevant documents, and the LLM uses those documents to create the final answer.

RAG is useful in many real-world applications. Customer support bots, company knowledge portals, and document search systems all benefit from it. For example, if an employee asks about the latest work-from-home policy, RAG can retrieve the newest policy document before the AI responds.

However, RAG is not a guarantee of correct answers. If the retrieval step finds the wrong document or misses important information, the response can still be inaccurate. Poor chunking is another common issue. When related information is split into different chunks, the model may not understand the complete context. Context window limitations also matter because the model can only process a limited amount of retrieved information at once.

Even with good retrieval, LLMs may still hallucinate and generate details that are not present in the provided documents. Another challenge is keeping the knowledge base updated, since outdated information will lead to outdated answers.

RAG is a great choice when an application depends on external knowledge. However, if a task does not require document retrieval, using a standard LLM is often simpler.

RAG makes AI responses more relevant and useful, but its success depends on good retrieval, proper chunking, updated knowledge, and careful system design.

How ChatGPT Understands Your Questions

Kshitij Nehete — Tue, 30 Jun 2026 15:39:43 GMT

Yesterday, we completed the first class of the GenAI with JavaScript Cohort by Chai Code.

The session served as an introduction to some of the core concepts behind modern AI, including Large Language Models (LLMs), tokenization, Transformers, and how ChatGPT generates responses.

What is an LLM?

Let's start with the most important concept: LLM, which stands for Large Language Model.

An LLM is an AI model trained on an enormous amount of text from books, articles, websites, documentation, and many other publicly available sources. Instead of memorizing every sentence, it learns the patterns and relationships between words so it can generate human-like text.

The primary job of an LLM is surprisingly simple:

Predict the next most likely word (or token) based on the context it has already seen.

Although this sounds simple, repeatedly predicting the next token allows an LLM to generate paragraphs, solve problems, write code, and answer questions.

Problems LLMs Can Solve

Answering questions
Writing articles and emails
Summarizing long documents
Translating between languages
Assisting with programming
Brainstorming ideas

Some of the most popular LLMs available today include ChatGPT, Gemini, Claude, Llama, and DeepSeek.

What Happens When You Send a Message to ChatGPT?

Now let's understand what actually happens when you send a prompt to ChatGPT.

Step 1: You Type a Prompt

Everything starts with a prompt.

For example:

"Explain JavaScript Promises."

Step 2: Your Message Is Converted into Tokens

Computers don't understand complete sentences the way humans do.

Instead, your text is broken into smaller pieces called tokens, which the model can process efficiently.

Step 3: The Model Processes the Tokens

Once tokenized, the model looks at all the tokens together instead of reading them one by one.

Using the Transformer architecture, it understands the relationships between different words and keeps the conversation's context in mind before generating an answer.

Step 4: The Response Is Generated

The model predicts one token at a time.

Each newly generated token becomes part of the context for predicting the next one. This process continues until the entire response is completed.

Although the response appears instantly, it is actually generated token by token behind the scenes.

Why Isn't the Response Copied from the Internet?

A common misconception is that ChatGPT searches Google every time you ask a question.

That's not how it works.

By default, ChatGPT does not search the web for every prompt. Instead, it generates responses using the patterns it learned during training.

You can think of it as an incredibly advanced autocomplete system—it predicts the next most likely token based on the context, but at a much larger scale.

Tokenization

The next concept we learned was Tokenization.

Tokenization is the process of breaking text into smaller units called tokens.

A token can be:

A complete word
Part of a word
A punctuation mark
A special character

For example,

Hello world!

↓

Hello
world
!

Another example:

playing

↓

play
ing

Why Do We Need Tokenization?

Computers work with numbers—not natural language.

Before an AI model can understand your text, it must first convert it into smaller pieces that can later be represented as numbers.

It's also important to remember that words and tokens are not always the same thing.

A short word like "cat" may become a single token, while a longer word like "unbelievable" might be split into multiple tokens.

This is one reason why token counts are different from word counts.

Transformers

The final major topic from the session was Transformers, the architecture that powers almost every modern Large Language Model.

Before Transformers, language models processed text mostly in sequence, making it difficult to understand long conversations or relationships between distant words.

Transformers changed this by allowing the model to look at the entire context at once.

One of the key ideas behind Transformers is Attention.

Instead of treating every word equally, the model learns which words are most relevant when understanding the current token.

For example, consider the sentence:

"After Maya finished her presentation, everyone congratulated her."

The Transformer understands that "her" refers to Maya, even though several other words appear between them.

Another example:

"John placed the laptop inside the backpack because it was too heavy."

Here, the model understands that "it" refers to the laptop, not the backpack.

This ability to understand relationships between words makes Transformers much better at reasoning, translation, summarization, coding, and conversation.

This architecture was introduced in the famous research paper:

Attention Is All You Need (2017)

Today, nearly every popular LLM—including GPT, Gemini, Claude, and Llama—is built using the Transformer architecture.

Conclusion

In this first class, we learned that ChatGPT doesn't magically understand language the way humans do.

Instead, it breaks text into tokens, converts them into numerical representations, processes them using the Transformer architecture, and predicts one token at a time to generate meaningful responses.

Understanding these fundamentals provides a strong foundation for exploring Generative AI and building AI-powered applications in the future.

Thanks for reading! This article is based on my notes from the first GenAI with JavaScript cohort session by Chai Code. More learning notes and practical implementations coming soon.

#GenerativeAI #LLM #ChatGPT #JavaScript #AI #Transformers