RAG: How AI Learns from Things It Was Never Trained On

✦ The Scholar and the Archive

Imagine a wise scholar sitting in a room.

The scholar is exceptional with language. They can explain, compare, summarize, simplify, and reason. They can turn dense material into clear answers.

But there is a problem.

Outside the room, there is an archive full of material your organization depends on: product manuals, company policies, meeting notes, support articles, technical documentation, research papers, learner progress records.

The scholar has never seen any of it.

A visitor walks in and asks:

What should I do if my device stops syncing after firmware update 2.7?

The scholar may respond beautifully. But without the archive, the scholar is guessing. The answer may sound confident. The source may be missing.

Now imagine a librarian standing beside the scholar.

Before the scholar speaks, the librarian runs to the archive, finds the three most relevant pages, and places them on the scholar's desk.

Now the scholar answers. Not from vague memory. Not from general knowledge. From the retrieved pages.

That is RAG.

The model is the scholar. The retrieval system is the librarian. Your documents are the archive. The final answer is grounded in the material brought into the room.

✦ The Problem RAG Solves

A language model is trained on a large amount of text. That training gives it remarkable capabilities. It learns how language works. It learns how explanations are formed. It absorbs patterns from vast amounts of human writing.

But it does not automatically know your private world.

It does not know:

• your company's current refund policy
• your internal onboarding document
• your personal notes and research
• your PDF from this morning
• your codebase
• your support articles from last week
• your learner's progress history

A user asks:

What is our refund policy for annual subscriptions after renewal?

The model may have encountered thousands of refund policies during training. But it has not seen yours. So if it answers from memory, it may sound completely correct and still be wrong.

That is the danger.

The answer may be fluent. The structure may be polished. The tone may be helpful. But when the source is missing, confidence becomes dangerous.

RAG exists to solve this. It does not make the model all-knowing. It teaches your application to bring the right material before asking for an answer.

✦ Three Words, One Loop

Let us slow down and unpack the name itself.

Retrieval

First, the system searches for relevant information.

A user asks a question. The system finds the closest matching chunks from your documents, notes, articles, or database. This is exactly where embeddings from Day 7 do their work.

The user may ask:

How can AI answer from my own files?

Your document may say:

Retrieval-Augmented Generation grounds model responses in external context.

The words are different. The meaning is close. Embeddings retrieve the right passage.

Augmented

In step 1, relevant chunks were retrieved. Now the retrieved information is added to the prompt. The original user question does not travel alone. Instead of sending this:

You send something like this:

The model's view has been augmented. It can now see material it was never trained on.

Generation

Finally, the model generates the answer. But now that answer is shaped by the retrieved context. The model is not relying only on what it remembers. It is reading what you provided.

That is the full loop:

Retrieve the right context. Add it to the prompt. Generate the answer from that context.

✦ Two Phases: Preparing and Answering

Day 7 introduced this idea briefly. Day 8 is where it becomes concrete.

A RAG system separates its work into two distinct moments. Understanding this separation is one of the most important things you can take from this lesson.

Phase 1: Indexing (happens before the user asks anything)

You start with your source material. PDFs, articles, help docs, markdown files, database records, transcripts, company knowledge, learner notes. But you do not send whole documents to the model. That would be too large, too noisy, too expensive.

Instead, you split content into smaller pieces. These are called chunks. A chunk may be a paragraph, a section, a few related lines, or a page from a PDF. Each chunk should carry one reasonably complete idea.

Then each chunk is converted into an embedding and stored alongside the original text and metadata.

A stored record looks like this:

{
  "text": "The context window is the fixed container of tokens the model can see during a request.",
  "embedding": [0.021, -0.088, 0.134, "..."],
  "source": "tokens-and-context",
  "section": "The Context Window",
  "page": 3
}

Notice what is stored: the actual text, the embedding, the source, and metadata. The metadata matters. Without it, your system may retrieve the right answer but not know where it came from. With it, the final answer can say:

According to the context window section from the tokens lesson...

That traceability is not decoration. It is part of a trustworthy product.

Phase 2: Retrieval and Generation (happens when the user asks a question)

The system does not immediately send the user's question to the LLM.

First, it creates an embedding for the question. Then it compares that query embedding with all stored chunk embeddings. The closest chunks rise to the top.

For a question like "Why does the model forget what I said earlier?" the system might retrieve:

Now those chunks are placed into the prompt alongside the user's question. The model generates its answer from that context.

The model did not search. Your system searched. The model did not know your documents. Your system brought the documents into the model's view.

That distinction is the heart of RAG.

✦ What This Looks Like in Code

In Day 7, you built semantic search. Your code embedded documents, embedded a query, and returned the closest chunks. That search is the retrieval layer of RAG.

Today, you add the missing piece: take those retrieved chunks and generate a grounded answer from them.

Create a new file called rag-demo.js. You can start from the Day 7 code or follow along from scratch.

Step 1: Setup

import "dotenv/config";
import OpenAI from "openai";

const openai = new OpenAI({
  apiKey: process.env.OPENAI_API_KEY,
});

This is identical to Day 7. One client, one API key from your .env file.

If you are setting up a fresh folder, add "type": "module" to your package.json so Node.js handles the import statements correctly:

{ "type": "module" }

Step 2: Converting text into meaning, and measuring similarity

This step builds the retrieval layer. Two functions handle the work: one converts any piece of text into a vector, the other measures how close two vectors are. If you completed Day 7, you have seen both before. If you are starting here, here is what each one does.

Converting text to a vector

Before your system can search by meaning, every piece of text must be converted into a list of numbers. That list is called an embedding. It places the text at a specific position in a large mathematical space, and the rule governing that space is: texts with similar meaning land close together.

async function getEmbedding(text) {
  const response = await openai.embeddings.create({
    model: "text-embedding-3-small",
    input: text,
    encoding_format: "float",
  });
  return response.data[0].embedding;
}

openai.embeddings.create calls the embedding endpoint. This is different from the chat endpoint. It does not produce a response. It converts your text into a vector: an array of 1,536 numbers that represents its meaning.

response.data[0].embedding reaches into the response and returns just that array. You will call this function once for every chunk during indexing, and once for the user's question at query time.

Measuring how close two meanings are

Once two pieces of text have been embedded, you can measure the distance between them mathematically.

function dotProduct(a, b) {
  return a.reduce((sum, val, i) => sum + val * b[i], 0);
}

This takes two vectors and returns a single number. Higher means the two vectors point in a similar direction, which means the two texts carry similar meaning. Lower means they diverge. You do not need to follow the arithmetic. What matters is the question it answers: are these two meanings close?

In a production system, a vector database runs this comparison automatically across millions of stored chunks in milliseconds. Here we are writing it out so the logic is fully visible.

Searching for the closest chunks

With those two primitives in place, the search function assembles them into something useful.

async function findSimilarChunks(query, documents, limit = 3) {
  const queryEmbedding = await getEmbedding(query);

  const results = [];
  for (const doc of documents) {
    const docEmbedding = await getEmbedding(doc.text);
    results.push({
      ...doc,
      score: dotProduct(queryEmbedding, docEmbedding),
    });
  }

  return results.sort((a, b) => b.score - a.score).slice(0, limit);
}

getEmbedding(query) converts the user's question into a vector.

The for loop then embeds each chunk and scores it against the query using dotProduct. Every chunk gets a similarity score.

results.sort(...) arranges all chunks from highest score to lowest. .slice(0, limit) returns only the top three.

What comes back is not the whole document. It is the three specific passages that carried the closest meaning to the question, each one still carrying its source label. Those are what the model will read.

Step 3: The prompt builder

This is the new function. It takes the user's question and the retrieved chunks and assembles them into a grounded prompt.

function buildPrompt(question, chunks) {
  const context = chunks
    .map((chunk, i) => `${i + 1}. [${chunk.source}]\n${chunk.text}`)
    .join("\n\n");

  return `You are a helpful assistant. Answer using only the context below.
If the answer is not present in the context, say: "I do not have enough information in the provided context."
Do not invent facts. Do not use outside knowledge.
Where relevant, mention which source the answer came from.

Context:
${context}

Question:
${question}`;
}

What this does:

chunks.map(...) formats each retrieved chunk with its source label and index. The context becomes readable to both the model and a human reading the logs.

The instructions in the prompt do three things: they limit the model to the retrieved context, they give the model a safe exit when the context is insufficient, and they ask for source awareness.

These are not nice-to-haves. They are the difference between a grounded system and a confident-sounding one that invents answers when retrieval falls short.

Step 4: The answer generator

async function generateAnswer(prompt) {
  const response = await openai.chat.completions.create({
    model: "gpt-4o-mini",
    messages: [{ role: "user", content: prompt }],
  });
  return response.choices[0].message.content;
}

This is the same chat completion pattern from Day 5, just wired into a function. The prompt it receives already contains the retrieved context. The model generates an answer from what it was given.

Step 5: From document to chunks

So far the code assumes the knowledge base already exists as a tidy array of text objects. But in a real system, you start with a document, and something has to turn that document into chunks before any embedding can happen.

Here is the simplest version of that transformation.

Why split on paragraphs?

This is one of the simpler chunking strategies, and a good one to start with for clean text and markdown files. A paragraph tends to hold one idea. Splitting at blank lines gives you pieces that are meaningful on their own, without being so large that the embedding becomes a blurred average of too many ideas.

In practice, chunking strategies vary quite a bit depending on the document type and the use case. A PDF with no blank lines needs a different approach. A long paragraph that covers five ideas may need to be split further. Production systems often split by token count with some overlap between adjacent chunks, so a sentence that falls at a boundary does not lose its context in either half. For code, splits happen at function boundaries. For legal documents, at clause boundaries.

Choosing the right strategy, including chunk size, overlap, and structure awareness, is explored properly on Day 15 when we build the full document pipeline.

The chunking function

import fs from "fs";

function chunkDocument(filePath, sourceName) {
  const text = fs.readFileSync(filePath, "utf8");

  return text
    .replace(/\r\n/g, "\n")
    .split("\n\n")
    .map((paragraph) => paragraph.trim())
    .filter((paragraph) => paragraph.length > 30)
    .map((paragraph) => ({ text: paragraph, source: sourceName }));
}

fs.readFileSync reads the file as a plain string.

.replace(/\r\n/g, "\n") normalizes line endings. Windows saves files with \r\n instead of \n, so a blank line on Windows is \r\n\r\n. Without this step, the split finds nothing and the entire file comes back as one chunk.

.split("\n\n") cuts it at every blank line. That is the paragraph boundary. One chunk per paragraph.

.map((p) => p.trim()) removes leading and trailing whitespace from each piece.

.filter((p) => p.length > 30) drops fragments that are too short to be meaningful: headings, dividers, blank lines that slipped through.

.map((p) => ({ text: p, source: sourceName })) wraps each chunk in the same shape the rest of the code already expects: a text field and a source label.

Create a document to chunk

Create a file called knowledge.md in the same folder as your script. Each paragraph must be separated by a blank line, not just a line break. That blank line is what the function splits on.

The context window is the fixed container of tokens the model can see during a request. Everything the model reasons from must fit inside it.

The model has no memory between API calls unless previous messages are sent again. Each call starts fresh with only what you provide.

Structured outputs make AI responses predictable objects that applications can use directly. Instead of parsing a paragraph, your code receives a field it can act on.

Embeddings convert text into vectors so similar meanings can be compared mathematically. Two sentences can be close in meaning even when they share no words.

RAG retrieves relevant chunks from an external knowledge source and passes them into the model as context before generating an answer. The model does not search. Your system searches, and the model reads what it finds.

Build the knowledge base from the file

const knowledgeBase = chunkDocument("./knowledge.md", "My Notes");

That is it. The rest of the code stays exactly the same. findSimilarChunks receives the same shape it always expected. The only thing that changed is where the chunks came from.

Step 6: Run it

console.log(`\n--- Knowledge Base ---`);
console.log(`Loaded ${knowledgeBase.length} chunks from document`);

const question = "How can AI answer from my own notes?";

const topChunks = await findSimilarChunks(question, knowledgeBase);
const prompt = buildPrompt(question, topChunks);
const answer = await generateAnswer(prompt);

console.log("\n--- Question ---");
console.log(question);

console.log("\n--- Retrieved Context ---");
topChunks.forEach((chunk, i) => {
  console.log(`\n  ${i + 1}. [${chunk.source}] (score: ${chunk.score.toFixed(4)})`);
  console.log(`     ${chunk.text}`);
});

console.log("\n--- Answer ---");
console.log(answer);
console.log("");

The first two lines log how many chunks were created from the document. If you see Loaded 1 chunks, it means the file does not have blank lines between paragraphs. Each paragraph in knowledge.md must be separated by an empty line, not just a newline, for the split to work correctly.

Run it:

node --env-file=.env rag-demo.js

The answer is not hallucinated. It came from retrieved context. The model read what your system found and answered from it.

You may notice the scores are lower than you might expect. The top chunk scores around 0.49, not 0.85 or 0.90. That is normal for a small, simple knowledge base.

Scores scale with the richness and volume of content. A knowledge base with hundreds of detailed paragraphs produces tighter, more confident matches because the embedding model has more material to find a close neighbor in.

With five short chunks, the best available match is still the right chunk, it just has less competition to distinguish itself from.

That is a working RAG system in under 80 lines.

The complete file

import "dotenv/config";
import fs from "fs";
import OpenAI from "openai";

const openai = new OpenAI({
  apiKey: process.env.OPENAI_API_KEY,
});

function chunkDocument(filePath, sourceName) {
  const text = fs.readFileSync(filePath, "utf8");

  return text
    .replace(/\r\n/g, "\n")
    .split("\n\n")
    .map((paragraph) => paragraph.trim())
    .filter((paragraph) => paragraph.length > 30)
    .map((paragraph) => ({ text: paragraph, source: sourceName }));
}

async function getEmbedding(text) {
  const response = await openai.embeddings.create({
    model: "text-embedding-3-small",
    input: text,
    encoding_format: "float",
  });
  return response.data[0].embedding;
}

function dotProduct(a, b) {
  return a.reduce((sum, val, i) => sum + val * b[i], 0);
}

async function findSimilarChunks(query, documents, limit = 3) {
  const queryEmbedding = await getEmbedding(query);
  const results = [];

  for (const doc of documents) {
    const docEmbedding = await getEmbedding(doc.text);
    results.push({ ...doc, score: dotProduct(queryEmbedding, docEmbedding) });
  }

  return results.sort((a, b) => b.score - a.score).slice(0, limit);
}

function buildPrompt(question, chunks) {
  const context = chunks
    .map((chunk, i) => `${i + 1}. [${chunk.source}]\n${chunk.text}`)
    .join("\n\n");

  return `You are a helpful assistant. Answer using only the context below.
If the answer is not present in the context, say: "I do not have enough information in the provided context."
Do not invent facts. Do not use outside knowledge.
Where relevant, mention which source the answer came from.

Context:
${context}

Question:
${question}`;
}

async function generateAnswer(prompt) {
  const response = await openai.chat.completions.create({
    model: "gpt-4o-mini",
    messages: [{ role: "user", content: prompt }],
  });
  return response.choices[0].message.content;
}

const knowledgeBase = chunkDocument("./knowledge.md", "My Notes");

console.log(`\n--- Knowledge Base ---`);
console.log(`Loaded ${knowledgeBase.length} chunks from document`);

const question = "How can AI answer from my own notes?";

const topChunks = await findSimilarChunks(question, knowledgeBase);
const prompt = buildPrompt(question, topChunks);
const answer = await generateAnswer(prompt);

console.log("\n--- Question ---");
console.log(question);

console.log("\n--- Retrieved Context ---");
topChunks.forEach((chunk, i) => {
  console.log(`\n  ${i + 1}. [${chunk.source}] (score: ${chunk.score.toFixed(4)})`);
  console.log(`     ${chunk.text}`);
});

console.log("\n--- Answer ---");
console.log(answer);
console.log("");

What chunkDocument makes possible

The knowledge base is not a text you wrote inside the code. chunkDocument reads any markdown file, splits it at paragraph boundaries, and produces exactly the same chunk shape the rest of the system expects.

Swap knowledge.md for any markdown file: your own notes, a lesson from this series, a support article, a product spec. The rest of the code does not change.

A note on what comes next

Paragraph splitting is the right starting point. It is simple, it respects natural idea boundaries, and it works well for clean text.

But real documents are messier. A PDF may have no blank lines at all. A long paragraph may contain five separate ideas. A very short paragraph may not carry enough context on its own. And splitting without any overlap means a thought that straddles two paragraphs gets cut in half.

On Day 15, we build the full document pipeline: proper chunk sizing, overlap between adjacent chunks, PDF extraction, and Supabase vector search for production-scale retrieval. This version gets you to a working system that you can reason about. That one gets you to something you could ship.

✦ RAG Does Not Retrain the Model

This distinction matters.

RAG does not change the model's weights. It does not permanently teach the model new facts. It does not say:

Here is a PDF. Please absorb this forever.

Instead, it says:

For this question, here are the most relevant passages. Use them while answering.

The model learns temporarily, inside the current context. Think of a person who has not memorized a book but can still answer well when the correct page is open in front of them.

That is why the title of this article says "learns from things it was never trained on." It is not learning in the training sense. It is learning in the moment.

This distinction separates three different approaches:

For most AI applications that MERN developers build, RAG comes first. Most real products need access to external, changing knowledge. A company knowledge base changes monthly. Fine-tuning it monthly is expensive and unnecessary. RAG handles it naturally.

✦ Where RAG Breaks

A mature builder does not only ask: how does this work?

They also ask: where does this fail?

RAG introduces new failure modes that do not exist in a simple chat application. Understanding them is part of being ready to ship a system that uses them.

Bad chunking

If chunks are too large, they contain too many ideas. The embedding becomes a blurred average of unrelated content. If chunks are too small, they may lose context. A single line may not carry enough meaning on its own.

Good RAG begins before the model is called. It begins when you decide how knowledge should be split.

Wrong retrieval

Sometimes the system retrieves something related but not sufficient.

A user asks: Can I cancel after renewal and still get a refund?

The system retrieves: How to cancel your subscription.

That is related. It may not answer the refund question. The answer needs refund policy, renewal terms, and cancellation details. A simple similarity search may retrieve only one part of the picture.

Missing context

If the answer is not in the knowledge base, retrieval cannot find it. The model should say:

I do not have enough information in the provided context.

But many systems do not enforce this. So the model tries to fill the gap from general training. That is where hallucination returns. The buildPrompt function we wrote above explicitly adds this instruction. It matters more than it looks.

Too much context

It is tempting to retrieve ten or twenty chunks just to be safe. But more context is not always better. Too many chunks can confuse the model. Unrelated passages may blend into the answer. The most relevant passage may get buried.

Retrieval should be selective. A good system retrieves enough, not everything.

Stale knowledge

Documents change. Policies update. Prices change. APIs evolve. If your embeddings were created three months ago but the document changed yesterday, your vector database still contains old meaning.

A RAG system needs a re-indexing strategy. When source content changes, the stored chunks and embeddings must be updated. Otherwise, your system may confidently answer from stale knowledge.

No permission boundaries

This is critical in real organizations.

Suppose a company has HR documents, finance documents, engineering documents, and executive documents. Not every user should access every document.

A RAG system must never retrieve context the user is not allowed to see. Security must happen before generation. The moment sensitive context enters the prompt, the damage is already done.

Permissions belong in the retrieval layer, not as an afterthought.

No evaluation

A RAG system may look correct in a demo and fail in production. Teams read the final answer and assume everything is working. But the retrieval layer may be quietly broken.

Test the full pipeline. Ask not only whether the answer sounds right. Ask whether the right chunks were retrieved, whether irrelevant chunks were ignored, whether the model refuses when context is missing, and whether sources are cited correctly.

RAG needs evaluation from the beginning, not someday.

✦ Why Not Just Paste Everything?

A beginner may ask: why retrieve at all? Why not paste the entire document into the prompt?

For very small knowledge bases, you can. If your context is a few short paragraphs, paste them. RAG may be unnecessary.

But real systems grow.

A company knowledge base may have 2,000 support articles. A legal document may have 300 pages. A course platform may have hundreds of lessons. A codebase may have thousands of files.

Even if the model supports a large context window, sending everything causes problems:

• it costs more per request
• it is slower to process
• it adds noise that can bury the important content
• it may reduce answer quality for information in the middle of a long context
• it may exceed token limits entirely

The goal is not to give the model more context. The goal is to give it better context.

A bigger context window lets you carry more. RAG helps you decide what belongs inside.

These are not the same capability. A large context window is useful. But it does not replace the discipline of choosing what enters the prompt. RAG is that discipline.

✦ How the Pieces Connect

By Day 8, the concepts from this series are no longer separate ideas. They are layers.

Prompting taught us that the way we ask matters. Structured outputs taught us that applications need predictable shape. Embeddings taught us that meaning can be searched. RAG brings these together into a single design:

Search by meaning. Put the right context into the prompt. Ask the model to answer in a useful shape.

That is the beginning of real AI application design.

✦ What This Opens Up for Builders

Once you understand RAG, your idea of AI applications changes.

You stop thinking only in terms of prompts. You start thinking in systems.

A resume reviewer can retrieve the job description, compare it with the resume, and produce grounded improvement suggestions.

A learning coach can retrieve previous mistakes, weak concepts, and relevant lessons before giving the next task.

A support assistant can retrieve the latest policy before answering a customer.

A document assistant can read a PDF, chunk it, embed it, and answer from the most relevant sections.

A code assistant can retrieve the right files before explaining how a feature works.

A research assistant can retrieve papers, summarize them, compare them, and cite where each idea came from.

A company assistant can answer from internal documents without exposing all data to everyone.

None of these are "chat with PDF." RAG is not a demo feature. It is a pattern. The same pattern appears in every serious AI application:

Find the right knowledge. Give it to the model. Ask the model to answer from it.

The chat window is the surface. Underneath, there is a retrieval system.

✦ Try It Yourself

Use the code from this lesson and test these questions against the knowledge base:

How can AI answer from my own files?
Why does the model forget previous messages?
How do I make the AI response usable in my frontend?
What is the best mutual fund to invest in today?

The first three should retrieve relevant Tapovan-related chunks and produce grounded answers. The fourth should not be answerable from the knowledge base.

When you run the fourth question, you will notice something: chunks are still returned, with scores around 0.15 or lower. That is expected. findSimilarChunks is a ranking function, not a filter. It always returns the top 3, regardless of how low the scores are. Given any question, it finds the least-wrong matches in your knowledge base and returns them.

The model still correctly refused because the prompt instructed it to. But in a production system, you would add a minimum score threshold. If the top chunk scores below say 0.3, skip the model call entirely and return the refusal directly. That saves an API call and makes the system more honest about what it does not know.

When the fourth question triggers the refusal message, pause for a moment. That is not a failure. That is the system working correctly. A good RAG system does not answer everything. It answers what it has grounding for.

Then extend the knowledge base. Add your own notes. Add content from any of the previous seven lessons. Ask questions you would actually need answered about something you are building.

When the right chunk rises to the top and shapes the answer, you will feel the difference between a general chatbot and a system that knows your material.

✦ Takeaway Summary

✦ Learn More

• Anthropic: Building with RAG: how Anthropic recommends structuring retrieval and context for Claude, with practical guidance on grounding and prompt design
• OpenAI Retrieval and Vector Stores documentation: the official guide to building retrieval systems with OpenAI's APIs
• Supabase Vector Search: how to store embeddings and run similarity searches using pgvector in Supabase, which is what the projects in this series use
• Google Cloud: What is Retrieval-Augmented Generation?: a clear conceptual overview of RAG from a production perspective
• OpenAI Cookbook: working examples of embeddings, semantic search, and RAG pipelines