What Is RAG and How Do You Build a Free RAG Stack for Document Q&A?
RAG (Retrieval-Augmented Generation) is a pattern where you retrieve relevant chunks of your own documents and pass them to an LLM as context, so the model answers from your data instead of its training memory. You can build a working document Q&A stack entirely from free-tier tools: a free LLM API for generation, an open embedding model, and a local or free-tier vector store. This guide covers the mechanism, the components, and a minimal build path — plus where these pipelines usually break.
The core mechanism in one pass
A plain LLM call sends your question and nothing else. A RAG call does two things first:
- Retrieve — search your document collection for the passages most semantically similar to the question.
- Augment — insert those passages into the prompt as context.
The model then generates an answer grounded in the retrieved text. This is why RAG is often described as "giving the model an open-book exam" rather than testing what it memorized.
RAG vs. fine-tuning vs. plain prompting
| Approach | What changes | Best when | Main cost |
|---|---|---|---|
| Plain prompting | Nothing — model uses training knowledge | General questions, no private data | Can't answer about your documents; hallucinates |
| RAG | Adds retrieved context at query time | Your data changes often; you need citations; data is too large for a prompt | Retrieval quality becomes the bottleneck |
| Fine-tuning | Adjusts model weights on your examples | You need a consistent style, format, or domain behavior | Expensive, slow to update, doesn't add fresh facts reliably |
For document Q&A over a knowledge base that changes, RAG is usually the right first move. Fine-tuning teaches how to respond; RAG supplies what to respond with.
The components of a minimal RAG pipeline
Every RAG stack, free or paid, has the same six stages. The free-tool directory at freeaitoolslist.vercel.app organizes its RAG Stack category (12 tools) around exactly these pieces.
1. Document loading
Pull raw text out of your sources — PDFs, Markdown, HTML, Notion exports, database rows. This is usually a small script, not a service.
2. Chunking
Split documents into passages small enough to embed and retrieve precisely. Typical starting point: 300–800 tokens per chunk with some overlap (e.g., 10–20%) so sentences aren't cut mid-thought.
3. Embedding
Convert each chunk into a vector using an embedding model. Open-weight embedding models can run locally for free; hosted embedding endpoints are also common.
4. Vector store
Store the vectors plus their source text and metadata. Options range from an in-process library (no server) to a hosted free-tier vector database. The directory lists vector database as a tracked category.
5. Retrieval
At query time, embed the question the same way and find the top-k nearest chunks. This is the step that most determines answer quality.
6. Generation
Send the retrieved chunks plus the question to an LLM. Free-tier LLM APIs from the directory's LLM APIs category (20 tools) fit here — for example:
- OpenRouter — unified API across 29 free models, 20 RPM and 50 requests/day (1,000/day with $10+ credits), no credit card required.
- Groq — 1,000–14,400 requests/day depending on model, no credit card required.
- Google AI Studio — free Gemini API, roughly 250 requests/day on Tier 1 for most models, up to 1,500 RPD for Gemini 3 Flash.
- Cerebras — 1.5M tokens/day, 30 req/min, 8,192 context.
Check each provider's current limits before committing, since free tiers change.
A minimal build path
Here's the shortest route from a folder of documents to a working Q&A loop. The exact libraries are your choice; the sequence is what matters.
- Load your documents into plain text and keep a
sourcefield on each one. - Chunk each document into ~500-token passages with overlap. Store
{text, source, chunk_id}. - Embed every chunk with an embedding model and write the vectors into your vector store alongside the metadata.
- Query: embed the user's question, retrieve the top 3–5 chunks by similarity.
- Prompt: build a message like "Answer using only the context below. If the answer isn't present, say so. Context: … Question: …"
- Generate with a free-tier LLM API and return the answer plus the source chunks so the user can verify.
Verify it works: ask a question whose answer exists verbatim in one document, and a second question whose answer does not exist anywhere. A correct build answers the first with the right source and declines the second instead of inventing one.
Where free RAG stacks break
- Chunking too coarse or too fine. Big chunks dilute the signal and blow your context budget; tiny chunks lose the surrounding meaning. Tune this first when answers feel vague.
- Poor retrieval recall. If the right passage never appears in the top-k, no LLM can fix it. Try more chunks, a better embedding model, or hybrid keyword + vector search.
- Context overflow. Free tiers have hard context caps — Cerebras lists 8,192 context, for instance. Stuffing too many chunks will truncate or error.
- Hallucination despite context. The model answers from memory when retrieval is weak. An explicit "answer only from context" instruction plus returning sources reduces this.
- Stale index. If documents change, re-embed the changed chunks. A RAG system is only as current as its vector store.
Choosing your free components
Match tools to your constraints rather than chasing the biggest name:
- Data must stay local → local embedding model + in-process vector store + a locally run open-weight model.
- Fastest to prototype → hosted free-tier embedding + a free-tier vector database + a free LLM API like Groq or Google AI Studio.
- Highest request volume → compare daily request and token caps across providers (Groq's 1,000–14,400 req/day and Cerebras's 1.5M tokens/day are the generous end of the directory's listings).
The directory's Recommended Stacks section groups these into ready-made combinations if you'd rather not assemble each piece yourself.