Guide
RAG vs Fine-Tuning for Startups
First-hand comparison from Agentic Academy Labs: when RAG beats fine-tuning for changing docs, what week 1 looks like, and the failure modes we see in Indian startups.
Published 2026-08-08 · Updated 2026-08-14
At Agentic Academy Labs in Sikar, most founders arrive asking which model to fine-tune. After internships and client work on Next.js, Bun, PostgreSQL, and Flutter products, our default is different: start with retrieval-augmented generation (RAG) whenever the answers live in documents that still change. Fine-tuning is a later tool for style, routing, and classification, not a substitute for a knowledge base you can update on Friday afternoon.
What RAG and fine-tuning actually do
RAG fetches relevant chunks from your store (Postgres with pgvector, or a dedicated index) and stuffs them into the prompt at query time. The model does not permanently learn your refund policy; it reads the latest indexed copy. Fine-tuning updates weights on labeled examples so the model repeats a format or tone. It does not automatically stay current when legal rewrites a PDF. Mixing them later is common. Starting with fine-tuning for a moving handbook is how we have watched teams retrain every sprint.
Side-by-side for a startup product
| Factor | RAG first | Fine-tuning first |
|---|---|---|
| Time to a demo on real docs | Days to 2 weeks | 2-6+ weeks plus labeling |
| Policy or SKU change on Friday | Re-index; answers move | Retrain or the model lies |
| Citations for legal or ops | Natural if you log sources | Weak unless you still retrieve |
| Tone and JSON shape | Prompt plus few-shot | Strong once you have examples |
| Data you need in week 1 | Readable documents and eval questions | Hundreds of labeled pairs |
| Ongoing cost driver | Retrieval plus tokens | Training runs plus serving |
| Best first use case | Internal Q&A, support, sales enablement | Classifier, extractor, fixed format |
What we do in week 1
- Inventory the real corpus: Notion exports, Google Drive PDFs, Slack pins, and the one spreadsheet nobody mentioned. Interns in Sikar usually find duplicate versions in the first afternoon.
- Write 25-40 evaluation questions with expected source files. If the founder cannot name a wrong answer they fear, we are not ready to pick a model.
- Chunk a sample (headings, tables, and scanned pages separately) and measure hit rate before we talk about fine-tunes.
- Stand up a thin Next.js admin on Bun so a non-engineer can upload a doc and see citations. Flutter clients, if they exist, call the same API.
- Log every turn: query, retrieved ids, tokens, latency. Without that log, fine-tuning later is guesswork.
When we recommend RAG first
- Handbooks, product specs, and pricing sheets change weekly or monthly.
- Someone in compliance will ask which paragraph the answer came from.
- You need a founder or investor demo inside a month, not a training pipeline.
- The team can maintain Postgres and an upload UI, but not labeling ops yet.
- You already failed a chatbot that sounded confident on last quarter's policy. See the Bangalore fintech RAG case study.
When fine-tuning is worth the wait
- You have thousands of labeled tickets or form fields, not a folder of PDFs.
- The bottleneck is consistent JSON or a short classification, not facts that move.
- Long RAG contexts are blowing token cost and p95 latency on your eval set.
- You already have an evaluation harness and someone who owns retraining.
Failure modes we have already paid for
- Fine-tuning on chat logs that contain outdated prices. The model memorizes the wrong number and retrieval never gets a chance to correct it.
- RAG over scanned PDFs with no OCR. Retrieval looks healthy; every answer is empty confidence.
- No tenant filter on pgvector. Two internships taught us to put org_id in the query before the first demo.
- Evaluating with trivia instead of the questions users actually ask in Slack. Quality scores look fine; adoption does not.
Sequence we ship: RAG on real documents with citations and an eval set, then measure faithfulness, refusals, latency, and cost per successful answer (not per 1K tokens), then add a small classifier or fine-tune only for routing, extraction, or tone that prompts cannot stabilize. Read LangChain vs a custom RAG pipeline if you are choosing scaffolding. Agentic Academy Labs builds this path for Indian startups from Sikar and remote. Start with custom AI development, compare AI MVP cost and timeline in India, or schedule a call.
Frequently asked questions
- Should a startup start with RAG or fine-tuning?
- Start with RAG when documents change and answers must be cited. Fine-tune later for narrow format or classification tasks. That is the default we use at Agentic Academy Labs.
- How long does a RAG MVP take?
- A thin slice on real docs can ship in two to four weeks. Production with admin, auth, and eval often takes six to twelve weeks after discovery. See custom AI MVP cost and timeline.
- Can RAG and fine-tuning be combined?
- Yes. We often keep RAG for knowledge and add a small fine-tune or classifier for routing and output shape. Do not fine-tune facts that still live in Drive.
- What eval questions should we write in week 1?
- Write questions that would embarrass you in front of a customer: conflicting policy versions, missing tables, and 'I do not know' cases. Include the expected source filename next to each question.