LangChain vs Custom RAG Pipeline

Interns at Agentic Academy Labs can stand up a LangChain demo in a day. That is the point of the framework. The Bangalore fintech assistant, and most customer-facing work we ship from Sikar, did not stay on default chains. We prototype fast, freeze a retrieve-assemble-generate-log interface, then keep or replace internals based on eval, tenancy, and cost. LangChain is not the villain. Unowned abstraction is.

Comparison from delivery, not Twitter

What week 1 looks like either way

1. Write eval questions before installing a package. Framework choice does not invent quality.
2. Index a real folder, not the README. Measure recall on those questions.
3. Log query, chunk ids, tokens, and latency to Postgres. If the framework hides that, wrap it.
4. Put org_id on retrieval. We have failed this on a LangChain spike and on a hand-rolled path; the bug is human.
5. Decide the service interface: retrieve, assemble, generate, log. Names stay stable when we rip out a chain.

When we keep the framework

• Internal tool, one tenant, founder is the only user this quarter.
• The intern project is a learning spike, not a billed SLA.
• We need loaders and splitters this week and will throw the code away after discovery.

When we extract a custom pipeline

• Answers go to paying users or to Slack in a regulated fintech. See the RAG fintech case study.
• Eval harnesses fight the chain more than they help (hidden retries, duplicate embeddings).
• We need Next.js admin to show the exact chunks, not a chain dump.
• Token cost per successful answer is unexplainable because tools nested extra calls.

Failure modes we have hit

• Default text splitters cutting tables in half. Custom chunking around headings and rows fixed recall more than a new model.
• Memory classes that stored the wrong tenant's chat. We now persist threads in Postgres ourselves.
• Upgrading a patch version that changed retriever defaults. Pin versions and re-run the golden set on every bump.
• Copy-pasting a cookbook agent into production. That is an agentic workflow vs chatbot problem, not a RAG problem.

Practical path we recommend

1. Prototype retrieval and prompts with whatever gets you to eval fastest.
2. Freeze the four-step interface and the log schema.
3. Replace internals only where latency, tenancy, or cost demand it. Do not rewrite for taste.
4. Keep RAG for changing docs; do not fine-tune facts. RAG vs fine-tuning.

Agentic Academy Labs ships both shapes under custom AI development, usually on Next.js, Bun, and PostgreSQL. Schedule a call if you want a review of an existing chain before it becomes the product.