Case Study: RAG Assistant for a Bangalore Fintech

Agentic Academy Labs shipped a document-grounded RAG assistant for a Bangalore Series A fintech in six weeks: Next.js admin, Slack for the team, and an evaluation set before investor demo day. Work ran from our Sikar lab with the client's ops and compliance in Bangalore. The stack was Next.js, Bun APIs, PostgreSQL with a vector index, and production LLM providers behind an adapter. Flutter was out of scope; the mobile app stayed on the existing backend and did not call the assistant.

Client context

• Location: Bangalore, Karnataka. Delivery: Agentic Academy Labs, Sikar, with daily standups over video.
• Stage: Series A fintech preparing an investor demo on proprietary policy and product docs.
• Problem: Search across Drive, Notion, and email attachments was tribal knowledge. New hires pinged founders.
• Constraint: Six calendar weeks to a live demo on real documents, not a slide with mock answers.

Week-by-week delivery

1. Week 1: inventory, OCR on scans, 40 eval questions with expected filenames. We refused to pick a model until the questions existed.
2. Week 2: chunking experiments and a Next.js admin so ops could upload without waiting on engineering. Postgres stored files, chunks, and citations.
3. Week 3: Slack bot plus web chat against the same Bun API. Tenant and role filters on retrieval from day one.
4. Week 4: eval failures on tables and stale PDFs. Re-chunked, killed duplicate versions, added 'I do not know' refusals.
5. Week 5: Slack thread context, screenshot refusal, and a review queue in admin for answers compliance flagged.
6. Week 6: monitoring, token caps, dress rehearsal, demo day. Handoff notes for the client's engineer.

What broke: document quality

Model choice was not the bottleneck in week 1. Document quality was. Drive held three generations of the same credit-policy PDF. Notion had the current version; a scanned KYC SOP had no text layer until OCR. Tables in product-limit PDFs split across chunks so retrieval returned a header without the rupee cap. Interns labeled each file as current, superseded, or unreadable. Until that pass finished, the assistant cited last year's fee schedule with perfect confidence. We now treat 'clean the corpus' as a delivery milestone, not a nice-to-have.

What broke: Slack vs web chat

Adoption went to Slack. The investor demo needed the web chat. Those two surfaces lied in different ways. Slack users asked follow-ups in a thread while the bot only saw the last message, so 'what about NRI accounts?' looked like a new question with no policy in context. People pasted screenshots of emails; the bot guessed. Web chat had session memory and a citation panel Slack did not. Demo day looked polished on a laptop and messy in the channel until we added thread history, a short-circuit for image-only messages, and a link back to the admin test page for anything that needed a source list. If we did this again, Slack thread context would ship in week 3, not week 5.

Eval set (the questions that actually moved quality)

• Cite the current cooling-off clause and name the file. Wrong file = fail, even if the prose sounds right.
• What is the fee cap in the latest product table? (This failed until table chunking was fixed.)
• Can we do X for a non-resident client? Must refuse if the corpus is silent.
• Two policies disagree: which one wins, and can you show both sources?
• Repeat the user's last Slack follow-up without the parent message (regression for thread context).

Outcomes

We reuse three lessons: RAG first (fine-tuning would have baked in the stale fee PDF; see RAG vs fine-tuning), admin for non-engineers is part of the product, and Slack wins adoption while web chat wins citations. Interns were effective on OCR QA and eval scoring; they were not on-call for demo day. This pattern is how we staff custom AI development. If you are choosing scaffolding, read LangChain vs custom RAG. For calendar and budget shape (not a fake rupee quote), see AI MVP cost and timeline in India. Schedule a call if you want the same six-week slice.