The story
Ask why a piece of old code works the way it does. Backstory answers from the project's own history, shows where each answer came from, and says so when it doesn't know.
Making refusal honest: retrieval that is mostly right is easy, but guaranteeing every sentence traces to a real citation needed a gate before generation, a citation check after it, and an eval suite (recall@5, citation correctness) to prove the rule held.
How it’s built
Inside Backstory
System architecture
- Next.js app
- → Clerk multi-tenant auth (sign-in)
- → FastAPI (question)
- Next.js app
- Clerk multi-tenant auth
- FastAPI
- → Redis queue (ingest jobs)
- → Hybrid retrieval
- Redis queue
- → Celery ingestion workers
- Hybrid retrieval
- → Postgres + pgvector (search)
- → Evidence gate
- Redis queue
- Celery ingestion workers
- → Whisper interview transcription (interviews)
- → Postgres + pgvector (chunks + embeddings)
- Evidence gate
- → Groq LLM generation (enough evidence)
- Celery ingestion workers
- Whisper interview transcription
- Postgres + pgvector
- Groq LLM generation
- → Citation check or honest refusal (draft answer)
- Citation check or honest refusal
Engineering decisions
Decision 01
Two gates, not one prompt
Evidence is checked before the LLM is called (is there enough to answer at all?) and every sentence is checked after (does it trace to a real citation?). Hallucinations are stopped by the pipeline, not by asking the model nicely.
Decision 02
Refusal is a feature
When the evidence isn't there, the answer is "I don't have this", shown as a real answer with a next step. A confident wrong answer about legacy code costs more than an honest gap.
Decision 03
Hybrid retrieval
Vector search on pgvector finds meaning; keyword search catches exact function names and ticket IDs that embeddings blur. Code needs both.
Decision 04
Slow work off the request path
Ingestion and Whisper transcription run as Celery workers on a Redis queue, so the chat stays fast while large repos and interviews are processed in the background.
Decision 05
Quality is a release gate
A pytest eval suite measures retrieval recall and citation correctness on every release. If recall@5 drops below 80% or citations below 90%, it doesn't ship.