Under the hood
Architecture
A small retrieval-augmented chat over one public-domain book. Every question flows through five stages: a trimmed client payload, a cached API route, hybrid retrieval over a pre-computed corpus, a three-provider LLM cascade, and a grounded response with citations. All numbers below are from the committed source code.
1Client
- Framework
- Next.js 16 · App Router · Tailwind v4
- Components
- ChatPanel · VisitCounter · SiteHeader
- Message trim
- Last 8 turns sent to server
- Auto-retry
- Up to 3 attempts, exponential backoff
- Visit counter
- Abacus API, no signup required
2API Layer
- Routes
- /api/chat · /api/visits
- Message handling
- Graceful truncation to last 8
- Char limit
- 2000 per message
- Response cache
- 5-min TTL, 64-entry LRU
- Dynamic rendering
- connection() from next/server
3Retrieval
- Embedding model
- sentence-transformers/static-retrieval-mrl-en-v1
- Dimensions
- 128 (Matryoshka truncation)
- Corpus
- 119 chunks across 12 chapters
- Max chunk size
- 1400 chars, paragraph-aware
- Hybrid scoring
- cosine + 0.4 × word coverage
- Refusal gate
- Score floor 0.4 · cosine-no-overlap 0.62
- Pipeline
- BERT tokenize → mean-pool → L2-normalize
- Storage
- Pre-computed .f16 + .json, committed to repo
4LLM Cascade
- Primary
- DeepSeek (deepseek-flash, $0.14/M input)
- Secondary
- Groq (llama-3.3-70b-versatile, free)
- Fallback
- OpenRouter — 4 trios, 12 free models, 8 providers
- Backoff
- 2s → 4s → 8s + jitter, Retry-After aware
- Timeout
- 90s per request
5Response
- Refusal label
- NOT IN THE BOOK
- Citations
- Chapter + title + excerpt per passage
- Caching
- Refusals and answers both cached
- Delivery
- JSON to client, rendered with passage links
Step-by-step walkthrough
1. Corpus
The ingest script downloads Project Gutenberg eBook #11 and keeps the twelve chapters. The Gutenberg header, footer, and license are not stored.
2. Chunks
Paragraphs are unwrapped and packed to about 1,400 characters, still inside a chapter. Each chunk has an id such as ch07-p06. The corpus is 119 chunks.
3. Local embeddings
Token vectors come from sentence-transformers/static-retrieval-mrl-en-v1, mean-pooled with the google-bert/bert-base-uncased WordPiece tokenizer, then cut to the first 128 Matryoshka dimensions and L2-normalized. Ingest writes them once. The server does not call an embedding API and does not load a transformer.
4. Retrieve
A question is embedded the same way. Chunks are ranked by cosine plus 0.4 times the share of the question’s content words found in the chunk, including its chapter title. The lexical term keeps names like “Tea-Party” from being outranked by a vague semantic neighbor. The top 5 chunks are the context.
5. Refuse
If the best hybrid score is below 0.4, or the passages share no content words with the question and the cosine is below 0.62, the route does not call the model. It returns a plain refusal. If the model itself answers NOT_IN_BOOK, the same refusal is shown and the citations are dropped.
6. Generate
Otherwise the passages and the question go to a multi-provider LLM cascade. DeepSeek is tried first (deepseek-flash via https://api.deepseek.com/chat/completions); if it is unavailable, Groq is tried next (llama-3.3-70b-versatile via https://api.groq.com/openai/v1/chat/completions); if both are busy, OpenRouter's free-tier rotation kicks in as the last resort. All three are OpenAI-compatible. The reply is prose. Citations are the retrieved chunks — id, chapter, and a short quote — not spans invented by the model.
Provider cascade
- Chat — Provider Cascade
- The app tries three OpenAI-compatible providers in order:
- DeepSeek —
deepseek-flashviahttps://api.deepseek.com/chat/completions. Paid API but extremely cheap ($0.14/M input, $0.28/M output — ~$0.0002 per answer). No rate-limit pressure; the primary provider. - Groq —
llama-3.3-70b-versatileviahttps://api.groq.com/openai/v1/chat/completions. Free tier: 30 RPM / 1,000 RPD. Ultra-fast LPU inference; the secondary fallback. - OpenRouter — 4 trios of free models via
https://openrouter.ai/api/v1/chat/completions. The last resort. One request triesnvidia/nemotron-3.5-lightning:free, then liquid/lfm-2.5-2.6b:free and recursive/flint-v1.0:free. If that free trio is rate-limited or returns nothing, the server waits 2s and tries a second free-only trio (thinkingmachines/inkling:free + nvidia/nemotron-3-ultra-550b-a55b:free, xingchen-agi/xing4.0-29b-a4b:free), then a third (google/gemma-4-31b-it:free + dots-studio/dots-3-note-preview:free, cohere/north-mini-code:free), then a fourth (thinkingmachines/inkling-small:free + google/gemma-4-26b-a4b-it:free, nvidia/nemotron-3-nano-omni-30b-a3b-reasoning:free), with the backoff doubling each round. The server also respects theRetry-Afterheader. The client refuses any OpenRouter model id that does not end in:free.
choices[0].message.content. A separate reasoning trace is ignored unless content is empty, in which case the server looks for a final-answer marker inside the reasoning field. If the text itself opens with a thinking-process preamble, only the final answer is shown.The OpenRouter free tier retires models regularly, so a retired id returns 404. That is treated as a model-level failure and the request falls through to the next trio instead of failing outright. Thenpm run check:modelsscript compares every configured OpenRouter id against the live catalog and fails when one has been retired.The client also retries automatically: on a 502provider_error, it waits and sends the request again (up to 3 retries with 2s/4s/8s backoff), showing a “Retrying…” indicator instead of an error.Identical questions within a 5-minute window are served from an in-memory cache, so repeat questions do not hit any provider at all. - DeepSeek —
- Embeddings
sentence-transformers/static-retrieval-mrl-en-v1Static retrieval model: a table lookup and a mean pool, about as accurate as a small transformer encoder and cheap enough to run inside a serverless function. The full 1,024-dimensional matrix is downloaded only bynpm run ingest. The repo ships the first 128 dimensions as float16, plusdata/vocab.txtand the chunk index. Query embedding is that same table, so a cold start does not download or initialize ONNX. Retrieval does not call any external API.
What the server stores
data/alice-index.json— chunk id, chapter, title, textdata/chunk-embeddings.f16— one normalized vector per chunkdata/token-embeddings.f16— the truncated token tabledata/vocab.txt— bert-base-uncased vocabulary
Source text: https://www.gutenberg.org/cache/epub/11/pg11.txt. Nothing about a visitor is written to disk.
Limits
English only. One book. The model can still miss a passage that retrieval found, or refuse with NOT_IN_BOOK when the passages do not actually answer. A question that shares a word with the book, such as “queen,” may be sent to the model; the model is instructed to refuse if the passages do not contain the answer.