Alicegrounded chat

Under the hood

Architecture

A small retrieval-augmented chat over one public-domain book. Every question flows through five stages: a trimmed client payload, a cached API route, hybrid retrieval over a pre-computed corpus, a three-provider LLM cascade, and a grounded response with citations. All numbers below are from the committed source code.

Alice RAG — Full Request Flowuser question → grounded answer (or refusal)1 · CLIENTNext.js 16 · App RouterChatPanel · VisitCounterTrims to last 8 turnsAuto-retry ×3, backoff2 · API LAYER/api/chat · /api/visits5-min LRU cache (64)Truncates to last 8 msgs2000-char limit3 · RETRIEVAL119 chunks · 12 chapters128-dim Matryoshka vectorscosine + 0.4 × word overlapScore floor 0.4 · gate 0.62embedText(query)1. BERT tokenize2. Mean-pool token vectors3. L2-normalize → 128-dim query vectorstatic-retrieval-mrl-en-v1 · pre-computed corpus: .f16 + .jsonassess(query)hybrid < 0.4 → refuse ("NOT IN THE BOOK")no overlap + cosine < 0.62 → refuseotherwise → top-5 passages + citations4 · LLM CASCADE① DeepSeek $0.14/M② Groq (free, 30 RPM)③ OpenRouter 4-trio12 models · 8 providers5 · RESPONSEAnswer + citationsor NOT IN THE BOOKCached 5-min TTLJSON → client renderbackoff: 2s → 4s → 8s + jitterRetry-After aware · 90s timeoutBuilt by Rick Low, 2026passage citations:id · chapter · excerptcorpus: Alice's Adventures in Wonderland · Project Gutenberg #11 · public domain

1Client

Framework
Next.js 16 · App Router · Tailwind v4
Components
ChatPanel · VisitCounter · SiteHeader
Message trim
Last 8 turns sent to server
Auto-retry
Up to 3 attempts, exponential backoff
Visit counter
Abacus API, no signup required

2API Layer

Routes
/api/chat · /api/visits
Message handling
Graceful truncation to last 8
Char limit
2000 per message
Response cache
5-min TTL, 64-entry LRU
Dynamic rendering
connection() from next/server

3Retrieval

Embedding model
sentence-transformers/static-retrieval-mrl-en-v1
Dimensions
128 (Matryoshka truncation)
Corpus
119 chunks across 12 chapters
Max chunk size
1400 chars, paragraph-aware
Hybrid scoring
cosine + 0.4 × word coverage
Refusal gate
Score floor 0.4 · cosine-no-overlap 0.62
Pipeline
BERT tokenize → mean-pool → L2-normalize
Storage
Pre-computed .f16 + .json, committed to repo

4LLM Cascade

Primary
DeepSeek (deepseek-flash, $0.14/M input)
Secondary
Groq (llama-3.3-70b-versatile, free)
Fallback
OpenRouter — 4 trios, 12 free models, 8 providers
Backoff
2s → 4s → 8s + jitter, Retry-After aware
Timeout
90s per request

5Response

Refusal label
NOT IN THE BOOK
Citations
Chapter + title + excerpt per passage
Caching
Refusals and answers both cached
Delivery
JSON to client, rendered with passage links

Step-by-step walkthrough

  1. 1. Corpus

    The ingest script downloads Project Gutenberg eBook #11 and keeps the twelve chapters. The Gutenberg header, footer, and license are not stored.

  2. 2. Chunks

    Paragraphs are unwrapped and packed to about 1,400 characters, still inside a chapter. Each chunk has an id such as ch07-p06. The corpus is 119 chunks.

  3. 3. Local embeddings

    Token vectors come from sentence-transformers/static-retrieval-mrl-en-v1, mean-pooled with the google-bert/bert-base-uncased WordPiece tokenizer, then cut to the first 128 Matryoshka dimensions and L2-normalized. Ingest writes them once. The server does not call an embedding API and does not load a transformer.

  4. 4. Retrieve

    A question is embedded the same way. Chunks are ranked by cosine plus 0.4 times the share of the question’s content words found in the chunk, including its chapter title. The lexical term keeps names like “Tea-Party” from being outranked by a vague semantic neighbor. The top 5 chunks are the context.

  5. 5. Refuse

    If the best hybrid score is below 0.4, or the passages share no content words with the question and the cosine is below 0.62, the route does not call the model. It returns a plain refusal. If the model itself answers NOT_IN_BOOK, the same refusal is shown and the citations are dropped.

  6. 6. Generate

    Otherwise the passages and the question go to a multi-provider LLM cascade. DeepSeek is tried first (deepseek-flash via https://api.deepseek.com/chat/completions); if it is unavailable, Groq is tried next (llama-3.3-70b-versatile via https://api.groq.com/openai/v1/chat/completions); if both are busy, OpenRouter's free-tier rotation kicks in as the last resort. All three are OpenAI-compatible. The reply is prose. Citations are the retrieved chunks — id, chapter, and a short quote — not spans invented by the model.

Provider cascade

Chat — Provider Cascade
The app tries three OpenAI-compatible providers in order:
  1. DeepSeek — deepseek-flash via https://api.deepseek.com/chat/completions. Paid API but extremely cheap ($0.14/M input, $0.28/M output — ~$0.0002 per answer). No rate-limit pressure; the primary provider.
  2. Groq — llama-3.3-70b-versatile via https://api.groq.com/openai/v1/chat/completions. Free tier: 30 RPM / 1,000 RPD. Ultra-fast LPU inference; the secondary fallback.
  3. OpenRouter — 4 trios of free models via https://openrouter.ai/api/v1/chat/completions. The last resort. One request tries nvidia/nemotron-3.5-lightning:free, then liquid/lfm-2.5-2.6b:free and recursive/flint-v1.0:free. If that free trio is rate-limited or returns nothing, the server waits 2s and tries a second free-only trio (thinkingmachines/inkling:free + nvidia/nemotron-3-ultra-550b-a55b:free, xingchen-agi/xing4.0-29b-a4b:free), then a third (google/gemma-4-31b-it:free + dots-studio/dots-3-note-preview:free, cohere/north-mini-code:free), then a fourth (thinkingmachines/inkling-small:free + google/gemma-4-26b-a4b-it:free, nvidia/nemotron-3-nano-omni-30b-a3b-reasoning:free), with the backoff doubling each round. The server also respects the Retry-After header. The client refuses any OpenRouter model id that does not end in :free.
Temperature is 0.2 across all providers. The reply is the OpenAI-style choices[0].message.content. A separate reasoning trace is ignored unless content is empty, in which case the server looks for a final-answer marker inside the reasoning field. If the text itself opens with a thinking-process preamble, only the final answer is shown.The OpenRouter free tier retires models regularly, so a retired id returns 404. That is treated as a model-level failure and the request falls through to the next trio instead of failing outright. The npm run check:models script compares every configured OpenRouter id against the live catalog and fails when one has been retired.The client also retries automatically: on a 502 provider_error, it waits and sends the request again (up to 3 retries with 2s/4s/8s backoff), showing a “Retrying…” indicator instead of an error.Identical questions within a 5-minute window are served from an in-memory cache, so repeat questions do not hit any provider at all.
Embeddings
sentence-transformers/static-retrieval-mrl-en-v1Static retrieval model: a table lookup and a mean pool, about as accurate as a small transformer encoder and cheap enough to run inside a serverless function. The full 1,024-dimensional matrix is downloaded only by npm run ingest. The repo ships the first 128 dimensions as float16, plus data/vocab.txt and the chunk index. Query embedding is that same table, so a cold start does not download or initialize ONNX. Retrieval does not call any external API.

What the server stores

Source text: https://www.gutenberg.org/cache/epub/11/pg11.txt. Nothing about a visitor is written to disk.

Limits

English only. One book. The model can still miss a passage that retrieval found, or refuse with NOT_IN_BOOK when the passages do not actually answer. A question that shares a word with the book, such as “queen,” may be sent to the model; the model is instructed to refuse if the passages do not contain the answer.