Architecture & Technical Choices

TL;DR

MIRROR is a production RAG application, not a static portfolio. Generation runs on the xAI Grok API, retrieval is hybrid BM25 (SQLite FTS5) with optional API embeddings, state lives in SQLite (WAL), and the whole system is deployed with Docker Compose + Caddy (automatic TLS) on a small VPS. Nothing heavier than a Python process runs on the server, and a hard daily budget caps LLM spend below $0.50/day.

1. From Sovereign Stack to API-First: an Architecture Decision

The most important engineering decision in this project was killing my own first architecture. V1 ran a fully sovereign stack; V2 runs entirely on APIs. Both are legitimate patterns - I have shipped both professionally (sovereign LLM on Scaleway for an enterprise client with data-residency constraints; API-first for products where speed of iteration wins). The point of this page is showing how to choose.

Why the switch was right for this product

• The user experience is the product. A recruiter waits 2 seconds, not 20. CPU inference of a 14B model could not meet that bar.
• Quality signals competence. A quantized 14B answering imprecisely about my own CV was a worse showcase than a frontier model answering well.
• Cost scales to zero. A portfolio has bursty, low traffic. Paying for 64 GB of RAM 24/7 to serve a handful of daily conversations is the wrong shape of cost. Pay-per-token matches the traffic profile exactly.
• No sovereignty constraint applies. The corpus is my public CV and published articles. When the data is confidential (as with my enterprise clients), the trade-off flips - and I have deployed the sovereign version of this exact stack for that case.

2. LLM Serving: Grok API with a Hard Cost Guardrail

Generation uses the OpenAI-compatible xAI endpoint with grok-4.20-non-reasoning: for short conversational answers, a reasoning model burns output tokens (and seconds) thinking without improving a CV answer. The non-reasoning variant is the right latency/cost point.

Cost guardrail design

• Hard daily budget (default $0.50/day) persisted to disk and enforced before each API call - a public chatbot without a spend cap is an open wallet.
• Real usage accounting - token counts come from the API's usage field (including on streams via stream_options.include_usage), not from estimates; pricing is configuration, not a hardcoded constant.
• Graceful degradation - when the cap is hit, the assistant says so and comes back tomorrow. The rest of the site is fully server-rendered and unaffected.
• Context discipline - retrieval context is capped (~6K tokens) and conversation history is truncated; the biggest LLM cost driver in RAG is unbounded context assembly.

3. Retrieval: Hybrid BM25, Right-Sized

The knowledge base (CV, 7 technical articles in 2 languages, architecture docs, visitor uploads) is a few hundred chunks. At this scale, a dedicated vector database is over-engineering - the honest engineering answer is BM25 on SQLite FTS5, in-process, with a design that upgrades to hybrid dense retrieval without a migration.

How it works

• Indexing - documents are chunked (768 chars, 128 overlap, sentence-boundary aware) and stored in a chunks table; an FTS5 virtual table (unicode61 tokenizer, diacritics-insensitive - matters for French) is kept in sync by triggers.
• Query - free text is sanitized into an OR-term FTS5 match; bm25() scores are mapped to (0,1].
• Hybrid upgrade path - each chunk has an optional embedding BLOB. If EMBEDDINGS_API_KEY is set (any OpenAI-compatible provider), vectors are fetched at index time and search fuses scores: 0.6 × cosine + 0.4 × BM25 over a 4× candidate pool. One env var, no re-architecture. xAI does not ship an embeddings model yet; the client is pre-wired for the day it does.
• Per-user scoping - portfolio knowledge is global; visitor uploads and scraped pages are scoped to an anonymous cookie session. Every query sees global + own sources, never someone else's.
• Self-indexing - at startup the site indexes its own content (profile, articles, these docs), so the assistant answers questions about me with citations, with zero visitor setup.

Why no reranker or vector DB here

• Cross-encoder reranking earns its latency when first-stage recall is noisy across a large corpus. Over a small, curated corpus, BM25 + a frontier LLM that reads 8 chunks is empirically enough - the LLM is the reranker.
• Qdrant (which I run in production elsewhere - HNSW tuning, INT8 quantization, payload indexes) starts paying for itself around 10⁵-10⁶ vectors with real QPS. Running it here would be resume-driven infrastructure.

4. RAG Pipeline & Behavior Control

• Adaptive routing - a zero-latency heuristic classifier (query length, question type, keyword density) routes greetings and small talk to direct chat and substantive questions through retrieval; no LLM call is spent on routing.
• Citation-aware prompting - retrieved chunks are labeled with their source; the system prompt requires [Source: ...] attributions and explicitly permits "the documents don't cover this" - hallucination control by design, verifiable by the user in the UI.
• Streaming end-to-end - SSE from Flask through Caddy to the browser; sources are emitted as the first event so the UI can show provenance before the first token.
• Fallback chain - retrieval miss → direct chat with the personal context; budget exhausted → explicit message; API error → surfaced in the stream, never a silent failure.
• Extra modes - visitors can upload PDF/DOCX/TXT/MD (PyMuPDF parsing) or scrape any URL (trafilatura extraction) and query those sources with the same pipeline - the point is to let technical visitors kick the tires of a real multi-tenant RAG system.

5. Data Layer & Sessions

• SQLite + WAL - concurrent readers with a single writer fit this workload; thread-local connections avoid lock contention across gunicorn threads.
• CASCADE deletes - removing a conversation removes its messages; removing a source removes its chunks.

6. Deployment & Operations

7. What Changes at Scale (and What I'd Keep)

Right-sizing cuts both ways: this architecture is deliberately minimal, and I know exactly where it stops being sufficient. This is the same judgment I apply on client systems - several of these "at scale" boxes are things I already run in production elsewhere.

8. Full Stack Summary

These docs are themselves indexed into the assistant's knowledge base - you can ask the AI why any of these decisions were made.