Pablan, as it stands

Self-hosted knowledge management for SMEs: a split-screen Markdown editor
whose sections an LLM refines while you write, and RAG question answering
over the documents that result. FastAPI + Postgres/pgvector on the back,
SvelteKit on the front, everything OpenAI-compatible and self-hostable.

Squashed into a single commit; the development history stays local.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CA43ZJda8Rbp2hKXNy8f6b

docs/decisions.md

@@ -0,0 +1,311 @@
+# Decision log
+
+Short ADR-style records. Context for all: team of 2 developers + 1
+legal/marketing co-founder; product must be self-hostable by SME IT admins;
+bias toward boring, simple, understandable technology.
+
+## D1 — PostgreSQL + pgvector, no dedicated vector DB
+One database for relational data, documents, vectors, full-text, jobs and
+sessions. `docker compose up` starts 3 containers; nothing extra to operate,
+patch and back up. HNSW handles our scale (SMEs, hundreds of thousands of
+chunks) easily. Retrieval is capsuled behind `rag/retrieval.py`, so swapping
+in Qdrant later (Variant B) is a contained refactor. Bonus: Postgres
+full-text (`german` config) enables hybrid search with zero extra
+infrastructure.
+
+## D2 — SQLAlchemy 2.0 + Alembic, not Prisma
+Prisma is TypeScript-first; the DB belongs to the FastAPI backend (the
+Python client is a community project — not production-grade). Prisma treats
+pgvector as `Unsupported("vector")`, forcing raw SQL anyway. The official
+pgvector package ships a first-class SQLAlchemy type.
+
+## D3 — No LangChain / LlamaIndex
+Thin custom LLM client (3 functions) + our own mode engine. For a focused
+product the framework abstraction costs more than it saves; prompts and
+retrieval are our core competence and must stay fully controllable.
+
+## D4 — OpenAI-compatible API as the only LLM contract
+llama.cpp server, vLLM, Ollama and all major clouds speak it. Config is just
+base_url/api_key/model — per model role (chat / utility / embedding).
+The serving layer applies each model's chat template; we never hand-build
+prompt formats.
+
+## D5 — Server-side sessions, no JWT; auth in FastAPI, not Auth.js
+Auth.js would put identity in the SvelteKit server while enforcement
+(document permissions, retrieval filtering) lives in FastAPI — two systems
+to keep in sync. Server-side sessions are instantly revocable ("employee
+leaves" is our core use case), trivial to reason about, and have zero
+drawbacks for single-backend self-hosted deployments.
+
+## D6 — Sessions naming
+`conversations` (chat threads) vs `auth_sessions` (logins). Never "session"
+unqualified.
+
+## D7 — Background jobs via Postgres table, no Redis/Celery
+`jobs` table + asyncio loop with `FOR UPDATE SKIP LOCKED`. ~80 lines,
+survives restarts, built-in retry, no new infrastructure for customers.
+Moves unchanged into a worker container when scale demands it.
+
+## D8 — SvelteKit is pure UI
+All `/api` traffic goes through the reverse proxy directly to FastAPI (same
+origin: no CORS, cookies and SSE just work, no stream buffering through
+Node). SvelteKit renders pages and gates routes via `hooks.server.ts`.
+
+## D9 — Own components on Bits UI, no component library imports
+Bits UI provides the hard invisible parts (focus traps, keyboard
+conventions, ARIA state, portal/positioning, outside-click) as a headless
+layer; we own 100% of markup and styling. shadcn-svelte was considered (it
+copies source rather than importing) but we skip its styling layer entirely.
+Discipline required: small base component set, everything from design
+tokens.
+
+## D10 — Caddy as the only supported reverse proxy (for now)
+Argument is config ergonomics for foreign IT admins (5-line Caddyfile,
+automatic TLS) — not capability. The app is proxy-agnostic (one origin,
+`/api/*` to the backend, rest to the frontend), so other proxies can be
+added to `deploy/` when a customer actually needs one; until then we
+maintain exactly one config.
+
+## D11 — uv + ruff (lint AND format), pre-commit, Makefile as task runner
+One Python toolchain, no Black. `make` because it is universally installed
+and gives CI and both developers identical commands.
+
+## D12 — Markdown as canonical knowledge format
+Human-readable, versionable, exportable — no data prison (sales argument).
+Chunks/embeddings are disposable derivatives; re-indexing from documents
+must always be possible (e.g. embedding model switch).
+
+## D13 — Step-wise prompt injection for interviews (SUPERSEDED by D19)
+Only the current interview point enters the prompt; a separate `chat_json`
+bookkeeper call tracks the checklist. Keeps Gemma-class 12B models reliable
+and shrinks the local-vs-cloud quality gap. **Superseded:** capture is no
+longer a dialog — the interview engine and bookkeeper were removed. The
+successor keeps small models fast a different way: FIM section refinement (D19).
+
+## D14 — Fair Source (FSL) + proprietary `ee/`
+See `licensing.md`.
+
+## D15 — Privacy defaults are product features
+Approval-gated publishing, user-deletable conversations, retention job,
+aggregation-only insights. Designed for the German works-council/GDPR
+conversation from day one — retrofitting this would be painful; leading with
+it is a sales argument.
+
+## D16 — Hybrid dev stack: Postgres in Docker, app processes native
+`make dev` starts only Postgres via `docker-compose.dev.yml` (own compose
+project name `pablan-dev`) and runs uvicorn `--reload` and vite natively on
+the host — fast reloads, IDE-friendly `.venv`/`node_modules`, no bind-mount
+quirks. The fully containerized `docker-compose.yml` is the customer
+deployment and doubles as the pre-release check (`docker compose up -d
+--build` with `PABLAN_DOMAIN=localhost`); separate compose project names let
+both run on one machine.
+
+## D17 — Display rename "Guided documentation" without id migration (SUPERSEDED by D19)
+The feature formerly presented as "Interview" was renamed **"Guided
+documentation"** in user-facing copy (M8), keeping the `interview` mode id and
+module. **Superseded:** the interview mode itself was removed — capture is now
+a writing editor (D19), so there is no interview engine to name.
+
+## D18 — LLM endpoint configuration in the DB, env as fallback
+Endpoints move during a pilot (local llama.cpp → cloud, or a new model),
+and asking a customer to edit `.env` and restart for that is a support
+call. Admins now override `base_url` / `model` / `api_key` per role in the
+UI; the environment stays the bootstrap and the per-field fallback, so a
+fresh deployment works with no database rows at all. Changes apply without
+a restart by rebuilding the cached clients. The key is stored in plaintext
+because it must be replayed to the endpoint — it is never returned by the
+API and never logged, and the database is the customer's own.
+
+## D19 — Capture inverted to a writing-first editor
+Capture was a dialog: an interviewer asked, a bookkeeper extracted notes, and
+the document was a render of those notes — the user never wrote it. That reads
+like an interrogation and fights the reality that someone *has* something to
+write down. Markdown is already the source of truth (D12/rule 1), so the user
+should edit that source directly, not watch a derived render. Capture is now a
+**writing editor**: the user writes Markdown into a `Document` (`app/authoring/`),
+and after a typing pause the model matures only the section at the cursor.
+Consequences that make this cheap and safe on small local models:
+- **Drafts are author-only and never indexed.** A captured document starts as
+  a `draft`, private to its author and outside retrieval, so it can never reach
+  another user or an LLM prompt (rule 2) before its author publishes it.
+- **FIM section refinement.** The whole document is prefix/suffix context but
+  the model regenerates ONLY the active section, so a large document is never
+  re-emitted whole. With the reasoning model's thinking disabled
+  (`enable_thinking = false`) the call returns in ~1s — fast enough for a
+  "after pause" trigger and reliable on Gemma-class 12B.
+- **One editor for capture and edit, with a pre-save diff.** The same editor
+  writes new drafts and revises existing documents, always showing a VSCode-style
+  line diff before saving.
+- **CodeMirror 6 + `@codemirror/merge` as a deliberate dependency.** CodeMirror
+  is an editor/diff engine (like marked/dompurify), not a UI component library,
+  so it is compatible with D9; `@codemirror/merge`'s `unifiedMergeView` provides
+  the pre-save diff, which is why no separate diff library (jsdiff) is pulled in.
+
+This removed the interview mode, bookkeeper, checklist, the `Draft`/`Navigable`
+protocols, and the conversation capture endpoints; `query` is the only core
+mode left. See `architecture.md` and `authoring-templates.md`.
+
+## D20 — Reviewer access as a read clause, not a new permission layer
+An author can ask a colleague to check a document, and that colleague must be
+able to read it even while it is an unpublished draft — without anything
+leaking into chat/search. So the grant is a single clause (an open
+`review_requests` row addressed to the user) added to
+`readable_documents_filter` **only** — never to `searchable_documents_filter` —
+reusing the existing filter split rather than introducing a status or a grant
+table. Reviewer candidates are derived department-granular from the document's
+would-be read set (author + public/department/`doc_permissions`) via a
+dedicated non-admin, permission-safe query returning id + name only — not by
+reusing `/admin/users`. (Superseded in part by D24: the clause is now the open
+request, not a `documents.reviewer_id` column, and being asked also grants the
+right to EDIT.)
+
+## D21 — Topic-summary retrieval, in the capture picker only (for now)
+When a capture starts from a chat, the picker suggests existing documents to
+extend. Retrieving over the raw last chat message is unreliable (the last turn is
+often a topic-less follow-up like "thanks"), so the match runs over a short **LLM
+topic summary** of the conversation (`chat_json`, utility) instead. An eval
+(`tests/evals/test_topic_retrieval_eval.py`) compares topic-summary vs.
+raw-message recall on a fixture; measured 4/4 vs. 1/4. On that evidence, query
+mode now uses it too, but only as a **fallback**: it searches on the raw message
+first and re-searches with a topic summary of the conversation ONLY when the raw
+search is low-confidence and there is earlier context (`query.py::_topic_transcript`
+returns "" for a first message). So a clear question pays no extra latency and a
+first-message miss stays a genuine no-answer, while a topic-losing follow-up
+recovers the subject. The same topic summary is stored as `meta.context` on a
+chat-originated draft and passed to the refine prompt as background, so refinement
+is chat-aware without re-summarizing per call.
+
+## D22 — Cache-friendly query prompt: static system, volatile excerpts last
+Local llama.cpp and cloud OpenAI-compatible endpoints both reuse a request's
+longest identical prefix (prompt/KV caching). The old query prompt injected the
+per-question excerpts INTO the system message, so everything after the static
+rules — including the conversation history — was reprocessed every turn. The
+message order is now `[static system rules] [conversation history] [this turn's
+excerpts + question]` (`modes/prompts.py::render_context_turn`): the system rules
+are byte-identical for every question (cached across all conversations and users)
+and the history is byte-identical across a conversation's turns (cached within
+it), so only the volatile excerpts + question are new work. The excerpts are
+inherently per-question and never cacheable; keeping them out of the shared
+prefix is what makes the rest cacheable. No endpoint config is needed — the win
+is purely in prompt structure.
+
+
+## D23 — API modules are packages cut by caller intent, with the gate on the router
+`api/documents.py` had grown to 1137 lines and `api/admin.py` to 615, which is
+what happens when "one module per resource" meets a resource with five different
+jobs. They are packages now, cut by what a caller is DOING — browse, the life of
+one document, its audit trail, the publish/review workflow, sharing — rather
+than by HTTP verb or by schema-vs-route. Two consequences are the actual point:
+
+- **Shared rules exist once, and are stated.** `documents/access.py` holds every
+  gate (including the Python mirror of the read filter used to judge a change
+  before it commits) and `documents/view.py` every response shape. Modules that
+  used to reach into another module's private helpers (`_readable_document`,
+  `_role_config`, `_excerpt`, `_HEADING_RE`) now import a named interface.
+- **A dependency belongs on the router, not on each endpoint.** Each package has
+  a `routing.py` that builds its `APIRouter`; `admin` puts `require_admin`
+  there, and `templates` has two constructors because reading is for everyone
+  and writing is not. A new endpoint is gated by the router it is added to,
+  which is not something an author can forget.
+
+The cost is that route order becomes explicit: FastAPI matches in registration
+order, so the module with static paths is included before the one with
+`/{id}`. That is stated in each package's `__init__` rather than discovered.
+
+
+
+## D24 — Review is a question about a document, not a status of it
+Publishing used to run through an approval queue: `draft → pending_approval →
+published`, with a delegated reviewer who could approve but not edit. Two things
+were wrong with it. It made every author wait for someone else before their
+knowledge was findable at all — the opposite of "writing it down must be
+cheap" — and it could only express doubt BEFORE publishing, while the doubt that
+actually matters ("do the 14 holiday days still hold?") turns up on a document
+that has been published for months.
+
+So the two ideas were separated:
+
+- **Status says where a document stands**: `draft` (being written, author-only,
+  unindexed), `published`, `archived`. Publishing is the author's own one-click
+  action; `pending_approval` is gone.
+- **A `ReviewRequest` says what is unsettled about its CONTENT** — one question,
+  addressed to one colleague, open until answered. It can sit on a draft or on a
+  document published months ago, and it is orthogonal to the status.
+
+Consequences that make it work:
+
+- **An open request marks the document everywhere**, including the sources under
+  a chat answer (`review_pending` on every search hit, snapshotted with the
+  citation). A reader who is handed an answer sees that the passage behind it is
+  not confirmed — which is the case the whole feature exists for.
+- **Being asked grants the right to edit** (`can_edit` = author, admin, or open
+  reviewer), not just to read: a reviewer who spots a wrong number should fix it
+  rather than file a second question about it. The grant ends with the answer,
+  and for a draft the read access ends with it too. The owner-only decisions
+  (delete, sharing, handing out a request) stay with the author or an admin.
+- **The audit trail keeps who checked what**: `review_requested` /
+  `review_resolved` events, with the answering colleague as `actor_id`. The
+  `review_requests` row alone would only tell you that nothing is open.
+
+The migration maps `pending_approval` → `draft`, `approved` events → `published`,
+drops `submitted` events and `documents.reviewer_id`.
+
+
+
+## D25 — Three fields removed rather than kept "for later"
+`documents.verified_until`, `meta.tags` and `documents.source_type` all
+existed, were all written on every publish, and none of them did anything.
+They were removed together.
+
+- **`verified_until`** was a freshness horizon: 180 days after publishing, the
+  UI put "Verifizierung fällig" on the document and offered a Re-verify button.
+  It marked documents and reminded nobody — no mail, no queue, no name attached
+  — so the badge said only "time has passed", which the `updated_at` in the
+  same line already said. A `ReviewRequest` (D24) is the thing that actually
+  asks: it names a person, carries a question, and can sit on a document
+  published months ago.
+- **`tags`** had been write-only since M8: free text with no way to see which
+  tags already existed, so the values that arrived were too inconsistent to
+  filter by, and hybrid search covers finding things anyway. Keeping the column
+  "because the data is real" only kept the template form asking for it.
+- **`source_type`** had two members and one of them (`upload`) was unreachable:
+  nothing can be uploaded. Everything in the table was written in Pablan, so
+  the column classified nothing.
+
+The general rule this records: a field nothing reads is not neutral. It costs a
+form row, a table column, a line in the export frontmatter, an entry in the ER
+diagram, and a paragraph of documentation explaining why it is there. Import
+(roadmap epic 19) will want provenance, and can reintroduce a column that
+distinguishes values it actually produces.
+
+
+
+## D26 — The queue in front of the endpoint is ours, not the server's
+A self-hosted llama.cpp server takes every request it is offered and serves
+`--parallel` of them at a time; the rest wait inside it. That is a queue Pablan
+can neither see, bound, nor explain — and every request waiting in it is still
+burning the client's HTTP timeout. Two colleagues chatting while a reindex runs
+is enough to turn "slow" into "everything times out at once".
+
+So the waiting happens in `app/llm/gate.py`, in front of the endpoint, and it
+is bounded twice: a caller waits at most `PABLAN_LLM_QUEUE_WAIT_SECONDS` for a
+slot, and past `PABLAN_LLM_MAX_QUEUED` waiters the gate stops admitting at all.
+An unbounded wait would be the same failure as no gate, just quieter.
+
+Three details that make it work:
+
+- **Keyed by base_url, not by role.** chat and utility point at the same server
+  in the shipped configuration; a per-role limit would let one server be given
+  twice its slots.
+- **Both refusals are `llm_busy`.** A queue timeout and a provider's 429 mean
+  the same thing to the person waiting, so they get the same sentence.
+- **A stream holds its slot to the last token**, because that is how long it
+  occupies the server's slot.
+
+What it buys beyond fairness: a saturated EMBEDDING endpoint now degrades query
+mode to full-text retrieval (that path already existed for an unreachable
+endpoint) instead of failing, and the chat can say "the model is busy" as a
+`queued` phase rather than blinking a cursor for twenty seconds.
+
+`probe()` and `list_models()` stay outside the gate: an admin has to be able to
+test an endpoint precisely when it is saturated.