Pablan, as it stands

Self-hosted knowledge management for SMEs: a split-screen Markdown editor
whose sections an LLM refines while you write, and RAG question answering
over the documents that result. FastAPI + Postgres/pgvector on the back,
SvelteKit on the front, everything OpenAI-compatible and self-hostable.

Squashed into a single commit; the development history stays local.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CA43ZJda8Rbp2hKXNy8f6b

docs/api-protocol.md

@@ -0,0 +1,415 @@
+# API protocol
+
+Status: rough contract — Claude refines this doc as endpoints are
+implemented; it must always match the actual FastAPI routes.
+
+## General
+
+- All API routes live under `/api/*`, served by FastAPI. The reverse proxy
+  (prod) / Vite dev proxy (dev) makes frontend and API same-origin.
+- Types flow one way: FastAPI OpenAPI → `make types` → `openapi-typescript`
+  → typed `openapi-fetch` client. Never hand-write response types.
+- Errors: consistent JSON problem shape `{ "detail": string, "code": string }`.
+  `detail` is a developer-facing English sentence; the frontend renders the
+  message from `code` (`lib/api/errors.ts`), never from `detail`.
+- **LLM endpoint failures share one vocabulary.** `LLMError.code`
+  (`llm/client.py`) classifies every endpoint failure once, and every surface
+  that talks to a model reports it unchanged, over REST (503) or as an SSE
+  `error` frame:
+
+  | code | when | what the user is told |
+  |---|---|---|
+  | `llm_unreachable` | connection refused / DNS / reset | the endpoint is not running |
+  | `llm_busy` | 408, 429, 503, 504, a timeout, or Pablan's own endpoint gate refusing to queue further (`llm/gate.py`) | it is loaded, retry in a moment |
+  | `llm_misconfigured` | 401, 403, 404 | wrong key, wrong model, wrong URL |
+  | `llm_failed` | anything else | it did not answer |
+
+  A busy endpoint queues rather than refuses, so a turn can simply stay
+  silent: the chat says so after 12s (`slow` in `chat/state.svelte.ts`)
+  instead of leaving the user guessing. Waiting for a free slot is bounded
+  by Pablan itself (`llm/gate.py`), so a turn that cannot be served soon
+  ends in `llm_busy` in seconds rather than at the HTTP timeout.
+- `GET /api/health` — unauthenticated liveness probe, returns
+  `{"status": "ok"}`. Used by reverse proxies / monitoring.
+
+## Auth flow
+
+Sequence incl. the 401 → redirect branch:
+[`diagrams/auth-sequence.svg`](diagrams/auth-sequence.svg)
+
+- `POST /api/auth/login` `{email, password}` → verifies argon2 hash, creates
+  `auth_sessions` row, sets httpOnly + Secure + SameSite=Lax cookie
+  `pablan_session`. Returns the user object.
+- `POST /api/auth/logout` → deletes the session row, clears the cookie.
+- `GET /api/auth/me` → current user or 401.
+- SvelteKit `hooks.server.ts` calls `/api/auth/me` server-side (forwarding
+  the cookie header) and fills `locals.user`; the `(app)` layout group
+  redirects to `/login` without it.
+- `POST /api/account/password` `{current_password, new_password}` → 204.
+  Self-service: the current password must verify (403
+  `invalid_current_password`), the new one is at least 8 characters. Every
+  OTHER session of that user is revoked; the session making the change
+  survives, so the user is not thrown out of the app they are standing in.
+- `PUT /api/account/locale` `{locale: "de" | "en" | null}` → 204. Pins the
+  interface language on `users.locale`, so it follows the person across
+  devices; `null` goes back to following the browser. The backend only
+  stores the choice — it never renders UI-language strings (see
+  `architecture.md`). Anything else → 422.
+- `GET /api/account/document` → `{document_id, title, status, template_id}`,
+  the caller's own document about themselves: the one they authored from the
+  person blueprint (`person`), or nulls plus the `template_id` to
+  start it from. Self-scoped, and AUTHORSHIP is the whole rule — a document
+  someone else wrote about you is not this one. The blueprint id lives here, so
+  the frontend needs to know no ids; `template_id` is null too when an admin
+  removed the blueprint, and the ordinary template picker takes over.
+- Later: OIDC (Entra ID) via authlib ends in the same `auth_sessions`
+  mechanism — no parallel auth system.
+
+## People
+
+- `GET /api/people` → `[{id, name, role, department}]`, the member-visible
+  colleague directory, ordered by name. Any authenticated user; **no email or
+  password hash** ever leaves it — the same permission-safe, non-admin shape as
+  `ReviewerCandidate`, deliberately separate from the admin-only
+  `/api/admin/users`.
+- `GET /api/people/{id}` → one colleague's `{id, name, role, department}`;
+  unknown id → 404.
+
+## Conversations & streaming
+
+Conversations are chat threads. The only core mode is `query` (RAG Q&A); EE
+registers `insight`. **Capture is no longer a conversation** — it writes a
+Document directly (see the Documents section and `authoring-templates.md`), so
+there are no interview/draft/checklist endpoints. All conversation endpoints
+are owner-scoped; another user's conversation returns 404. The `mode` must be
+registered (`modes/registry.py`) or creation returns 400 `unknown_mode`.
+
+- `POST /api/conversations` `{mode}` → creates a conversation.
+- `GET /api/conversations` / `GET /api/conversations/{id}` → list / detail
+  incl. messages; list carries a `title` derived from the first user message.
+- `DELETE /api/conversations/{id}` → user deletes own conversation (GDPR);
+  messages cascade.
+- `POST /api/conversations/{id}/messages` `{content}` → **SSE stream**
+  response (`text/event-stream`). The user message is persisted before
+  streaming; the assistant message is persisted when the stream ends —
+  complete on normal end, as a partial if the client aborts (stop button).
+
+### SSE events (mirror of `ModeEvent`)
+
+| event | data | meaning |
+|---|---|---|
+| `token` | `{"text": "..."}` | next fragment of the assistant reply |
+| `sources` | `{"chunks": [{document_id, title, heading_path, excerpt, used, review_pending}]}` | every retrieved passage; `used` marks the ones that grounded the answer (the "?" inspector shows all, the badges only `used`), `review_pending` that the cited document has an unanswered question about it |
+| `state` | `{"phase": "...", "count": 3\|null}` | progress updates |
+| `error` | `{"code"}` | why the turn failed, e.g. `llm_unreachable`; the frontend phrases it |
+| `fallback` | `{"code"}` | no model was reachable: the preceding `sources` are a plain full-text result list to open, and no answer follows |
+| `done` | `{"message_id": "..."}` | reply persisted, stream ends |
+
+`state` is **metadata only, never content** (rule 12): the query text and
+retrieved passages never appear in a `state` payload. Query mode's phases are
+`searching` → `results` \| `no_answer` → `queued`? → `answering`; `queued`
+appears only when every slot on the chat endpoint is taken as the turn is about
+to stream, and is followed by `answering` when the first token arrives. `count`
+is the number of
+passages (0 for `no_answer`). The two-field frame is structurally incapable of
+carrying document text, and one shape serves every mode — an EE mode adding a
+phase does not change it.
+
+`fallback` is the no-model path, and deliberately not an `error`: the turn
+still has a reply, just not a generated one. Retrieval drops to the German
+full-text index alone (`rag/retrieval.text_search`, no embedding call — and
+with the query's terms ORed rather than ANDed, so a whole typed question still
+finds something), the
+`sources` frame is re-sent with every passage `used: false` (nothing reached a
+prompt), and the frontend renders the hits as a list the reader opens
+themselves. It is persisted like any other reply — `messages.meta.fallback`
+holds the code, and `GET /api/conversations/{id}` returns it as
+`messages[].fallback` — so a reload replays the turn instead of showing an
+empty assistant bubble. A dead EMBEDDING endpoint alone does not trigger it:
+the roles are configured separately, so retrieval falls back to full text and
+the chat model still answers.
+
+`no_answer` is the low-confidence path: retrieval found nothing solid, the
+model answers without sources, and the frontend offers to capture the missing
+knowledge — which opens the template picker and starts a new draft
+(`POST /api/documents`), not a conversation.
+
+`review_pending` travels with every hit, from the retrieval SQL through the
+mode to the citation: a document can be published and still carry an open
+question about it, and the answer that leans on it says so (a warning mark on
+the source badge, in the "?" inspector, and in the fallback list). It is
+snapshotted with the citation like the rest, so reopening a conversation shows
+what was true when the answer was given.
+
+The `excerpt` on a citation is a short (≤280 char) preview of the cited
+chunk — the requesting user already passed the permission filter for it.
+Assistant messages snapshot their citations into `messages.meta`, so
+`GET /api/conversations/{id}` returns them under `messages[].sources`
+(empty for user turns) and citations survive reload and re-indexing.
+
+Frontend consumes this via `fetch` + `ReadableStream` (NOT `EventSource` —
+it can't POST) with an `AbortController` wired to the stop button; see
+`lib/api/stream.ts`.
+
+## Documents
+
+Listing and detail are permission-scoped (same filter as retrieval, plus the
+unpublished documents this user owns **or was asked to check**); unreadable
+documents return 404 — their existence must not leak. **Editing** requires the
+author, an admin, or a colleague with an open review request on the document —
+being asked to check something is what grants the right to fix it. The
+owner-only decisions (delete, sharing, handing out a review request) stay with
+the author or an admin. Every `DocumentSummary`/`DocumentDetail` carries the
+per-request `can_edit`, `open_reviews` (the number of unanswered questions) and
+`access_reason` (`author | public | department | granted | review`), so the UI
+predicts the gate rather than guessing it, and can mark a document that is
+readable but not settled. `review` is the reason that ENDS: it means an open
+request is the only thing letting this caller in, so answering it takes the
+access away — a draft goes back to being its author's alone.
+
+This is where knowledge is captured: the user writes Markdown into a document
+that starts as a `draft`, matures it with AI section refinement, and publishes
+it themselves — one action, no approval queue. A `draft` is author-only
+(`readable_documents_filter` shows it to no one else, bar a colleague asked to
+check it) and never indexed (only `published` documents are searched), so it
+never reaches another user or an LLM prompt (rule 2).
+
+- `POST /api/documents` `{template_id?, title?, visibility?, conversation_id?}`
+  → 201 with the new `draft`'s `DocumentDetail`. With a `template_id` the draft
+  opens on that template's Markdown skeleton and rendered title (see
+  `authoring-templates.md`); a bad template is 404 `not_found` / 422
+  `invalid_template`. Without one it starts blank and `title` is required
+  (422 `title_required`). When
+  the capture started from a chat, an optional `conversation_id` (owner-scoped;
+  a foreign or unknown id is ignored) has that conversation's subject summarized
+  once by the LLM and stored on the draft as `meta.context`, background the
+  refine prompt then uses to stay on topic.
+- `POST /api/documents/suggest-similar` `{conversation_id}` → existing documents
+  that match the conversation a capture is starting from, as
+  `[{document_id, title}]` (best first). The match runs over an **LLM
+  topic summary** of the chat (not the raw last message), then hybrid retrieval
+  (`rag/similarity.similar_documents`, `exclude_builtin`), so it is filtered by
+  construction (rule 2) and never offers a help page. Owner-scoped; an empty
+  list when the conversation is unknown/foreign/empty or nothing is close
+  enough. Feeds the "matches your conversation" block in the picker so the user
+  can extend an existing document instead of starting a new one.
+- `POST /api/documents/{id}/refine` `{content_md, cursor_line}` → **SSE
+  stream** that matures the section the cursor sits in. Owner-scoped (author
+  or admin over a readable document). The server computes the active section
+  (`app/authoring/sections.py`), streams a refined version of **only** that
+  section (FIM-style: the rest of the document is prefix/suffix context the
+  model must not re-emit), and disables the reasoning model's thinking for
+  ~1s latency. The refinement is **retrieval-aware**: the server first searches
+  the permission-filtered knowledge base for related, already-published
+  material the author may read (the current document and help pages excluded,
+  and only once the section carries enough of its own text) and passes it to
+  the prompt as grounding, so a suggestion stays consistent with what the
+  company already documented. Frames:
+  - `section` `{start_line, end_line}` — 1-based inclusive range the
+    suggestion will replace, sent first so the client can bind "Accept" to an
+    exact range before any token arrives.
+  - `grounding` `{references: [{title, heading_path}]}` — what the suggestion
+    drew from (the author's own readable material), for the "?" inspector; only
+    sent when the section matched something.
+  - `token` `{text}` — next fragment of the refined section.
+  - `done` `{}` — stream complete.
+  - `error` `{code}` — an `llm_*` code (see General).
+
+  The request body and the streamed response carry document text; that is fine
+  on this owner-scoped endpoint (same trust boundary as `GET /api/documents/{id}`)
+  but nothing here logs content — metadata only (rule 12).
+- `POST /api/documents/{id}/suggest-title` → `{title}`, a concise title
+  suggested from the document's content via `chat_json` (utility). Used at the
+  review step for a new document that still carries its generic template title;
+  owner-scoped (author or admin), content in / title out, nothing logged. Falls
+  back to the current title for an empty document; 503 with the `llm_*` code
+  when the endpoint fails.
+- `POST /api/documents/{id}/publish` → draft → published + `index_document`
+  job. Author or admin, deliberately NOT every editor: a
+  colleague asked to check a draft may fix what is wrong in it, but whether the
+  company gets to read it at all is not their call. An open question about the
+  content does not block it, it travels with the document instead. A dedicated
+  transition because PATCH deliberately refuses to publish. Any other status →
+  409 `invalid_status`.
+- `GET /api/documents/{id}/reviewers` → `[{id, name}]`, the colleagues who can
+  be asked to check this document: everyone who could read it once published
+  (author excluded). Editor-only, non-admin, and permission-safe — a dedicated
+  query, not `/admin/users` — so only id + name leave the server.
+- `POST /api/documents/{id}/reviews` `{reviewer_id, question?}` → the document
+  detail. Asks one colleague to check it, optionally about something specific
+  ("do the 14 holiday days still hold?", ≤2000 chars). Author/admin only; the
+  reviewer must be in that read set (422 `invalid_reviewer`), cannot be the
+  caller (422 `invalid_reviewer`), and cannot already have an open request on
+  the document (409 `review_already_open`). Until it is answered, the request
+  lets that colleague read AND edit the document (never search it), and marks
+  it wherever it appears.
+- `POST /api/documents/{id}/reviews/{review_id}/resolve` → the document detail.
+  The answer: the content was checked. The reviewer answers their own request;
+  the author or an admin can close one that has become moot, so a question
+  nobody will answer does not mark a document forever. Unknown request → 404,
+  already answered → 409 `already_resolved`.
+- `PUT /api/documents/{id}/departments` `{department_ids, confirm_lockout?}` →
+  the document detail. **Multi-department sharing**: replaces the full set of
+  ADDITIONAL departments a document is shared with (its `doc_permissions`
+  grants), on top of the owning department. Author/admin; unknown department →
+  404; no reindex (grants are evaluated live). Unsharing can drop the editing
+  admin's own access, so it runs the same self-lockout guard as PATCH (409
+  `self_lockout_warning` unless `confirm_lockout`). The owning department is
+  never part of the set. `GET /api/documents/{id}` returns the current set as
+  `shared_departments: [{id, name}]`; the `?department=` list filter counts a
+  shared document under each department it reaches.
+
+- `GET /api/documents` (filters: department, status, assigned_to_me,
+  search — the `search` param here is a plain title match, used for cheap
+  filtering)
+- `GET /api/documents/search?q=` → ranked hits through the SAME hybrid
+  retrieval the chat uses (`rag/retrieval.search`), so the permission
+  filter is identical by construction (rule 2). Each hit carries the
+  document plus `heading_path` — the best-matching section, empty when the
+  match was on the title. Drafts are readable but never chunked, so a title
+  fallback covers them — the one asymmetry between this endpoint and chat
+  retrieval.
+- `GET /api/documents/{id}` — includes Markdown content, plus `reviews`:
+  every request on this document oldest-first, open and answered, as
+  `{id, question, requester_name, reviewer_id, reviewer_name, created_at,
+  resolved_at, resolved_by_name, is_mine}`. The answered ones are the record of
+  what was already checked; `is_mine` tells the UI to offer the answer rather
+  than just show the question. Built-in help documents (`is_builtin`, sourced
+  from `help/*.md`) are read-only: PATCH and DELETE return 409
+  `builtin_readonly`, and their `can_edit` is always false.
+- `GET /api/documents` — browse. Returns an envelope
+  `{items, total, per_page}`, not a bare list: the document list is
+  the one screen that grows without bound. Query params `department`,
+  `status`, `search` (title substring), `assigned_to_me` (bool — only
+  documents with an open review request addressed to this user, the "waiting
+  for my check" queue), `sort` (`updated` | `created`), `page` (≥1), `per_page`
+  (1–100, default 30). `total` is
+  computed over the same permission filter as the page, so it never
+  reveals how much exists beyond what the caller may read. Built-in help
+  sorts last in SQL, so it stays last across page boundaries.
+- `PATCH /api/documents/{id}` — title, content_md, visibility, and
+  the archive transition (`status`: only published ↔ archived; publishing has
+  its own endpoint, because it indexes the document). `visibility` needs the
+  author or an admin (like sharing and deleting) while title/content only need
+  an editor — who may READ a document is the owner's decision, a reviewer
+  corrects the text. Content-affecting edits to
+  published documents and status transitions enqueue `index_document`. A
+  visibility change that would remove the editing user's own access runs the
+  self-lockout guard (409 `self_lockout_warning` for an admin unless
+  `confirm_lockout` is set; an author always keeps access).
+- `GET /api/documents/{id}/history` → `[{id, action, actor_id, actor_name,
+  visibility, created_at, has_snapshot}]`, the audit trail newest-first: who
+  changed or checked the document, when. `action` ∈ `created | edited |
+  published | archived | visibility_changed | review_requested |
+  review_resolved`;
+  `has_snapshot` marks the content-bearing events whose frozen version can be
+  fetched. Same read gate as `GET /api/documents/{id}` — history never leaks to a
+  user who cannot read the document.
+- `GET /api/documents/{id}/versions/{event_id}` → `{id, action, actor_id,
+  actor_name, created_at, title, content_md, previous_content_md, visibility}`,
+  one past version's frozen Markdown plus the content it replaced, so the caller
+  can show what THIS event changed (the detail page reuses the
+  `unifiedMergeView` line diff). A snapshot is written after its event, so
+  `previous_content_md` is the closest earlier snapshot, and null for the first
+  one. Unknown/foreign event → 404. Same read gate.
+- `GET /api/documents/export` — a streamed ZIP of the readable knowledge base
+  as Markdown files with YAML frontmatter (title, status, visibility,
+  departments (owning + shared)). Permission-filtered by construction
+  (`readable_documents_filter`), built-in help pages excluded; stdlib
+  `zipfile`/`io` only, no temp files. Declared before `/{document_id}`.
+- `DELETE /api/documents/{id}` — chunks go with it (cascade)
+- `GET /api/documents/stats` → `{documents_total, departments_total}`, read by
+  the landing page's first-run guide to tell a fresh install from a filled one.
+  `documents_total` counts published documents only. Aggregates carry no titles
+  and no per-user data (D15), so they are deliberately not permission-filtered.
+
+## Templates & admin
+
+- `GET /api/templates`, `GET /api/templates/{id}` — any authenticated user
+  (the template picker needs them) → `{id, config_id, name, version,
+  description}`. `id` is the row id used to start a document; `config_id` is
+  the blueprint id from the YAML (e.g. `prozess`), stable across
+  installs, so a surface that wants to offer ONE known blueprint can find it
+  (the profile page's "write about yourself"). The detail carries `yaml`, the
+  editable source rendered server-side, since the frontend has no YAML
+  library. Admin-only from here on:
+  - `POST /api/templates/build` `{template_id, config}` saves a template
+    from the **form builder**: `config` is the structured
+    `AuthoringTemplate` (the same schema pasted YAML would parse into, so
+    the form and the YAML editor share one validation — the frontend has no
+    YAML library and sends the config instead of serializing it).
+    `template_id` null creates a row (the derived config id is uniquified
+    server-side, so a second template with the same name never overwrites
+    the first); a UUID updates that row in place (409 `id_taken` only if the
+    config id collides with a *different* row). 422 on schema violations.
+  - `PUT /api/templates/{id}` `{yaml}` replaces a template, validated on
+    save; 409 `id_taken` if another template already uses that config id.
+  - `POST /api/templates/{id}/duplicate` forks one — the copy gets a fresh
+    config id (`<id>-kopie`), so the two never collide.
+  - `DELETE /api/templates/{id}` removes a template; documents created from
+    it are independent and survive (a template is only a starting point).
+  - **Every template is editable.** There is no `builtin_readonly` here —
+    that applies only to the built-in help *documents*
+    (`/api/documents/*`). See `authoring-templates.md` for the reasoning.
+- The shipped **catalog** of blueprints, admin-only. A blueprint lives in
+  `templates/` and has no row until it is added, so `catalog_id` is the
+  config id, not a UUID:
+  - `GET /api/templates/catalog` → `{id, name, description, sections, added}`
+    per blueprint (`sections` = how many skeleton headings carry hints);
+    `added` is true when a template with that config id already exists.
+  - `GET /api/templates/catalog/{catalog_id}` → the same plus `yaml`, for
+    reading the skeleton before adding. 404 `not_found` if unknown.
+  - `POST /api/templates/catalog/{catalog_id}` copies it into this
+    instance and returns the new `TemplateDetail`. 409 `already_added`
+    rather than overwriting an admin's edits.
+- `GET /api/departments` — names + ids for filters and pickers, any
+  authenticated user.
+- Admin (requires admin role):
+  - Users CRUD under `/api/admin/users`: create (409 `email_taken` on
+    duplicates), update (setting `password` IS the reset and **revokes all
+    sessions of that user** — including the current one when an admin
+    changes their own password; admins cannot change their own role or
+    delete themselves — 409 `self_modification`), delete
+    (sessions/conversations cascade, documents survive authorless).
+  - Departments CRUD under `/api/admin/departments` (409 `name_taken`;
+    deletion leaves members and documents without a department).
+  - `POST /api/admin/llm/test` — pings all three model roles and reports
+    which endpoint is broken (first-line support tool). Per role:
+    `{role, ok, base_url, model, latency_ms, code, error}` — `code` is the
+    shared `llm_*` code the frontend phrases, `error` the sanitized technical
+    detail (exception class + role) an admin needs to act. With a body
+    `{role, base_url?, model?, api_key?}` it probes ONE candidate
+    configuration without persisting it, so an endpoint can be validated
+    before saving.
+  - `GET /api/admin/llm/settings` → stored config per role plus
+    `*_from_env` / `api_key_set` flags. The key itself is never returned
+    (rule 12). Values are seeded from `.env` at first start; after that
+    the DB is authoritative (see `architecture.md`).
+  - `PUT /api/admin/llm/settings/{role}` `{base_url?, model?, api_key?,
+    reset_*?}` → stores the value and marks that field as changed here;
+    `reset_<field>: true` writes back what `.env` currently says and marks
+    it as coming from the environment again. Applies without a restart.
+    An omitted `api_key` keeps the stored one.
+  - `POST /api/admin/llm/models/{role}` `{base_url?, api_key?}` →
+    `{models, supported, error}`. Calls the endpoint's `GET /v1/models`
+    server-side so the credentials never reach the browser, letting the UI
+    offer a dropdown. `supported: false` means the endpoint has no such
+    route (common, not an error) and the UI keeps free-text entry.
+  - `GET /api/admin/prompts` → `[{key, content, is_default}]`, every editable
+    system prompt with its effective text (an override, or the code default)
+    and whether it is still the default. The UI labels each key from i18n.
+  - `PUT /api/admin/prompts/{key}` `{content?}` or `{reset: true}` → overrides a
+    system prompt (applied without a restart) or drops the override back to the
+    code default. Empty content → 422 `empty_prompt`; unknown key → 404.
+- `GET /api/admin/metrics` — in-process metrics snapshot as JSON (LLM calls
+  per role, job queue depth/duration/retries). Admin only; a Prometheus
+  text-format exporter is planned for M10.
+
+## Security notes
+
+- All LLM output and document content rendered in the frontend passes
+  through the sanitizing Markdown renderer (DOMPurify). Model output is
+  untrusted input (stored-XSS vector via document content).
+- Retrieval endpoints never accept a user id from the client; the user comes
+  from the session, and permission filtering happens inside
+  `rag/retrieval.search` (see `data-model.md`).