ProfessorNova/pablan
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Pablan, as it stands

Browse Source 
Self-hosted knowledge management for SMEs: a split-screen Markdown editor
whose sections an LLM refines while you write, and RAG question answering
over the documents that result. FastAPI + Postgres/pgvector on the back,
SvelteKit on the front, everything OpenAI-compatible and self-hostable.

Squashed into a single commit; the development history stays local.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CA43ZJda8Rbp2hKXNy8f6b
This commit is contained in:
ProfessorNova
2026-09-04 09:21:37 +02:00
co-authored by Claude Opus 5
parent 68d3a43191
commit 784b76baf7
346 changed files with 43430 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/api-protocol.md
+415
View File
@@ -0,0 +1,415 @@
# API protocol
Status: rough contract — Claude refines this doc as endpoints are
implemented; it must always match the actual FastAPI routes.
## General
- All API routes live under `/api/*`, served by FastAPI. The reverse proxy
(prod) / Vite dev proxy (dev) makes frontend and API same-origin.
- Types flow one way: FastAPI OpenAPI → `make types` → `openapi-typescript`
→ typed `openapi-fetch` client. Never hand-write response types.
- Errors: consistent JSON problem shape `{ "detail": string, "code": string }`.
`detail` is a developer-facing English sentence; the frontend renders the
message from `code` (`lib/api/errors.ts`), never from `detail`.
- **LLM endpoint failures share one vocabulary.** `LLMError.code`
(`llm/client.py`) classifies every endpoint failure once, and every surface
that talks to a model reports it unchanged, over REST (503) or as an SSE
`error` frame:
| code | when | what the user is told |
|---|---|---|
| `llm_unreachable` | connection refused / DNS / reset | the endpoint is not running |
| `llm_busy` | 408, 429, 503, 504, a timeout, or Pablan's own endpoint gate refusing to queue further (`llm/gate.py`) | it is loaded, retry in a moment |
| `llm_misconfigured` | 401, 403, 404 | wrong key, wrong model, wrong URL |
| `llm_failed` | anything else | it did not answer |
A busy endpoint queues rather than refuses, so a turn can simply stay
silent: the chat says so after 12s (`slow` in `chat/state.svelte.ts`)
instead of leaving the user guessing. Waiting for a free slot is bounded
by Pablan itself (`llm/gate.py`), so a turn that cannot be served soon
ends in `llm_busy` in seconds rather than at the HTTP timeout.
- `GET /api/health` — unauthenticated liveness probe, returns
`{"status": "ok"}`. Used by reverse proxies / monitoring.
## Auth flow
Sequence incl. the 401 → redirect branch:
[`diagrams/auth-sequence.svg`](diagrams/auth-sequence.svg)
- `POST /api/auth/login` `{email, password}` → verifies argon2 hash, creates
`auth_sessions` row, sets httpOnly + Secure + SameSite=Lax cookie
`pablan_session`. Returns the user object.
- `POST /api/auth/logout` → deletes the session row, clears the cookie.
- `GET /api/auth/me` → current user or 401.
- SvelteKit `hooks.server.ts` calls `/api/auth/me` server-side (forwarding
the cookie header) and fills `locals.user`; the `(app)` layout group
redirects to `/login` without it.
- `POST /api/account/password` `{current_password, new_password}` → 204.
Self-service: the current password must verify (403
`invalid_current_password`), the new one is at least 8 characters. Every
OTHER session of that user is revoked; the session making the change
survives, so the user is not thrown out of the app they are standing in.
- `PUT /api/account/locale` `{locale: "de" | "en" | null}` → 204. Pins the
interface language on `users.locale`, so it follows the person across
devices; `null` goes back to following the browser. The backend only
stores the choice — it never renders UI-language strings (see
`architecture.md`). Anything else → 422.
- `GET /api/account/document` → `{document_id, title, status, template_id}`,
the caller's own document about themselves: the one they authored from the
person blueprint (`person`), or nulls plus the `template_id` to
start it from. Self-scoped, and AUTHORSHIP is the whole rule — a document
someone else wrote about you is not this one. The blueprint id lives here, so
the frontend needs to know no ids; `template_id` is null too when an admin
removed the blueprint, and the ordinary template picker takes over.
- Later: OIDC (Entra ID) via authlib ends in the same `auth_sessions`
mechanism — no parallel auth system.
## People
- `GET /api/people` → `[{id, name, role, department}]`, the member-visible
colleague directory, ordered by name. Any authenticated user; **no email or
password hash** ever leaves it — the same permission-safe, non-admin shape as
`ReviewerCandidate`, deliberately separate from the admin-only
`/api/admin/users`.
- `GET /api/people/{id}` → one colleague's `{id, name, role, department}`;
unknown id → 404.
## Conversations & streaming
Conversations are chat threads. The only core mode is `query` (RAG Q&A); EE
registers `insight`. **Capture is no longer a conversation** — it writes a
Document directly (see the Documents section and `authoring-templates.md`), so
there are no interview/draft/checklist endpoints. All conversation endpoints
are owner-scoped; another user's conversation returns 404. The `mode` must be
registered (`modes/registry.py`) or creation returns 400 `unknown_mode`.
- `POST /api/conversations` `{mode}` → creates a conversation.
- `GET /api/conversations` / `GET /api/conversations/{id}` → list / detail
incl. messages; list carries a `title` derived from the first user message.
- `DELETE /api/conversations/{id}` → user deletes own conversation (GDPR);
messages cascade.
- `POST /api/conversations/{id}/messages` `{content}` → **SSE stream**
response (`text/event-stream`). The user message is persisted before
streaming; the assistant message is persisted when the stream ends —
complete on normal end, as a partial if the client aborts (stop button).
### SSE events (mirror of `ModeEvent`)
| event | data | meaning |
|---|---|---|
| `token` | `{"text": "..."}` | next fragment of the assistant reply |
| `sources` | `{"chunks": [{document_id, title, heading_path, excerpt, used, review_pending}]}` | every retrieved passage; `used` marks the ones that grounded the answer (the "?" inspector shows all, the badges only `used`), `review_pending` that the cited document has an unanswered question about it |
| `state` | `{"phase": "...", "count": 3\|null}` | progress updates |
| `error` | `{"code"}` | why the turn failed, e.g. `llm_unreachable`; the frontend phrases it |
| `fallback` | `{"code"}` | no model was reachable: the preceding `sources` are a plain full-text result list to open, and no answer follows |
| `done` | `{"message_id": "..."}` | reply persisted, stream ends |
`state` is **metadata only, never content** (rule 12): the query text and
retrieved passages never appear in a `state` payload. Query mode's phases are
`searching` → `results` \| `no_answer` → `queued`? → `answering`; `queued`
appears only when every slot on the chat endpoint is taken as the turn is about
to stream, and is followed by `answering` when the first token arrives. `count`
is the number of
passages (0 for `no_answer`). The two-field frame is structurally incapable of
carrying document text, and one shape serves every mode — an EE mode adding a
phase does not change it.
`fallback` is the no-model path, and deliberately not an `error`: the turn
still has a reply, just not a generated one. Retrieval drops to the German
full-text index alone (`rag/retrieval.text_search`, no embedding call — and
with the query's terms ORed rather than ANDed, so a whole typed question still
finds something), the
`sources` frame is re-sent with every passage `used: false` (nothing reached a
prompt), and the frontend renders the hits as a list the reader opens
themselves. It is persisted like any other reply — `messages.meta.fallback`
holds the code, and `GET /api/conversations/{id}` returns it as
`messages[].fallback` — so a reload replays the turn instead of showing an
empty assistant bubble. A dead EMBEDDING endpoint alone does not trigger it:
the roles are configured separately, so retrieval falls back to full text and
the chat model still answers.
`no_answer` is the low-confidence path: retrieval found nothing solid, the
model answers without sources, and the frontend offers to capture the missing
knowledge — which opens the template picker and starts a new draft
(`POST /api/documents`), not a conversation.
`review_pending` travels with every hit, from the retrieval SQL through the
mode to the citation: a document can be published and still carry an open
question about it, and the answer that leans on it says so (a warning mark on
the source badge, in the "?" inspector, and in the fallback list). It is
snapshotted with the citation like the rest, so reopening a conversation shows
what was true when the answer was given.
The `excerpt` on a citation is a short (≤280 char) preview of the cited
chunk — the requesting user already passed the permission filter for it.
Assistant messages snapshot their citations into `messages.meta`, so
`GET /api/conversations/{id}` returns them under `messages[].sources`
(empty for user turns) and citations survive reload and re-indexing.
Frontend consumes this via `fetch` + `ReadableStream` (NOT `EventSource` —
it can't POST) with an `AbortController` wired to the stop button; see
`lib/api/stream.ts`.
## Documents
Listing and detail are permission-scoped (same filter as retrieval, plus the
unpublished documents this user owns **or was asked to check**); unreadable
documents return 404 — their existence must not leak. **Editing** requires the
author, an admin, or a colleague with an open review request on the document —
being asked to check something is what grants the right to fix it. The
owner-only decisions (delete, sharing, handing out a review request) stay with
the author or an admin. Every `DocumentSummary`/`DocumentDetail` carries the
per-request `can_edit`, `open_reviews` (the number of unanswered questions) and
`access_reason` (`author | public | department | granted | review`), so the UI
predicts the gate rather than guessing it, and can mark a document that is
readable but not settled. `review` is the reason that ENDS: it means an open
request is the only thing letting this caller in, so answering it takes the
access away — a draft goes back to being its author's alone.
This is where knowledge is captured: the user writes Markdown into a document
that starts as a `draft`, matures it with AI section refinement, and publishes
it themselves — one action, no approval queue. A `draft` is author-only
(`readable_documents_filter` shows it to no one else, bar a colleague asked to
check it) and never indexed (only `published` documents are searched), so it
never reaches another user or an LLM prompt (rule 2).
- `POST /api/documents` `{template_id?, title?, visibility?, conversation_id?}`
→ 201 with the new `draft`'s `DocumentDetail`. With a `template_id` the draft
opens on that template's Markdown skeleton and rendered title (see
`authoring-templates.md`); a bad template is 404 `not_found` / 422
`invalid_template`. Without one it starts blank and `title` is required
(422 `title_required`). When
the capture started from a chat, an optional `conversation_id` (owner-scoped;
a foreign or unknown id is ignored) has that conversation's subject summarized
once by the LLM and stored on the draft as `meta.context`, background the
refine prompt then uses to stay on topic.
- `POST /api/documents/suggest-similar` `{conversation_id}` → existing documents
that match the conversation a capture is starting from, as
`[{document_id, title}]` (best first). The match runs over an **LLM
topic summary** of the chat (not the raw last message), then hybrid retrieval
(`rag/similarity.similar_documents`, `exclude_builtin`), so it is filtered by
construction (rule 2) and never offers a help page. Owner-scoped; an empty
list when the conversation is unknown/foreign/empty or nothing is close
enough. Feeds the "matches your conversation" block in the picker so the user
can extend an existing document instead of starting a new one.
- `POST /api/documents/{id}/refine` `{content_md, cursor_line}` → **SSE
stream** that matures the section the cursor sits in. Owner-scoped (author
or admin over a readable document). The server computes the active section
(`app/authoring/sections.py`), streams a refined version of **only** that
section (FIM-style: the rest of the document is prefix/suffix context the
model must not re-emit), and disables the reasoning model's thinking for
~1s latency. The refinement is **retrieval-aware**: the server first searches
the permission-filtered knowledge base for related, already-published
material the author may read (the current document and help pages excluded,
and only once the section carries enough of its own text) and passes it to
the prompt as grounding, so a suggestion stays consistent with what the
company already documented. Frames:
- `section` `{start_line, end_line}` — 1-based inclusive range the
suggestion will replace, sent first so the client can bind "Accept" to an
exact range before any token arrives.
- `grounding` `{references: [{title, heading_path}]}` — what the suggestion
drew from (the author's own readable material), for the "?" inspector; only
sent when the section matched something.
- `token` `{text}` — next fragment of the refined section.
- `done` `{}` — stream complete.
- `error` `{code}` — an `llm_*` code (see General).
The request body and the streamed response carry document text; that is fine
on this owner-scoped endpoint (same trust boundary as `GET /api/documents/{id}`)
but nothing here logs content — metadata only (rule 12).
- `POST /api/documents/{id}/suggest-title` → `{title}`, a concise title
suggested from the document's content via `chat_json` (utility). Used at the
review step for a new document that still carries its generic template title;
owner-scoped (author or admin), content in / title out, nothing logged. Falls
back to the current title for an empty document; 503 with the `llm_*` code
when the endpoint fails.
- `POST /api/documents/{id}/publish` → draft → published + `index_document`
job. Author or admin, deliberately NOT every editor: a
colleague asked to check a draft may fix what is wrong in it, but whether the
company gets to read it at all is not their call. An open question about the
content does not block it, it travels with the document instead. A dedicated
transition because PATCH deliberately refuses to publish. Any other status →
409 `invalid_status`.
- `GET /api/documents/{id}/reviewers` → `[{id, name}]`, the colleagues who can
be asked to check this document: everyone who could read it once published
(author excluded). Editor-only, non-admin, and permission-safe — a dedicated
query, not `/admin/users` — so only id + name leave the server.
- `POST /api/documents/{id}/reviews` `{reviewer_id, question?}` → the document
detail. Asks one colleague to check it, optionally about something specific
("do the 14 holiday days still hold?", ≤2000 chars). Author/admin only; the
reviewer must be in that read set (422 `invalid_reviewer`), cannot be the
caller (422 `invalid_reviewer`), and cannot already have an open request on
the document (409 `review_already_open`). Until it is answered, the request
lets that colleague read AND edit the document (never search it), and marks
it wherever it appears.
- `POST /api/documents/{id}/reviews/{review_id}/resolve` → the document detail.
The answer: the content was checked. The reviewer answers their own request;
the author or an admin can close one that has become moot, so a question
nobody will answer does not mark a document forever. Unknown request → 404,
already answered → 409 `already_resolved`.
- `PUT /api/documents/{id}/departments` `{department_ids, confirm_lockout?}` →
the document detail. **Multi-department sharing**: replaces the full set of
ADDITIONAL departments a document is shared with (its `doc_permissions`
grants), on top of the owning department. Author/admin; unknown department →
404; no reindex (grants are evaluated live). Unsharing can drop the editing
admin's own access, so it runs the same self-lockout guard as PATCH (409
`self_lockout_warning` unless `confirm_lockout`). The owning department is
never part of the set. `GET /api/documents/{id}` returns the current set as
`shared_departments: [{id, name}]`; the `?department=` list filter counts a
shared document under each department it reaches.
- `GET /api/documents` (filters: department, status, assigned_to_me,
search — the `search` param here is a plain title match, used for cheap
filtering)
- `GET /api/documents/search?q=` → ranked hits through the SAME hybrid
retrieval the chat uses (`rag/retrieval.search`), so the permission
filter is identical by construction (rule 2). Each hit carries the
document plus `heading_path` — the best-matching section, empty when the
match was on the title. Drafts are readable but never chunked, so a title
fallback covers them — the one asymmetry between this endpoint and chat
retrieval.
- `GET /api/documents/{id}` — includes Markdown content, plus `reviews`:
every request on this document oldest-first, open and answered, as
`{id, question, requester_name, reviewer_id, reviewer_name, created_at,
resolved_at, resolved_by_name, is_mine}`. The answered ones are the record of
what was already checked; `is_mine` tells the UI to offer the answer rather
than just show the question. Built-in help documents (`is_builtin`, sourced
from `help/*.md`) are read-only: PATCH and DELETE return 409
`builtin_readonly`, and their `can_edit` is always false.
- `GET /api/documents` — browse. Returns an envelope
`{items, total, per_page}`, not a bare list: the document list is
the one screen that grows without bound. Query params `department`,
`status`, `search` (title substring), `assigned_to_me` (bool — only
documents with an open review request addressed to this user, the "waiting
for my check" queue), `sort` (`updated` | `created`), `page` (≥1), `per_page`
(1–100, default 30). `total` is
computed over the same permission filter as the page, so it never
reveals how much exists beyond what the caller may read. Built-in help
sorts last in SQL, so it stays last across page boundaries.
- `PATCH /api/documents/{id}` — title, content_md, visibility, and
the archive transition (`status`: only published ↔ archived; publishing has
its own endpoint, because it indexes the document). `visibility` needs the
author or an admin (like sharing and deleting) while title/content only need
an editor — who may READ a document is the owner's decision, a reviewer
corrects the text. Content-affecting edits to
published documents and status transitions enqueue `index_document`. A
visibility change that would remove the editing user's own access runs the
self-lockout guard (409 `self_lockout_warning` for an admin unless
`confirm_lockout` is set; an author always keeps access).
- `GET /api/documents/{id}/history` → `[{id, action, actor_id, actor_name,
visibility, created_at, has_snapshot}]`, the audit trail newest-first: who
changed or checked the document, when. `action` ∈ `created | edited |
published | archived | visibility_changed | review_requested |
review_resolved`;
`has_snapshot` marks the content-bearing events whose frozen version can be
fetched. Same read gate as `GET /api/documents/{id}` — history never leaks to a
user who cannot read the document.
- `GET /api/documents/{id}/versions/{event_id}` → `{id, action, actor_id,
actor_name, created_at, title, content_md, previous_content_md, visibility}`,
one past version's frozen Markdown plus the content it replaced, so the caller
can show what THIS event changed (the detail page reuses the
`unifiedMergeView` line diff). A snapshot is written after its event, so
`previous_content_md` is the closest earlier snapshot, and null for the first
one. Unknown/foreign event → 404. Same read gate.
- `GET /api/documents/export` — a streamed ZIP of the readable knowledge base
as Markdown files with YAML frontmatter (title, status, visibility,
departments (owning + shared)). Permission-filtered by construction
(`readable_documents_filter`), built-in help pages excluded; stdlib
`zipfile`/`io` only, no temp files. Declared before `/{document_id}`.
- `DELETE /api/documents/{id}` — chunks go with it (cascade)
- `GET /api/documents/stats` → `{documents_total, departments_total}`, read by
the landing page's first-run guide to tell a fresh install from a filled one.
`documents_total` counts published documents only. Aggregates carry no titles
and no per-user data (D15), so they are deliberately not permission-filtered.
## Templates & admin
- `GET /api/templates`, `GET /api/templates/{id}` — any authenticated user
(the template picker needs them) → `{id, config_id, name, version,
description}`. `id` is the row id used to start a document; `config_id` is
the blueprint id from the YAML (e.g. `prozess`), stable across
installs, so a surface that wants to offer ONE known blueprint can find it
(the profile page's "write about yourself"). The detail carries `yaml`, the
editable source rendered server-side, since the frontend has no YAML
library. Admin-only from here on:
- `POST /api/templates/build` `{template_id, config}` saves a template
from the **form builder**: `config` is the structured
`AuthoringTemplate` (the same schema pasted YAML would parse into, so
the form and the YAML editor share one validation — the frontend has no
YAML library and sends the config instead of serializing it).
`template_id` null creates a row (the derived config id is uniquified
server-side, so a second template with the same name never overwrites
the first); a UUID updates that row in place (409 `id_taken` only if the
config id collides with a *different* row). 422 on schema violations.
- `PUT /api/templates/{id}` `{yaml}` replaces a template, validated on
save; 409 `id_taken` if another template already uses that config id.
- `POST /api/templates/{id}/duplicate` forks one — the copy gets a fresh
config id (`<id>-kopie`), so the two never collide.
- `DELETE /api/templates/{id}` removes a template; documents created from
it are independent and survive (a template is only a starting point).
- **Every template is editable.** There is no `builtin_readonly` here —
that applies only to the built-in help *documents*
(`/api/documents/*`). See `authoring-templates.md` for the reasoning.
- The shipped **catalog** of blueprints, admin-only. A blueprint lives in
`templates/` and has no row until it is added, so `catalog_id` is the
config id, not a UUID:
- `GET /api/templates/catalog` → `{id, name, description, sections, added}`
per blueprint (`sections` = how many skeleton headings carry hints);
`added` is true when a template with that config id already exists.
- `GET /api/templates/catalog/{catalog_id}` → the same plus `yaml`, for
reading the skeleton before adding. 404 `not_found` if unknown.
- `POST /api/templates/catalog/{catalog_id}` copies it into this
instance and returns the new `TemplateDetail`. 409 `already_added`
rather than overwriting an admin's edits.
- `GET /api/departments` — names + ids for filters and pickers, any
authenticated user.
- Admin (requires admin role):
- Users CRUD under `/api/admin/users`: create (409 `email_taken` on
duplicates), update (setting `password` IS the reset and **revokes all
sessions of that user** — including the current one when an admin
changes their own password; admins cannot change their own role or
delete themselves — 409 `self_modification`), delete
(sessions/conversations cascade, documents survive authorless).
- Departments CRUD under `/api/admin/departments` (409 `name_taken`;
deletion leaves members and documents without a department).
- `POST /api/admin/llm/test` — pings all three model roles and reports
which endpoint is broken (first-line support tool). Per role:
`{role, ok, base_url, model, latency_ms, code, error}` — `code` is the
shared `llm_*` code the frontend phrases, `error` the sanitized technical
detail (exception class + role) an admin needs to act. With a body
`{role, base_url?, model?, api_key?}` it probes ONE candidate
configuration without persisting it, so an endpoint can be validated
before saving.
- `GET /api/admin/llm/settings` → stored config per role plus
`*_from_env` / `api_key_set` flags. The key itself is never returned
(rule 12). Values are seeded from `.env` at first start; after that
the DB is authoritative (see `architecture.md`).
- `PUT /api/admin/llm/settings/{role}` `{base_url?, model?, api_key?,
reset_*?}` → stores the value and marks that field as changed here;
`reset_<field>: true` writes back what `.env` currently says and marks
it as coming from the environment again. Applies without a restart.
An omitted `api_key` keeps the stored one.
- `POST /api/admin/llm/models/{role}` `{base_url?, api_key?}` →
`{models, supported, error}`. Calls the endpoint's `GET /v1/models`
server-side so the credentials never reach the browser, letting the UI
offer a dropdown. `supported: false` means the endpoint has no such
route (common, not an error) and the UI keeps free-text entry.
- `GET /api/admin/prompts` → `[{key, content, is_default}]`, every editable
system prompt with its effective text (an override, or the code default)
and whether it is still the default. The UI labels each key from i18n.
- `PUT /api/admin/prompts/{key}` `{content?}` or `{reset: true}` → overrides a
system prompt (applied without a restart) or drops the override back to the
code default. Empty content → 422 `empty_prompt`; unknown key → 404.
- `GET /api/admin/metrics` — in-process metrics snapshot as JSON (LLM calls
per role, job queue depth/duration/retries). Admin only; a Prometheus
text-format exporter is planned for M10.
## Security notes
- All LLM output and document content rendered in the frontend passes
through the sanitizing Markdown renderer (DOMPurify). Model output is
untrusted input (stored-XSS vector via document content).
- Retrieval endpoints never accept a user id from the client; the user comes
from the session, and permission filtering happens inside
`rag/retrieval.search` (see `data-model.md`).
docs/architecture.md
+465
View File
@@ -0,0 +1,465 @@
# Architecture
Pablan is a self-hosted knowledge management application for SMEs. Employees
capture knowledge by **writing Markdown directly** into a document in an
editor that matures the section at the cursor with the LLM (a writing-first
flow, not a chatbot interview — D19); the results are Markdown documents. It
answers questions about the knowledge base via RAG, and (as a paid feature)
generates aggregated insights for management.
Core pain points addressed: knowledge leaves the company when employees do
("tacit knowledge" is never written down), and departments solve problems in
isolation without learning from each other.
## Deployment model (Variant A — the pragmatic monolith)
Everything runs on the customer's infrastructure via Docker Compose:
```
[Browser] ⇄ [Reverse proxy :443]
├── /api/* → backend (FastAPI, :8000)
└── /* → frontend (SvelteKit node adapter, :3000)
[backend] ⇄ [PostgreSQL + pgvector]
[backend] ⇄ [LLM endpoints] (OpenAI-compatible: llama.cpp local OR cloud API)
```
- **One origin.** The reverse proxy makes frontend and API same-origin: no
CORS, httpOnly session cookies just work, SSE streams flow directly from
FastAPI to the browser without passing through Node.
- **One database.** PostgreSQL holds relational data, documents, embeddings
(pgvector), full-text indexes, background jobs, and auth sessions. No
dedicated vector DB, no Redis, no message broker.
- **Reverse proxy** is a reference config in `deploy/`, not part of the app.
Caddy is the supported customer setup (5-line config, automatic TLS).
Component overview (built vs. planned):
[`diagrams/components.svg`](diagrams/components.svg)
## Backend structure (`backend/app/`)
| Module | Responsibility |
|---|---|
| `models/` | SQLAlchemy 2.0 models — see `data-model.md` |
| `auth/` | Server-side sessions, argon2, permission dependencies |
| `api/` | Thin HTTP routers; no business logic. The big ones (`documents/`, `admin/`, `conversations/`, `authoring/`, `templates/`) are packages split by what a caller is doing |
| `modes/` | Mode engine: query (+ ee: insights) |
| `authoring/` | Capture: authoring-template schema + document/section refinement for the writing editor |
| `llm/` | The ONLY code speaking to LLM endpoints |
| `rag/` | Chunking, indexing, hybrid retrieval with permission filter |
| `ingestion/` | Background jobs via a Postgres `jobs` table |
| `log.py` | Structured JSON logging, correlation ids; content-free by policy (rule 12) |
| `metrics.py` | In-process metrics registry, exposed via `GET /api/admin/metrics` |
| `ee_hooks.py` | Optional import of `pablan_ee`, registers EE modules |
### How an API package is cut
By what a caller is doing, not by HTTP verb: `documents/` has `browse`, `crud`,
`history`, `workflow` and `sharing`, with the gates in `access.py` and the
response shapes in `view.py`, so a rule like "an author keeps access to their
own document" exists once. Each module builds its router through the package's
`routing.py`, which is also where a shared dependency lives — the admin gate
sits on the router, so a new admin endpoint is gated by where it is added
rather than by remembering a decorator.
**Route order matters**: FastAPI matches in registration order, so the module
with the static paths (`/search`, `/export`, `/catalog`) is included before the
one with `/{id}`.
### Mode engine
Every interaction type implements the `Mode` protocol:
```python
class Mode(Protocol):
name: str
async def handle_turn(conv, user_message, db) -> AsyncIterator[ModeEvent]
```
`ModeEvent` = `Token | Sources | StateChanged | Done | Error | Degraded`
(`StateChanged` carries a `phase` and an optional `count` — metadata only,
rule 12; `Degraded` says no model could be reached, so the accompanying
`Sources` are a plain full-text result list rather than citations). The
conversations router converts events 1:1 into SSE events (see
`api-protocol.md`). Modes know nothing about HTTP; routers know nothing about
mode logic. Modes register in `modes/registry.py` — this is also the EE
extension point (the insights mode lives entirely in `ee/backend`).
**`query` (RAG Q&A) is the only core mode.** Capture is deliberately NOT a
mode: it is writing into a Document directly (`app/authoring/`), not a
conversation, so it never touches `handle_turn` or the SSE turn loop. That
keeps rule 5 clean — `modes/` stays the home of the turn protocol.
### LLM abstraction
Three independently configured model roles, each just `base_url` +
`api_key` + `model` (env vars `PABLAN_CHAT_*`, `PABLAN_UTILITY_*`,
`PABLAN_EMBEDDING_*`):
- **chat** — conversation turns (quality matters; Gemma-class 12B+ locally,
Claude/cloud in production)
- **utility** — summarization, entity extraction (cheap + fast)
- **embedding** — multilingual model (e.g. bge-m3); German quality is a
first-class requirement
The client exposes exactly three functions: `chat_stream`, `chat_json`
(structured output enforced via `response_format` JSON schema + Pydantic
validation + one retry), and `embed`. llama.cpp's server implements all
required endpoints including grammar-constrained JSON. `chat_stream` takes an
optional `extra_body` passed through to the endpoint verbatim — used only by
section refinement to send `chat_template_kwargs.enable_thinking = false`,
which turns off a reasoning model's hidden thinking so the refine call returns
in ~1s instead of ~10s (a mechanical rewrite needs no chain of thought).
**Configuration: bootstrap from env, then the DB owns it.** At first start
`bootstrap_llm_settings()` copies the `PABLAN_CHAT_*` / `PABLAN_UTILITY_*` /
`PABLAN_EMBEDDING_*` values into `llm_settings`, one row per role. From then
on **the table is authoritative and later `.env` edits are ignored** — a
configuration the admin can change in the UI and the deployment can change
underneath them cannot both be the truth, and silently losing a UI change
on the next restart is the worse of the two failures.
The environment stays reachable as the value a field can be *reset* to.
Each field carries a `*_from_env` flag recording where its current value
came from, so the UI labels every field "from .env" or "changed here" and
offers a per-field reset that writes back what `.env` currently says.
Provenance is tracked explicitly rather than inferred by comparing against
the environment, which would mislabel every field the moment someone edits
`.env` after install.
`_role_config()` reads a module-level cache (`app/llm/overrides.py`)
because it is synchronous and hot; the cache is filled at startup and
refreshed on every admin write, which also calls `rebuild_clients()` — the
cached `AsyncOpenAI` instances hold the old URL and key, so changes apply
without a restart. This works because the deployment pins one uvicorn
worker: exactly one cache to refresh. A multi-worker setup would need a
notification channel.
The `api_key` is stored in plaintext — it must be replayed to the endpoint,
so there is nothing to compare a hash against. It is never returned by the
API (only `api_key_set` / `api_key_from_env` booleans) and never logged
(rule 12). Admins validate a candidate endpoint with `probe()` before
saving, so a wrong URL fails in the settings screen rather than on the next
user question, and `list_models()` asks the endpoint's `GET /v1/models`
**server-side** so the model field can become a dropdown without the
browser ever holding the credentials. Endpoints that do not implement that
route are common; the response carries `supported: false` and the UI keeps
its free-text field rather than showing an error.
**One gate per endpoint, in front of every call** (`app/llm/gate.py`). A
self-hosted server has a fixed number of parallel slots; sending more than that
does not make it faster, it just moves the queue somewhere Pablan cannot see or
bound, where every waiting request still burns the HTTP timeout. Two colleagues
chatting while a reindex runs is enough. So `chat_stream`, `chat_json` and
`embed` each take a slot first, keyed by **base_url** rather than by role —
chat and utility usually share one server, and the server is what has the
slots. A caller waits at most `PABLAN_LLM_QUEUE_WAIT_SECONDS`, and past
`PABLAN_LLM_MAX_QUEUED` waiters the gate stops admitting at all; both refusals
come back as `llm_busy`, the same code a cloud provider's 429 produces, so the
frontend has one sentence to say. A streaming completion holds its slot until
the last token, because that is how long it occupies the server's. Two
consequences worth naming: a saturated EMBEDDING endpoint makes query mode fall
back to full-text retrieval (it already handles `LLMError` there) rather than
fail, and the query mode asks `endpoint_busy()` before it streams, so a waiting
turn says "the model is busy" instead of blinking a cursor. `probe()` and
`list_models()` deliberately bypass the gate: an admin has to be able to test an
endpoint precisely when it is saturated. `PABLAN_LLM_MAX_PARALLEL` should match
the server's own parallelism (llama.cpp `--parallel`).
**Admin-editable prompts** mirror this pattern with two simplifications. Every
shipped system prompt (query, no-sources, refinement persona/rules, grounding
framing, topic summary, title) has a CODE default in
`app/prompts/defaults.py`; an admin may override it in `prompt_settings`, applied
without a restart via a module cache (`app/prompts/overrides.py::get_prompt`,
refreshed on every write). The render functions read `get_prompt(key)`, so an
override wins and a missing row falls back to the default — the reset target is
the code default, since prompts have no `.env` layer, and there is no bootstrap.
Prompt-cache friendliness (D22) holds: a value only changes on an admin write, so
`query_system` stays byte-identical between a conversation's turns.
**Inspectable working context.** Behind a "?" in the chat, the context inspector
shows every passage retrieval surfaced for an answer, marking which grounded it
(`used`) versus which were retrieved but dropped as too weak — so a no-answer is
explained rather than silent. The editor shows the same for a refinement: the
grounding references it drew from, sent as a `grounding` SSE frame. Both surface
only material the requesting user may already read (rule 2) — titles/headings and
short excerpts, never new content.
### Ingestion queue
Sequence: [`diagrams/queue-sequence.svg`](diagrams/queue-sequence.svg)
A `jobs` table + an asyncio loop in the app lifespan, claiming jobs with
`SELECT … FOR UPDATE SKIP LOCKED`. The claim transaction stays open while
the handler runs: a crash rolls back claim and handler writes together, so
jobs are atomic and claimable again after a restart. Retry with exponential
backoff, `last_error` sanitized. Job types: `index_document`, `reindex_all`,
`retention_cleanup` (daily, reschedules itself).
Designed so the loop can later move into a separate worker container
(Variant B) without changing any imports.
**Constraint: exactly one uvicorn worker per deployment.** One process =
one queue loop = one in-process metrics registry. Scaling beyond this is
the Variant B worker split, NOT more uvicorn workers (the customer compose
pins `--workers 1`).
## Frontend structure (`frontend/src/`)
SvelteKit is pure UI — no database access, no auth logic beyond passing the
session cookie through. `hooks.server.ts` validates the session against
`GET /api/auth/me` and fills `locals.user`; the `(app)` route group redirects
to `/login` without it (sequence:
[`diagrams/auth-sequence.svg`](diagrams/auth-sequence.svg)). Chat state lives in Svelte 5 runes classes. Custom
components on top of Bits UI (headless behavior layer); design tokens in
`app.css` (see CLAUDE.md). API types are generated from FastAPI's OpenAPI
schema (`make types`).
The shell is a collapsible left sidebar (`lib/nav/Sidebar.svelte`): one
primary action on top, recent conversations in the middle,
documents/admin/account anchored at the bottom. Collapse state is
per-device (`localStorage`). The account entry opens a dialog
(`lib/nav/SettingsDialog.svelte`) rather than a menu — settings hold forms
(password, theme, language), which a menu cannot.
The conversation list is shared between sidebar and chat page through one
module-scope runes store (`lib/chat/conversations.svelte.ts`).
### One thing per screen
Every screen was gone over with two questions: what has to be seen without
looking for it, and what has to be findable without being in the way. The
rules that came out of it, applied everywhere:
- **One primary action, labelled.** The action a visitor most likely came for
is a button with a word on it. Everything else that has to be reachable goes
into `lib/components/Menu.svelte` — a labelled overflow menu, because a row
of bare icons makes the reader guess.
- **Only exceptions are marked.** A document list where every card says
"published · public · yours" says nothing; the one row that carries an
unanswered question disappears into the pattern. Badges are for the
abnormal: open questions, drafts, archived, built-in help.
- **Metadata is one quiet line, not a stack of rows.** The document page shows
status, who may read it, and when it changed in a single grey row above the
text, with the details a click behind the chip that states them
(`lib/documents/AccessPopover.svelte` — visibility and department sharing in
one place, since they are the same question).
- **Controls that are used monthly hide behind a toggle, and say when they are
on.** The document list is a search field and results; the six ways to narrow
it sit behind "Filter", and any filter that IS active stays visible as a
removable chip, because a quietly filtered list lies about what exists.
- **Unrelated jobs become tabs, not a longer scroll.** `/admin` is four
unrelated jobs (users and departments, templates, endpoints, prompts); as
tabs each is named up front and one click away.
- **A page with nothing to do is a page with something missing.** The landing
page volunteers the drafts nobody published and the checks waiting for this
person (`lib/documents/OpenWork.svelte`); the profile answers "what have I
written?".
The landing page itself is one input (focused on arrival), the two other
things anyone comes for, and then that open work.
A conversation lives at **`/chat/[id]`**, resolved in a server load so a
pasted link renders the right thing on first paint. The API is
owner-scoped, so an unknown id and someone else's id both come back 404 and
are indistinguishable from the outside — existence must not leak, same
semantics as the documents API. `/chat` is the empty composer and accepts
`?q=<text>` (ask straight away), consumed once.
Both routes render the same `lib/chat/ChatView.svelte` — the shell that owns
the route-driven conversation switch, the URL sync and the scroll follow, while
one reply is `lib/chat/AssistantTurn.svelte` (streaming, cited, uncovered, or
answered by a plain search) and the input is `lib/chat/Composer.svelte`. The
chat state
is a **module-level singleton** (`chatState` in `lib/chat/state.svelte.ts`)
rather than per-component. The first message on `/chat` creates the
conversation and moves the URL to `/chat/[id]` via `replaceState` — which
unmounts one page component and mounts another. Per-component state would
take the in-flight stream and the messages already on screen down with it.
Reacting to the route parameter rather than to clicks is also the single
place where switching conversations aborts the previous stream.
Chat is a split view with ONE right-hand slot for a cited document. Citations
render as labelled badges ("Sources:") that reveal the cited passage on hover
and open the document in the slot on click (`lib/chat/SourceBadge.svelte`,
`lib/chat/DocumentPanel.svelte`); a document whose title repeats as its first
heading is rendered without that heading (`bodyWithoutTitle`), because every
surface already shows the title above the text; below `lg` the panel stacks under the
conversation instead. When retrieval takes the low-confidence path, the answer
is followed by an invitation to capture the missing knowledge; it opens the
template picker, which creates a draft (`POST /api/documents`) and navigates to
the writing editor — the router never touches capture.
### Writing editor
Capture is a **writing-first editor**, not a chat panel. It starts at
`/documents/new` — a template picker that lists `GET /api/templates`, creates a
draft (`POST /api/documents {template_id}`) and navigates to the editor (the
landing "capture" chip and the chat no-answer/gap links all point here). The
chat links carry `?conversation=<id>`, which makes the picker
**retrieval-aware**: it calls `POST /api/documents/suggest-similar` and, under
the templates, offers existing documents that match the conversation ("matches
your conversation") so the user can **extend** one instead of starting fresh —
picking a match routes to that document's editor with the same
`?conversation=`. The matching runs over an **LLM topic summary** of the chat
(`app/authoring/context.py::summarize_conversation` → `chat_json`), not the raw
last message, then permission-filtered hybrid retrieval
(`rag/similarity.similar_documents`, help pages excluded). Creating a draft with
that `conversation_id` also stores the topic summary as `meta.context`, which
the refine prompt passes as background so refinement is chat-aware. Refinement
is also **retrieval-aware**: before it runs, the server searches the
permission-filtered knowledge base for related, already-published material the
author may read (`rag/similarity.similar_chunks`, the current document and help
pages excluded, and only once the section carries enough of its own text to
match on) and passes it to the prompt as grounding — so a suggestion stays
consistent with what the company already documented, while the prompt keeps it
a reference rather than a source of new facts. Query mode
uses the same topic summary too, but only as a **fallback** — it searches on the
raw message first and re-searches with a conversation topic summary only when the
raw search is low-confidence and there is earlier context (so a clear question
pays no extra latency; see `modes/query.py::_topic_transcript` and decision D21,
backed by `tests/evals/test_topic_retrieval_eval.py`, 4/4 vs. 1/4). The query
prompt is also structured for **prompt caching** (D22): a byte-identical static
system message, the conversation history, then this turn's excerpts + question
last (`modes/prompts.py::render_context_turn`), so the endpoint reuses the cached
system + history prefix and only reprocesses the volatile excerpts. The editor
lives at
`/documents/[id]/edit` (component
`lib/documents/WritingEditor.svelte`), loads the document in a server load
(same 404 semantics as chat), and is the SAME editor a document's detail page
opens for editing (`can_edit` → the edit button routes to `/edit`). It is built
on **CodeMirror 6** (`@codemirror/state`/`view`/`commands`/`language`/
`lang-markdown`), styled through the design tokens rather than a CodeMirror
theme.
- **A single full-width CodeMirror editor** over the Markdown, two-way synced to
the document string. It is not continuously autosaved (that would defeat the
review-before-save diff); instead a `beforeNavigate` guard keeps things tidy on
the way out: an abandoned, never-filled template draft (still only its skeleton)
is **discarded** so empty drafts do not pile up in the list, and an edited but
unsaved draft is **saved** (a draft is private and unindexed, so this is cheap)
so leaving never loses work.
- **Section refinement — shown INLINE at the section you are editing.** A typing
pause (debounced on document change) fires `POST /api/documents/{id}/refine`
through an SSE consumer (`lib/api/refine.ts`, the same `fetch` +
`ReadableStream` + `parseFrame` pattern as `stream.ts`). The suggestion streams
into a **CodeMirror block widget inserted right below the active section** (not
a separate pane), so it appears exactly where the cursor is; its body renders
through the sanitizing `$lib/markdown` renderer (rule 10) and it is aborted the
moment the user resumes typing (`AbortController`, like the chat stop button).
**Accept overwrites that section**: the exact `[start_line, end_line]` from the
server's `section` frame is replaced via a CodeMirror `dispatch`, and the widget
clears. The server owns that boundary; the client mirrors it
(`lib/documents/sections.ts`) only to place the widget. A short cooldown after
accept blocks the next refine until enough new editing has accumulated, and a
suggestion computed against a since-edited region is discarded rather than
applied.
- **Save shows the diff** — the editor's one action sits bottom right where
the writing ends: **Save** opens a `Dialog` with a read-only VSCode-style line
diff via `@codemirror/merge`'s `unifiedMergeView` (original = last saved
state, modified = the buffer), and asks again — save, cancel, or, for a draft,
**save and publish**. There is no autosave, because that look at the diff is
the point: you see what you are about to put your name on and can still back
out. (Leaving the page still persists an unsaved draft rather than losing
work; a never-filled template skeleton is discarded instead.) **The title is
edited in that dialog**, next to the diff: this is the moment you notice the
document is still called "Neuer Ablauf", and a ✨ button fills
the field from `POST /api/documents/{id}/suggest-title` when asked (a model
call on demand, not on every save). `lib/documents/SaveDialog.svelte`.
- **The draft says what state it is in.** Picking a documentation type creates
the document immediately (rule 6: the document IS the state) and leaving an
untouched skeleton deletes it again — both right, and both invisible, which
is what made "the document does not exist yet" confusing. The line under the
editor now says which of the three it is: a new draft that will be discarded
if nothing is written, a draft that exists and is private, or the time of the
last save.
- **A dead endpoint is knocked on once, not every two seconds.** After an
`llm_*` failure the suggestions go quiet, say so in one line, and wait before
trying again (two minutes for unreachable, five for misconfigured, thirty
seconds for busy) — with a "try again now" for the writer who knows the
endpoint is back. A successful suggestion clears it. Writing is untouched
either way: the assistant is the optional half of this editor.
- **The editor edits title and text, nothing else.** Who may READ a document is
a property of the document as it stands, not of the text being typed, so
visibility lives on the document page next to the department sharing it
belongs with (`lib/documents/AccessControls.svelte`) — and both are the
owner's decision, unlike the text itself.
- **Publishing is the author's own action.** One click, no queue: from the
save dialog, from the draft card on the document page, or from the drafts
card on the landing page. A reward modal (a check animation) confirms the
knowledge is captured, and offers "view it" or **"have it checked"**.
- **Review requests** — "please check this", addressed to one colleague,
optionally about something specific ("do the 14 holiday days still hold?").
Orthogonal to the status: it can sit on a draft the author is unsure about or
on a document published months ago, and it marks the document everywhere it
appears — the list card, the detail page, and the sources under a chat answer
(`review_pending`, warning-marked in `SourceBadge`, `ContextInspector`,
`FallbackResults` and the document panel). The author picks from
`GET /api/documents/{id}/reviewers` (candidates who can read the document, id
+ name only) and calls `POST /api/documents/{id}/reviews`
(`lib/documents/ReviewRequestForm.svelte`, shared by the detail page and the
publish reward). Being asked grants read AND edit until it is answered, so
the colleague can fix a wrong number instead of filing a second question;
they answer with **"that's correct"**
(`POST /api/documents/{id}/reviews/{review_id}/resolve`), and the author can
close a question that has become moot. The open questions sit above the
document text (`lib/documents/ReviewPanel.svelte`), because a reader has to
see them before trusting it; answered ones stay as a one-line record of who
checked what. A reviewer finds their queue through the `assigned_to_me`
filter — surfaced both as a list pill and, with their own unpublished drafts,
on the landing page (`lib/documents/OpenWork.svelte`, deep-linking to
`/documents?review=1` and `/documents?status=draft`).
The component set in `lib/components/` is deliberately small: `Button`,
`Input`, `FormField`, `Card`, `Badge`, `Markdown` (the only sanitized
`{@html}` site), plus the Bits UI wrappers `Popover`, `Tooltip`, `Dialog`
and `Tabs`. Tooltip content is plain text by contract, so untrusted strings
(document excerpts, model output) can never become markup.
## In-product help
`help/*.md` are the help pages users read inside Pablan (German product
content, YAML frontmatter with `key` and `title`). `app/help_import.py`
upserts them into `documents` on every start, keyed by `key`, flagged
`is_builtin`, published and public, with no author or department. Unchanged
files are skipped; a changed file is rewritten in place and re-indexed, so a
release always ships current help — and Pablan can answer questions about
itself from its own retrieval path. The API refuses to edit or delete them
(409 `builtin_readonly`), because the next deploy would overwrite the edit.
The **template catalog** is shipped content too, but works the other way
round: `templates/*.yaml` are blueprints that stay on disk until an admin
adds one, and what lands in the `templates` table is then the customer's,
editable and never re-imported (`app/template_catalog.py`). The dividing
line: content describing how *Pablan* works belongs to the product and stays
locked; content describing how *this company* works belongs to the customer.
`docs/authoring-templates.md` records the open questions around both.
Admins edit templates through a **form builder** (`lib/admin/TemplateBuilder`),
not raw YAML: fields for the persona, per-section heading + hint (add,
remove, reorder), title, language, visibility and model hints. The
skeleton the editor opens with is *derived* from the section headings (one
`## heading` each), so headings and hints never drift apart. The builder
posts the structured config to `POST /api/templates/build` — the frontend
has no YAML library, so it sends the `AuthoringTemplate` rather than
serializing it, and the backend validates the same schema a pasted YAML
goes through. Raw YAML stays one click away as an advanced escape hatch for
existing templates (`PUT /api/templates/{id}`), reachable and reversible
from the builder.
## EE plugin mechanism
- Core NEVER imports from `ee/` (enforced by import-linter in CI).
- `ee/backend` is an installable package `pablan_ee` exposing
`register(app, registry)`; activated only if installed AND a license key is
configured.
- `ee/frontend` fills named slots in `lib/ee/registry.ts` via a Vite alias;
without EE the alias points to an empty registry.
## Growth path
- **Variant B** (when needed): move the ingestion loop to a worker container;
optionally swap pgvector for Qdrant behind the retrieval interface.
- **Phase 2 — cross-department discovery**: entity/relation extraction during
ingestion into a simple edge table; a nightly job compares new knowledge
against other departments' documents and generates proactive hints. This is
the key market differentiator (see market research summary in README/docs)
but explicitly NOT part of the MVP.
docs/authoring-templates.md
+226
View File
@@ -0,0 +1,226 @@
# Authoring templates
A template is **not** an interview guide. It is a **Markdown skeleton** — a
starting document with headings the author fills in — plus a persona and
optional per-section hints that steer the section-refinement model. Templates
are **declarative configuration, not code**: a new document type (a machine
write-up, a decision record, a process) is a new YAML file, not a new
feature.
`templates/` ships a **catalog of blueprints** that an admin adds from — see
"Catalog and lifecycle" below; the `templates` table holds only what a
customer actually uses, and every row in it is editable. The full template is
stored in `templates.config` (JSONB) and validated on load against
`AuthoringTemplate` (`app/authoring/schema.py`, schema version 1.0). The schema
is versioned (`version` field) and expected to evolve after real use.
## Format
```yaml
id: prozess
name: "Ablauf: wie wir das machen"
version: "1.0"
kind: authoring
locale: de
description: >
Ein wiederkehrender Ablauf, Schritt für Schritt — so, dass jemand anderes
ihn allein schafft.
model:
temperature: 0.4
min_class_hint: "12b" # UI warning if the endpoint is weaker
persona: |
Du bist ein präziser Fachredakteur für Arbeitsanweisungen. Du schreibst
sachlich, in vollständigen Sätzen, und machst aus einer Abfolge eine
nummerierte Liste. Zahlen, Fristen, Systemnamen und Zuständigkeiten
behältst du exakt bei und erfindest keine dazu.
title_template: "Neuer Ablauf"
skeleton: |
## Wann das gilt
## Schritt für Schritt
## Wenn es klemmt
## Wer zuständig ist
sections:
- heading: "Wann das gilt"
hint: >
Der Auslöser: in welcher Situation dieser Ablauf greift, und wo er
nicht gilt.
- heading: "Schritt für Schritt"
hint: >
Die Schritte in ihrer Reihenfolge, jeder als eine Handlung — mit den
Systemen, Formularen und Fristen, die dazugehören.
- heading: "Wenn es klemmt"
hint: >
Die Sonderfälle und die Stellen, an denen es erfahrungsgemäß hakt.
- heading: "Wer zuständig ist"
hint: >
Wer den Ablauf verantwortet und wen man bei Rückfragen anspricht.
metadata:
visibility: department
```
**The headings are the questions a colleague actually asks.** That is the
whole craft in a template: "Wann das gilt / Schritt für Schritt / Wenn es
klemmt" gets filled in, "Worum es geht / Details / Was andere wissen müssen"
does not — it is a blank page wearing a structure. An `skeleton: ""` with no
sections is a legitimate template (`notiz`), and better than headings that ask
for nothing in particular.
## Fields
| Field | Meaning |
|---|---|
| `id` | Config id, stable across edits; the catalog and the picker key on it. Not the row UUID. |
| `name` | Human title in the picker. |
| `version` | Revision string, raised by hand when the skeleton or hints change. Read as a string even when the YAML looks like a float (`1.0`). |
| `kind` | Always `authoring`. Replaces the old `mode: interview` — it distinguishes an authoring template from any legacy config still shaped like an interview. |
| `locale` | The language this template's **content** is written in (`de` \| `en` \| null), not a UI string. The picker lists templates in the reader's language first. |
| `description` | One line shown in the picker and catalog. |
| `model.temperature` | Sampling temperature for the refinement call (default `0.4`). |
| `model.min_class_hint` | Advisory model class (e.g. `"12b"`); the UI warns when the configured endpoint looks weaker. |
| `persona` | The **editor voice** — how the refinement model should write. Injected as the system persona for every section of this document. |
| `skeleton` | The Markdown the editor **opens with**: headings the author fills in. This IS the starting content, not a description of it. An empty skeleton yields a blank document. |
| `sections` | Per-heading hints: `[{heading, hint}]`. |
| `title_template` | Draft title, with `{{user.name}}` and `{{date}}` substituted at creation. |
| `metadata.visibility` | Default visibility of the resulting document (`public` \| `department` \| `restricted`), changeable in the editor before publishing. |
## How a template drives the editor
1. **Creating a draft.** `POST /api/documents {template_id}` loads the
`AuthoringTemplate`, renders the skeleton (`render_skeleton`, verbatim with
a single trailing newline) and the title (`render_title`, with `{{user.name}}`
/ `{{date}}` substituted), and writes a `draft` document authored by the user
(`app/authoring/document.py`). A draft is author-only and never indexed — see
[`data-model.md`](data-model.md).
2. **Writing.** The editor opens on the skeleton. The author writes Markdown
directly; Markdown is the source of truth (rule 1), so there is no derived
render to watch.
3. **Section refinement.** After a typing pause the client asks the model to
mature the section at the cursor (`POST /api/documents/{id}/refine`, SSE).
The whole document travels as prefix/suffix context, but the model
regenerates **only** that section (FIM-style) so a large document is never
re-emitted whole. The prompt (`app/authoring/prompts.py`, natural language —
rule 7) is `persona` + the matching section `hint` + a fixed rule block
("return only the refined section, keep the heading, invent no facts").
**`sections[].heading` must match a skeleton heading exactly.** The server finds
the section the cursor sits in (`app/authoring/sections.py::active_section`,
sharing the heading/fence logic with `rag/chunking.py`), reads the heading text
the section starts with, and looks up its hint by exact string match
(`AuthoringTemplate.hint_for`). A hint whose `heading` does not appear in the
skeleton simply never reaches the model. A section with no hint still gets
refined — persona and the rule block are enough.
## Language policy
Shipped template **content** — persona, hints, descriptions, title templates,
skeleton headings — is **German**. Templates are product content for the German
market, exactly like the fixture corpus, NOT UI copy. Schema keys, structure
and section ids stay **English**. File naming is `<id>.<locale>.yaml`
(`prozess.de.yaml`); a file with no locale suffix belongs to the
default locale, so a customer can drop their own YAML in without learning the
convention.
## The catalog
`templates/` ships blueprints for common SME situations. A blueprint is inert:
it has no row in `templates` and cannot be used until an admin adds it.
| id | Purpose | Visibility of the result | Starter |
|---|---|---|---|
| `notiz` | Anything at all — **no skeleton**, an empty document | department | yes |
| `prozess` | How something is done here, step by step, with what snags | department | yes |
| `stoerung` | What broke, what caused it, what fixed it | department | yes |
| `person` | What someone does and what to ask them about | public | yes |
| `anlage` | A machine: running it, maintaining it, its quirks | department | |
| `entscheidung` | What was decided, why, and what follows | department | |
| `projekt-debrief` | What a project taught, in three questions | department | |
Two of these deserve their reasoning written down:
- **`notiz` has no skeleton on purpose.** Three generic headings ("What this
is about / Details / What others need to know") are what a blank page looks
like when it is trying to be helpful, and nobody writes a document that way.
Someone reaching for the open template already knows what they want to say.
- **`person` is written by the person themselves** (the profile page starts
it, `api/account.py PERSONAL_BLUEPRINT`) and is `public`: a directory that
half the company cannot read answers nobody's "who knows about X?". It
replaced an "onboarding" blueprint somebody else was supposed to fill in
FOR a new colleague — nobody does that, and the colleague knows the answers.
A draft stays private to its author until they publish it, so anything
sensitive can be removed first.
## Catalog and lifecycle
**A fresh instance seeds four starter blueprints** — `notiz`, `prozess`,
`stoerung`, `person` (`STARTER_TEMPLATE_IDS` in `app/template_catalog.py`) —
and only while the `templates` table is empty. They are the four occasions on
which anyone actually writes something down: write it down, how we do this,
what broke, who you are. Everything more specific — a machine, a decision, a
project review — is a deliberate add from the catalog, because a picker of ten
options is a picker nobody reads. `seed_starter_templates` is the ONLY automatic
write to the table, and it runs exclusively against an empty table; a startup
upsert would silently discard an admin's edits.
**Nothing in `templates` is read-only.** A template describes how a company
documents its own knowledge, so the company owns it — including the seeded ones
and every blueprint added later. Admins can edit the YAML, duplicate (fork gets
`<id>-kopie`), and delete; deleting is safe because the catalog can always
supply the blueprint again, and documents created from the template are
independent and survive.
This is the opposite of the built-in **help documents** (`app/help_import.py`),
which are re-imported on every start and refuse edits (409 `builtin_readonly`).
The rule behind both: content that describes *how Pablan works* belongs to the
product; content that describes *how this company works* belongs to the customer.
Adding a blueprint twice is refused (409 `already_added`) rather than
overwriting — the second add would silently discard the admin's edits.
Versioning stays manual: the `version` string in the YAML identifies a revision,
and the editor reminds the admin to raise it when the skeleton or hints change.
There is no version history table — a template is content under the customer's
control, not an audit trail.
## Validation approach
Cheap schema validation before any UI work: create a draft from the template,
write a rough section under each heading, and check that refinement matures the
prose without inventing facts, keeps the heading, and answers in the section's
language. A broken blueprint is logged and skipped on load rather than taking
the catalog down (`catalog_invalid`). The refinement prompt is covered by an
eval in `tests/evals`; extend it whenever the prompt or the rule block changes.
## Open problem: where built-in content lives
Pablan ships two kinds of content it did not get from the customer: the help
documents and the template catalog. The current split (help documents
re-imported and locked, templates offered and owned) is a decision we are
comfortable defending, but the surrounding questions are open:
- **How much should a fresh instance start with?** We seed four templates and a
handful of help documents. Should the seed be larger (a demo document set that
shows what a good captured document looks like), configurable at install time,
or nothing at all?
- **Where should the shipped content live?** Today: `templates/` and `help/` as
files in the repo, imported at startup. Files-in-repo is right for two
developers; it is not obviously right once the catalog has thirty entries in
several languages.
- **How does catalog content reach existing instances?** A blueprint improved in
a later release does not reach anyone who already added it, by design. There is
no "update available" signal — a diff view against the current blueprint is the
obvious feature, and it is not built.
- **Should deleting the last template be possible?** It is today. A member who
opens "Capture knowledge" in that state sees an empty picker.
This is deliberately recorded, not solved. Decide it with a real customer install
in front of us rather than from first principles.
docs/data-model.md
+289
View File
@@ -0,0 +1,289 @@
# Data model
All primary keys are UUIDs. All tables get `created_at` / `updated_at` via a
shared mixin (omitted in the ER diagram). Enum-like columns are stored as
`varchar(32)` backed by Python `StrEnum`s — no native Postgres enum types.
Naming rule: chat threads are **conversations**, login sessions are
**auth_sessions** — never mix these up.
The ER diagram lives in [`diagrams/data-model.svg`](diagrams/data-model.svg)
— one maintained source, updated with every model or migration change.
## Tables (13)
(Plus `alembic_version`, Alembic's migration bookkeeping table.)
### departments
`id, name (unique)`
### users
`id, email (unique), name, role (member|admin), password_hash (argon2),
locale (varchar(5), nullable), department_id → departments`
A person is described by a DOCUMENT, not by a profile column: the one started
from the person blueprint (`GET /api/account/document`). It is versioned,
approvable and findable by the search, which a text column on `users` was not.
`locale` pins the interface language (`de` | `en`); NULL follows the
browser's `Accept-Language`. One column rather than a preferences table: it
is the only preference that has to follow the person across devices. The
colour theme deliberately does NOT live here — it is per-device and sits in
the browser's `localStorage`.
`department_id` is nullable (`ON DELETE SET NULL`).
### auth_sessions
`id, user_id → users, expires_at, created_at`
Server-side sessions; the row id doubles as the httpOnly cookie token
(UUIDv4). Sessions expire after 14 days (`PABLAN_AUTH_SESSION_TTL_DAYS`) and
cascade when the user is deleted. Instantly revocable (deleting rows logs the
user out everywhere) — important because "employee leaves the company" is
literally our core use case. No JWT.
### templates
`id, name, version, config (jsonb)`
`config` holds an authoring template — a Markdown skeleton plus per-section
hints (see `authoring-templates.md`). Every row is the customer's own and
fully editable — there is no read-only template. The YAML files in
`templates/` are a *catalog* of blueprints that an admin adds from; a
blueprint has no row until it is added, and a fresh instance is seeded with
four starter templates.
### conversations
`id, mode (query|insight), status (active|completed|abandoned),
user_id → users`
One generic table for all modes. The only core mode is `query` (RAG Q&A); EE
adds `insight`. Adding a mode requires no migration. **Capture is no longer a
conversation** — it writes a Document directly (see `app/authoring/`), so this
table holds no per-turn engine state, template link, or result-document link
any more (the `state`, `template_id` and `result_document_id` columns were
dropped). Deleting a user cascades to their conversations.
### messages
`id, conversation_id → conversations, role (user|assistant|system),
content (text), meta (jsonb), created_at`
Full history is stored: state reconstruction after restarts, debugging/eval
material, and (aggregated only) knowledge-gap reporting. Messages cascade with
their conversation. Privacy rules below.
`meta` holds an assistant turn's citation snapshot
(`{"sources": [{document_id, title, heading_path, excerpt, used}]}`, `{}` for
user turns). Chunks are disposable derivatives that re-indexing replaces,
so the rendered citation is stored with the message rather than referenced
— reopening an old conversation still shows what was cited. `used` marks
whether a passage grounded the answer or was only retrieved and dropped as too
weak (a no-answer turn); the "?" context inspector shows the full retrieved set,
the cited-source badges only the `used` ones.
### documents
`id, title, status (draft|published|archived), is_builtin (bool),
visibility (public|department|restricted), content_md (text), meta (jsonb),
author_id → users, department_id → departments`
**Markdown is the source of truth.** `content_md` holds the canonical
document; everything else (chunks, embeddings) is derived and disposable.
`meta` holds the summary, extracted entities, and — for a captured
document — the `template` config id it was created from, plus (when the
capture started from a chat) the LLM topic summary of that conversation as
`context` and its `conversation_id`, used as background for section refinement.
**Capture writes here directly.** A captured document is authored by the
user, starting in `draft` (author-only, never indexed), and becoming
`published` when its author says so — one action, no queue. There is no
`source_type`: everything in the table was written in Pablan. The column and
its `upload` member went with the tags removal — the day an import lands, a
provenance column earns its place again, and can be reintroduced then.
**Three statuses, and only three.** `draft` is being written, `published` is
readable and searchable, `archived` is retired. Doubt about the CONTENT is
deliberately not a status: it is a `review_request` (below), because a
document can be published for months and still carry an unanswered question
about a number in it — which is exactly when readers need to be told.
**No tags, and no freshness date.** Both were removed rather than kept
around: tags were write-only (free text, no way to know which ones existed,
too inconsistent to filter by, and hybrid search already finds things), and
`verified_until` marked documents without ever asking a person anything. A
review request does that job — it names someone, carries a question, and can
sit on a document published months ago. Treat a UI that displays or edits
tags, or a date that flags a document as stale, as a regression.
`author_id` and `department_id` are nullable with `ON DELETE SET NULL` —
documents survive their author leaving the company.
`is_builtin` marks the shipped help pages that describe Pablan itself
(source: `help/*.md`, re-imported on every start). They are published,
public and authorless, and the API refuses to edit or delete them
(409 `builtin_readonly`) — the next deploy would overwrite the change.
### chunks
`id, document_id → documents, chunk_index (int), content (text),
embedding (vector(1024) — bge-m3; dimension change = migration + reindex),
tsv (tsvector, stored generated column, german config over the content AND
the heading path), meta (jsonb)`
Chunks are slices of a document (~400 tokens, split along Markdown heading
hierarchy; each chunk stores its heading path in `meta`, e.g.
"Ablauf: Reklamation › Schritt für Schritt", included as context in prompts and
citations). The heading path is not decoration: it is embedded WITH the chunk
(`rag/indexing.embedding_text`) and included in `tsv`, because a section saying
"Solldruck 180 bar" never repeats which machine it belongs to and is otherwise
unreachable by the name the asker uses. What is STORED as `content` stays the
raw section. `meta` also carries denormalized filter fields (department_id,
visibility) so retrieval doesn't need triple joins. Chunks are deleted
and regenerated whenever the document or the embedding model changes, and
cascade on document deletion.
Indexes: HNSW on `embedding` (cosine), GIN on `tsv`;
`(document_id, chunk_index)` is unique.
### doc_permissions
`document_id → documents, department_id → departments, level (read)`
Additional department read grants on top of `documents.visibility` — the
mechanism behind **multi-department sharing** (a document reaches departments
beyond its owning one). Managed through `PUT /api/documents/{id}/departments`
(author/admin), which replaces the full set of extra departments; the permission
filter's `EXISTS doc_permission` branch already unions them into reads and
retrieval, so sharing is management, not new permission logic. Composite PK
(document_id, department_id); both FKs `CASCADE`. Note the asymmetry with
`documents.department_id`, which is `SET NULL`: deleting a department **orphans**
its owned documents (they keep existing, department set NULL) but **drops** its
grant rows here, silently removing that department's shared access — an admin
lockout vector to weigh before deleting a department.
### review_requests
`id, document_id → documents, requester_id → users, reviewer_id → users,
question (text, nullable), resolved_at (timestamptz, nullable),
resolved_by_id → users, created_at, updated_at`
"Please check this", addressed to one colleague, optionally about something
specific ("do the 14 holiday days still hold?"). Open while `resolved_at` is
NULL; the pair (`resolved_at`, `resolved_by_id`) is the answer. All three user
FKs are `ON DELETE SET NULL` (a request outlives the accounts on either side of
it); rows `CASCADE` with their document.
It is deliberately **not** a status, and it is orthogonal to one: a request can
sit on a draft the author is unsure about OR on a document published months
ago. An open request marks the document wherever it appears — the list, the
detail page, and the sources under a chat answer (`review_pending` on every
search hit).
Being asked is what **grants the right to edit**: a reviewer who spots a wrong
number should fix it rather than file a second question about it. The grant is
scoped to the open request and disappears with the answer — for a draft that
also means the reviewer's read access ends there. The relationship is loaded
with every document (`lazy="selectin"`), because whether a question is open
decides both who may edit and how the document is marked.
### document_events
`id, document_id → documents, actor_id → users, action, content_md,
title, visibility, meta, created_at`
An append-only audit trail: who changed or reviewed a document, and when.
`action` is one of `created | edited | published | archived |
visibility_changed | review_requested | review_resolved`. Content-bearing events (`created`, `edited`)
snapshot the Markdown source of truth (`content_md`, `title`, `meta`) so a past
version can be viewed or diffed — the disposable chunks are never snapshotted
(rule 1); pure transitions carry no content snapshot. `visibility` is small, so
it is recorded on every event. `actor_id` is `ON DELETE SET NULL` so the trail
outlives the actor's account (like `author_id`); events `CASCADE` with their
document.
This is where **"who checked it"** lives: `review_resolved` records the
colleague who answered as `actor_id`, next to the `review_requested` that asked
— the two events are the record that someone with the knowledge looked at this,
which the `review_requests` row alone (one open flag) does not preserve once it
is answered. History is read through the document's own permission gate
(`GET /api/documents/{id}/history`), so it never leaks to a user who cannot read
the document.
### llm_settings
`id, role (unique), base_url, model, api_key, created_at, updated_at`
Per-role endpoint overrides edited in the admin UI; at most three rows.
NULL means "inherit from the environment" (precedence env < DB, per
field — see `architecture.md`). `api_key` is plaintext because it is
replayed to the endpoint; it is never returned by the API and never
logged.
### prompt_settings
`id, key (unique), content, created_at, updated_at`
Admin overrides for shipped system prompts. Every prompt has a CODE default
(`app/prompts/defaults.py`); a row exists here only when an admin has changed
one, and `content` is the full replacement. Applied without a restart via a
module cache (`app/prompts/overrides.py`), refreshed on every write — like
`llm_settings`, and single-process by design (`--workers 1`). Resetting a prompt
deletes its row so the code default takes over. Unlike LLM settings there is no
`.env` layer: prompts have no environment representation, so the reset target is
the code default. Keys: `query_system`, `query_no_sources`, `refine_persona`,
`refine_rules`, `grounding_framing`, `topic_summary`, `title`, `bio_persona`,
`bio_rules`.
### jobs
`id, type, payload (jsonb), status (pending|running|done|failed),
run_after, attempts, last_error`
Postgres-backed background queue, claimed via `FOR UPDATE SKIP LOCKED`.
Indexed on `(status, run_after)` for the claim query.
## Permission model
Two layers:
1. `documents.visibility` as the fast default:
- `public` — whole company
- `department` — author's department only
- `restricted` — only explicit grants
2. `doc_permissions` for additional department grants.
Authors always see their own documents: any status through the API (writing
needs drafts), published-only in retrieval — unpublished documents are never
searchable, not even for their author. A colleague with an **open review
request** sees the document the same way: an open-request clause in
`readable_documents_filter` but never in `searchable_documents_filter`, so
being asked to check a draft reaches the API without ever leaking into
chat/search. That clause is also the edit gate (`can_edit` = author, admin, or
open reviewer): whoever is asked may fix what they find, until they answer.
**Self-lockout guard.** Before a visibility or grant change is committed, the
proposed state is evaluated against the read rules (`_guard_self_lockout`, a
Python mirror of `readable_documents_filter`). An author always keeps access as
author, so this only bites an **admin** editing a document they do not own: the
change is refused (409 `self_lockout_warning`) until they resend it with
`confirm_lockout`, after which they may knowingly give up their own access.
**The filter runs BEFORE the LLM ever sees anything.** Retrieval
(`rag/retrieval.search`) requires a `user` argument and applies the
visibility CTE (visibility rules + doc_permissions + `status = published`)
before vector/full-text search. It is structurally impossible to retrieve a
chunk the requesting user may not read. The filter is shared with the
documents API (`rag/permissions.py`) and always reads the live documents
table — never the denormalized copies in chunk meta, which can be stale
between an edit and the next reindex.
## Retrieval (hybrid)
Sequence: [`diagrams/retrieval-sequence.svg`](diagrams/retrieval-sequence.svg)
One SQL query: visibility CTE → in parallel (a) vector top-20 via
`embedding <=> :qvec` (HNSW) and (b) full-text top-20 via
`tsv @@ websearch_to_tsquery('german', :q)` → merged with Reciprocal Rank
Fusion → top-k. Rationale: pure vector search underperforms on German
technical terms, product names, and error codes; hybrid is the default.
**The full-text half also works alone.** `chunks.tsv` is a stored generated
column (`to_tsvector('german', content || ' ' || heading path)`) behind a GIN
index — an inverted
index that needs no model at all. `text_search()` uses exactly that, with the
same permission CTE, and query mode falls back to it when the embedding
endpoint is unreachable. Keyword matching finds less than the hybrid path, so
the user is told (`fallback` in `api-protocol.md`), but the knowledge base
stays searchable when nothing is running behind the LLM config.
## Privacy / GDPR defaults
- Deleting a user (offboarding) deletes their conversations and messages
via `ON DELETE CASCADE` — the transcript is personal data of the departed
employee; the published document is the legitimate artifact and survives
authorless (`author_id` SET NULL).
- Users can delete their own conversations.
- Configurable retention: query-mode conversations are auto-deleted after a
default of 90 days (`retention_cleanup` job).
- A captured document is private until its author publishes it
(`draft → published`, the author's own action) — nothing enters the
knowledge base without the writer's consent.
- Insights features (EE) only ever receive aggregated, anonymized data —
never per-user raw messages.
docs/decisions.md
+311
View File
@@ -0,0 +1,311 @@
# Decision log
Short ADR-style records. Context for all: team of 2 developers + 1
legal/marketing co-founder; product must be self-hostable by SME IT admins;
bias toward boring, simple, understandable technology.
## D1 — PostgreSQL + pgvector, no dedicated vector DB
One database for relational data, documents, vectors, full-text, jobs and
sessions. `docker compose up` starts 3 containers; nothing extra to operate,
patch and back up. HNSW handles our scale (SMEs, hundreds of thousands of
chunks) easily. Retrieval is capsuled behind `rag/retrieval.py`, so swapping
in Qdrant later (Variant B) is a contained refactor. Bonus: Postgres
full-text (`german` config) enables hybrid search with zero extra
infrastructure.
## D2 — SQLAlchemy 2.0 + Alembic, not Prisma
Prisma is TypeScript-first; the DB belongs to the FastAPI backend (the
Python client is a community project — not production-grade). Prisma treats
pgvector as `Unsupported("vector")`, forcing raw SQL anyway. The official
pgvector package ships a first-class SQLAlchemy type.
## D3 — No LangChain / LlamaIndex
Thin custom LLM client (3 functions) + our own mode engine. For a focused
product the framework abstraction costs more than it saves; prompts and
retrieval are our core competence and must stay fully controllable.
## D4 — OpenAI-compatible API as the only LLM contract
llama.cpp server, vLLM, Ollama and all major clouds speak it. Config is just
base_url/api_key/model — per model role (chat / utility / embedding).
The serving layer applies each model's chat template; we never hand-build
prompt formats.
## D5 — Server-side sessions, no JWT; auth in FastAPI, not Auth.js
Auth.js would put identity in the SvelteKit server while enforcement
(document permissions, retrieval filtering) lives in FastAPI — two systems
to keep in sync. Server-side sessions are instantly revocable ("employee
leaves" is our core use case), trivial to reason about, and have zero
drawbacks for single-backend self-hosted deployments.
## D6 — Sessions naming
`conversations` (chat threads) vs `auth_sessions` (logins). Never "session"
unqualified.
## D7 — Background jobs via Postgres table, no Redis/Celery
`jobs` table + asyncio loop with `FOR UPDATE SKIP LOCKED`. ~80 lines,
survives restarts, built-in retry, no new infrastructure for customers.
Moves unchanged into a worker container when scale demands it.
## D8 — SvelteKit is pure UI
All `/api` traffic goes through the reverse proxy directly to FastAPI (same
origin: no CORS, cookies and SSE just work, no stream buffering through
Node). SvelteKit renders pages and gates routes via `hooks.server.ts`.
## D9 — Own components on Bits UI, no component library imports
Bits UI provides the hard invisible parts (focus traps, keyboard
conventions, ARIA state, portal/positioning, outside-click) as a headless
layer; we own 100% of markup and styling. shadcn-svelte was considered (it
copies source rather than importing) but we skip its styling layer entirely.
Discipline required: small base component set, everything from design
tokens.
## D10 — Caddy as the only supported reverse proxy (for now)
Argument is config ergonomics for foreign IT admins (5-line Caddyfile,
automatic TLS) — not capability. The app is proxy-agnostic (one origin,
`/api/*` to the backend, rest to the frontend), so other proxies can be
added to `deploy/` when a customer actually needs one; until then we
maintain exactly one config.
## D11 — uv + ruff (lint AND format), pre-commit, Makefile as task runner
One Python toolchain, no Black. `make` because it is universally installed
and gives CI and both developers identical commands.
## D12 — Markdown as canonical knowledge format
Human-readable, versionable, exportable — no data prison (sales argument).
Chunks/embeddings are disposable derivatives; re-indexing from documents
must always be possible (e.g. embedding model switch).
## D13 — Step-wise prompt injection for interviews (SUPERSEDED by D19)
Only the current interview point enters the prompt; a separate `chat_json`
bookkeeper call tracks the checklist. Keeps Gemma-class 12B models reliable
and shrinks the local-vs-cloud quality gap. **Superseded:** capture is no
longer a dialog — the interview engine and bookkeeper were removed. The
successor keeps small models fast a different way: FIM section refinement (D19).
## D14 — Fair Source (FSL) + proprietary `ee/`
See `licensing.md`.
## D15 — Privacy defaults are product features
Approval-gated publishing, user-deletable conversations, retention job,
aggregation-only insights. Designed for the German works-council/GDPR
conversation from day one — retrofitting this would be painful; leading with
it is a sales argument.
## D16 — Hybrid dev stack: Postgres in Docker, app processes native
`make dev` starts only Postgres via `docker-compose.dev.yml` (own compose
project name `pablan-dev`) and runs uvicorn `--reload` and vite natively on
the host — fast reloads, IDE-friendly `.venv`/`node_modules`, no bind-mount
quirks. The fully containerized `docker-compose.yml` is the customer
deployment and doubles as the pre-release check (`docker compose up -d
--build` with `PABLAN_DOMAIN=localhost`); separate compose project names let
both run on one machine.
## D17 — Display rename "Guided documentation" without id migration (SUPERSEDED by D19)
The feature formerly presented as "Interview" was renamed **"Guided
documentation"** in user-facing copy (M8), keeping the `interview` mode id and
module. **Superseded:** the interview mode itself was removed — capture is now
a writing editor (D19), so there is no interview engine to name.
## D18 — LLM endpoint configuration in the DB, env as fallback
Endpoints move during a pilot (local llama.cpp → cloud, or a new model),
and asking a customer to edit `.env` and restart for that is a support
call. Admins now override `base_url` / `model` / `api_key` per role in the
UI; the environment stays the bootstrap and the per-field fallback, so a
fresh deployment works with no database rows at all. Changes apply without
a restart by rebuilding the cached clients. The key is stored in plaintext
because it must be replayed to the endpoint — it is never returned by the
API and never logged, and the database is the customer's own.
## D19 — Capture inverted to a writing-first editor
Capture was a dialog: an interviewer asked, a bookkeeper extracted notes, and
the document was a render of those notes — the user never wrote it. That reads
like an interrogation and fights the reality that someone *has* something to
write down. Markdown is already the source of truth (D12/rule 1), so the user
should edit that source directly, not watch a derived render. Capture is now a
**writing editor**: the user writes Markdown into a `Document` (`app/authoring/`),
and after a typing pause the model matures only the section at the cursor.
Consequences that make this cheap and safe on small local models:
- **Drafts are author-only and never indexed.** A captured document starts as
a `draft`, private to its author and outside retrieval, so it can never reach
another user or an LLM prompt (rule 2) before its author publishes it.
- **FIM section refinement.** The whole document is prefix/suffix context but
the model regenerates ONLY the active section, so a large document is never
re-emitted whole. With the reasoning model's thinking disabled
(`enable_thinking = false`) the call returns in ~1s — fast enough for a
"after pause" trigger and reliable on Gemma-class 12B.
- **One editor for capture and edit, with a pre-save diff.** The same editor
writes new drafts and revises existing documents, always showing a VSCode-style
line diff before saving.
- **CodeMirror 6 + `@codemirror/merge` as a deliberate dependency.** CodeMirror
is an editor/diff engine (like marked/dompurify), not a UI component library,
so it is compatible with D9; `@codemirror/merge`'s `unifiedMergeView` provides
the pre-save diff, which is why no separate diff library (jsdiff) is pulled in.
This removed the interview mode, bookkeeper, checklist, the `Draft`/`Navigable`
protocols, and the conversation capture endpoints; `query` is the only core
mode left. See `architecture.md` and `authoring-templates.md`.
## D20 — Reviewer access as a read clause, not a new permission layer
An author can ask a colleague to check a document, and that colleague must be
able to read it even while it is an unpublished draft — without anything
leaking into chat/search. So the grant is a single clause (an open
`review_requests` row addressed to the user) added to
`readable_documents_filter` **only** — never to `searchable_documents_filter` —
reusing the existing filter split rather than introducing a status or a grant
table. Reviewer candidates are derived department-granular from the document's
would-be read set (author + public/department/`doc_permissions`) via a
dedicated non-admin, permission-safe query returning id + name only — not by
reusing `/admin/users`. (Superseded in part by D24: the clause is now the open
request, not a `documents.reviewer_id` column, and being asked also grants the
right to EDIT.)
## D21 — Topic-summary retrieval, in the capture picker only (for now)
When a capture starts from a chat, the picker suggests existing documents to
extend. Retrieving over the raw last chat message is unreliable (the last turn is
often a topic-less follow-up like "thanks"), so the match runs over a short **LLM
topic summary** of the conversation (`chat_json`, utility) instead. An eval
(`tests/evals/test_topic_retrieval_eval.py`) compares topic-summary vs.
raw-message recall on a fixture; measured 4/4 vs. 1/4. On that evidence, query
mode now uses it too, but only as a **fallback**: it searches on the raw message
first and re-searches with a topic summary of the conversation ONLY when the raw
search is low-confidence and there is earlier context (`query.py::_topic_transcript`
returns "" for a first message). So a clear question pays no extra latency and a
first-message miss stays a genuine no-answer, while a topic-losing follow-up
recovers the subject. The same topic summary is stored as `meta.context` on a
chat-originated draft and passed to the refine prompt as background, so refinement
is chat-aware without re-summarizing per call.
## D22 — Cache-friendly query prompt: static system, volatile excerpts last
Local llama.cpp and cloud OpenAI-compatible endpoints both reuse a request's
longest identical prefix (prompt/KV caching). The old query prompt injected the
per-question excerpts INTO the system message, so everything after the static
rules — including the conversation history — was reprocessed every turn. The
message order is now `[static system rules] [conversation history] [this turn's
excerpts + question]` (`modes/prompts.py::render_context_turn`): the system rules
are byte-identical for every question (cached across all conversations and users)
and the history is byte-identical across a conversation's turns (cached within
it), so only the volatile excerpts + question are new work. The excerpts are
inherently per-question and never cacheable; keeping them out of the shared
prefix is what makes the rest cacheable. No endpoint config is needed — the win
is purely in prompt structure.
## D23 — API modules are packages cut by caller intent, with the gate on the router
`api/documents.py` had grown to 1137 lines and `api/admin.py` to 615, which is
what happens when "one module per resource" meets a resource with five different
jobs. They are packages now, cut by what a caller is DOING — browse, the life of
one document, its audit trail, the publish/review workflow, sharing — rather
than by HTTP verb or by schema-vs-route. Two consequences are the actual point:
- **Shared rules exist once, and are stated.** `documents/access.py` holds every
gate (including the Python mirror of the read filter used to judge a change
before it commits) and `documents/view.py` every response shape. Modules that
used to reach into another module's private helpers (`_readable_document`,
`_role_config`, `_excerpt`, `_HEADING_RE`) now import a named interface.
- **A dependency belongs on the router, not on each endpoint.** Each package has
a `routing.py` that builds its `APIRouter`; `admin` puts `require_admin`
there, and `templates` has two constructors because reading is for everyone
and writing is not. A new endpoint is gated by the router it is added to,
which is not something an author can forget.
The cost is that route order becomes explicit: FastAPI matches in registration
order, so the module with static paths is included before the one with
`/{id}`. That is stated in each package's `__init__` rather than discovered.
## D24 — Review is a question about a document, not a status of it
Publishing used to run through an approval queue: `draft → pending_approval →
published`, with a delegated reviewer who could approve but not edit. Two things
were wrong with it. It made every author wait for someone else before their
knowledge was findable at all — the opposite of "writing it down must be
cheap" — and it could only express doubt BEFORE publishing, while the doubt that
actually matters ("do the 14 holiday days still hold?") turns up on a document
that has been published for months.
So the two ideas were separated:
- **Status says where a document stands**: `draft` (being written, author-only,
unindexed), `published`, `archived`. Publishing is the author's own one-click
action; `pending_approval` is gone.
- **A `ReviewRequest` says what is unsettled about its CONTENT** — one question,
addressed to one colleague, open until answered. It can sit on a draft or on a
document published months ago, and it is orthogonal to the status.
Consequences that make it work:
- **An open request marks the document everywhere**, including the sources under
a chat answer (`review_pending` on every search hit, snapshotted with the
citation). A reader who is handed an answer sees that the passage behind it is
not confirmed — which is the case the whole feature exists for.
- **Being asked grants the right to edit** (`can_edit` = author, admin, or open
reviewer), not just to read: a reviewer who spots a wrong number should fix it
rather than file a second question about it. The grant ends with the answer,
and for a draft the read access ends with it too. The owner-only decisions
(delete, sharing, handing out a request) stay with the author or an admin.
- **The audit trail keeps who checked what**: `review_requested` /
`review_resolved` events, with the answering colleague as `actor_id`. The
`review_requests` row alone would only tell you that nothing is open.
The migration maps `pending_approval` → `draft`, `approved` events → `published`,
drops `submitted` events and `documents.reviewer_id`.
## D25 — Three fields removed rather than kept "for later"
`documents.verified_until`, `meta.tags` and `documents.source_type` all
existed, were all written on every publish, and none of them did anything.
They were removed together.
- **`verified_until`** was a freshness horizon: 180 days after publishing, the
UI put "Verifizierung fällig" on the document and offered a Re-verify button.
It marked documents and reminded nobody — no mail, no queue, no name attached
— so the badge said only "time has passed", which the `updated_at` in the
same line already said. A `ReviewRequest` (D24) is the thing that actually
asks: it names a person, carries a question, and can sit on a document
published months ago.
- **`tags`** had been write-only since M8: free text with no way to see which
tags already existed, so the values that arrived were too inconsistent to
filter by, and hybrid search covers finding things anyway. Keeping the column
"because the data is real" only kept the template form asking for it.
- **`source_type`** had two members and one of them (`upload`) was unreachable:
nothing can be uploaded. Everything in the table was written in Pablan, so
the column classified nothing.
The general rule this records: a field nothing reads is not neutral. It costs a
form row, a table column, a line in the export frontmatter, an entry in the ER
diagram, and a paragraph of documentation explaining why it is there. Import
(roadmap epic 19) will want provenance, and can reintroduce a column that
distinguishes values it actually produces.
## D26 — The queue in front of the endpoint is ours, not the server's
A self-hosted llama.cpp server takes every request it is offered and serves
`--parallel` of them at a time; the rest wait inside it. That is a queue Pablan
can neither see, bound, nor explain — and every request waiting in it is still
burning the client's HTTP timeout. Two colleagues chatting while a reindex runs
is enough to turn "slow" into "everything times out at once".
So the waiting happens in `app/llm/gate.py`, in front of the endpoint, and it
is bounded twice: a caller waits at most `PABLAN_LLM_QUEUE_WAIT_SECONDS` for a
slot, and past `PABLAN_LLM_MAX_QUEUED` waiters the gate stops admitting at all.
An unbounded wait would be the same failure as no gate, just quieter.
Three details that make it work:
- **Keyed by base_url, not by role.** chat and utility point at the same server
in the shipped configuration; a per-role limit would let one server be given
twice its slots.
- **Both refusals are `llm_busy`.** A queue timeout and a provider's 429 mean
the same thing to the person waiting, so they get the same sentence.
- **A stream holds its slot to the last token**, because that is how long it
occupies the server's slot.
What it buys beyond fairness: a saturated EMBEDDING endpoint now degrades query
mode to full-text retrieval (that path already existed for an unreachable
endpoint) instead of failing, and the chat can say "the model is busy" as a
`queued` phase rather than blinking a cursor for twenty seconds.
`probe()` and `list_models()` stay outside the gate: an admin has to be able to
test an endpoint precisely when it is saturated.
docs/diagrams/README.md
+24
View File
@@ -0,0 +1,24 @@
# Architecture diagrams
Hand-authored SVG, one per subject. The SVG **is** the artifact: no diagram
toolchain, no build step, and it renders anywhere a browser or an IDE preview
does. Each file carries its own stylesheet and follows the reader's light or
dark mode through `prefers-color-scheme`.
**Maintenance rule:** diagrams are as-is documentation like the rest of
`docs/`. When a change lands that affects them, the diagram is edited in the
same change: a stale diagram is a bug. Where docs and code disagree, the code
wins and the doc gets fixed in the same commit.
**Editing:** every box is a `<rect>` plus its `<text>` lines at explicit
coordinates on a plain grid, so a label change is a text edit and a new box is
a copied block with new numbers. Keep the shared `<style>` block identical
across the five files so they stay one visual system.
| Diagram | Shows |
|---|---|
| [data-model.svg](data-model.svg) | ER diagram of all 13 tables, FK semantics, uniques (mirrors the actual migrations) |
| [components.svg](components.svg) | System components and their dependencies; built vs. planned (dashed) |
| [auth-sequence.svg](auth-sequence.svg) | Login (argon2, session row, httpOnly cookie) and per-request validation incl. the 401 to redirect branch |
| [queue-sequence.svg](queue-sequence.svg) | Job queue: claim via FOR UPDATE SKIP LOCKED, handler inside the open claim transaction, backoff, crash safety |
| [retrieval-sequence.svg](retrieval-sequence.svg) | Hybrid retrieval, the text-only fallback, and the similarity path: permission CTE first in every one of them |
docs/diagrams/auth-sequence.svg
+158
View File
@@ -0,0 +1,158 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1000 1290" width="1000" height="1290" role="img" aria-label="Login and every page request">
<title>Login and every page request</title>
<style>
:root {
--bg: #fbfaf9;
--surface: #ffffff;
--sunken: #f2f0ed;
--ink: #1a1815;
--muted: #63605a;
--line: #c5bfb6;
--line-soft: #e5e1db;
--accent: #b8770a;
--secondary: #b04a00;
--note: #faf3e6;
}
@media (prefers-color-scheme: dark) {
:root {
--bg: #0f0e0d;
--surface: #1a1817;
--sunken: #232120;
--ink: #efece8;
--muted: #a49d94;
--line: #4a443d;
--line-soft: #332f2b;
--accent: #e6b422;
--secondary: #e08030;
--note: #241f16;
}
}
text { font-family: ui-sans-serif, system-ui, -apple-system, 'Segoe UI', Roboto, sans-serif; fill: var(--ink); }
.t-title { font-size: 15px; font-weight: 650; }
.t-sub { font-size: 11.5px; fill: var(--muted); }
.t-node { font-size: 12.5px; font-weight: 600; }
.t-body { font-size: 11.5px; }
.t-muted { font-size: 11px; fill: var(--muted); }
.t-edge { font-size: 10.5px; fill: var(--muted); }
.t-mono { font-family: ui-monospace, SFMono-Regular, Menlo, Consolas, monospace; font-size: 11px; }
.box { fill: var(--surface); stroke: var(--line); stroke-width: 1.2; }
.box-sunken { fill: var(--sunken); stroke: var(--line-soft); stroke-width: 1; }
.box-accent { fill: var(--surface); stroke: var(--accent); stroke-width: 1.6; }
.note { fill: var(--note); stroke: var(--line-soft); stroke-width: 1; }
.group { fill: none; stroke: var(--line-soft); stroke-width: 1.2;
stroke-dasharray: 5 4; }
.lifeline { stroke: var(--line); stroke-width: 1; stroke-dasharray: 4 4; }
.edge { stroke: var(--line); stroke-width: 1.3; fill: none; }
.edge-accent { stroke: var(--secondary); stroke-width: 1.5; fill: none; }
.edge-soft { stroke: var(--line); stroke-width: 1.1; fill: none;
stroke-dasharray: 5 4; }
.edge-soft-accent { stroke: var(--secondary); stroke-width: 1.3; fill: none;
stroke-dasharray: 5 4; }
.planned { stroke-dasharray: 6 4; opacity: 0.62; }
</style>
<defs>
<marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5"
markerWidth="7" markerHeight="7" orient="auto-start-reverse">
<path d="M 0 1 L 9 5 L 0 9 z" fill="var(--line)"/>
</marker>
<marker id="arrow-accent" viewBox="0 0 10 10" refX="9" refY="5"
markerWidth="7" markerHeight="7" orient="auto-start-reverse">
<path d="M 0 1 L 9 5 L 0 9 z" fill="var(--secondary)"/>
</marker>
<marker id="arrow-open" viewBox="0 0 10 10" refX="9" refY="5"
markerWidth="8" markerHeight="8" orient="auto-start-reverse">
<path d="M 0 1 L 9 5 L 0 9" fill="none" stroke="var(--line)"
stroke-width="1.3"/>
</marker>
</defs>
<rect width="1000" height="1290" fill="var(--bg)"/>
<text x="28" y="30" class="t-title" text-anchor="start">Login and every page request</text>
<text x="28" y="50" class="t-sub" text-anchor="start">Server-side sessions: the cookie token IS the auth_sessions row id.</text>
<rect x="98" y="74" width="96" height="30" rx="8" class="box" />
<text x="146" y="93" class="t-node" text-anchor="middle">Browser</text>
<line x1="146" y1="104" x2="146" y2="1272" class="lifeline"/>
<rect x="316.2" y="74" width="131.6" height="30" rx="8" class="box" />
<text x="382" y="93" class="t-node" text-anchor="middle">SvelteKit server</text>
<line x1="382" y1="104" x2="382" y2="1272" class="lifeline"/>
<rect x="570" y="74" width="96" height="30" rx="8" class="box" />
<text x="618" y="93" class="t-node" text-anchor="middle">FastAPI</text>
<line x1="618" y1="104" x2="618" y2="1272" class="lifeline"/>
<rect x="806" y="74" width="96" height="30" rx="8" class="box" />
<text x="854" y="93" class="t-node" text-anchor="middle">Postgres</text>
<line x1="854" y1="104" x2="854" y2="1272" class="lifeline"/>
<line x1="28" y1="138" x2="972" y2="138" class="edge-soft"/>
<rect x="336.325" y="127" width="327.35" height="22" rx="11" class="box-sunken" />
<text x="500" y="142" class="t-muted" text-anchor="middle">Login: the browser talks to FastAPI directly (D8)</text>
<text x="382" y="170" class="t-edge" text-anchor="middle">POST /api/auth/login {email, password}</text>
<line x1="146" y1="187" x2="618" y2="187" class="edge" marker-end="url(#arrow)"/>
<text x="736" y="209" class="t-edge" text-anchor="middle">SELECT user by email</text>
<line x1="618" y1="226" x2="854" y2="226" class="edge" marker-end="url(#arrow)"/>
<rect x="644" y="268" width="321.2" height="40" rx="6" class="note" />
<text x="657" y="285" class="t-body" text-anchor="start">Unknown email still burns a dummy argon2 verify,</text>
<text x="657" y="298" class="t-body" text-anchor="start">so timing reveals nothing about who exists.</text>
<text x="382" y="326" class="t-edge" text-anchor="middle">401 {detail, code: invalid_credentials}</text>
<line x1="618" y1="343" x2="146" y2="343" class="edge-soft" marker-end="url(#arrow-open)"/>
<text x="736" y="407" class="t-edge" text-anchor="middle">INSERT auth_sessions</text>
<text x="736" y="420" class="t-edge" text-anchor="middle">(id = cookie token, expires_at = now + 14d)</text>
<line x1="618" y1="437" x2="854" y2="437" class="edge" marker-end="url(#arrow)"/>
<text x="382" y="459" class="t-edge" text-anchor="middle">200 user + Set-Cookie pablan_session</text>
<text x="382" y="472" class="t-edge" text-anchor="middle">(httpOnly, SameSite=Lax, Secure per PABLAN_COOKIE_SECURE)</text>
<line x1="618" y1="489" x2="146" y2="489" class="edge-soft-accent" marker-end="url(#arrow-accent)"/>
<line x1="28" y1="533" x2="972" y2="533" class="edge-soft"/>
<rect x="336.325" y="522" width="327.35" height="22" rx="11" class="box-sunken" />
<text x="500" y="537" class="t-muted" text-anchor="middle">Every page request: handle = sequence(auth, i18n)</text>
<text x="264" y="565" class="t-edge" text-anchor="middle">GET /some-page (Cookie: pablan_session, Accept-Language)</text>
<line x1="146" y1="582" x2="382" y2="582" class="edge" marker-end="url(#arrow)"/>
<text x="500" y="604" class="t-edge" text-anchor="middle">GET /api/auth/me (absolute URL, cookie forwarded)</text>
<line x1="382" y1="621" x2="618" y2="621" class="edge" marker-end="url(#arrow)"/>
<rect x="408" y="637" width="351.95" height="40" rx="6" class="note" />
<text x="421" y="654" class="t-body" text-anchor="start">The base URL must be absolute: a relative server-side</text>
<text x="421" y="667" class="t-body" text-anchor="start">fetch never reaches the proxy.</text>
<text x="736" y="695" class="t-edge" text-anchor="middle">SELECT auth_sessions + user, check expires_at</text>
<line x1="618" y1="712" x2="854" y2="712" class="edge" marker-end="url(#arrow)"/>
<text x="500" y="760" class="t-edge" text-anchor="middle">200 user (incl. locale)</text>
<line x1="618" y1="777" x2="382" y2="777" class="edge-soft" marker-end="url(#arrow-open)"/>
<rect x="408" y="793" width="444.2" height="53" rx="6" class="note" />
<text x="421" y="810" class="t-body" text-anchor="start">Auth runs FIRST and stashes user.locale in a WeakMap keyed by the</text>
<text x="421" y="823" class="t-body" text-anchor="start">Request: the locale strategy only receives the request, and a second</text>
<text x="421" y="836" class="t-body" text-anchor="start">/me would ask a question we already asked.</text>
<rect x="408" y="858" width="450.35" height="40" rx="6" class="note" />
<text x="421" y="875" class="t-body" text-anchor="start">i18n: paraglideMiddleware resolves userPreference &gt; cookie &gt;</text>
<text x="421" y="888" class="t-body" text-anchor="start">Accept-Language &gt; base, then stamps %lang% / %dir% into the document.</text>
<text x="264" y="916" class="t-edge" text-anchor="middle">render with locals.user, in the resolved locale</text>
<line x1="382" y1="933" x2="146" y2="933" class="edge-soft" marker-end="url(#arrow-open)"/>
<text x="500" y="997" class="t-edge" text-anchor="middle">401 {code: not_authenticated}</text>
<line x1="618" y1="1014" x2="382" y2="1014" class="edge-soft" marker-end="url(#arrow-open)"/>
<rect x="408" y="1030" width="438.05" height="27" rx="6" class="note" />
<text x="421" y="1047" class="t-body" text-anchor="start">No user, so the locale falls back to the cookie or Accept-Language.</text>
<text x="264" y="1075" class="t-edge" text-anchor="middle">303 redirect to /login</text>
<line x1="382" y1="1092" x2="146" y2="1092" class="edge-soft" marker-end="url(#arrow-open)"/>
<line x1="28" y1="1136" x2="972" y2="1136" class="edge-soft"/>
<rect x="437.8" y="1125" width="124.4" height="22" rx="11" class="box-sunken" />
<text x="500" y="1140" class="t-muted" text-anchor="middle">Ending a session</text>
<rect x="290.2" y="1162" width="419.6" height="92" rx="6" class="note" />
<text x="303.2" y="1179" class="t-body" text-anchor="start">Logout deletes the session row: revoked everywhere, instantly.</text>
<text x="303.2" y="1192" class="t-body" text-anchor="start"></text>
<text x="303.2" y="1205" class="t-body" text-anchor="start">Password change (POST /api/account/password): verify the current</text>
<text x="303.2" y="1218" class="t-body" text-anchor="start">hash, store the new one, then DELETE auth_sessions WHERE</text>
<text x="303.2" y="1231" class="t-body" text-anchor="start">user_id = me AND id &lt;&gt; my session. Other devices are logged out,</text>
<text x="303.2" y="1244" class="t-body" text-anchor="start">this one keeps its cookie.</text>
<rect x="28" y="242" width="944" height="125" rx="8" class="group" />
<rect x="28" y="242" width="38.45" height="20" rx="6" class="box-sunken" />
<text x="47.225" y="256" class="t-muted" text-anchor="middle">alt</text>
<text x="78.45" y="256" class="t-muted" text-anchor="start">unknown email or wrong password</text>
<rect x="28" y="375" width="944" height="138" rx="8" class="group" />
<rect x="28" y="375" width="44.6" height="20" rx="6" class="box-sunken" />
<text x="50.3" y="389" class="t-muted" text-anchor="middle">else</text>
<text x="84.6" y="389" class="t-muted" text-anchor="start">credentials valid</text>
<rect x="28" y="728" width="944" height="229" rx="8" class="group" />
<rect x="28" y="728" width="38.45" height="20" rx="6" class="box-sunken" />
<text x="47.225" y="742" class="t-muted" text-anchor="middle">alt</text>
<text x="78.45" y="742" class="t-muted" text-anchor="start">session valid</text>
<rect x="28" y="965" width="944" height="151" rx="8" class="group" />
<rect x="28" y="965" width="44.6" height="20" rx="6" class="box-sunken" />
<text x="50.3" y="979" class="t-muted" text-anchor="middle">else</text>
<text x="84.6" y="979" class="t-muted" text-anchor="start">missing, expired or invalid</text>
</svg>
Side by Side

After

Width:  |  Height:  |  Size: 11 KiB

docs/diagrams/components.svg
+179
View File
@@ -0,0 +1,179 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1240 880" width="1240" height="880" role="img" aria-label="Pablan components">
<title>Pablan components</title>
<style>
:root {
--bg: #fbfaf9;
--surface: #ffffff;
--sunken: #f2f0ed;
--ink: #1a1815;
--muted: #63605a;
--line: #c5bfb6;
--line-soft: #e5e1db;
--accent: #b8770a;
--secondary: #b04a00;
--note: #faf3e6;
}
@media (prefers-color-scheme: dark) {
:root {
--bg: #0f0e0d;
--surface: #1a1817;
--sunken: #232120;
--ink: #efece8;
--muted: #a49d94;
--line: #4a443d;
--line-soft: #332f2b;
--accent: #e6b422;
--secondary: #e08030;
--note: #241f16;
}
}
text { font-family: ui-sans-serif, system-ui, -apple-system, 'Segoe UI', Roboto, sans-serif; fill: var(--ink); }
.t-title { font-size: 15px; font-weight: 650; }
.t-sub { font-size: 11.5px; fill: var(--muted); }
.t-node { font-size: 12.5px; font-weight: 600; }
.t-body { font-size: 11.5px; }
.t-muted { font-size: 11px; fill: var(--muted); }
.t-edge { font-size: 10.5px; fill: var(--muted); }
.t-mono { font-family: ui-monospace, SFMono-Regular, Menlo, Consolas, monospace; font-size: 11px; }
.box { fill: var(--surface); stroke: var(--line); stroke-width: 1.2; }
.box-sunken { fill: var(--sunken); stroke: var(--line-soft); stroke-width: 1; }
.box-accent { fill: var(--surface); stroke: var(--accent); stroke-width: 1.6; }
.note { fill: var(--note); stroke: var(--line-soft); stroke-width: 1; }
.group { fill: none; stroke: var(--line-soft); stroke-width: 1.2;
stroke-dasharray: 5 4; }
.lifeline { stroke: var(--line); stroke-width: 1; stroke-dasharray: 4 4; }
.edge { stroke: var(--line); stroke-width: 1.3; fill: none; }
.edge-accent { stroke: var(--secondary); stroke-width: 1.5; fill: none; }
.edge-soft { stroke: var(--line); stroke-width: 1.1; fill: none;
stroke-dasharray: 5 4; }
.edge-soft-accent { stroke: var(--secondary); stroke-width: 1.3; fill: none;
stroke-dasharray: 5 4; }
.planned { stroke-dasharray: 6 4; opacity: 0.62; }
</style>
<defs>
<marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5"
markerWidth="7" markerHeight="7" orient="auto-start-reverse">
<path d="M 0 1 L 9 5 L 0 9 z" fill="var(--line)"/>
</marker>
<marker id="arrow-accent" viewBox="0 0 10 10" refX="9" refY="5"
markerWidth="7" markerHeight="7" orient="auto-start-reverse">
<path d="M 0 1 L 9 5 L 0 9 z" fill="var(--secondary)"/>
</marker>
<marker id="arrow-open" viewBox="0 0 10 10" refX="9" refY="5"
markerWidth="8" markerHeight="8" orient="auto-start-reverse">
<path d="M 0 1 L 9 5 L 0 9" fill="none" stroke="var(--line)"
stroke-width="1.3"/>
</marker>
</defs>
<rect width="1240" height="880" fill="var(--bg)"/>
<text x="30" y="32" class="t-title" text-anchor="start">Components</text>
<text x="30" y="52" class="t-sub" text-anchor="start">One deployment: browser, one proxy, two app processes, one database, and whatever speaks the OpenAI API.</text>
<rect x="40" y="80" width="170" height="38" rx="7" class="box" />
<text x="51" y="99" class="t-node" text-anchor="start">Browser</text>
<rect x="40" y="148" width="1160" height="46" rx="7" class="box" />
<text x="51" y="167" class="t-node" text-anchor="start">Reverse proxy: Caddy</text>
<text x="51" y="182" class="t-muted" text-anchor="start">/api/* to the backend, everything else to the frontend; TLS for PABLAN_DOMAIN</text>
<line x1="125" y1="118" x2="125" y2="146" class="edge" marker-end="url(#arrow)"/>
<line x1="240" y1="194" x2="240" y2="240" class="edge" marker-end="url(#arrow)"/>
<line x1="860" y1="194" x2="860" y2="240" class="edge" marker-end="url(#arrow)"/>
<rect x="40" y="250" width="410" height="330" rx="10" class="group" />
<rect x="54" y="239" width="284.4" height="22" rx="11" class="box-sunken" />
<text x="196.2" y="254" class="t-muted" text-anchor="middle">SvelteKit frontend (no DB, no auth logic)</text>
<rect x="56" y="272" width="378" height="48" rx="7" class="box" />
<text x="67" y="291" class="t-node" text-anchor="start">hooks.server.ts</text>
<text x="67" y="306" class="t-muted" text-anchor="start">sequence(auth gate, paraglide i18n); cookie passthrough</text>
<rect x="56" y="332" width="378" height="48" rx="7" class="box" />
<text x="67" y="351" class="t-node" text-anchor="start">typed API client</text>
<text x="67" y="366" class="t-muted" text-anchor="start">openapi-fetch over generated types, never hand-written</text>
<rect x="56" y="392" width="183" height="62" rx="7" class="box" />
<text x="67" y="411" class="t-node" text-anchor="start">chat</text>
<text x="67" y="426" class="t-muted" text-anchor="start">/chat, /chat/[id],</text>
<text x="67" y="439" class="t-muted" text-anchor="start">one ChatView, SSE</text>
<rect x="251" y="392" width="183" height="62" rx="7" class="box" />
<text x="262" y="411" class="t-node" text-anchor="start">writing editor</text>
<text x="262" y="426" class="t-muted" text-anchor="start">documents/[id]/edit,</text>
<text x="262" y="439" class="t-muted" text-anchor="start">CodeMirror + refine SSE</text>
<rect x="56" y="466" width="183" height="62" rx="7" class="box" />
<text x="67" y="485" class="t-node" text-anchor="start">documents, people,</text>
<text x="67" y="500" class="t-muted" text-anchor="start">admin, account</text>
<text x="67" y="513" class="t-muted" text-anchor="start"></text>
<rect x="251" y="466" width="183" height="62" rx="7" class="box" />
<text x="262" y="485" class="t-node" text-anchor="start">paraglide messages</text>
<text x="262" y="500" class="t-muted" text-anchor="start">de source, en in sync;</text>
<text x="262" y="513" class="t-muted" text-anchor="start">app.css design tokens</text>
<rect x="520" y="250" width="690" height="350" rx="10" class="group" />
<rect x="534" y="239" width="444.4" height="22" rx="11" class="box-sunken" />
<text x="756.2" y="254" class="t-muted" text-anchor="middle">FastAPI backend (single worker: queue and metrics are per process)</text>
<rect x="536" y="272" width="658" height="48" rx="7" class="box" />
<text x="547" y="291" class="t-node" text-anchor="start">api/ routers</text>
<text x="547" y="306" class="t-muted" text-anchor="start">auth, account, people, documents, templates, admin, conversations (SSE), authoring (SSE)</text>
<rect x="536" y="334" width="160" height="58" rx="7" class="box" />
<text x="547" y="353" class="t-node" text-anchor="start">auth/</text>
<text x="547" y="368" class="t-muted" text-anchor="start">argon2,</text>
<text x="547" y="381" class="t-muted" text-anchor="start">auth_sessions</text>
<rect x="708" y="334" width="160" height="58" rx="7" class="box" />
<text x="719" y="353" class="t-node" text-anchor="start">modes/</text>
<text x="719" y="368" class="t-muted" text-anchor="start">registry, query;</text>
<text x="719" y="381" class="t-muted" text-anchor="start">yields ModeEvents</text>
<rect x="880" y="334" width="160" height="58" rx="7" class="box" />
<text x="891" y="353" class="t-node" text-anchor="start">authoring/</text>
<text x="891" y="368" class="t-muted" text-anchor="start">template schema,</text>
<text x="891" y="381" class="t-muted" text-anchor="start">section refinement</text>
<rect x="1044" y="334" width="150" height="58" rx="7" class="box" />
<text x="1055" y="353" class="t-node" text-anchor="start">ingestion/</text>
<text x="1055" y="368" class="t-muted" text-anchor="start">jobs queue</text>
<text x="1055" y="381" class="t-muted" text-anchor="start">+ handlers</text>
<rect x="536" y="406" width="658" height="52" rx="7" class="box" />
<text x="547" y="425" class="t-node" text-anchor="start">rag/</text>
<text x="547" y="440" class="t-muted" text-anchor="start">chunking, indexing, permissions, retrieval (hybrid + text-only + similarity)</text>
<rect x="536" y="470" width="494" height="52" rx="7" class="box" />
<text x="547" y="489" class="t-node" text-anchor="start">llm/client.py</text>
<text x="547" y="504" class="t-muted" text-anchor="start">the ONLY caller of an endpoint: chat_stream, chat_json, embed, probe</text>
<rect x="1042" y="470" width="152" height="52" rx="7" class="box" />
<text x="1053" y="489" class="t-node" text-anchor="start">llm/overrides.py</text>
<text x="1053" y="504" class="t-muted" text-anchor="start">env bootstrap,</text>
<text x="1053" y="517" class="t-muted" text-anchor="start">then the DB wins</text>
<rect x="536" y="534" width="323" height="48" rx="7" class="box" />
<text x="547" y="553" class="t-node" text-anchor="start">config, log, metrics</text>
<text x="547" y="568" class="t-muted" text-anchor="start">content never reaches a log line</text>
<rect x="871" y="534" width="323" height="48" rx="7" class="box planned" />
<text x="882" y="553" class="t-node" text-anchor="start">ee_hooks.py</text>
<text x="882" y="568" class="t-muted" text-anchor="start">optional pablan_ee import; core never imports ee/</text>
<line x1="616" y1="320" x2="616" y2="332" class="edge-soft" marker-end="url(#arrow)"/>
<line x1="783" y1="320" x2="783" y2="332" class="edge-soft" marker-end="url(#arrow)"/>
<line x1="950" y1="320" x2="950" y2="332" class="edge-soft" marker-end="url(#arrow)"/>
<line x1="1111" y1="320" x2="1111" y2="332" class="edge-soft" marker-end="url(#arrow)"/>
<line x1="783" y1="392" x2="783" y2="404" class="edge" marker-end="url(#arrow)"/>
<line x1="950" y1="392" x2="950" y2="404" class="edge" marker-end="url(#arrow)"/>
<line x1="1111" y1="392" x2="1111" y2="404" class="edge" marker-end="url(#arrow)"/>
<line x1="616" y1="458" x2="616" y2="468" class="edge" marker-end="url(#arrow)"/>
<line x1="950" y1="458" x2="950" y2="468" class="edge" marker-end="url(#arrow)"/>
<path d="M 450 296 H 520" class="edge-accent" marker-end="url(#arrow)"/>
<text x="485" y="288" class="t-edge" text-anchor="middle">/api/auth/me</text>
<path d="M 450 356 H 520" class="edge-accent" marker-end="url(#arrow)"/>
<text x="485" y="348" class="t-edge" text-anchor="middle">same origin</text>
<rect x="40" y="760" width="560" height="70" rx="7" class="box" />
<text x="51" y="779" class="t-node" text-anchor="start">PostgreSQL + pgvector</text>
<text x="51" y="794" class="t-muted" text-anchor="start">documents (Markdown, the source of truth), chunks (embedding + tsvector),</text>
<text x="51" y="807" class="t-muted" text-anchor="start">auth_sessions, conversations, jobs, llm_settings, templates</text>
<rect x="640" y="760" width="560" height="70" rx="7" class="box" />
<text x="651" y="779" class="t-node" text-anchor="start">OpenAI-compatible endpoints</text>
<text x="651" y="794" class="t-muted" text-anchor="start">three roles, each its own base_url + key + model:</text>
<text x="651" y="807" class="t-muted" text-anchor="start">chat, utility, embedding (bge-m3). llama.cpp locally, any cloud API in production</text>
<line x1="560" y1="600" x2="560" y2="758" class="edge" marker-end="url(#arrow)"/>
<text x="566" y="668" class="t-edge" text-anchor="start">every table</text>
<line x1="1185" y1="600" x2="1185" y2="758" class="edge" marker-end="url(#arrow)"/>
<text x="1179" y="745" class="t-edge" text-anchor="end">llm/client.py only</text>
<rect x="660" y="630" width="240" height="78" rx="7" class="box" />
<text x="671" y="649" class="t-node" text-anchor="start">templates/</text>
<text x="671" y="664" class="t-muted" text-anchor="start">blueprint catalog, de + en;</text>
<text x="671" y="677" class="t-muted" text-anchor="start">inert until an admin adds one</text>
<rect x="920" y="630" width="240" height="78" rx="7" class="box" />
<text x="931" y="649" class="t-node" text-anchor="start">help/</text>
<text x="931" y="664" class="t-muted" text-anchor="start">built-in help pages, German;</text>
<text x="931" y="677" class="t-muted" text-anchor="start">re-imported on every start</text>
<path d="M 780 630 V 604" class="edge-soft" marker-end="url(#arrow)"/>
<path d="M 1040 630 V 604" class="edge-soft" marker-end="url(#arrow)"/>
</svg>
Side by Side

After

Width:  |  Height:  |  Size: 12 KiB

docs/diagrams/data-model.svg
+381
View File
@@ -0,0 +1,381 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1290 1090" width="1290" height="1090" role="img" aria-label="Pablan data model">
<title>Pablan data model</title>
<style>
:root {
--bg: #fbfaf9;
--surface: #ffffff;
--sunken: #f2f0ed;
--ink: #1a1815;
--muted: #63605a;
--line: #c5bfb6;
--line-soft: #e5e1db;
--accent: #b8770a;
--secondary: #b04a00;
--note: #faf3e6;
}
@media (prefers-color-scheme: dark) {
:root {
--bg: #0f0e0d;
--surface: #1a1817;
--sunken: #232120;
--ink: #efece8;
--muted: #a49d94;
--line: #4a443d;
--line-soft: #332f2b;
--accent: #e6b422;
--secondary: #e08030;
--note: #241f16;
}
}
text { font-family: ui-sans-serif, system-ui, -apple-system, 'Segoe UI', Roboto, sans-serif; fill: var(--ink); }
.t-title { font-size: 15px; font-weight: 650; }
.t-sub { font-size: 11.5px; fill: var(--muted); }
.t-node { font-size: 12.5px; font-weight: 600; }
.t-body { font-size: 11.5px; }
.t-muted { font-size: 11px; fill: var(--muted); }
.t-edge { font-size: 10.5px; fill: var(--muted); }
.t-mono { font-family: ui-monospace, SFMono-Regular, Menlo, Consolas, monospace; font-size: 11px; }
.box { fill: var(--surface); stroke: var(--line); stroke-width: 1.2; }
.box-sunken { fill: var(--sunken); stroke: var(--line-soft); stroke-width: 1; }
.box-accent { fill: var(--surface); stroke: var(--accent); stroke-width: 1.6; }
.note { fill: var(--note); stroke: var(--line-soft); stroke-width: 1; }
.group { fill: none; stroke: var(--line-soft); stroke-width: 1.2;
stroke-dasharray: 5 4; }
.lifeline { stroke: var(--line); stroke-width: 1; stroke-dasharray: 4 4; }
.edge { stroke: var(--line); stroke-width: 1.3; fill: none; }
.edge-accent { stroke: var(--secondary); stroke-width: 1.5; fill: none; }
.edge-soft { stroke: var(--line); stroke-width: 1.1; fill: none;
stroke-dasharray: 5 4; }
.edge-soft-accent { stroke: var(--secondary); stroke-width: 1.3; fill: none;
stroke-dasharray: 5 4; }
.planned { stroke-dasharray: 6 4; opacity: 0.62; }
</style>
<defs>
<marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5"
markerWidth="7" markerHeight="7" orient="auto-start-reverse">
<path d="M 0 1 L 9 5 L 0 9 z" fill="var(--line)"/>
</marker>
<marker id="arrow-accent" viewBox="0 0 10 10" refX="9" refY="5"
markerWidth="7" markerHeight="7" orient="auto-start-reverse">
<path d="M 0 1 L 9 5 L 0 9 z" fill="var(--secondary)"/>
</marker>
<marker id="arrow-open" viewBox="0 0 10 10" refX="9" refY="5"
markerWidth="8" markerHeight="8" orient="auto-start-reverse">
<path d="M 0 1 L 9 5 L 0 9" fill="none" stroke="var(--line)"
stroke-width="1.3"/>
</marker>
<marker id="many" viewBox="0 0 12 12" refX="11" refY="6" markerWidth="11"
markerHeight="11" orient="auto">
<path d="M 1 1 L 11 6 M 1 6 L 11 6 M 1 11 L 11 6" fill="none"
stroke="var(--line)" stroke-width="1.2"/>
</marker>
<marker id="one" viewBox="0 0 12 12" refX="2" refY="6" markerWidth="11"
markerHeight="11" orient="auto">
<path d="M 3 1 L 3 11" fill="none" stroke="var(--line)" stroke-width="1.4"/>
</marker>
</defs>
<rect width="1290" height="1090" fill="var(--bg)"/>
<text x="30" y="32" class="t-title" text-anchor="start">Data model</text>
<text x="30" y="52" class="t-sub" text-anchor="start">Markdown in documents.content_md is the source of truth; chunks are a disposable derivative.</text>
<path d="M 90 167 V 200" class="edge" marker-end="url(#many)" marker-start="url(#one)"/>
<text x="98" y="187.5" class="t-edge" text-anchor="start">employs</text>
<path d="M 90 356 V 404" class="edge" marker-end="url(#many)" marker-start="url(#one)"/>
<text x="98" y="384" class="t-edge" text-anchor="start">logs in</text>
<path d="M 130 356 V 520" class="edge" marker-end="url(#many)" marker-start="url(#one)"/>
<text x="138" y="442" class="t-edge" text-anchor="start">starts</text>
<path d="M 90 625 V 653" class="edge" marker-end="url(#many)" marker-start="url(#one)"/>
<text x="98" y="643" class="t-edge" text-anchor="start">contains</text>
<path d="M 530 303 V 404" class="edge" marker-end="url(#many)" marker-start="url(#one)"/>
<text x="538" y="374.5" class="t-edge" text-anchor="start">grants</text>
<path d="M 590 303 V 520" class="edge" marker-end="url(#many)" marker-start="url(#one)"/>
<text x="598" y="441" class="t-edge" text-anchor="start">audited by</text>
<path d="M 650 303 V 724" class="edge" marker-end="url(#many)" marker-start="url(#one)"/>
<text x="658" y="543" class="t-edge" text-anchor="start">derived into</text>
<path d="M 410 136 H 422 V 130 H 470" class="edge" marker-end="url(#many)" marker-start="url(#one)"/>
<text x="462" y="130" class="t-edge" text-anchor="end">owns</text>
<path d="M 410 158 H 434 V 464 H 470" class="edge" marker-end="url(#many)" marker-start="url(#one)"/>
<text x="462" y="152" class="t-edge" text-anchor="end">granted to</text>
<path d="M 410 284 H 446 V 182 H 470" class="edge" marker-end="url(#many)" marker-start="url(#one)"/>
<text x="462" y="278" class="t-edge" text-anchor="end">authors</text>
<path d="M 410 328 H 470 V 658 H 470" class="edge" marker-end="url(#many)" marker-start="url(#one)"/>
<text x="462" y="322" class="t-edge" text-anchor="end">acted</text>
<path d="M 30 306 H 18 V 950 H 470" class="edge" marker-end="url(#many)" marker-start="url(#one)"/>
<text x="26" y="940" class="t-edge" text-anchor="start">is asked / answers</text>
<path d="M 850 270 H 866 V 950 H 850" class="edge" marker-end="url(#many)" marker-start="url(#one)"/>
<text transform="rotate(-90 870 700)" x="870" y="700" class="t-edge" text-anchor="middle">is questioned by</text>
<rect x="30" y="96" width="380" height="71" rx="8" class="box" />
<rect x="30" y="96" width="380" height="28" rx="8" class="box-sunken" />
<text x="42" y="115" class="t-node" text-anchor="start">departments</text>
<text x="42" y="137" class="t-mono" text-anchor="start">id</text>
<text x="182" y="137" class="t-mono" text-anchor="start">uuid</text>
<text x="245" y="137" class="t-muted" text-anchor="start">PK</text>
<text x="42" y="154" class="t-mono" text-anchor="start">name</text>
<text x="182" y="154" class="t-mono" text-anchor="start">string</text>
<text x="245" y="154" class="t-muted" text-anchor="start">unique</text>
<rect x="30" y="200" width="380" height="156" rx="8" class="box" />
<rect x="30" y="200" width="380" height="28" rx="8" class="box-sunken" />
<text x="42" y="219" class="t-node" text-anchor="start">users</text>
<text x="42" y="241" class="t-mono" text-anchor="start">id</text>
<text x="182" y="241" class="t-mono" text-anchor="start">uuid</text>
<text x="245" y="241" class="t-muted" text-anchor="start">PK</text>
<text x="42" y="258" class="t-mono" text-anchor="start">email</text>
<text x="182" y="258" class="t-mono" text-anchor="start">string</text>
<text x="245" y="258" class="t-muted" text-anchor="start">unique</text>
<text x="42" y="275" class="t-mono" text-anchor="start">name</text>
<text x="182" y="275" class="t-mono" text-anchor="start">string</text>
<text x="42" y="292" class="t-mono" text-anchor="start">role</text>
<text x="182" y="292" class="t-mono" text-anchor="start">string</text>
<text x="245" y="292" class="t-muted" text-anchor="start">member | admin</text>
<text x="42" y="309" class="t-mono" text-anchor="start">locale</text>
<text x="182" y="309" class="t-mono" text-anchor="start">varchar</text>
<text x="245" y="309" class="t-muted" text-anchor="start">de | en, null = browser</text>
<text x="42" y="326" class="t-mono" text-anchor="start">password_hash</text>
<text x="182" y="326" class="t-mono" text-anchor="start">string</text>
<text x="245" y="326" class="t-muted" text-anchor="start">argon2</text>
<text x="42" y="343" class="t-mono" text-anchor="start">department_id</text>
<text x="182" y="343" class="t-mono" text-anchor="start">uuid</text>
<text x="245" y="343" class="t-muted" text-anchor="start">FK, null, SET NULL</text>
<rect x="30" y="404" width="380" height="88" rx="8" class="box" />
<rect x="30" y="404" width="380" height="28" rx="8" class="box-sunken" />
<text x="42" y="423" class="t-node" text-anchor="start">auth_sessions</text>
<text x="42" y="445" class="t-mono" text-anchor="start">id</text>
<text x="182" y="445" class="t-mono" text-anchor="start">uuid</text>
<text x="245" y="445" class="t-muted" text-anchor="start">PK, = cookie token</text>
<text x="42" y="462" class="t-mono" text-anchor="start">user_id</text>
<text x="182" y="462" class="t-mono" text-anchor="start">uuid</text>
<text x="245" y="462" class="t-muted" text-anchor="start">FK, CASCADE</text>
<text x="42" y="479" class="t-mono" text-anchor="start">expires_at</text>
<text x="182" y="479" class="t-mono" text-anchor="start">timestamp</text>
<text x="245" y="479" class="t-muted" text-anchor="start">TTL 14d, configurable</text>
<rect x="30" y="520" width="380" height="105" rx="8" class="box" />
<rect x="30" y="520" width="380" height="28" rx="8" class="box-sunken" />
<text x="42" y="539" class="t-node" text-anchor="start">conversations</text>
<text x="42" y="561" class="t-mono" text-anchor="start">id</text>
<text x="182" y="561" class="t-mono" text-anchor="start">uuid</text>
<text x="245" y="561" class="t-muted" text-anchor="start">PK</text>
<text x="42" y="578" class="t-mono" text-anchor="start">mode</text>
<text x="182" y="578" class="t-mono" text-anchor="start">string</text>
<text x="245" y="578" class="t-muted" text-anchor="start">query | insight</text>
<text x="42" y="595" class="t-mono" text-anchor="start">status</text>
<text x="182" y="595" class="t-mono" text-anchor="start">string</text>
<text x="245" y="595" class="t-muted" text-anchor="start">active | completed</text>
<text x="42" y="612" class="t-mono" text-anchor="start">user_id</text>
<text x="182" y="612" class="t-mono" text-anchor="start">uuid</text>
<text x="245" y="612" class="t-muted" text-anchor="start">FK, CASCADE</text>
<rect x="30" y="653" width="380" height="122" rx="8" class="box" />
<rect x="30" y="653" width="380" height="28" rx="8" class="box-sunken" />
<text x="42" y="672" class="t-node" text-anchor="start">messages</text>
<text x="42" y="694" class="t-mono" text-anchor="start">id</text>
<text x="182" y="694" class="t-mono" text-anchor="start">uuid</text>
<text x="245" y="694" class="t-muted" text-anchor="start">PK</text>
<text x="42" y="711" class="t-mono" text-anchor="start">conversation_id</text>
<text x="182" y="711" class="t-mono" text-anchor="start">uuid</text>
<text x="245" y="711" class="t-muted" text-anchor="start">FK, CASCADE</text>
<text x="42" y="728" class="t-mono" text-anchor="start">role</text>
<text x="182" y="728" class="t-mono" text-anchor="start">string</text>
<text x="245" y="728" class="t-muted" text-anchor="start">user/assistant/system</text>
<text x="42" y="745" class="t-mono" text-anchor="start">content</text>
<text x="182" y="745" class="t-mono" text-anchor="start">text</text>
<text x="42" y="762" class="t-mono" text-anchor="start">meta</text>
<text x="182" y="762" class="t-mono" text-anchor="start">jsonb</text>
<text x="245" y="762" class="t-muted" text-anchor="start">citations, fallback code</text>
<rect x="470" y="96" width="380" height="207" rx="8" class="box" />
<rect x="470" y="96" width="380" height="28" rx="8" class="box-sunken" />
<text x="482" y="115" class="t-node" text-anchor="start">documents</text>
<text x="482" y="137" class="t-mono" text-anchor="start">id</text>
<text x="622" y="137" class="t-mono" text-anchor="start">uuid</text>
<text x="685" y="137" class="t-muted" text-anchor="start">PK</text>
<text x="482" y="154" class="t-mono" text-anchor="start">title</text>
<text x="622" y="154" class="t-mono" text-anchor="start">string</text>
<text x="482" y="171" class="t-mono" text-anchor="start">status</text>
<text x="622" y="171" class="t-mono" text-anchor="start">string</text>
<text x="685" y="171" class="t-muted" text-anchor="start">draft | published | archived</text>
<text x="482" y="188" class="t-mono" text-anchor="start">is_builtin</text>
<text x="622" y="188" class="t-mono" text-anchor="start">bool</text>
<text x="685" y="188" class="t-muted" text-anchor="start">shipped help page</text>
<text x="482" y="205" class="t-mono" text-anchor="start">visibility</text>
<text x="622" y="205" class="t-mono" text-anchor="start">string</text>
<text x="685" y="205" class="t-muted" text-anchor="start">public | department |</text>
<text x="685" y="222" class="t-muted" text-anchor="start">restricted</text>
<text x="482" y="239" class="t-mono" text-anchor="start">content_md</text>
<text x="622" y="239" class="t-mono" text-anchor="start">text</text>
<text x="685" y="239" class="t-muted" text-anchor="start">THE source of truth</text>
<text x="482" y="256" class="t-mono" text-anchor="start">meta</text>
<text x="622" y="256" class="t-mono" text-anchor="start">jsonb</text>
<text x="685" y="256" class="t-muted" text-anchor="start">template, context, summary</text>
<text x="482" y="273" class="t-mono" text-anchor="start">author_id</text>
<text x="622" y="273" class="t-mono" text-anchor="start">uuid</text>
<text x="685" y="273" class="t-muted" text-anchor="start">FK, null, SET NULL</text>
<text x="482" y="290" class="t-mono" text-anchor="start">department_id</text>
<text x="622" y="290" class="t-mono" text-anchor="start">uuid</text>
<text x="685" y="290" class="t-muted" text-anchor="start">FK, null, SET NULL</text>
<rect x="470" y="404" width="380" height="88" rx="8" class="box" />
<rect x="470" y="404" width="380" height="28" rx="8" class="box-sunken" />
<text x="482" y="423" class="t-node" text-anchor="start">doc_permissions</text>
<text x="482" y="445" class="t-mono" text-anchor="start">document_id</text>
<text x="622" y="445" class="t-mono" text-anchor="start">uuid</text>
<text x="685" y="445" class="t-muted" text-anchor="start">PK, FK, CASCADE</text>
<text x="482" y="462" class="t-mono" text-anchor="start">department_id</text>
<text x="622" y="462" class="t-mono" text-anchor="start">uuid</text>
<text x="685" y="462" class="t-muted" text-anchor="start">PK, FK, CASCADE</text>
<text x="482" y="479" class="t-mono" text-anchor="start">level</text>
<text x="622" y="479" class="t-mono" text-anchor="start">string</text>
<text x="685" y="479" class="t-muted" text-anchor="start">read</text>
<rect x="470" y="520" width="380" height="173" rx="8" class="box" />
<rect x="470" y="520" width="380" height="28" rx="8" class="box-sunken" />
<text x="482" y="539" class="t-node" text-anchor="start">document_events</text>
<text x="482" y="561" class="t-mono" text-anchor="start">id</text>
<text x="622" y="561" class="t-mono" text-anchor="start">uuid</text>
<text x="685" y="561" class="t-muted" text-anchor="start">PK</text>
<text x="482" y="578" class="t-mono" text-anchor="start">document_id</text>
<text x="622" y="578" class="t-mono" text-anchor="start">uuid</text>
<text x="685" y="578" class="t-muted" text-anchor="start">FK, CASCADE</text>
<text x="482" y="595" class="t-mono" text-anchor="start">actor_id</text>
<text x="622" y="595" class="t-mono" text-anchor="start">uuid</text>
<text x="685" y="595" class="t-muted" text-anchor="start">FK, null, SET NULL</text>
<text x="482" y="612" class="t-mono" text-anchor="start">action</text>
<text x="622" y="612" class="t-mono" text-anchor="start">string</text>
<text x="685" y="612" class="t-muted" text-anchor="start">created | edited | published |</text>
<text x="685" y="629" class="t-muted" text-anchor="start">archived | visibility_changed |</text>
<text x="685" y="646" class="t-muted" text-anchor="start">review_requested / _resolved</text>
<text x="482" y="663" class="t-mono" text-anchor="start">content_md</text>
<text x="622" y="663" class="t-mono" text-anchor="start">text</text>
<text x="685" y="663" class="t-muted" text-anchor="start">snapshot AFTER the event</text>
<text x="482" y="680" class="t-mono" text-anchor="start">title / visibility / meta</text>
<text x="685" y="680" class="t-muted" text-anchor="start">snapshot</text>
<rect x="470" y="724" width="380" height="156" rx="8" class="box" />
<rect x="470" y="724" width="380" height="28" rx="8" class="box-sunken" />
<text x="482" y="743" class="t-node" text-anchor="start">chunks</text>
<text x="482" y="765" class="t-mono" text-anchor="start">id</text>
<text x="622" y="765" class="t-mono" text-anchor="start">uuid</text>
<text x="685" y="765" class="t-muted" text-anchor="start">PK</text>
<text x="482" y="782" class="t-mono" text-anchor="start">document_id</text>
<text x="622" y="782" class="t-mono" text-anchor="start">uuid</text>
<text x="685" y="782" class="t-muted" text-anchor="start">FK, CASCADE</text>
<text x="482" y="799" class="t-mono" text-anchor="start">chunk_index</text>
<text x="622" y="799" class="t-mono" text-anchor="start">int</text>
<text x="685" y="799" class="t-muted" text-anchor="start">unique per document</text>
<text x="482" y="816" class="t-mono" text-anchor="start">content</text>
<text x="622" y="816" class="t-mono" text-anchor="start">text</text>
<text x="482" y="833" class="t-mono" text-anchor="start">embedding</text>
<text x="622" y="833" class="t-mono" text-anchor="start">vector</text>
<text x="685" y="833" class="t-muted" text-anchor="start">1024 bge-m3, HNSW cosine</text>
<text x="482" y="850" class="t-mono" text-anchor="start">tsv</text>
<text x="622" y="850" class="t-mono" text-anchor="start">tsvector</text>
<text x="685" y="850" class="t-muted" text-anchor="start">generated german, GIN</text>
<text x="482" y="867" class="t-mono" text-anchor="start">meta</text>
<text x="622" y="867" class="t-mono" text-anchor="start">jsonb</text>
<text x="685" y="867" class="t-muted" text-anchor="start">heading path, filters</text>
<rect x="470" y="904" width="380" height="156" rx="8" class="box" />
<rect x="470" y="904" width="380" height="28" rx="8" class="box-sunken" />
<text x="482" y="923" class="t-node" text-anchor="start">review_requests</text>
<text x="482" y="945" class="t-mono" text-anchor="start">id</text>
<text x="622" y="945" class="t-mono" text-anchor="start">uuid</text>
<text x="685" y="945" class="t-muted" text-anchor="start">PK</text>
<text x="482" y="962" class="t-mono" text-anchor="start">document_id</text>
<text x="622" y="962" class="t-mono" text-anchor="start">uuid</text>
<text x="685" y="962" class="t-muted" text-anchor="start">FK, CASCADE</text>
<text x="482" y="979" class="t-mono" text-anchor="start">requester_id</text>
<text x="622" y="979" class="t-mono" text-anchor="start">uuid</text>
<text x="685" y="979" class="t-muted" text-anchor="start">FK, null, SET NULL</text>
<text x="482" y="996" class="t-mono" text-anchor="start">reviewer_id</text>
<text x="622" y="996" class="t-mono" text-anchor="start">uuid</text>
<text x="685" y="996" class="t-muted" text-anchor="start">FK, null, grants read+edit</text>
<text x="482" y="1013" class="t-mono" text-anchor="start">question</text>
<text x="622" y="1013" class="t-mono" text-anchor="start">text</text>
<text x="685" y="1013" class="t-muted" text-anchor="start">null, "still 14 days?"</text>
<text x="482" y="1030" class="t-mono" text-anchor="start">resolved_at</text>
<text x="622" y="1030" class="t-mono" text-anchor="start">timestamp</text>
<text x="685" y="1030" class="t-muted" text-anchor="start">null while open</text>
<text x="482" y="1047" class="t-mono" text-anchor="start">resolved_by_id</text>
<text x="622" y="1047" class="t-mono" text-anchor="start">uuid</text>
<text x="685" y="1047" class="t-muted" text-anchor="start">FK, null, who checked it</text>
<rect x="890" y="96" width="380" height="105" rx="8" class="box" />
<rect x="890" y="96" width="380" height="28" rx="8" class="box-sunken" />
<text x="902" y="115" class="t-node" text-anchor="start">templates</text>
<text x="902" y="137" class="t-mono" text-anchor="start">id</text>
<text x="1042" y="137" class="t-mono" text-anchor="start">uuid</text>
<text x="1105" y="137" class="t-muted" text-anchor="start">PK</text>
<text x="902" y="154" class="t-mono" text-anchor="start">name</text>
<text x="1042" y="154" class="t-mono" text-anchor="start">string</text>
<text x="902" y="171" class="t-mono" text-anchor="start">version</text>
<text x="1042" y="171" class="t-mono" text-anchor="start">string</text>
<text x="1105" y="171" class="t-muted" text-anchor="start">schema version, e.g. 1.0</text>
<text x="902" y="188" class="t-mono" text-anchor="start">config</text>
<text x="1042" y="188" class="t-mono" text-anchor="start">jsonb</text>
<text x="1105" y="188" class="t-muted" text-anchor="start">authoring template</text>
<rect x="890" y="226" width="380" height="173" rx="8" class="box" />
<rect x="890" y="226" width="380" height="28" rx="8" class="box-sunken" />
<text x="902" y="245" class="t-node" text-anchor="start">llm_settings</text>
<text x="902" y="267" class="t-mono" text-anchor="start">id</text>
<text x="1042" y="267" class="t-mono" text-anchor="start">uuid</text>
<text x="1105" y="267" class="t-muted" text-anchor="start">PK</text>
<text x="902" y="284" class="t-mono" text-anchor="start">role</text>
<text x="1042" y="284" class="t-mono" text-anchor="start">string</text>
<text x="1105" y="284" class="t-muted" text-anchor="start">unique: chat/utility/embed</text>
<text x="902" y="301" class="t-mono" text-anchor="start">base_url</text>
<text x="1042" y="301" class="t-mono" text-anchor="start">string</text>
<text x="1105" y="301" class="t-muted" text-anchor="start">seeded from env at start</text>
<text x="902" y="318" class="t-mono" text-anchor="start">model</text>
<text x="1042" y="318" class="t-mono" text-anchor="start">string</text>
<text x="1105" y="318" class="t-muted" text-anchor="start">seeded from env at start</text>
<text x="902" y="335" class="t-mono" text-anchor="start">api_key</text>
<text x="1042" y="335" class="t-mono" text-anchor="start">text</text>
<text x="1105" y="335" class="t-muted" text-anchor="start">plaintext, never returned</text>
<text x="902" y="352" class="t-mono" text-anchor="start">base_url_from_env</text>
<text x="1042" y="352" class="t-mono" text-anchor="start">bool</text>
<text x="1105" y="352" class="t-muted" text-anchor="start">provenance, drives reset</text>
<text x="902" y="369" class="t-mono" text-anchor="start">model_from_env</text>
<text x="1042" y="369" class="t-mono" text-anchor="start">bool</text>
<text x="1105" y="369" class="t-muted" text-anchor="start">provenance, drives reset</text>
<text x="902" y="386" class="t-mono" text-anchor="start">api_key_from_env</text>
<text x="1042" y="386" class="t-mono" text-anchor="start">bool</text>
<text x="1105" y="386" class="t-muted" text-anchor="start">provenance, drives reset</text>
<rect x="890" y="430" width="380" height="88" rx="8" class="box" />
<rect x="890" y="430" width="380" height="28" rx="8" class="box-sunken" />
<text x="902" y="449" class="t-node" text-anchor="start">prompt_settings</text>
<text x="902" y="471" class="t-mono" text-anchor="start">id</text>
<text x="1042" y="471" class="t-mono" text-anchor="start">uuid</text>
<text x="1105" y="471" class="t-muted" text-anchor="start">PK</text>
<text x="902" y="488" class="t-mono" text-anchor="start">key</text>
<text x="1042" y="488" class="t-mono" text-anchor="start">string</text>
<text x="1105" y="488" class="t-muted" text-anchor="start">unique, e.g. query_system</text>
<text x="902" y="505" class="t-mono" text-anchor="start">content</text>
<text x="1042" y="505" class="t-mono" text-anchor="start">text</text>
<text x="1105" y="505" class="t-muted" text-anchor="start">full replacement text</text>
<rect x="890" y="530" width="380" height="156" rx="8" class="box" />
<rect x="890" y="530" width="380" height="28" rx="8" class="box-sunken" />
<text x="902" y="549" class="t-node" text-anchor="start">jobs</text>
<text x="902" y="571" class="t-mono" text-anchor="start">id</text>
<text x="1042" y="571" class="t-mono" text-anchor="start">uuid</text>
<text x="1105" y="571" class="t-muted" text-anchor="start">PK</text>
<text x="902" y="588" class="t-mono" text-anchor="start">type</text>
<text x="1042" y="588" class="t-mono" text-anchor="start">string</text>
<text x="902" y="605" class="t-mono" text-anchor="start">payload</text>
<text x="1042" y="605" class="t-mono" text-anchor="start">jsonb</text>
<text x="902" y="622" class="t-mono" text-anchor="start">status</text>
<text x="1042" y="622" class="t-mono" text-anchor="start">string</text>
<text x="1105" y="622" class="t-muted" text-anchor="start">pending/running/done/failed</text>
<text x="902" y="639" class="t-mono" text-anchor="start">run_after</text>
<text x="1042" y="639" class="t-mono" text-anchor="start">timestamp</text>
<text x="1105" y="639" class="t-muted" text-anchor="start">idx (status, run_after)</text>
<text x="902" y="656" class="t-mono" text-anchor="start">attempts</text>
<text x="1042" y="656" class="t-mono" text-anchor="start">int</text>
<text x="902" y="673" class="t-mono" text-anchor="start">last_error</text>
<text x="1042" y="673" class="t-mono" text-anchor="start">text</text>
<text x="1105" y="673" class="t-muted" text-anchor="start">sanitized, never content</text>
<rect x="890" y="904" width="380" height="156" rx="8" class="note" />
<text x="904" y="926" class="t-node" text-anchor="start">Reading the lines</text>
<text x="904" y="945" class="t-muted" text-anchor="start">A bar is the one side, a fan the many side.</text>
<text x="904" y="958" class="t-muted" text-anchor="start">Optionality is on the column instead: a note</text>
<text x="904" y="971" class="t-muted" text-anchor="start">saying null (and SET NULL vs CASCADE) is what</text>
<text x="904" y="984" class="t-muted" text-anchor="start">actually matters when a row is deleted.</text>
<text x="904" y="997" class="t-muted" text-anchor="start">templates, llm_settings, prompt_settings and</text>
<text x="904" y="1010" class="t-muted" text-anchor="start">jobs stand alone: config and work, not knowledge.</text>
<text x="904" y="1029" class="t-muted" text-anchor="start">An open review_request is not a document status:</text>
<text x="904" y="1042" class="t-muted" text-anchor="start">it can hang on a draft or on one published long ago,</text>
<text x="904" y="1055" class="t-muted" text-anchor="start">and it marks the document everywhere it appears.</text>
</svg>
Side by Side

After

Width:  |  Height:  |  Size: 26 KiB

docs/diagrams/queue-sequence.svg
+126
View File
@@ -0,0 +1,126 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 860 810" width="860" height="810" role="img" aria-label="Background jobs: one Postgres table, no broker">
<title>Background jobs: one Postgres table, no broker</title>
<style>
:root {
--bg: #fbfaf9;
--surface: #ffffff;
--sunken: #f2f0ed;
--ink: #1a1815;
--muted: #63605a;
--line: #c5bfb6;
--line-soft: #e5e1db;
--accent: #b8770a;
--secondary: #b04a00;
--note: #faf3e6;
}
@media (prefers-color-scheme: dark) {
:root {
--bg: #0f0e0d;
--surface: #1a1817;
--sunken: #232120;
--ink: #efece8;
--muted: #a49d94;
--line: #4a443d;
--line-soft: #332f2b;
--accent: #e6b422;
--secondary: #e08030;
--note: #241f16;
}
}
text { font-family: ui-sans-serif, system-ui, -apple-system, 'Segoe UI', Roboto, sans-serif; fill: var(--ink); }
.t-title { font-size: 15px; font-weight: 650; }
.t-sub { font-size: 11.5px; fill: var(--muted); }
.t-node { font-size: 12.5px; font-weight: 600; }
.t-body { font-size: 11.5px; }
.t-muted { font-size: 11px; fill: var(--muted); }
.t-edge { font-size: 10.5px; fill: var(--muted); }
.t-mono { font-family: ui-monospace, SFMono-Regular, Menlo, Consolas, monospace; font-size: 11px; }
.box { fill: var(--surface); stroke: var(--line); stroke-width: 1.2; }
.box-sunken { fill: var(--sunken); stroke: var(--line-soft); stroke-width: 1; }
.box-accent { fill: var(--surface); stroke: var(--accent); stroke-width: 1.6; }
.note { fill: var(--note); stroke: var(--line-soft); stroke-width: 1; }
.group { fill: none; stroke: var(--line-soft); stroke-width: 1.2;
stroke-dasharray: 5 4; }
.lifeline { stroke: var(--line); stroke-width: 1; stroke-dasharray: 4 4; }
.edge { stroke: var(--line); stroke-width: 1.3; fill: none; }
.edge-accent { stroke: var(--secondary); stroke-width: 1.5; fill: none; }
.edge-soft { stroke: var(--line); stroke-width: 1.1; fill: none;
stroke-dasharray: 5 4; }
.edge-soft-accent { stroke: var(--secondary); stroke-width: 1.3; fill: none;
stroke-dasharray: 5 4; }
.planned { stroke-dasharray: 6 4; opacity: 0.62; }
</style>
<defs>
<marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5"
markerWidth="7" markerHeight="7" orient="auto-start-reverse">
<path d="M 0 1 L 9 5 L 0 9 z" fill="var(--line)"/>
</marker>
<marker id="arrow-accent" viewBox="0 0 10 10" refX="9" refY="5"
markerWidth="7" markerHeight="7" orient="auto-start-reverse">
<path d="M 0 1 L 9 5 L 0 9 z" fill="var(--secondary)"/>
</marker>
<marker id="arrow-open" viewBox="0 0 10 10" refX="9" refY="5"
markerWidth="8" markerHeight="8" orient="auto-start-reverse">
<path d="M 0 1 L 9 5 L 0 9" fill="none" stroke="var(--line)"
stroke-width="1.3"/>
</marker>
</defs>
<rect width="860" height="810" fill="var(--bg)"/>
<text x="28" y="30" class="t-title" text-anchor="start">Background jobs: one Postgres table, no broker</text>
<text x="28" y="50" class="t-sub" text-anchor="start">FOR UPDATE SKIP LOCKED, and the handler runs inside the claim transaction.</text>
<rect x="114" y="74" width="96" height="30" rx="8" class="box" />
<text x="162" y="93" class="t-node" text-anchor="middle">Queue loop</text>
<line x1="162" y1="104" x2="162" y2="792" class="lifeline"/>
<rect x="382" y="74" width="96" height="30" rx="8" class="box" />
<text x="430" y="93" class="t-node" text-anchor="middle">Postgres</text>
<line x1="430" y1="104" x2="430" y2="792" class="lifeline"/>
<rect x="648.7" y="74" width="98.6" height="30" rx="8" class="box" />
<text x="698" y="93" class="t-node" text-anchor="middle">Job handler</text>
<line x1="698" y1="104" x2="698" y2="792" class="lifeline"/>
<text x="296" y="158" class="t-edge" text-anchor="middle">BEGIN</text>
<line x1="162" y1="175" x2="430" y2="175" class="edge" marker-end="url(#arrow)"/>
<text x="296" y="197" class="t-edge" text-anchor="middle">SELECT job WHERE status=pending AND run_after&lt;=now()</text>
<text x="296" y="210" class="t-edge" text-anchor="middle">ORDER BY run_after LIMIT 1 FOR UPDATE SKIP LOCKED</text>
<line x1="162" y1="227" x2="430" y2="227" class="edge" marker-end="url(#arrow)"/>
<text x="296" y="275" class="t-edge" text-anchor="middle">ROLLBACK</text>
<line x1="162" y1="292" x2="430" y2="292" class="edge" marker-end="url(#arrow)"/>
<text x="208" y="316" class="t-edge" text-anchor="start">wait one poll interval</text>
<path d="M 162.0 319 h 34 v 22 h -34" class="edge" marker-end="url(#arrow)"/>
<text x="430" y="403" class="t-edge" text-anchor="middle">handler(db, job), INSIDE the open claim transaction</text>
<line x1="162" y1="420" x2="698" y2="420" class="edge-accent" marker-end="url(#arrow-accent)"/>
<text x="296" y="468" class="t-edge" text-anchor="middle">status=done, attempts+1, COMMIT</text>
<text x="296" y="481" class="t-edge" text-anchor="middle">(the handler&#x27;s writes commit atomically with the job)</text>
<line x1="162" y1="498" x2="430" y2="498" class="edge" marker-end="url(#arrow)"/>
<text x="296" y="562" class="t-edge" text-anchor="middle">ROLLBACK (the handler&#x27;s writes go too)</text>
<line x1="162" y1="579" x2="430" y2="579" class="edge" marker-end="url(#arrow)"/>
<text x="296" y="601" class="t-edge" text-anchor="middle">follow-up tx: attempts+1, last_error (sanitized),</text>
<text x="296" y="614" class="t-edge" text-anchor="middle">run_after = now + 30s * 2^(attempts-1)</text>
<line x1="162" y1="631" x2="430" y2="631" class="edge" marker-end="url(#arrow)"/>
<rect x="456" y="647" width="222.8" height="27" rx="6" class="note" />
<text x="469" y="664" class="t-body" text-anchor="start">After 5 attempts: status=failed.</text>
<rect x="214.05" y="734" width="431.9" height="40" rx="6" class="note" />
<text x="227.05" y="751" class="t-body" text-anchor="start">Crash mid-job: the open transaction aborts, the row lock releases,</text>
<text x="227.05" y="764" class="t-body" text-anchor="start">and the job is still pending and claimable after the restart.</text>
<rect x="28" y="243" width="804" height="120" rx="8" class="group" />
<rect x="28" y="243" width="38.45" height="20" rx="6" class="box-sunken" />
<text x="47.225" y="257" class="t-muted" text-anchor="middle">alt</text>
<text x="78.45" y="257" class="t-muted" text-anchor="start">no claimable job</text>
<rect x="28" y="436" width="804" height="86" rx="8" class="group" />
<rect x="28" y="436" width="38.45" height="20" rx="6" class="box-sunken" />
<text x="47.225" y="450" class="t-muted" text-anchor="middle">alt</text>
<text x="78.45" y="450" class="t-muted" text-anchor="start">handler succeeds</text>
<rect x="28" y="530" width="804" height="164" rx="8" class="group" />
<rect x="28" y="530" width="44.6" height="20" rx="6" class="box-sunken" />
<text x="50.3" y="544" class="t-muted" text-anchor="middle">else</text>
<text x="84.6" y="544" class="t-muted" text-anchor="start">handler raises</text>
<rect x="28" y="371" width="804" height="339" rx="8" class="group" />
<rect x="28" y="371" width="44.6" height="20" rx="6" class="box-sunken" />
<text x="50.3" y="385" class="t-muted" text-anchor="middle">else</text>
<text x="84.6" y="385" class="t-muted" text-anchor="start">job claimed, row lock held</text>
<rect x="28" y="126" width="804" height="600" rx="8" class="group" />
<rect x="28" y="126" width="44.6" height="20" rx="6" class="box-sunken" />
<text x="50.3" y="140" class="t-muted" text-anchor="middle">loop</text>
<text x="84.6" y="140" class="t-muted" text-anchor="start">every poll interval, default 1s</text>
</svg>
Side by Side

After

Width:  |  Height:  |  Size: 7.7 KiB

docs/diagrams/retrieval-sequence.svg
+157
View File
@@ -0,0 +1,157 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1040 1356" width="1040" height="1356" role="img" aria-label="Retrieval: hybrid search, and the similarity path next to it">
<title>Retrieval: hybrid search, and the similarity path next to it</title>
<style>
:root {
--bg: #fbfaf9;
--surface: #ffffff;
--sunken: #f2f0ed;
--ink: #1a1815;
--muted: #63605a;
--line: #c5bfb6;
--line-soft: #e5e1db;
--accent: #b8770a;
--secondary: #b04a00;
--note: #faf3e6;
}
@media (prefers-color-scheme: dark) {
:root {
--bg: #0f0e0d;
--surface: #1a1817;
--sunken: #232120;
--ink: #efece8;
--muted: #a49d94;
--line: #4a443d;
--line-soft: #332f2b;
--accent: #e6b422;
--secondary: #e08030;
--note: #241f16;
}
}
text { font-family: ui-sans-serif, system-ui, -apple-system, 'Segoe UI', Roboto, sans-serif; fill: var(--ink); }
.t-title { font-size: 15px; font-weight: 650; }
.t-sub { font-size: 11.5px; fill: var(--muted); }
.t-node { font-size: 12.5px; font-weight: 600; }
.t-body { font-size: 11.5px; }
.t-muted { font-size: 11px; fill: var(--muted); }
.t-edge { font-size: 10.5px; fill: var(--muted); }
.t-mono { font-family: ui-monospace, SFMono-Regular, Menlo, Consolas, monospace; font-size: 11px; }
.box { fill: var(--surface); stroke: var(--line); stroke-width: 1.2; }
.box-sunken { fill: var(--sunken); stroke: var(--line-soft); stroke-width: 1; }
.box-accent { fill: var(--surface); stroke: var(--accent); stroke-width: 1.6; }
.note { fill: var(--note); stroke: var(--line-soft); stroke-width: 1; }
.group { fill: none; stroke: var(--line-soft); stroke-width: 1.2;
stroke-dasharray: 5 4; }
.lifeline { stroke: var(--line); stroke-width: 1; stroke-dasharray: 4 4; }
.edge { stroke: var(--line); stroke-width: 1.3; fill: none; }
.edge-accent { stroke: var(--secondary); stroke-width: 1.5; fill: none; }
.edge-soft { stroke: var(--line); stroke-width: 1.1; fill: none;
stroke-dasharray: 5 4; }
.edge-soft-accent { stroke: var(--secondary); stroke-width: 1.3; fill: none;
stroke-dasharray: 5 4; }
.planned { stroke-dasharray: 6 4; opacity: 0.62; }
</style>
<defs>
<marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5"
markerWidth="7" markerHeight="7" orient="auto-start-reverse">
<path d="M 0 1 L 9 5 L 0 9 z" fill="var(--line)"/>
</marker>
<marker id="arrow-accent" viewBox="0 0 10 10" refX="9" refY="5"
markerWidth="7" markerHeight="7" orient="auto-start-reverse">
<path d="M 0 1 L 9 5 L 0 9 z" fill="var(--secondary)"/>
</marker>
<marker id="arrow-open" viewBox="0 0 10 10" refX="9" refY="5"
markerWidth="8" markerHeight="8" orient="auto-start-reverse">
<path d="M 0 1 L 9 5 L 0 9" fill="none" stroke="var(--line)"
stroke-width="1.3"/>
</marker>
</defs>
<rect width="1040" height="1356" fill="var(--bg)"/>
<text x="28" y="30" class="t-title" text-anchor="start">Retrieval: hybrid search, and the similarity path next to it</text>
<text x="28" y="50" class="t-sub" text-anchor="start">The permission CTE is part of the one statement. There is no search without a user.</text>
<rect x="75.3" y="74" width="151.4" height="30" rx="8" class="box" />
<text x="151" y="93" class="t-node" text-anchor="middle">Caller (mode / API)</text>
<line x1="151" y1="104" x2="151" y2="1338" class="lifeline"/>
<rect x="298.2" y="74" width="197.6" height="30" rx="8" class="box" />
<text x="397" y="93" class="t-node" text-anchor="middle">rag/retrieval + similarity</text>
<line x1="397" y1="104" x2="397" y2="1338" class="lifeline"/>
<rect x="577.2" y="74" width="131.6" height="30" rx="8" class="box" />
<text x="643" y="93" class="t-node" text-anchor="middle">llm/client.embed</text>
<line x1="643" y1="104" x2="643" y2="1338" class="lifeline"/>
<rect x="841" y="74" width="96" height="30" rx="8" class="box" />
<text x="889" y="93" class="t-node" text-anchor="middle">Postgres</text>
<line x1="889" y1="104" x2="889" y2="1338" class="lifeline"/>
<line x1="28" y1="138" x2="1012" y2="138" class="edge-soft"/>
<rect x="417.825" y="127" width="204.35" height="22" rx="11" class="box-sunken" />
<text x="520" y="142" class="t-muted" text-anchor="middle">search(): hybrid, the default</text>
<text x="274" y="170" class="t-edge" text-anchor="middle">search(db, query, user=..., top_k=5)</text>
<line x1="151" y1="187" x2="397" y2="187" class="edge" marker-end="url(#arrow)"/>
<text x="520" y="209" class="t-edge" text-anchor="middle">embed([query])</text>
<line x1="397" y1="226" x2="643" y2="226" class="edge" marker-end="url(#arrow)"/>
<text x="520" y="248" class="t-edge" text-anchor="middle">query vector (1024, bge-m3)</text>
<line x1="643" y1="265" x2="397" y2="265" class="edge-soft" marker-end="url(#arrow-open)"/>
<text x="643" y="287" class="t-edge" text-anchor="middle">ONE statement:</text>
<line x1="397" y1="304" x2="889" y2="304" class="edge-accent" marker-end="url(#arrow-accent)"/>
<rect x="423" y="320" width="419.6" height="53" rx="6" class="note" />
<text x="436" y="337" class="t-body" text-anchor="start">1. CTE &quot;allowed&quot;: the LIVE documents table. status=published AND</text>
<text x="436" y="350" class="t-body" text-anchor="start"> (public | own department | doc_permissions grant | author).</text>
<text x="436" y="363" class="t-body" text-anchor="start"> Never chunk meta, so a change applies without reindexing.</text>
<rect x="423" y="385" width="364.25" height="40" rx="6" class="note" />
<text x="436" y="402" class="t-body" text-anchor="start">2. vector candidates: top-20 by cosine distance (HNSW),</text>
<text x="436" y="415" class="t-body" text-anchor="start"> joined to allowed.</text>
<rect x="423" y="437" width="395" height="40" rx="6" class="note" />
<text x="436" y="454" class="t-body" text-anchor="start">3. FTS candidates: top-20 by ts_rank_cd over</text>
<text x="436" y="467" class="t-body" text-anchor="start"> websearch_to_tsquery(&#x27;german&#x27;, query), joined to allowed.</text>
<rect x="423" y="489" width="450.35" height="27" rx="6" class="note" />
<text x="436" y="506" class="t-body" text-anchor="start">4. RRF merge: score = sum of 1/(60+rank), ORDER BY score LIMIT top_k.</text>
<text x="643" y="534" class="t-edge" text-anchor="middle">rows: chunk, document, title, meta, score,</text>
<text x="643" y="547" class="t-edge" text-anchor="middle">vector distance, fts rank</text>
<line x1="889" y1="564" x2="397" y2="564" class="edge-soft" marker-end="url(#arrow-open)"/>
<text x="274" y="586" class="t-edge" text-anchor="middle">SearchResult[] with citation metadata</text>
<text x="274" y="599" class="t-edge" text-anchor="middle">(title, heading_path, score, vector_distance, fts_match)</text>
<line x1="397" y1="616" x2="151" y2="616" class="edge-soft" marker-end="url(#arrow-open)"/>
<rect x="304.05" y="632" width="431.9" height="53" rx="6" class="note" />
<text x="317.05" y="649" class="t-body" text-anchor="start">No-answer signal: the top result has fts_match=false AND</text>
<text x="317.05" y="662" class="t-body" text-anchor="start">vector_distance &gt;= 0.45 (calibrated for bge-m3). Callers treat the</text>
<text x="317.05" y="675" class="t-body" text-anchor="start">whole set as low confidence rather than citing it.</text>
<line x1="28" y1="709" x2="1012" y2="709" class="edge-soft"/>
<rect x="353.25" y="698" width="333.5" height="22" rx="11" class="box-sunken" />
<text x="520" y="713" class="t-muted" text-anchor="middle">text_search(): the same query with no model at all</text>
<text x="274" y="741" class="t-edge" text-anchor="middle">text_search(db, query, user=...)</text>
<line x1="151" y1="758" x2="397" y2="758" class="edge" marker-end="url(#arrow)"/>
<text x="643" y="780" class="t-edge" text-anchor="middle">SAME allowed CTE, then the german tsvector GIN index alone</text>
<line x1="397" y1="797" x2="889" y2="797" class="edge" marker-end="url(#arrow)"/>
<rect x="300.975" y="813" width="438.05" height="53" rx="6" class="note" />
<text x="313.975" y="830" class="t-body" text-anchor="start">The fallback for a dead embedding endpoint: keywords, not meaning,</text>
<text x="313.975" y="843" class="t-body" text-anchor="start">so the user is told it was a search without a model. No embed call,</text>
<text x="313.975" y="856" class="t-body" text-anchor="start">which is why it works when nothing runs behind the LLM config.</text>
<line x1="28" y1="890" x2="1012" y2="890" class="edge-soft"/>
<rect x="390.15" y="879" width="259.7" height="22" rx="11" class="box-sunken" />
<text x="520" y="894" class="t-muted" text-anchor="middle">rag/similarity: threshold, not ranking</text>
<text x="274" y="922" class="t-edge" text-anchor="middle">similar_chunks(db, text, user=..., max_distance=&lt;one of two constants&gt;)</text>
<line x1="151" y1="939" x2="397" y2="939" class="edge" marker-end="url(#arrow)"/>
<text x="520" y="961" class="t-edge" text-anchor="middle">embed([text])</text>
<line x1="397" y1="978" x2="643" y2="978" class="edge" marker-end="url(#arrow)"/>
<text x="643" y="1000" class="t-edge" text-anchor="middle">SAME allowed CTE, then pure vector:</text>
<text x="643" y="1013" class="t-edge" text-anchor="middle">ORDER BY cosine distance LIMIT top_k</text>
<line x1="397" y1="1030" x2="889" y2="1030" class="edge" marker-end="url(#arrow)"/>
<text x="643" y="1052" class="t-edge" text-anchor="middle">rows with a distance that is ALWAYS present</text>
<line x1="889" y1="1069" x2="397" y2="1069" class="edge-soft" marker-end="url(#arrow-open)"/>
<text x="274" y="1091" class="t-edge" text-anchor="middle">SimilarChunk[] filtered to distance &lt;= max_distance</text>
<text x="274" y="1104" class="t-edge" text-anchor="middle">(similar_documents groups per document, keeping the min)</text>
<line x1="397" y1="1121" x2="151" y2="1121" class="edge-soft" marker-end="url(#arrow-open)"/>
<rect x="288.675" y="1137" width="462.65" height="92" rx="6" class="note" />
<text x="301.675" y="1154" class="t-body" text-anchor="start">Why not the RRF path: its score is a fusion rank, not a similarity,</text>
<text x="301.675" y="1167" class="t-body" text-anchor="start">and vector_distance is null for FTS-only hits. A threshold needs a</text>
<text x="301.675" y="1180" class="t-body" text-anchor="start">comparable number. The permission filter is shared; only the ranking</text>
<text x="301.675" y="1193" class="t-body" text-anchor="start">differs. Two calibrated constants exist, CAPTURE_CONTEXT_MAX_DISTANCE</text>
<text x="301.675" y="1206" class="t-body" text-anchor="start">(loose) and DUPLICATE_MAX_DISTANCE (tight); exclude_builtin drops</text>
<text x="301.675" y="1219" class="t-body" text-anchor="start">Pablan&#x27;s own help pages, because a capture asks what the COMPANY knows.</text>
<rect x="294.825" y="1241" width="450.35" height="79" rx="6" class="note" />
<text x="307.825" y="1258" class="t-body" text-anchor="start">Document search (GET /api/documents/search) is the same call with a</text>
<text x="307.825" y="1271" class="t-body" text-anchor="start">larger top_k, grouped per document: one code path, so browsing and</text>
<text x="307.825" y="1284" class="t-body" text-anchor="start">chat can never disagree on access. Query mode surfaces the run as SSE</text>
<text x="307.825" y="1297" class="t-body" text-anchor="start">state events (searching, results(count) | no_answer, answering).</text>
<text x="307.825" y="1310" class="t-body" text-anchor="start">Counts only, never the query text or a passage (rule 12).</text>
</svg>
Side by Side

After

Width:  |  Height:  |  Size: 12 KiB

docs/i18n.md
+214
View File
@@ -0,0 +1,214 @@
# Internationalization
The interface ships in German and English. **German is the source
language**: messages are authored in `messages/de.json` first, and English
is written in the same change. Product CONTENT stays out of this system
entirely (see "What is not translated").
Compiler: [Paraglide JS](https://paraglidejs.com). Messages compile to
tree-shakable functions, so an unused string costs nothing at runtime and
there is no runtime i18n library in the bundle.
## Where things live
| Path | What |
|---|---|
| `frontend/project.inlang/settings.json` | locales, base locale, message file pattern |
| `frontend/messages/{de,en}.json` | the messages themselves |
| `frontend/src/lib/paraglide/` | compiler output, generated on every build, gitignored |
| `frontend/src/lib/i18n/locale.svelte.ts` | the reactive locale store and the client strategy |
| `frontend/src/lib/i18n/strategy.server.ts` | the server strategy that reads `users.locale` |
| `frontend/scripts/check-messages.py` | the two lint rules, run by `make lint` |
`npm run messages` compiles the message files into
`src/lib/paraglide/`. `npm run check` and `npm run lint` both run it first,
because `svelte-kit sync` does not execute vite plugins: without it, a
newly added message is a type error against a stale build, and the fix
looks like it belongs in the component. `npm run dev` and `npm run build`
compile through the vite plugin as usual.
## Message keys
Keys are **feature-scoped**, `snake_case`, and read as
`<feature>_<thing>_<role>`:
```
settings_theme_light good
settings_password_repeat good
common_cancel good, for genuinely shared words
light bad, collides across features
button_label_2 bad, says nothing
```
A key belongs to the surface that owns the string. Reach for `common_*`
only when a word is shared by unrelated features and would be identical in
both languages in every one of them, which is rarer than it looks:
"Cancel" qualifies, "Name" usually does not, because the noun it labels
differs by context in German.
Never reuse a key just because two strings happen to match in German
today. Translation splits them apart sooner than you expect.
**Keys are never constructed at runtime.** `m['status_' + doc.status]` is
forbidden, however tempting it looks next to a status enum. Paraglide is a
compiler: it can only check that a key exists, and only tree-shake the ones
you do not use, if every key appears literally in the source. A computed
key silently ships every message in the bundle and turns a typo into a
runtime blank instead of a build error. Write the mapping out:
```ts
const STATUS_LABELS = $derived({
draft: m.documents_status_draft(),
published: m.documents_status_published()
});
```
## Using messages
```svelte
<script lang="ts">
import { m } from '$lib/paraglide/messages';
</script>
<h2>{m.settings_title()}</h2>
```
Interpolation passes an object; the placeholder name is the key:
```json
"landing_greeting_morning": "Guten Morgen, {name}"
```
```svelte
{m.landing_greeting_morning({ name })}
```
Pluralization lives in the message rather than in the component, as a list
of **variants**. Inline ICU (`{count, plural, one {...}}`) inside a plain
string is NOT parsed by the message-format plugin: it compiles into a
placeholder with a nonsense name, which type-checks against nothing and
renders as broken text. Use the variant form:
```json
"landing_review_pending": [
{
"declarations": ["input count", "local countPlural = count: plural"],
"selectors": ["countPlural"],
"match": {
"countPlural=one": "Ein Dokument wartet auf deine Prüfung.",
"countPlural=other": "{count} Dokumente warten auf deine Prüfung."
}
}
]
```
The compiler resolves the category through `Intl.PluralRules` for the
active locale, so a language with more categories than German or English
(Polish, Arabic) only needs more `match` entries, never a code change.
Never build a sentence by concatenating messages. Word order differs
between languages, so a message has to be a whole sentence with holes in
it, not a sentence assembled from fragments.
**Option lists must be `$derived`, not `const`.** A constant array of
labels is evaluated once and then never again, so it keeps the language it
was born in:
```svelte
const THEME_OPTIONS = $derived([{ value: 'light', label: m.settings_theme_light() }]);
```
Dates and numbers go through `Intl`, and **the locale is always passed
explicitly**:
```ts
const dateFormat = $derived(new Intl.DateTimeFormat(i18n.locale, { dateStyle: 'medium' }));
```
Not `new Intl.DateTimeFormat(undefined, ...)` and not
`date.toLocaleDateString()`. Both fall back to the *browser's* locale,
which is a different question from the interface language: someone running
an English browser who set the interface to German would get German labels
around English dates. Passing `i18n.locale` also makes the formatter
`$derived`, so it rebuilds when the language changes.
## Per-route migration checklist
A route is done when all of these are true, not just the visible sentences:
- [ ] Body copy, headings, button labels, empty states, error messages
- [ ] `aria-label` and the `label` prop on icon-only controls
- [ ] `title` attributes (tooltips, truncated text, disabled explanations)
- [ ] `alt` text on images
- [ ] `placeholder` on inputs and textareas
- [ ] `confirm()` / `alert()` text
- [ ] The page `<title>` in `<svelte:head>`
- [ ] Option lists converted from `const` to `$derived`
- [ ] `Intl` formatters take `i18n.locale` explicitly
- [ ] No computed message keys introduced
The invisible ones matter most in practice, because nothing on screen
reminds you they are wrong. A screen reader user on a German interface
hitting an English `aria-label` gets no visual cue that anything is off.
## How the locale is resolved
Strategy chain, highest precedence first
(`frontend/vite.config.ts`):
1. `custom-userPreference` reads `users.locale`. A language someone chose
should follow them to every device, so this outranks everything.
2. `cookie` keeps that decision available before `/me` has answered.
3. `preferredLanguage` is the browser's `Accept-Language`, the first
contact guess for someone who has never chosen.
4. `baseLocale` is the floor.
There is deliberately **no `url` strategy**. Pablan is one installation for
one company; `/de/` path prefixes would buy nothing and invalidate every
existing link.
Server side, `hooks.server.ts` runs `auth` before `i18n`: the auth handle
fetches `/me` anyway, so it stashes the user's locale in a `WeakMap` keyed
by the `Request`, and the custom strategy reads it from there. A strategy
only receives the request, and a second `/me` per page render to answer the
same question would be waste.
Client side, `i18n.init()` points Paraglide's `getLocale()` at a `$state`
rune. Message functions call `getLocale()` internally, so that read is
tracked by Svelte and **switching the language re-renders the strings in
place**: no reload, no flash of the previous language, and no lost form
input. `setLocale(..., { reload: false })` still runs the chain so the
cookie is right for the next server render.
## The two lint rules
`frontend/scripts/check-messages.py`, wired into `make lint`:
1. **Every locale carries every key.** Paraglide falls back to the base
locale for a missing message, which means an untranslated string ships
silently as German inside an English interface. A missing key fails the
build instead. Keys present in `en` but not in `de` fail too: the source
language defines the set.
2. **No em or en dashes in messages** (`—`, `–`). They are awkward to type,
drift in style when hand-written across a codebase, and in German they
collide with the Gedankenstrich convention. Use a comma, a colon, or two
sentences. The rule reads message files only, so prose in `docs/` and in
code comments is unaffected.
## What is not translated
- **Knowledge content.** Documents are whatever language their author
wrote them in. The embedding model is multilingual, so a German document
answers an English question; one eval case covers exactly that.
- **Template content.** Personas and section hints are customer content, not
UI. Blueprints carry a `locale:` field and a filename suffix
(`prozess.de.yaml`), the catalog collapses variants to one entry
per id, and the picker lists templates matching the reader's language
first. See `authoring-templates.md`.
- **Backend strings.** The backend never renders UI-language text. API
errors are `{detail, code}` and the frontend translates by `code`; SSE
`state` events carry counts and markers, and the frontend writes the
sentence.
- **Language names** in the switcher. "Deutsch" stays "Deutsch" in the
English interface, because the point of that entry is to be recognised
by someone who cannot read the language currently on screen.
docs/licensing.md
+50
View File
@@ -0,0 +1,50 @@
# Licensing model
**Decision: Fair Source core (FSL-1.1-ALv2) + proprietary `ee/` layer.**
## How it works
- Everything outside `ee/` is licensed under the **Functional Source License
1.1 with Apache 2.0 future grant** (`LICENSE` in repo root, text from
fsl.software). Anyone may read, audit, self-host and use Pablan internally
for free; offering a competing product/service based on the code is
prohibited. **Each released version automatically becomes Apache 2.0 two
years after its release** — the built-in trust promise: even if Pablan (the
company) disappears, the code inevitably becomes true open source.
- `ee/` is proprietary (`ee/LICENSE`), activated via license key. Planned EE
scope: OIDC/Entra ID SSO, fine-grained permissions beyond the basics, the
management insights module, audit log, priority support.
- The free core is the funnel: small teams start free, grow, need SSO and
permissions — that is where revenue begins (GitLab/Cal.com playbook,
Sentry licensing).
## Rules that follow from this
1. **Never call it "Open Source"** in public communication — FSL is not
OSI-approved. The correct, established term is **"Fair Source"**
(fair.io). Using "open source" invites justified openwashing criticism.
The two-year Apache conversion may and should be highlighted.
2. **Core must never import from `ee/`** — enforced by import-linter in CI.
The core only offers extension points (mode registry, frontend slot
registry, `ee_hooks.py`).
3. **CLA required from day one** if external contributions are ever
accepted — without it we can never change the license later.
4. **Register the "Pablan" trademark.** The license protects code; the brand
carries the trust. This also neutralizes forks: a fork may not be offered
commercially (FSL) and must be renamed (trademark).
5. `LICENSE` (root) and `ee/LICENSE` exist from the first commit so the
boundary is unambiguous in the entire history. The final wording of
`ee/LICENSE` and the FSL fine print are owned by the legal co-founder
with an IT lawyer.
## Rationale (short)
Considered alternatives: Open Core with MIT/Apache core (maximum trust, but
competitors may resell the core), AGPL + dual licensing (self-hosting
without distribution triggers no obligations → no revenue mechanism for our
deployment model, only FUD), fully proprietary (loses bottom-up adoption —
the only realistic sales channel for a three-person company). FSL + `ee/`
keeps the auditability that our target market actually means by "trust"
(every commit inspectable — critical when companies put their entire
internal knowledge into the system) while protecting against the one real
threat: a competitor commercializing our code.
docs/notes.md
+322
View File
@@ -0,0 +1,322 @@
# Implementation notes
Durable engineering learnings — decisions, calibrations and gotchas that are
still true of the system as it stands. Grouped by theme rather than by when they
were learned. The [roadmap](roadmap.md) says *what* exists; this says *why it is
the way it is* and *what will bite you*. For formal decisions see
[decisions.md](decisions.md); for the as-is contract see
[architecture.md](architecture.md) and [data-model.md](data-model.md).
## Retrieval & embeddings
- **One permission filter, evaluated live.** `rag/permissions.py` is the single
source both the API and retrieval use, and it runs against the current tables —
never against the denormalized visibility/department copy stored in chunk meta
(that copy is display-only and can lag a reindex). This is what makes "a chunk the
user may not read is structurally unreachable" true rather than aspirational.
- **Authors vs retrieval.** An author sees their own drafts through the API
(`readable_documents_filter`), but retrieval is published-only
(`searchable_documents_filter`), so a draft never reaches an LLM prompt. Admins
get **no** read-everything bypass — an admin only sees what the normal rules grant.
- **Hybrid search is one SQL statement.** pgvector top-20 + German FTS top-20 fused
with Reciprocal Rank Fusion, `RRF_K=60`. Keeping it one statement is what lets the
permission CTE gate everything before ranking.
- **Thresholds are calibrated against bge-m3 on the fixture corpus, and must be
re-measured if the embedding model changes.** No-answer fires at vector distance
≥ 0.45 (matched hits land ~0.34–0.45, unrelated ~0.50+). The similarity path
(`similar_chunks`) uses `max_distance` 0.45 for capture-context.
- **A chunk is embedded WITH its heading path** (`rag/indexing.embedding_text`).
A section reads "Solldruck 180 bar" and never repeats which machine it belongs
to, so on its own it is unreachable by the name the asker uses. Measured on the
dev corpus: "Welcher Solldruck gilt für die Hydraulikpresse?" went from a
no-answer (best distance 0.492, above the 0.45 gate) to a confident hit (0.375,
and the section that literally answers it at rank 2). The genuine no-answers
moved too little to matter (Deutschlandticket 0.503 → 0.488), so the gate keeps
separating them. What is STORED as `content` stays the raw section: the path is
context for the vector, not part of the document, and the excerpt a reader sees
must not grow a header. Tests that search for "the exact chunk text" have to
build it through `embedding_text` or they land at distance ~0.96.
- **The chunk `tsv` covers the heading path too** (generated column over
`content || ' ' || meta->>'heading_path'`). Same reason on the keyword side:
"Solldruck Hydraulikpresse" matched no chunk before, because only the document
title carries the machine name. Note what it does NOT fix: `websearch_to_tsquery`
is AND over every surviving lexeme, and German stopword lists keep verbs, so
"Welcher Solldruck **gilt** für die Hydraulikpresse?" still has no keyword match.
Treat `fts_match` as "the keywords really are in there", not as "this was a
keyword question".
- **Calibrate on what the engine renders, not on what a human writes.** Real capture
drafts are compressed notes and sit further from their source than a hand-written
paraphrase, so thresholds tuned on paraphrases miss real matches.
- **`rag/similarity` is deliberately not the hybrid path.** RRF produces a fusion
rank, not a similarity, and its vector distance is null for FTS-only hits — a
threshold needs a comparable number, so similarity is a pure-cosine query with the
threshold applied in Python *after* `ORDER BY … LIMIT` (a distance predicate in
WHERE fights the HNSW index).
- **HNSW post-filtering can under-fill.** For a heavily permission-filtered user, the
fixed candidate fetch can return fewer rows than expected; keep candidate counts
generous.
- **Hybrid retrieval is robust to statement-vs-question phrasing; the pure-cosine path
is not.** A declarative statement ("Die Zentralschmierung … muss aufgefüllt werden.")
retrieves the same top document as the equivalent question, because the German FTS
component matches the keywords regardless of phrasing — guarded by declarative-
statement cases in `golden_queries.yaml`. The pure-cosine similarity path
(`similar_documents`/`similar_chunks`, used by the capture picker and refinement
grounding) has no FTS to lean on and is therefore more sensitive to how a thought is
phrased. It stays pure-cosine on purpose (a comparable distance for its threshold),
so a phrasing-sensitive picker is a known trade-off, not a bug.
## Permissions & auth
- **FK delete semantics encode the product.** Authored documents survive their
author (`ON DELETE SET NULL`) — "an employee leaving is exactly the case the
product exists for". The same holds for a document's department.
- **The auth boundary is a full-page navigation.** Login and logout do a real
document load (`window.location.assign`), never a client `goto`. Module-level runes
singletons (conversation list, chat state, resolved locale) live for one page load,
so reloading at the boundary guarantees the next user starts clean. A client
navigation kept the previous user's conversation titles on screen — an information
disclosure. Logout also clears the `PARAGLIDE_LOCALE` cookie so a shared terminal
does not pin one user's language for the next.
- **A private helper imported across modules is a design smell, not a shortcut.**
Four of them had grown (`_readable_document`, `_role_config`, `_excerpt`,
`_HEADING_RE`): each marked a place where one module had quietly become part of
another's interface without saying so. Splitting the big API modules surfaced
all four at once. When a second module wants an underscore name, that name is
the interface and should be renamed, not imported.
- **A test double that patches by module name silently rots.** The fake-embedding
fixture patched `indexing` and `retrieval`; the moment the similarity path moved
into its own module, an "offline" test would have called a real endpoint. A
fixture that names the modules it patches has to be updated with every split, so
it says why in a comment right there.
- **A history snapshot is written AFTER its event.** So the content stored on an
`edited` event is the state that edit produced, not the one it replaced. Diffing
a snapshot against the live document therefore shows the change of the NEXT
entry, and the newest entry shows nothing at all. What a timeline entry has to
answer is "what did this one do", so the version endpoint returns the pair
(`content_md` + `previous_content_md`) and the dialog diffs those.
- **The product degrades to a search engine, not to an error page.** Postgres is
already an inverted index (`chunks.tsv`, a stored `to_tsvector('german', …)`
column with a GIN index), so "no model reachable" costs the generated answer and
the semantic half of retrieval, nothing else. The turn ends with the full-text
hits as a list, marked as a search without a model, and is persisted like a
normal reply so a reload replays it. Worth remembering when adding any other
model-dependent surface: ask what remains without the model.
- **Classify an endpoint failure once, in `LLMError.code`.** "Not reachable" and
"busy, try again" need different words to the user, and a timeout means the
second, not the first: the endpoint took the request and never came back.
Retrieval embeds before the model is ever called, so the failure often happens
in `search()`, not in `chat_stream` — the conversations router catches
`LLMError` centrally rather than every mode remembering to.
- **Enumeration-resistant login.** An unknown email still runs a constant-time dummy
verify, so timing does not reveal which accounts exist.
- **Admin CRUD guardrails.** An admin cannot delete themselves or change their own
role (409 `self_modification`); a password change or reset revokes all of that
user's other sessions.
- **Sharing is grants, not new permission logic.** Multi-department sharing plugs
straight into the `EXISTS doc_permission` branch that was already in
`searchable_documents_filter` — `PUT /documents/{id}/departments` just manages the
rows. No reindex: grants are evaluated live against the table, not denormalized into
chunk meta. Any per-department count therefore measures REACH, not a partition:
a shared document counts under every department it reaches, so such counts can
exceed the distinct document count.
- **Self-lockout only bites admins.** `readable_documents_filter` grants the author
read access unconditionally, so a normal author can never edit themselves out of
their own document; the guard (`_guard_self_lockout`, a Python mirror of the read
rules) only ever fires for an admin changing a document they do not own, who must
resend with `confirm_lockout`. When mirroring a SQL filter in Python for a
pre-commit check, keep the two in the same file so they cannot drift.
- **A `bool = False` Pydantic field serializes as *required* in the generated TS
client** (openapi-typescript), which breaks every existing caller that omitted it.
Use `bool | None = None` (like the other optional fields) and `bool(...)` at the use
site — then it stays optional in `schema.d.ts`.
- **The audit trail is where "who checked it" lives.** `document_events`
(`authoring/history.py::record_event`) appends who-did-what at every transition;
`review_resolved` writes the answering colleague as the event actor — once a
request is answered, the `review_requests` row only says that nothing is open
any more, so the trail is the only place that identity survives. Content-bearing events
(`created`, `edited`) snapshot the Markdown, never the chunks, so per-event
snapshots of the *resulting* state chain into a version diff. Gotcha: a freshly
created document must be `flush()`ed before recording its `created` event — the
UUID PK default lands at flush, not at object construction, so `document.id` is
None until then. History is served through the document's own read gate, so it
cannot leak to a user who may not read the document.
## LLM & prompts
- **`chat_template_kwargs.enable_thinking = false` is the big latency win.** On a
reasoning-capable local model it drops section refinement from ~10s to ~1s with no
quality loss, because refinement is a mechanical rewrite. Passed through
`extra_body`; ignored by endpoints that do not support it.
- **Prompt shape for caching.** Keep the system prompt byte-identical and the history
stable, and put the volatile part (retrieval excerpts, the grounding block) last —
the endpoint then reuses the cached prefix and only reprocesses what changed.
- **Content never reaches logs.** `LLMError` is raised `from None` so a model
response cannot ride out in a chained traceback; SQL bind params are stripped; the
openai SDK's DEBUG body-logging is capped unless `PABLAN_DEBUG_LOG_PROMPTS` is set
(never in production). This is canary-tested.
- **One SDK-level retry** covers the occasional llama.cpp `APIConnectionError`;
`chat_json` additionally validates and retries once on a schema mismatch.
- **Prompts are admin-editable, defaulting to code.** Every system prompt renders
through `app/prompts/overrides.py::get_prompt(key)`, which returns a DB override
(`prompt_settings`) or the code default (`app/prompts/defaults.py`). This mirrors the
LLM-settings cache but is simpler: the reset target is the code default (prompts have
no `.env`), and there is no bootstrap — a missing row just means "use the default".
Keep the module-cache clear in the test teardown, or an override leaks into the next
test. Prompt-cache byte-identity (D22) still holds because a value changes only on an
admin write.
- **The context inspector needs the FULL retrieval, so a no-answer keeps its results.**
Query mode used to discard the low-confidence hits (`results = []`); now it keeps them,
sends them in the `sources` event marked `used=False`, and passes only `used` ones to
the prompt. That is what lets the "?" inspector explain a no-answer ("these were close
but too weak") instead of showing nothing. The cited-source badges filter to `used`.
- **The shipped system prompt was picked by measurement, not by taste.** A bench
over 15 objective checks (grounding phase, the load-bearing fact, an honest
"not documented", answer language) on the dev corpus: the long rules-list
version scored 42/45 at 486 chars mean answer, a four-line version 45/45 at 279
chars. Everything the list spelled out, the model already did; what it was
missing was the one instruction that carries its weight — *say when something
is not documented* — without which the three no-answer cases all produced
plausible filler. Shorter prompt, shorter answers, better score. The bench lives
in the session scratch, not in the repo: it needs a seeded instance and a live
endpoint, which is what `make eval` is for — port a case there when it earns a
permanent floor.
- **Asking about the product is a retrieval question, not a special case.** The
`help/` pages are indexed like anything else, so "how do I share a document?"
is answered through the same path — and `tests/evals/test_self_knowledge_eval.py`
measures it (10/10, and asserts the help pages do NOT outrank company documents
for a company question, which is the failure mode that would matter).
- **Cloud validation is still owed.** Every shipped prompt (query, refinement,
grounding, topic summary, title) has only run against a local Gemma-class model.
The first cloud-model eval pass is roadmap epic 15 — expect to fix behaviour a 26B
model tolerated.
## Jobs & runtime
- **The claim transaction stays open for the whole handler.** Claiming a job
(`FOR UPDATE SKIP LOCKED`) and completing it happen in one transaction, so a crash
mid-handler rolls the claim back and the job is retried — at the cost of one pinned
connection per in-flight job.
- **Backoff** is 30s·2^n over 5 attempts; failure bookkeeping runs in its own
transaction. Retention cleanup reschedules itself daily.
- **The endpoint gate is per base_url, not per role** (`llm/gate.py`). chat and
utility point at the same server in the shipped config, so a per-role limit
would let one server take twice its slots. The waiting is bounded twice (a wait
timeout and a queue-length cap) because an unbounded wait in front of a
saturated endpoint is the same failure as no gate at all, just quieter. Both
refusals reuse `llm_busy`, so the UI has one sentence rather than two. The
background job worker is strictly sequential, so a big reindex contributes ONE
concurrent embedding call, not a burst.
- **Exactly one uvicorn worker.** The job loop, the metrics registry and the LLM
client cache are per-process; a second worker would double-run the queue and split
the metrics. The compose file pins `--workers 1`.
## Frontend & i18n
- **Citations are snapshotted into `messages.meta`.** Chunks are disposable and get
reindexed, so an answer freezes the exact passages it cited at answer time; the
excerpt is re-cleaned to plain prose on the way out (cleaning is idempotent).
- **SSE via fetch + ReadableStream, not `EventSource`.** The turn is a POST that must
be abortable (the stop button), which `EventSource` cannot do. A client abort still
persists the partial assistant message (`asyncio.shield` on the backend).
- **Theme per device, language per account, both applied pre-paint.** Theme lives in
localStorage and is stamped onto `<html>` by a boot script before first paint;
language comes from `users.locale` and is baked into the first SSR byte — otherwise
the page flashes the wrong theme/language on every load.
- **Paraglide pitfalls.** Source is **de** (informal "du"), en written in the same
change; two lint rules fail the build (a missing `en` key, an em/en dash). There is
no `url` locale strategy (locale is not in the path). `Intl` formatters must be
passed the resolved locale explicitly or they default to the server's. ICU plurals
have their own syntax trap. Recompiling messages under a running dev server can
wedge it — restart after adding keys.
- **`hooks.server.ts` needs an absolute backend URL.** A relative `event.fetch` never
reaches the vite proxy, which silently breaks server-side auth calls.
- **The chat renders Markdown LIVE while streaming.** Rendering plain-text tokens and
then re-rendering to Markdown on completion read as a snap; the turn now goes through
the sanitizing `Markdown` component every token, so the formatted output is unveiled
progressively. Partial constructs (an unclosed `**` or `$`) render as literal text
until closed — a small local flicker, far better than the whole-reply snap.
- **The local model emits real LaTeX**, inline `$...$` and display `$$...$$` with
`\frac`, `\sqrt`, superscripts. `$lib/markdown.ts` runs `marked-katex-extension` +
KaTeX BEFORE DOMPurify, and the KaTeX CSS/fonts are bundled locally (imported in the
root layout — no CDN, the artifacts self-host). DOMPurify keeps `<span class style>`
by default, so the typeset math survives sanitizing; `throwOnError: false` degrades a
malformed formula to text instead of throwing mid-answer.
- **Prefer `SvelteSet` over a plain `Set`** for reactive membership state — eslint
(`svelte/prefer-svelte-reactivity`) enforces it, and mutating it (`add`/`delete`)
is reactive without reassigning.
- **The refine suggestion is an inline CodeMirror block widget, not a side pane.**
It appears right under the section being edited (`WritingEditor.svelte`): a
`WidgetType` wrapping a persistent DOM element (built once, `eq()` compares by
element identity so CodeMirror reuses it) placed by a `StateField` +
`StateEffect`. Tokens stream by mutating that DOM directly; CodeMirror does not
learn a widget's height changed on its own, so call `view.requestMeasure()`
(rAF-throttled) after each update or the lines below overlap. `ignoreEvent()`
returns true so the accept/dismiss buttons handle their own clicks, and their
`mousedown` preventDefault keeps the editor from blurring first.
- **The editor is not continuously autosaved** (that would empty the
diff you are shown before saving). A `beforeNavigate` guard instead discards an
abandoned, never-filled template draft (only headings/blank lines) and saves an
edited-but-unsaved draft on the way out — fixing both the data-loss trap and the
empty-draft clutter without touching the save flow. Skip it once the document
was published from the editor, or it would re-save over the publish.
- **Seeded documents are built through the real process, not inserted finished**
(`app/seed.py`). Each one gets the history it would have had: the empty draft,
one `edited` snapshot per section as the author works down the page, the
publish, and the questions colleagues asked afterwards — plus a few documents
left as drafts and one archived, so every screen has something to show. Without
it the history view, the version diffs and "recently changed" are empty or lie
in every dev stack. Order the corpus deliberately (the overdue document has to
be an old one, drafts have to be the newest), and give `document_events` and
`review_requests` explicit timestamps: `server_default=now()` would stamp the
whole life of a document at seed time.
- **Bits UI keeps inactive tab panels mounted**, just hidden. On a page of
independent panels (`/admin`) that means all four fetch their data on
arrival, and controls nobody can see sit in the DOM where a click never
reaches them (Playwright finds the element, then times out waiting for it to
be visible). `lib/components/Tabs.svelte` renders only the open panel.
- **A grant that ends needs its own access reason.** Answering a review request
on someone else's draft takes the access away again — correct, and it dropped
the reviewer on a 404 for doing what they were asked. `AccessReason.review`
names the case ("the only thing letting you in is the request"), so the UI can
say thank you instead. It mirrors `readable_documents_filter`, where the
visibility rules only apply to a PUBLISHED document: on a draft, `public`
never explains access, the request does.
- **The classic-search fallback ORs its terms; the hybrid path ANDs them**
(`rag/retrieval._any_term_tsquery`). `websearch_to_tsquery` builds an AND over
every lexeme, which is fine when the vector half carries the recall — but alone
it answers a typed-out question ("Wie läuft die Qualitätsprüfung im
Wareneingang?") with nothing at all, because no single chunk contains every
word. Rewriting the operators between the groups (`' & '` → `' | '` on the
tsquery text) keeps quoted phrases and exclusions intact and leaves the ranking
to `ts_rank_cd`. `NULLIF` the result: a query of nothing but stop words
rewrites to an empty string, and `to_tsquery('')` is a syntax error rather than
an empty match.
- **The fallback needs chunks, so it needs a past embedding run.** `chunks.tsv`
is generated from `chunks.content`, and a chunk row is only written when
`reindex_document` succeeded — which calls the embedding endpoint. A document
published while the embedding endpoint is down has no chunks at all, so the
"classic search" fallback cannot find it either. It keeps an already-indexed
knowledge base usable without a model; it does not index without one.
- **The e2e suite pins English with Accept-Language, which an ACCOUNT language
outranks** — that is the product rule (a language a person picked follows
them to every device), so a dev stack where somebody switched the admin to
German failed every spec that reads a label. The login helper hands the
account back to "follow the browser"; specs that run as that user select
tabs by testid rather than by label.
- **`npm run check` compiles Paraglide with the CLI**, which used to write the
DEFAULT strategy chain over the one `vite.config.ts` configures — leaving the
dev server (and the e2e suite, which pins `Accept-Language`) resolving German
for everyone until the next vite restart. The `messages` script now passes the
same `--strategy` flags. The other half of the trap stands: recompiling while
`vite dev` runs breaks its module graph until restart, which is why `make lint`
skips the recompile when :5173 is listening.
## History
A dialogue-driven "interview" capture engine (a state machine that interviewed the
employee and assembled a document from the answers) was built first, then removed
and replaced by the writing-first editor that exists today. Nothing from it remains
in the code path; the roadmap describes building the current system directly. The
only ideas that carried across are generic: template import/upsert by config id, and
the `pyyaml` dependency.
docs/roadmap.md
+508
View File
@@ -0,0 +1,508 @@
# Roadmap
The feature backlog for Pablan, organized by feature area rather than by
delivery date. It doubles as a **rebuild manual**: follow the done epics top to
bottom and you arrive at the system as it stands today; the open epics are the
road ahead.
**How to read it.** Each epic is a feature area with a one-line intent and a
checklist of stories — `[x]` shipped, `[ ]` not yet. Epics are ordered as a
build sequence (foundation upward). File pointers name where a capability lives,
so a rebuilder can find the pattern to follow. Implementation learnings that
survive across features live in [notes.md](notes.md); the as-is contract lives
in [architecture.md](architecture.md), [data-model.md](data-model.md) and
[api-protocol.md](api-protocol.md).
---
## 1. Platform foundation *(done)*
The backend skeleton every other feature stands on.
- [x] Config via `pydantic-settings`, all `PABLAN_*` env vars; per-role LLM
config (base_url / api_key / model for chat, utility, embedding).
- [x] Async SQLAlchemy 2.0 (typed) + Alembic migrations on PostgreSQL + pgvector;
UUID primary keys and a shared timestamp mixin (`models/base.py`).
- [x] The table set — see [data-model.md](data-model.md) and
[diagrams/data-model.svg](diagrams/data-model.svg).
- [x] pytest harness against a real Postgres (docker); retrieval logic gets
SQL-level tests.
- [x] Seed script with a production guard; a shared German fixture corpus
(`tests/fixtures/corpus/` + `golden_queries.yaml`) that seeds and tests draw
from.
## 2. Authentication & sessions *(done)*
Server-side sessions, no JWT, and an auth boundary that cannot leak one user's
state to the next.
- [x] argon2 password hashing; `auth_sessions` table; httpOnly `pablan_session`
cookie (14-day TTL); `{detail, code}` error shape.
- [x] login / logout / me; enumeration-resistant login.
- [x] Session revocation (`auth/sessions.py::revoke_user_sessions`) — a password
change logs out every other session.
- [x] **Full-page auth boundary**: login and logout are document navigations,
never client `goto`, so module-singleton client state (conversation list,
chat state, resolved locale) is guaranteed dead at the boundary and cannot
leak the previous user's data.
## 3. Frontend foundation *(done)*
The SvelteKit shell, the design system, the typed API contract, and i18n.
- [x] Semantic design tokens as CSS variables mapped into Tailwind; the palette
is swappable by editing tokens only; a WCAG-AA contrast gate
(`frontend/scripts/contrast-check.py`) runs in `make lint`.
- [x] A small styled component set on Bits UI (Button, Input, Select, Dialog,
Card, Badge, Tabs, Tooltip, Popover, FormField) — no component-library deps.
- [x] Typed API client: FastAPI OpenAPI → openapi-typescript → openapi-fetch;
`make types` regenerates it after any backend API change.
- [x] `(app)` route group + `hooks.server.ts` session validation (absolute
backend URL, since a relative `event.fetch` never hits the vite proxy).
- [x] The sanitizing Markdown renderer (`components/Markdown.svelte`, DOMPurify) —
all model and document output renders only through it.
- [x] Paraglide i18n: **de** source, **en** written in the same change; two lint
rules (missing `en`, em/en dashes) in `check-messages.py`; the backend never
renders UI strings — it returns `{detail, code}` and the frontend translates.
- [x] Theme per device (localStorage, applied pre-paint by a boot script) and
language per account (`users.locale`).
## 4. LLM gateway & background jobs *(done)*
One place that talks to models, and a Postgres-only work queue.
- [x] `llm/client.py` is the ONLY code that calls LLM endpoints: `chat_stream`,
`chat_json` (structured output via a Pydantic `response_format`), `embed`;
roles chat / utility / embedding, each independently configured; `extra_body`
passthrough (e.g. `enable_thinking:false`).
- [x] In-app LLM config: bootstrap each field from env, then the DB, with
per-field provenance ("from .env" / "changed here"); `llm/test` pings the
three roles; model discovery is server-side.
- [x] Postgres job queue (`jobs` table, `FOR UPDATE SKIP LOCKED`, backoff
30s·2^n over 5 attempts); the handler runs inside the open claim transaction;
exactly one uvicorn worker (queue and metrics are per-process).
- [x] Scheduled retention cleanup (reschedules itself); an in-process metrics
registry.
- [x] Content-safe logging: metadata only, never prompts / responses / user text;
`LLMError` raised `from None`; an openai-SDK DEBUG-log guard
(`PABLAN_DEBUG_LOG_PROMPTS`, never in production).
- [x] **A failing endpoint says why.** `LLMError.code` classifies every failure
once (`llm_unreachable` / `llm_busy` / `llm_misconfigured` / `llm_failed`);
every surface reports it unchanged and the frontend phrases it
(`lib/api/errors.ts`). The conversations router catches `LLMError` centrally,
so a failure during retrieval (which embeds before the model is called) ends
the turn with the reason instead of a truncated stream. A busy endpoint
queues rather than refuses, so the chat also says so after 12s of silence.
## 5. Documents & the knowledge model *(done)*
Markdown documents with a permissioned lifecycle; the single source of truth.
- [x] Documents stored as Markdown (`content_md`); chunks and embeddings are
disposable derivatives, always re-indexable from the document.
- [x] Lifecycle draft → published → archived; `visibility` (public /
department / restricted).
- [x] Publishing is the author's own action (`POST /documents/{id}/publish`),
plus archive/republish. Whether the CONTENT is trusted is a review
request instead, never a status (epic 16).
- [x] The **single permission filter** (`rag/permissions.py`) shared by the API
and retrieval, evaluated live against the table (never on stale chunk meta):
`readable` adds a user's own drafts and the documents someone asked this
user to check; `searchable` is published-only, so a draft never reaches
another user or an LLM prompt.
- [x] ZIP export of the readable knowledge base (Markdown + YAML frontmatter),
permission-filtered, help pages excluded, stdlib `zipfile`/`io` only.
- [x] Built-in help pages (`help/*.md`) imported into `documents` on start
(`is_builtin`, read-only, not deletable).
- [x] Admin CRUD guardrails (no self-delete / self-demote, 409 `self_modification`);
GDPR surfaces (delete-own-conversation, the retention job).
## 6. RAG retrieval *(done)*
Permission-filtered hybrid search, German-first.
- [x] Heading-aware chunking (the active-section boundary is shared with the
editor); index / reindex jobs; bge-m3 multilingual embeddings
(`EMBEDDING_DIM=1024`).
- [x] Hybrid search: a permission CTE → pgvector top-20 + Postgres `german` FTS
top-20 → Reciprocal Rank Fusion (k=60), all in one SQL statement.
- [x] **Permission filter before the LLM**: there is no search without a user;
an unauthorized chunk is structurally impossible to retrieve.
- [x] Similarity path (`similar_chunks` / `similar_documents`): pure cosine,
permission-filtered, threshold applied in Python after `ORDER BY … LIMIT`
(a WHERE predicate fights the HNSW index); calibrated constants
(no-answer ≥ 0.45, capture-context 0.45).
- [x] Eval suite (`make eval`): recall@5 baseline against the fixture corpus and
the no-answer threshold.
- [x] **The knowledge base stays searchable without any model.** `text_search()`
is the full-text half alone (the `german` tsvector GIN index, no embedding
call), same permission CTE; query mode uses it when the embedding endpoint
is gone, and when no model answers at all the turn ends as a plain hit list
the user opens themselves, labelled as a search without a model.
## 7. Query chat *(done)*
Ask the knowledge base a question, get a streamed, cited answer.
- [x] The `Mode` protocol + registry: a mode yields `ModeEvent`s and the router
converts them to SSE; Query is the only core mode (EE registers insight).
- [x] Conversations + messages; the SSE streaming turn; a stop button (a client
abort persists the partial message via `asyncio.shield`).
- [x] Citations snapshotted into `messages.meta` (chunks are disposable) + the
source side panel; one badge per document, not per chunk.
- [x] No-answer handling lives in retrieval (the ≥0.45 threshold), not the prompt;
a no-answer offers a "capture this knowledge" hand-off into the editor.
- [x] Topic-summary retrieval fallback: on a low-confidence hit with prior
context, re-search on an LLM topic summary of the conversation.
- [x] A prompt-cache-friendly shape: a stable system prompt + history, with the
volatile excerpts + question last.
## 8. Writing-first capture *(done)*
The centerpiece: employees write knowledge; the model matures the section at the
cursor. (This replaced an earlier dialogue-driven approach — see
[notes.md](notes.md) History.)
- [x] A CodeMirror 6 split editor at `/documents/{id}/edit` (also the edit surface
for any existing document); the caret is visible (`drawSelection` + accent
caret color).
- [x] Section refinement `POST /documents/{id}/refine` (SSE): the server owns the
active-section boundary (shared with the chunker), sends a FIM-style natural-
language prompt, and streams a matured version of only that section into the
right pane; `enable_thinking:false` keeps it ~1s; only `delta.content` is read.
- [x] **Retrieval-aware grounding**: before refining, the server searches the
permission-filtered knowledge base for related published material the author
may read (`_grounding` via `similar_chunks`, the current doc and help pages
excluded) and passes it to the prompt as a reference, not a fact source.
- [x] A staleness guard and a post-accept cooldown on "Übernehmen"; a suggestion
computed against a since-edited region is discarded rather than applied.
- [x] A pre-save diff: `@codemirror/merge` `unifiedMergeView` (VSCode-style).
**Save** sits bottom right where the writing ends, shows what changed, and
asks again — save, cancel, or (for a draft) save and publish.
- [x] Authoring templates (`AuthoringTemplate`): a Markdown skeleton + persona +
per-section hints — declarative config, not a Mode. A shipped catalog plus a
**form template builder** (`POST /templates/build`; the skeleton is derived
from the section headings), with raw YAML behind an advanced toggle.
- [x] Capture from a chat: carries the conversation as background context
(`meta.context`) and shows a matching-documents picker (topic summary →
`similar_documents`) to extend an existing document instead.
- [x] A reward flow after publishing (a check animation → View / Have it
checked), a title suggestion in the save dialog, and a streaming token
fade-in.
## 9. Insight & growth *(open)*
Aggregate signals about the knowledge base — never per-user data.
- [ ] A growth surface, in a form that earns its place. The first version (a
`/dashboard` page with bar charts off `GET /documents/dashboard`) was
removed again: numbers nobody acts on are noise, and the charts said
nothing the document list does not. Rebuild only around a question someone
actually asks.
- [x] `GET /documents/stats` — company-wide counts, read by the landing page's
first-run guide.
- [x] The document detail page shows created / last-changed timestamps.
## 10. Product content & docs *(done)*
The content that ships with the product, and the docs that describe it.
- [x] In-product help (`help/*.md`, German) imported as read-only documents,
kept current in the same change as the workflow it documents.
- [x] The shipped template catalog: blueprints on disk, inert until an admin adds
one; an added template becomes the customer's, fully editable.
- [x] Claude-maintained developer docs (`docs/`), hand-authored SVG diagrams
(`docs/diagrams/`, theme-aware, no toolchain), this roadmap, and
[notes.md](notes.md).
---
## 11. Change history & review audit *(done)*
A git-like record of who changed what, when, and who checked it — including
**who confirmed the content**, which used to be stored nowhere.
- [x] A `document_events` table: `document_id`, `actor_id`, `action` (created /
edited / published / archived / visibility_changed /
review_requested / review_resolved),
`created_at`, and — for content-bearing events — a snapshot of `content_md`,
title, visibility and meta. Snapshots the Markdown (the source of truth), never
chunks; `actor_id` is SET NULL and events CASCADE with the document.
- [x] An event is recorded at every transition: the content branch of
`update_document`, and create / publish / archive plus asking and
answering a review in `api/documents/` (helper
`authoring/history.py::record_event`). `review_resolved` captures **who**
confirmed the content as the event actor.
- [x] `GET /api/documents/{id}/history` → the events newest-first (actor name,
action, timestamp, `has_snapshot`), plus
`GET /api/documents/{id}/versions/{event_id}` to fetch a past version's content
for viewing or diffing. Both behind the document's own read gate.
- [x] A history timeline on the document detail page
(`lib/documents/DocumentHistory.svelte`) and a version diff reusing the
`unifiedMergeView` (shared theme extracted to `lib/documents/editorTheme.ts`),
with restore of a previous version.
- [x] **An entry shows its OWN change.** A snapshot is written after its event, so
diffing it against the live document made every row show the next row's edit
(the newest edit showed nothing at all). `GET /versions/{event_id}` now also
returns `previous_content_md`, the closest earlier snapshot, and the dialog
diffs the pair.
- [x] The review queue is discoverable from the landing page, deep-linking to
`/documents?review=1` (see epic 16 for the review model itself).
## 12. People & profiles *(done)*
A directory colleagues can browse, and ONE way to describe a person: a document
like any other.
- [x] A member-visible directory: `GET /api/people` (+ `/api/people/{id}`) →
colleagues' `{id, name, role, department}` for any authenticated user (no
password/email leakage; mirrors the permission-scoped shape of
`ReviewerCandidate`), with a directory page (`/people`) and a person page
(`/people/{id}`) wired into the nav.
- [x] **The profile page is the document about you.** `GET /api/account/document`
answers whether the caller wrote one (from the person blueprint) and what to
start it from otherwise; the page opens it for editing or offers to create
it. The blueprint id lives in the backend, so the frontend knows no ids.
- [x] **The self-written bio was removed again.** A short profile blurb and a
document about yourself are the same thing said twice, and only one of them
is findable by the search, versioned and approvable. The `users.bio` column,
both endpoints, the editor and its two prompts are gone.
## 13. Access control & sharing *(done)*
Let a document reach more than one department, and stop anyone from accidentally
locking themselves out of it.
- [x] **Multi-department documents.** `documents.department_id` stays the owning
department; extra departments are shared through the existing `doc_permissions`
join (the `searchable_documents_filter` `EXISTS doc_permission` branch already
unions them in — this was management, not permission logic).
`PUT /api/documents/{id}/departments` replaces the set; a detail-page multi-select
(`lib/documents/DepartmentSharing.svelte`) manages it; and several departments are
reflected in `shared_departments`, the `?department=` filter and the export
frontmatter.
- [x] **Self-lockout guard.** Before committing a visibility or grant change, the
*proposed* state is evaluated against the read rules (`_guard_self_lockout`, a
Python mirror of `readable_documents_filter`). An author keeps access as author,
so this only bites an admin editing a document they do not own: 409
`self_lockout_warning` unless resent with `confirm_lockout` (surfaced as an
inline "save anyway" in the UI).
- [x] Close the adjacent admin lockout vectors. `delete_department` now refuses to
silently delete a department that still has members, owned documents or grants
(409 `department_in_use` unless `?confirm=true`) — its `doc_permissions` grants
CASCADE away invisibly otherwise. The `delete_user` / `update_user`
department-nulling is deliberate orphaning (documents and users survive
authorless/departmentless, `notes.md`), not a silent lockout, and is documented
in `data-model.md`.
## 14. Prompt & context transparency *(done)*
Give admins control over the model's instructions, and make what the model is
working from visible to the user.
- [x] **Admin-editable system prompts.** Every system prompt (query + no-sources,
refinement persona/rules, grounding framing, topic summary, title) is
editable on the admin UI, stored in `prompt_settings` and applied without a
restart — mirroring the per-role LLM-settings pattern (`app/prompts/overrides.py`
+ `api/admin/`, code defaults in `app/prompts/defaults.py`), with a
reset-to-default per prompt (`lib/admin/PromptSettings.svelte`).
- [x] **Always-inspectable working context.** A "?" inspector in the chat
(`lib/chat/ContextInspector.svelte`) shows every retrieved passage, marking
which grounded the answer (`used`) versus which were too weak — so a no-answer
is explained, not silent. The editor shows the grounding references a refinement
drew from, sent as a `grounding` SSE frame.
## 16. Publish and review *(done)*
Separating "where does this document stand" from "is what it says right" — see
[decisions.md](decisions.md) D24.
- [x] Statuses are draft / published / archived; `pending_approval` is gone and
publishing is the author's own one-click action (from the save dialog, the
document page, or the landing page).
- [x] A `review_requests` table: one question, addressed to one colleague, open
until answered (`resolved_at` + `resolved_by_id`). Orthogonal to the status —
it can sit on a draft or on a document published months ago.
- [x] Being asked grants read AND edit until the answer
(`readable_documents_filter`, `can_edit`), so a reviewer fixes a wrong number
instead of filing a second question. Owner-only decisions (delete, sharing,
handing out a request) stay with the author or an admin.
- [x] An open question marks the document everywhere: list card, detail page, the
document panel, and the sources under a chat answer (`review_pending` on
every search hit, snapshotted with the citation).
- [x] The landing page surfaces open work: your unpublished drafts (with publish)
and the documents waiting for your check (`lib/documents/OpenWork.svelte`).
- [x] Follow-through from use: `AccessReason.review` names the access a request
grants, the document page thanks a reviewer instead of 404ing when their
answer ends it, visibility moved out of the editor to the document page
next to department sharing (both owner-only, like publishing), and the
title is edited in the save dialog with a ✨ suggestion on demand.
- [x] Seed data is built through the real process — per-section edit history,
publish events, drafts and review requests — so every history and diff
surface has something true to show.
## 17. Interface pass *(done)*
One question asked of every screen: does the important thing catch the eye, and
is everything else still findable? See `architecture.md` "One thing per screen".
- [x] Ranked actions: one labelled primary button per screen, the rest in a
labelled overflow menu (`lib/components/Menu.svelte`) instead of a row of
bare icons.
- [x] Badges mark exceptions only — an open question, an overdue check, a
draft, an archived or built-in document. "Published / public / yours" is
the normal case and says nothing on every card.
- [x] The document page is the document: one quiet metadata line above the
text, visibility and sharing behind the chip that states them
(`AccessPopover`), history collapsed to the recent entries, and the
leading `# Title` dropped where it repeats the title.
- [x] The document list is a search field and results; filters live behind one
toggle and stay visible as removable chips while they are on.
- [x] `/admin` became four tabs; `/people` groups by department and filters;
the profile lists what you wrote; the landing page focuses its input.
## 18. A template set people actually use *(done)*
- [x] The catalog is four starter blueprints and three additions, replacing
eight that read like a taxonomy: `notiz` (no skeleton at all), `prozess`,
`stoerung`, `person`, plus `anlage`, `entscheidung`, `projekt-debrief`.
Headings are the questions a colleague asks ("Wann das gilt / Schritt für
Schritt / Wenn es klemmt"), not shapes of a document ("Worum es geht /
Details").
- [x] `onboarding-basis` became `person`, written by the person themselves and
public — the profile page starts it. `offboarding`, `kundentermin` and
`lieferantenwissen` are gone.
- [x] The picker is a list, not a grid: a handful of choices read top to
bottom, where the description that decides between them is legible.
## 19. Decided *(open)*
Decisions taken on 2026-08-27. The three removals landed on 2026-08-30 (D25);
what is left needs either a screen size or an endpoint to work against.
- [x] **Remove the verification workflow.** `documents.verified_until`, the
`document_review_days` setting, `POST /{id}/reverify`, the `reverified`
event and every "Verifizierung fällig" surface are gone. It marked
documents and reminded nobody; a review request is the mechanism that
actually asks a person something.
- [x] **Remove tags.** `meta.tags`, the `?tag=` filter, the chunk-meta copy,
`metadata.tags` in the template schema and the fixture frontmatter are
gone; they had been write-only since M8.
- [x] **Remove `source_type` / the upload half.** Nothing can be uploaded, and
`upload` was the value nobody wrote. The idea comes back with an import
(below), which is when a provenance column earns its place again.
- [ ] **Mobile.** Desktop is the main target, but the app has to be usable on a
phone: sidebar, the chat split view, the CodeMirror editor and the new
overflow menus are unverified below `lg`.
- [ ] **Import, as the counterpart to the export.** A self-hosted system has to
be able to read its own ZIP back in (Markdown + YAML frontmatter):
restore after a move, and the first honest answer to "we already have
documentation somewhere else".
- [x] **Carry the chat a capture started from.** A draft started out of a chat
keeps its subject as `meta.context`; extending an EXISTING document out of
the same chat now does too (`PATCH /api/documents/{id} {conversation_id}`),
which was the one path that dropped it.
- [ ] **Show the search query the model actually wrote** — the topic-summary
retry rewrites the question on the low-confidence path, and the context
inspector still shows only what came back, not what was asked.
## Polish & known issues *(open)*
Smaller fixes and rough edges to pick up opportunistically.
- [x] **Refinement suggestions appear inline at the section you are editing**, as
a CodeMirror block widget right below it, instead of in a disconnected
right-hand pane (`WritingEditor.svelte`).
- [x] **The editor no longer loses work or leaves clutter.** A `beforeNavigate`
guard saves an edited-but-unsaved draft on the way out (no autosave, so the
review-before-save diff is preserved) and discards an abandoned, never-filled
template draft so empty drafts do not pile up. The document list also has its
own "capture" entry, not only the landing page.
- [x] The chat input **stays focused after the first message is sent**: the
`/chat` → `/chat/[id]` navigation swaps the page component, so `ChatView`
re-focuses the composer in `afterNavigate`.
- [x] **Destructive confirmations use the app's modal**, not the browser's
`confirm()`: `lib/components/ConfirmDialog.svelte` backs delete-user,
delete-department, delete-template, delete-document and delete-conversation.
- [x] **The streaming reply no longer snaps.** The assistant turn renders as
sanitized Markdown LIVE as it streams (with a blinking cursor) instead of
fading in plain text and re-rendering to Markdown on completion.
- [x] **Model LaTeX/math renders as math.** The sanitizing renderer runs
`marked-katex-extension` + KaTeX before DOMPurify (`$lib/markdown.ts`), with
KaTeX styles/fonts bundled locally (no CDN) — the local model emits real
`$...$` / `$$...$$` LaTeX, now typeset.
- [x] **The classic-search fallback answers a whole question.** Its terms are
ORed rather than ANDed (`_any_term_tsquery`), so a typed-out sentence still
finds something when no model is reachable; ranking stays `ts_rank_cd`. The
hybrid path keeps the AND, where `fts_match` has to mean "the words are
really in there". Open: the fallback still needs chunks, so a document
published while the embedding endpoint was down is not findable at all
(`notes.md`).
- [~] **Retrieval quality on statements.** Investigated: the hybrid path (query +
search) retrieves the same top document for a declarative statement as for the
question, because the German full-text component catches the keywords
regardless of phrasing — verified and now guarded by declarative-statement
cases in the golden query eval. The pure-cosine similarity path (capture
picker / grounding) is more phrasing-sensitive by design (it needs a
comparable distance for its threshold, `notes.md`); switching it to hybrid is
left open.
## 15. Hardening & release readiness *(open)*
Get to "a stranger can deploy from the README alone", with the EE boundary in
place and every quality gate green.
- [ ] EE boundary: `app/ee_hooks.py` (optional `pablan_ee` import + `register`),
`lib/ee/registry.ts` (empty frontend slot registry + Vite alias), and an
import-linter contract (core never imports `ee/`) wired into `make lint`.
- [ ] Minimal login throttling: in-process, per-user+IP backoff on failed logins
(no new dependency, no Redis).
- [ ] A Prometheus text-format exporter at `/metrics` for the metrics registry,
documented for customer IT.
- [x] An endpoint concurrency gate (`llm/gate.py`): one semaphore per base_url in
front of `chat_stream` / `chat_json` / `embed`, a bounded wait and a bounded
queue, both refusals reported as `llm_busy`, wait time and queue depth
metered, and a `queued` chat phase so a waiting turn says so.
- [ ] SDK-retry visibility: meter actual HTTP attempts, or document the tradeoff
(SDK-internal retries are invisible to our metrics) in the ops docs.
- [ ] Customer ops docs: backup/restore (pg_dump, rehearsed once), the upgrade
path (`alembic upgrade`), the `/metrics` protection note, and the parked
`idle_in_transaction_session_timeout` note for managed Postgres.
- [ ] The full e2e suite green: login, chat streaming, the writing editor, document
approval, permission boundaries.
- [ ] **First cloud-model eval validation of every prompt shipped so far** — the
writing-first refinement, grounding, topic-summary and query prompts have
only run against a local Gemma-class model. Add a `PABLAN_EVAL_CLOUD_*` role to
`.env.example`; the eval picks it up when set and skips it otherwise. Budget
time to fix prompt behaviour a 26B model tolerated.
- [ ] Two anti-scaffolding assertions in the refinement eval: across the eval
set the replies must not share one opening prefix, and at most a third may
share their first word. A model that scaffolds every answer the same way
reads as a form letter, and nothing catches that today.
- [ ] Clean-clone verification: quickstart (dev) and customer deploy (compose +
Caddy) from the README on a fresh checkout; a final pass so every `docs/*.md`
matches the implementation.
---
## Non-goals
Deliberately out of scope; revisit only with a concrete need.
- **EE insights implementation** — only extension-point stubs in the core.
- **OIDC / Entra ID SSO.**
- **Cross-department knowledge discovery** as a distinct feature (the capture-time
"matching documents" panel is a deliberate precursor; this is different from epic
13, which shares one document across departments).
- **CI pipeline setup.**
- **File uploads / attachments** — Markdown is the source of truth; binary storage,
previews and scanning add complexity without serving the capture → RAG loop.
- **Comments / in-tool collaboration** — knowledge flows through conversations with
Pablan, not discussion threads inside the tool.
- **Real-time / concurrent editing** — no use case; SSE streaming is all the
liveness the product needs.
- **Email notifications** — needs SMTP and deliverability support on customer
infrastructure; a candidate for EE later.
_(Document version history was previously a non-goal; it is now planned as epic 11.)_