pablan/docs/roadmap.md

Roadmap

The feature backlog for Pablan, organized by feature area rather than by delivery date. It doubles as a rebuild manual: follow the done epics top to bottom and you arrive at the system as it stands today; the open epics are the road ahead.

How to read it. Each epic is a feature area with a one-line intent and a checklist of stories — [x] shipped, [ ] not yet. Epics are ordered as a build sequence (foundation upward). File pointers name where a capability lives, so a rebuilder can find the pattern to follow. Implementation learnings that survive across features live in notes.md; the as-is contract lives in architecture.md, data-model.md and api-protocol.md.

---

1. Platform foundation (done)

The backend skeleton every other feature stands on.

• Config via pydantic-settings, all PABLAN_* env vars; per-role LLM config (base_url / api_key / model for chat, utility, embedding).
• Async SQLAlchemy 2.0 (typed) + Alembic migrations on PostgreSQL + pgvector; UUID primary keys and a shared timestamp mixin (models/base.py).
• The table set — see data-model.md and diagrams/data-model.svg.
• pytest harness against a real Postgres (docker); retrieval logic gets SQL-level tests.
• Seed script with a production guard; a shared German fixture corpus (tests/fixtures/corpus/ + golden_queries.yaml) that seeds and tests draw from.

2. Authentication & sessions (done)

Server-side sessions, no JWT, and an auth boundary that cannot leak one user's state to the next.

• argon2 password hashing; auth_sessions table; httpOnly pablan_session cookie (14-day TTL); {detail, code} error shape.
• login / logout / me; enumeration-resistant login.
• Session revocation (auth/sessions.py::revoke_user_sessions) — a password change logs out every other session.
• Full-page auth boundary: login and logout are document navigations, never client goto, so module-singleton client state (conversation list, chat state, resolved locale) is guaranteed dead at the boundary and cannot leak the previous user's data.

3. Frontend foundation (done)

The SvelteKit shell, the design system, the typed API contract, and i18n.

• Semantic design tokens as CSS variables mapped into Tailwind; the palette is swappable by editing tokens only; a WCAG-AA contrast gate (frontend/scripts/contrast-check.py) runs in make lint.
• A small styled component set on Bits UI (Button, Input, Select, Dialog, Card, Badge, Tabs, Tooltip, Popover, FormField) — no component-library deps.
• Typed API client: FastAPI OpenAPI → openapi-typescript → openapi-fetch; make types regenerates it after any backend API change.
• (app) route group + hooks.server.ts session validation (absolute backend URL, since a relative event.fetch never hits the vite proxy).
• The sanitizing Markdown renderer (components/Markdown.svelte, DOMPurify) — all model and document output renders only through it.
• Paraglide i18n: de source, en written in the same change; two lint rules (missing en, em/en dashes) in check-messages.py; the backend never renders UI strings — it returns {detail, code} and the frontend translates.
• Theme per device (localStorage, applied pre-paint by a boot script) and language per account (users.locale).

4. LLM gateway & background jobs (done)

One place that talks to models, and a Postgres-only work queue.

• llm/client.py is the ONLY code that calls LLM endpoints: chat_stream, chat_json (structured output via a Pydantic response_format), embed; roles chat / utility / embedding, each independently configured; extra_body passthrough (e.g. enable_thinking:false).
• In-app LLM config: bootstrap each field from env, then the DB, with per-field provenance ("from .env" / "changed here"); llm/test pings the three roles; model discovery is server-side.
• Postgres job queue (jobs table, FOR UPDATE SKIP LOCKED, backoff 30s·2^n over 5 attempts); the handler runs inside the open claim transaction; exactly one uvicorn worker (queue and metrics are per-process).
• Scheduled retention cleanup (reschedules itself); an in-process metrics registry.
• Content-safe logging: metadata only, never prompts / responses / user text; LLMError raised from None; an openai-SDK DEBUG-log guard (PABLAN_DEBUG_LOG_PROMPTS, never in production).
• A failing endpoint says why. LLMError.code classifies every failure once (llm_unreachable / llm_busy / llm_misconfigured / llm_failed); every surface reports it unchanged and the frontend phrases it (lib/api/errors.ts). The conversations router catches LLMError centrally, so a failure during retrieval (which embeds before the model is called) ends the turn with the reason instead of a truncated stream. A busy endpoint queues rather than refuses, so the chat also says so after 12s of silence.

5. Documents & the knowledge model (done)

Markdown documents with a permissioned lifecycle; the single source of truth.

• Documents stored as Markdown (content_md); chunks and embeddings are disposable derivatives, always re-indexable from the document.
• Lifecycle draft → published → archived; visibility (public / department / restricted).
• Publishing is the author's own action (POST /documents/{id}/publish), plus archive/republish. Whether the CONTENT is trusted is a review request instead, never a status (epic 16).
• The single permission filter (rag/permissions.py) shared by the API and retrieval, evaluated live against the table (never on stale chunk meta): readable adds a user's own drafts and the documents someone asked this user to check; searchable is published-only, so a draft never reaches another user or an LLM prompt.
• ZIP export of the readable knowledge base (Markdown + YAML frontmatter), permission-filtered, help pages excluded, stdlib zipfile/io only.
• Built-in help pages (help/*.md) imported into documents on start (is_builtin, read-only, not deletable).
• Admin CRUD guardrails (no self-delete / self-demote, 409 self_modification); GDPR surfaces (delete-own-conversation, the retention job).

6. RAG retrieval (done)

Permission-filtered hybrid search, German-first.

• Heading-aware chunking (the active-section boundary is shared with the editor); index / reindex jobs; bge-m3 multilingual embeddings (EMBEDDING_DIM=1024).
• Hybrid search: a permission CTE → pgvector top-20 + Postgres german FTS top-20 → Reciprocal Rank Fusion (k=60), all in one SQL statement.
• Permission filter before the LLM: there is no search without a user; an unauthorized chunk is structurally impossible to retrieve.
• Similarity path (similar_chunks / similar_documents): pure cosine, permission-filtered, threshold applied in Python after ORDER BY … LIMIT (a WHERE predicate fights the HNSW index); calibrated constants (no-answer ≥ 0.45, capture-context 0.45).
• Eval suite (make eval): recall@5 baseline against the fixture corpus and the no-answer threshold.
• The knowledge base stays searchable without any model. text_search() is the full-text half alone (the german tsvector GIN index, no embedding call), same permission CTE; query mode uses it when the embedding endpoint is gone, and when no model answers at all the turn ends as a plain hit list the user opens themselves, labelled as a search without a model.

7. Query chat (done)

Ask the knowledge base a question, get a streamed, cited answer.

• The Mode protocol + registry: a mode yields ModeEvents and the router converts them to SSE; Query is the only core mode (EE registers insight).
• Conversations + messages; the SSE streaming turn; a stop button (a client abort persists the partial message via asyncio.shield).
• Citations snapshotted into messages.meta (chunks are disposable) + the source side panel; one badge per document, not per chunk.
• No-answer handling lives in retrieval (the ≥0.45 threshold), not the prompt; a no-answer offers a "capture this knowledge" hand-off into the editor.
• Topic-summary retrieval fallback: on a low-confidence hit with prior context, re-search on an LLM topic summary of the conversation.
• A prompt-cache-friendly shape: a stable system prompt + history, with the volatile excerpts + question last.

8. Writing-first capture (done)

The centerpiece: employees write knowledge; the model matures the section at the cursor. (This replaced an earlier dialogue-driven approach — see notes.md History.)

• A CodeMirror 6 split editor at /documents/{id}/edit (also the edit surface for any existing document); the caret is visible (drawSelection + accent caret color).
• Section refinement POST /documents/{id}/refine (SSE): the server owns the active-section boundary (shared with the chunker), sends a FIM-style natural- language prompt, and streams a matured version of only that section into the right pane; enable_thinking:false keeps it ~1s; only delta.content is read.
• Retrieval-aware grounding: before refining, the server searches the permission-filtered knowledge base for related published material the author may read (_grounding via similar_chunks, the current doc and help pages excluded) and passes it to the prompt as a reference, not a fact source.
• A staleness guard and a post-accept cooldown on "Übernehmen"; a suggestion computed against a since-edited region is discarded rather than applied.
• A pre-save diff: @codemirror/merge unifiedMergeView (VSCode-style). Save sits bottom right where the writing ends, shows what changed, and asks again — save, cancel, or (for a draft) save and publish.
• Authoring templates (AuthoringTemplate): a Markdown skeleton + persona + per-section hints — declarative config, not a Mode. A shipped catalog plus a form template builder (POST /templates/build; the skeleton is derived from the section headings), with raw YAML behind an advanced toggle.
• Capture from a chat: carries the conversation as background context (meta.context) and shows a matching-documents picker (topic summary → similar_documents) to extend an existing document instead.
• A reward flow after publishing (a check animation → View / Have it checked), a title suggestion in the save dialog, and a streaming token fade-in.

9. Insight & growth (open)

Aggregate signals about the knowledge base — never per-user data.

• A growth surface, in a form that earns its place. The first version (a /dashboard page with bar charts off GET /documents/dashboard) was removed again: numbers nobody acts on are noise, and the charts said nothing the document list does not. Rebuild only around a question someone actually asks.
• GET /documents/stats — company-wide counts, read by the landing page's first-run guide.
• The document detail page shows created / last-changed timestamps.

10. Product content & docs (done)

The content that ships with the product, and the docs that describe it.

• In-product help (help/*.md, German) imported as read-only documents, kept current in the same change as the workflow it documents.
• The shipped template catalog: blueprints on disk, inert until an admin adds one; an added template becomes the customer's, fully editable.
• Claude-maintained developer docs (docs/), hand-authored SVG diagrams (docs/diagrams/, theme-aware, no toolchain), this roadmap, and notes.md.

---

11. Change history & review audit (done)

A git-like record of who changed what, when, and who checked it — including who confirmed the content, which used to be stored nowhere.

• A document_events table: document_id, actor_id, action (created / edited / published / archived / visibility_changed / review_requested / review_resolved), created_at, and — for content-bearing events — a snapshot of content_md, title, visibility and meta. Snapshots the Markdown (the source of truth), never chunks; actor_id is SET NULL and events CASCADE with the document.
• An event is recorded at every transition: the content branch of update_document, and create / publish / archive plus asking and answering a review in api/documents/ (helper authoring/history.py::record_event). review_resolved captures who confirmed the content as the event actor.
• GET /api/documents/{id}/history → the events newest-first (actor name, action, timestamp, has_snapshot), plus GET /api/documents/{id}/versions/{event_id} to fetch a past version's content for viewing or diffing. Both behind the document's own read gate.
• A history timeline on the document detail page (lib/documents/DocumentHistory.svelte) and a version diff reusing the unifiedMergeView (shared theme extracted to lib/documents/editorTheme.ts), with restore of a previous version.
• An entry shows its OWN change. A snapshot is written after its event, so diffing it against the live document made every row show the next row's edit (the newest edit showed nothing at all). GET /versions/{event_id} now also returns previous_content_md, the closest earlier snapshot, and the dialog diffs the pair.
• The review queue is discoverable from the landing page, deep-linking to /documents?review=1 (see epic 16 for the review model itself).

12. People & profiles (done)

A directory colleagues can browse, and ONE way to describe a person: a document like any other.

• A member-visible directory: GET /api/people (+ /api/people/{id}) → colleagues' {id, name, role, department} for any authenticated user (no password/email leakage; mirrors the permission-scoped shape of ReviewerCandidate), with a directory page (/people) and a person page (/people/{id}) wired into the nav.
• The profile page is the document about you. GET /api/account/document answers whether the caller wrote one (from the person blueprint) and what to start it from otherwise; the page opens it for editing or offers to create it. The blueprint id lives in the backend, so the frontend knows no ids.
• The self-written bio was removed again. A short profile blurb and a document about yourself are the same thing said twice, and only one of them is findable by the search, versioned and approvable. The users.bio column, both endpoints, the editor and its two prompts are gone.

13. Access control & sharing (done)

Let a document reach more than one department, and stop anyone from accidentally locking themselves out of it.

• Multi-department documents. documents.department_id stays the owning department; extra departments are shared through the existing doc_permissions join (the searchable_documents_filter EXISTS doc_permission branch already unions them in — this was management, not permission logic). PUT /api/documents/{id}/departments replaces the set; a detail-page multi-select (lib/documents/DepartmentSharing.svelte) manages it; and several departments are reflected in shared_departments, the ?department= filter and the export frontmatter.
• Self-lockout guard. Before committing a visibility or grant change, the proposed state is evaluated against the read rules (_guard_self_lockout, a Python mirror of readable_documents_filter). An author keeps access as author, so this only bites an admin editing a document they do not own: 409 self_lockout_warning unless resent with confirm_lockout (surfaced as an inline "save anyway" in the UI).
• Close the adjacent admin lockout vectors. delete_department now refuses to silently delete a department that still has members, owned documents or grants (409 department_in_use unless ?confirm=true) — its doc_permissions grants CASCADE away invisibly otherwise. The delete_user / update_user department-nulling is deliberate orphaning (documents and users survive authorless/departmentless, notes.md), not a silent lockout, and is documented in data-model.md.

14. Prompt & context transparency (done)

Give admins control over the model's instructions, and make what the model is working from visible to the user.

• Admin-editable system prompts. Every system prompt (query + no-sources, refinement persona/rules, grounding framing, topic summary, title) is editable on the admin UI, stored in prompt_settings and applied without a restart — mirroring the per-role LLM-settings pattern (app/prompts/overrides.py + api/admin/, code defaults in app/prompts/defaults.py), with a reset-to-default per prompt (lib/admin/PromptSettings.svelte).
• Always-inspectable working context. A "?" inspector in the chat (lib/chat/ContextInspector.svelte) shows every retrieved passage, marking which grounded the answer (used) versus which were too weak — so a no-answer is explained, not silent. The editor shows the grounding references a refinement drew from, sent as a grounding SSE frame.

16. Publish and review (done)

Separating "where does this document stand" from "is what it says right" — see decisions.md D24.

• Statuses are draft / published / archived; pending_approval is gone and publishing is the author's own one-click action (from the save dialog, the document page, or the landing page).
• A review_requests table: one question, addressed to one colleague, open until answered (resolved_at + resolved_by_id). Orthogonal to the status — it can sit on a draft or on a document published months ago.
• Being asked grants read AND edit until the answer (readable_documents_filter, can_edit), so a reviewer fixes a wrong number instead of filing a second question. Owner-only decisions (delete, sharing, handing out a request) stay with the author or an admin.
• An open question marks the document everywhere: list card, detail page, the document panel, and the sources under a chat answer (review_pending on every search hit, snapshotted with the citation).
• The landing page surfaces open work: your unpublished drafts (with publish) and the documents waiting for your check (lib/documents/OpenWork.svelte).
• Follow-through from use: AccessReason.review names the access a request grants, the document page thanks a reviewer instead of 404ing when their answer ends it, visibility moved out of the editor to the document page next to department sharing (both owner-only, like publishing), and the title is edited in the save dialog with a ✨ suggestion on demand.
• Seed data is built through the real process — per-section edit history, publish events, drafts and review requests — so every history and diff surface has something true to show.

17. Interface pass (done)

One question asked of every screen: does the important thing catch the eye, and is everything else still findable? See architecture.md "One thing per screen".

• Ranked actions: one labelled primary button per screen, the rest in a labelled overflow menu (lib/components/Menu.svelte) instead of a row of bare icons.
• Badges mark exceptions only — an open question, an overdue check, a draft, an archived or built-in document. "Published / public / yours" is the normal case and says nothing on every card.
• The document page is the document: one quiet metadata line above the text, visibility and sharing behind the chip that states them (AccessPopover), history collapsed to the recent entries, and the leading # Title dropped where it repeats the title.
• The document list is a search field and results; filters live behind one toggle and stay visible as removable chips while they are on.
• /admin became four tabs; /people groups by department and filters; the profile lists what you wrote; the landing page focuses its input.

18. A template set people actually use (done)

• The catalog is four starter blueprints and three additions, replacing eight that read like a taxonomy: notiz (no skeleton at all), prozess, stoerung, person, plus anlage, entscheidung, projekt-debrief. Headings are the questions a colleague asks ("Wann das gilt / Schritt für Schritt / Wenn es klemmt"), not shapes of a document ("Worum es geht / Details").
• onboarding-basis became person, written by the person themselves and public — the profile page starts it. offboarding, kundentermin and lieferantenwissen are gone.
• The picker is a list, not a grid: a handful of choices read top to bottom, where the description that decides between them is legible.

19. Decided (open)

Decisions taken on 2026-08-27. The three removals landed on 2026-08-30 (D25); what is left needs either a screen size or an endpoint to work against.

• Remove the verification workflow. documents.verified_until, the document_review_days setting, POST /{id}/reverify, the reverified event and every "Verifizierung fällig" surface are gone. It marked documents and reminded nobody; a review request is the mechanism that actually asks a person something.
• Remove tags. meta.tags, the ?tag= filter, the chunk-meta copy, metadata.tags in the template schema and the fixture frontmatter are gone; they had been write-only since M8.
• Remove source_type / the upload half. Nothing can be uploaded, and upload was the value nobody wrote. The idea comes back with an import (below), which is when a provenance column earns its place again.
• Mobile. Desktop is the main target, but the app has to be usable on a phone: sidebar, the chat split view, the CodeMirror editor and the new overflow menus are unverified below lg.
• Import, as the counterpart to the export. A self-hosted system has to be able to read its own ZIP back in (Markdown + YAML frontmatter): restore after a move, and the first honest answer to "we already have documentation somewhere else".
• Carry the chat a capture started from. A draft started out of a chat keeps its subject as meta.context; extending an EXISTING document out of the same chat now does too (PATCH /api/documents/{id} {conversation_id}), which was the one path that dropped it.
• Show the search query the model actually wrote — the topic-summary retry rewrites the question on the low-confidence path, and the context inspector still shows only what came back, not what was asked.

Polish & known issues (open)

Smaller fixes and rough edges to pick up opportunistically.

• Refinement suggestions appear inline at the section you are editing, as a CodeMirror block widget right below it, instead of in a disconnected right-hand pane (WritingEditor.svelte).

• The editor no longer loses work or leaves clutter. A beforeNavigate guard saves an edited-but-unsaved draft on the way out (no autosave, so the review-before-save diff is preserved) and discards an abandoned, never-filled template draft so empty drafts do not pile up. The document list also has its own "capture" entry, not only the landing page.

• The chat input stays focused after the first message is sent: the /chat → /chat/[id] navigation swaps the page component, so ChatView re-focuses the composer in afterNavigate.

• Destructive confirmations use the app's modal, not the browser's confirm(): lib/components/ConfirmDialog.svelte backs delete-user, delete-department, delete-template, delete-document and delete-conversation.

• The streaming reply no longer snaps. The assistant turn renders as sanitized Markdown LIVE as it streams (with a blinking cursor) instead of fading in plain text and re-rendering to Markdown on completion.

• Model LaTeX/math renders as math. The sanitizing renderer runs marked-katex-extension + KaTeX before DOMPurify ($lib/markdown.ts), with KaTeX styles/fonts bundled locally (no CDN) — the local model emits real $...$ / $$...$$ LaTeX, now typeset.

• The classic-search fallback answers a whole question. Its terms are ORed rather than ANDed (_any_term_tsquery), so a typed-out sentence still finds something when no model is reachable; ranking stays ts_rank_cd. The hybrid path keeps the AND, where fts_match has to mean "the words are really in there". Open: the fallback still needs chunks, so a document published while the embedding endpoint was down is not findable at all (notes.md).

• [~] Retrieval quality on statements. Investigated: the hybrid path (query + search) retrieves the same top document for a declarative statement as for the question, because the German full-text component catches the keywords regardless of phrasing — verified and now guarded by declarative-statement cases in the golden query eval. The pure-cosine similarity path (capture picker / grounding) is more phrasing-sensitive by design (it needs a comparable distance for its threshold, notes.md); switching it to hybrid is left open.

15. Hardening & release readiness (open)

Get to "a stranger can deploy from the README alone", with the EE boundary in place and every quality gate green.

• EE boundary: app/ee_hooks.py (optional pablan_ee import + register), lib/ee/registry.ts (empty frontend slot registry + Vite alias), and an import-linter contract (core never imports ee/) wired into make lint.
• Minimal login throttling: in-process, per-user+IP backoff on failed logins (no new dependency, no Redis).
• A Prometheus text-format exporter at /metrics for the metrics registry, documented for customer IT.
• An endpoint concurrency gate (llm/gate.py): one semaphore per base_url in front of chat_stream / chat_json / embed, a bounded wait and a bounded queue, both refusals reported as llm_busy, wait time and queue depth metered, and a queued chat phase so a waiting turn says so.
• SDK-retry visibility: meter actual HTTP attempts, or document the tradeoff (SDK-internal retries are invisible to our metrics) in the ops docs.
• Customer ops docs: backup/restore (pg_dump, rehearsed once), the upgrade path (alembic upgrade), the /metrics protection note, and the parked idle_in_transaction_session_timeout note for managed Postgres.
• The full e2e suite green: login, chat streaming, the writing editor, document approval, permission boundaries.
• First cloud-model eval validation of every prompt shipped so far — the writing-first refinement, grounding, topic-summary and query prompts have only run against a local Gemma-class model. Add a PABLAN_EVAL_CLOUD_* role to .env.example; the eval picks it up when set and skips it otherwise. Budget time to fix prompt behaviour a 26B model tolerated.
• Two anti-scaffolding assertions in the refinement eval: across the eval set the replies must not share one opening prefix, and at most a third may share their first word. A model that scaffolds every answer the same way reads as a form letter, and nothing catches that today.
• Clean-clone verification: quickstart (dev) and customer deploy (compose + Caddy) from the README on a fresh checkout; a final pass so every docs/*.md matches the implementation.

---

Non-goals

Deliberately out of scope; revisit only with a concrete need.

• EE insights implementation — only extension-point stubs in the core.
• OIDC / Entra ID SSO.
• Cross-department knowledge discovery as a distinct feature (the capture-time "matching documents" panel is a deliberate precursor; this is different from epic 13, which shares one document across departments).
• CI pipeline setup.
• File uploads / attachments — Markdown is the source of truth; binary storage, previews and scanning add complexity without serving the capture → RAG loop.
• Comments / in-tool collaboration — knowledge flows through conversations with Pablan, not discussion threads inside the tool.
• Real-time / concurrent editing — no use case; SSE streaming is all the liveness the product needs.
• Email notifications — needs SMTP and deliverability support on customer infrastructure; a candidate for EE later.

(Document version history was previously a non-goal; it is now planned as epic 11.)