Pablan, as it stands
Self-hosted knowledge management for SMEs: a split-screen Markdown editor whose sections an LLM refines while you write, and RAG question answering over the documents that result. FastAPI + Postgres/pgvector on the back, SvelteKit on the front, everything OpenAI-compatible and self-hostable. Squashed into a single commit; the development history stays local. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CA43ZJda8Rbp2hKXNy8f6b
This commit is contained in:
ProfessorNova
co-authored by
Claude Opus 5
Claude Opus 5
parent
68d3a43191
commit
784b76baf7
+322
@@ -0,0 +1,322 @@
|
||||
# Implementation notes
|
||||
|
||||
Durable engineering learnings — decisions, calibrations and gotchas that are
|
||||
still true of the system as it stands. Grouped by theme rather than by when they
|
||||
were learned. The [roadmap](roadmap.md) says *what* exists; this says *why it is
|
||||
the way it is* and *what will bite you*. For formal decisions see
|
||||
[decisions.md](decisions.md); for the as-is contract see
|
||||
[architecture.md](architecture.md) and [data-model.md](data-model.md).
|
||||
|
||||
## Retrieval & embeddings
|
||||
|
||||
- **One permission filter, evaluated live.** `rag/permissions.py` is the single
|
||||
source both the API and retrieval use, and it runs against the current tables —
|
||||
never against the denormalized visibility/department copy stored in chunk meta
|
||||
(that copy is display-only and can lag a reindex). This is what makes "a chunk the
|
||||
user may not read is structurally unreachable" true rather than aspirational.
|
||||
- **Authors vs retrieval.** An author sees their own drafts through the API
|
||||
(`readable_documents_filter`), but retrieval is published-only
|
||||
(`searchable_documents_filter`), so a draft never reaches an LLM prompt. Admins
|
||||
get **no** read-everything bypass — an admin only sees what the normal rules grant.
|
||||
- **Hybrid search is one SQL statement.** pgvector top-20 + German FTS top-20 fused
|
||||
with Reciprocal Rank Fusion, `RRF_K=60`. Keeping it one statement is what lets the
|
||||
permission CTE gate everything before ranking.
|
||||
- **Thresholds are calibrated against bge-m3 on the fixture corpus, and must be
|
||||
re-measured if the embedding model changes.** No-answer fires at vector distance
|
||||
≥ 0.45 (matched hits land ~0.34–0.45, unrelated ~0.50+). The similarity path
|
||||
(`similar_chunks`) uses `max_distance` 0.45 for capture-context.
|
||||
- **A chunk is embedded WITH its heading path** (`rag/indexing.embedding_text`).
|
||||
A section reads "Solldruck 180 bar" and never repeats which machine it belongs
|
||||
to, so on its own it is unreachable by the name the asker uses. Measured on the
|
||||
dev corpus: "Welcher Solldruck gilt für die Hydraulikpresse?" went from a
|
||||
no-answer (best distance 0.492, above the 0.45 gate) to a confident hit (0.375,
|
||||
and the section that literally answers it at rank 2). The genuine no-answers
|
||||
moved too little to matter (Deutschlandticket 0.503 → 0.488), so the gate keeps
|
||||
separating them. What is STORED as `content` stays the raw section: the path is
|
||||
context for the vector, not part of the document, and the excerpt a reader sees
|
||||
must not grow a header. Tests that search for "the exact chunk text" have to
|
||||
build it through `embedding_text` or they land at distance ~0.96.
|
||||
- **The chunk `tsv` covers the heading path too** (generated column over
|
||||
`content || ' ' || meta->>'heading_path'`). Same reason on the keyword side:
|
||||
"Solldruck Hydraulikpresse" matched no chunk before, because only the document
|
||||
title carries the machine name. Note what it does NOT fix: `websearch_to_tsquery`
|
||||
is AND over every surviving lexeme, and German stopword lists keep verbs, so
|
||||
"Welcher Solldruck **gilt** für die Hydraulikpresse?" still has no keyword match.
|
||||
Treat `fts_match` as "the keywords really are in there", not as "this was a
|
||||
keyword question".
|
||||
- **Calibrate on what the engine renders, not on what a human writes.** Real capture
|
||||
drafts are compressed notes and sit further from their source than a hand-written
|
||||
paraphrase, so thresholds tuned on paraphrases miss real matches.
|
||||
- **`rag/similarity` is deliberately not the hybrid path.** RRF produces a fusion
|
||||
rank, not a similarity, and its vector distance is null for FTS-only hits — a
|
||||
threshold needs a comparable number, so similarity is a pure-cosine query with the
|
||||
threshold applied in Python *after* `ORDER BY … LIMIT` (a distance predicate in
|
||||
WHERE fights the HNSW index).
|
||||
- **HNSW post-filtering can under-fill.** For a heavily permission-filtered user, the
|
||||
fixed candidate fetch can return fewer rows than expected; keep candidate counts
|
||||
generous.
|
||||
- **Hybrid retrieval is robust to statement-vs-question phrasing; the pure-cosine path
|
||||
is not.** A declarative statement ("Die Zentralschmierung … muss aufgefüllt werden.")
|
||||
retrieves the same top document as the equivalent question, because the German FTS
|
||||
component matches the keywords regardless of phrasing — guarded by declarative-
|
||||
statement cases in `golden_queries.yaml`. The pure-cosine similarity path
|
||||
(`similar_documents`/`similar_chunks`, used by the capture picker and refinement
|
||||
grounding) has no FTS to lean on and is therefore more sensitive to how a thought is
|
||||
phrased. It stays pure-cosine on purpose (a comparable distance for its threshold),
|
||||
so a phrasing-sensitive picker is a known trade-off, not a bug.
|
||||
|
||||
## Permissions & auth
|
||||
|
||||
- **FK delete semantics encode the product.** Authored documents survive their
|
||||
author (`ON DELETE SET NULL`) — "an employee leaving is exactly the case the
|
||||
product exists for". The same holds for a document's department.
|
||||
- **The auth boundary is a full-page navigation.** Login and logout do a real
|
||||
document load (`window.location.assign`), never a client `goto`. Module-level runes
|
||||
singletons (conversation list, chat state, resolved locale) live for one page load,
|
||||
so reloading at the boundary guarantees the next user starts clean. A client
|
||||
navigation kept the previous user's conversation titles on screen — an information
|
||||
disclosure. Logout also clears the `PARAGLIDE_LOCALE` cookie so a shared terminal
|
||||
does not pin one user's language for the next.
|
||||
- **A private helper imported across modules is a design smell, not a shortcut.**
|
||||
Four of them had grown (`_readable_document`, `_role_config`, `_excerpt`,
|
||||
`_HEADING_RE`): each marked a place where one module had quietly become part of
|
||||
another's interface without saying so. Splitting the big API modules surfaced
|
||||
all four at once. When a second module wants an underscore name, that name is
|
||||
the interface and should be renamed, not imported.
|
||||
- **A test double that patches by module name silently rots.** The fake-embedding
|
||||
fixture patched `indexing` and `retrieval`; the moment the similarity path moved
|
||||
into its own module, an "offline" test would have called a real endpoint. A
|
||||
fixture that names the modules it patches has to be updated with every split, so
|
||||
it says why in a comment right there.
|
||||
- **A history snapshot is written AFTER its event.** So the content stored on an
|
||||
`edited` event is the state that edit produced, not the one it replaced. Diffing
|
||||
a snapshot against the live document therefore shows the change of the NEXT
|
||||
entry, and the newest entry shows nothing at all. What a timeline entry has to
|
||||
answer is "what did this one do", so the version endpoint returns the pair
|
||||
(`content_md` + `previous_content_md`) and the dialog diffs those.
|
||||
- **The product degrades to a search engine, not to an error page.** Postgres is
|
||||
already an inverted index (`chunks.tsv`, a stored `to_tsvector('german', …)`
|
||||
column with a GIN index), so "no model reachable" costs the generated answer and
|
||||
the semantic half of retrieval, nothing else. The turn ends with the full-text
|
||||
hits as a list, marked as a search without a model, and is persisted like a
|
||||
normal reply so a reload replays it. Worth remembering when adding any other
|
||||
model-dependent surface: ask what remains without the model.
|
||||
- **Classify an endpoint failure once, in `LLMError.code`.** "Not reachable" and
|
||||
"busy, try again" need different words to the user, and a timeout means the
|
||||
second, not the first: the endpoint took the request and never came back.
|
||||
Retrieval embeds before the model is ever called, so the failure often happens
|
||||
in `search()`, not in `chat_stream` — the conversations router catches
|
||||
`LLMError` centrally rather than every mode remembering to.
|
||||
- **Enumeration-resistant login.** An unknown email still runs a constant-time dummy
|
||||
verify, so timing does not reveal which accounts exist.
|
||||
- **Admin CRUD guardrails.** An admin cannot delete themselves or change their own
|
||||
role (409 `self_modification`); a password change or reset revokes all of that
|
||||
user's other sessions.
|
||||
- **Sharing is grants, not new permission logic.** Multi-department sharing plugs
|
||||
straight into the `EXISTS doc_permission` branch that was already in
|
||||
`searchable_documents_filter` — `PUT /documents/{id}/departments` just manages the
|
||||
rows. No reindex: grants are evaluated live against the table, not denormalized into
|
||||
chunk meta. Any per-department count therefore measures REACH, not a partition:
|
||||
a shared document counts under every department it reaches, so such counts can
|
||||
exceed the distinct document count.
|
||||
- **Self-lockout only bites admins.** `readable_documents_filter` grants the author
|
||||
read access unconditionally, so a normal author can never edit themselves out of
|
||||
their own document; the guard (`_guard_self_lockout`, a Python mirror of the read
|
||||
rules) only ever fires for an admin changing a document they do not own, who must
|
||||
resend with `confirm_lockout`. When mirroring a SQL filter in Python for a
|
||||
pre-commit check, keep the two in the same file so they cannot drift.
|
||||
- **A `bool = False` Pydantic field serializes as *required* in the generated TS
|
||||
client** (openapi-typescript), which breaks every existing caller that omitted it.
|
||||
Use `bool | None = None` (like the other optional fields) and `bool(...)` at the use
|
||||
site — then it stays optional in `schema.d.ts`.
|
||||
- **The audit trail is where "who checked it" lives.** `document_events`
|
||||
(`authoring/history.py::record_event`) appends who-did-what at every transition;
|
||||
`review_resolved` writes the answering colleague as the event actor — once a
|
||||
request is answered, the `review_requests` row only says that nothing is open
|
||||
any more, so the trail is the only place that identity survives. Content-bearing events
|
||||
(`created`, `edited`) snapshot the Markdown, never the chunks, so per-event
|
||||
snapshots of the *resulting* state chain into a version diff. Gotcha: a freshly
|
||||
created document must be `flush()`ed before recording its `created` event — the
|
||||
UUID PK default lands at flush, not at object construction, so `document.id` is
|
||||
None until then. History is served through the document's own read gate, so it
|
||||
cannot leak to a user who may not read the document.
|
||||
|
||||
## LLM & prompts
|
||||
|
||||
- **`chat_template_kwargs.enable_thinking = false` is the big latency win.** On a
|
||||
reasoning-capable local model it drops section refinement from ~10s to ~1s with no
|
||||
quality loss, because refinement is a mechanical rewrite. Passed through
|
||||
`extra_body`; ignored by endpoints that do not support it.
|
||||
- **Prompt shape for caching.** Keep the system prompt byte-identical and the history
|
||||
stable, and put the volatile part (retrieval excerpts, the grounding block) last —
|
||||
the endpoint then reuses the cached prefix and only reprocesses what changed.
|
||||
- **Content never reaches logs.** `LLMError` is raised `from None` so a model
|
||||
response cannot ride out in a chained traceback; SQL bind params are stripped; the
|
||||
openai SDK's DEBUG body-logging is capped unless `PABLAN_DEBUG_LOG_PROMPTS` is set
|
||||
(never in production). This is canary-tested.
|
||||
- **One SDK-level retry** covers the occasional llama.cpp `APIConnectionError`;
|
||||
`chat_json` additionally validates and retries once on a schema mismatch.
|
||||
- **Prompts are admin-editable, defaulting to code.** Every system prompt renders
|
||||
through `app/prompts/overrides.py::get_prompt(key)`, which returns a DB override
|
||||
(`prompt_settings`) or the code default (`app/prompts/defaults.py`). This mirrors the
|
||||
LLM-settings cache but is simpler: the reset target is the code default (prompts have
|
||||
no `.env`), and there is no bootstrap — a missing row just means "use the default".
|
||||
Keep the module-cache clear in the test teardown, or an override leaks into the next
|
||||
test. Prompt-cache byte-identity (D22) still holds because a value changes only on an
|
||||
admin write.
|
||||
- **The context inspector needs the FULL retrieval, so a no-answer keeps its results.**
|
||||
Query mode used to discard the low-confidence hits (`results = []`); now it keeps them,
|
||||
sends them in the `sources` event marked `used=False`, and passes only `used` ones to
|
||||
the prompt. That is what lets the "?" inspector explain a no-answer ("these were close
|
||||
but too weak") instead of showing nothing. The cited-source badges filter to `used`.
|
||||
- **The shipped system prompt was picked by measurement, not by taste.** A bench
|
||||
over 15 objective checks (grounding phase, the load-bearing fact, an honest
|
||||
"not documented", answer language) on the dev corpus: the long rules-list
|
||||
version scored 42/45 at 486 chars mean answer, a four-line version 45/45 at 279
|
||||
chars. Everything the list spelled out, the model already did; what it was
|
||||
missing was the one instruction that carries its weight — *say when something
|
||||
is not documented* — without which the three no-answer cases all produced
|
||||
plausible filler. Shorter prompt, shorter answers, better score. The bench lives
|
||||
in the session scratch, not in the repo: it needs a seeded instance and a live
|
||||
endpoint, which is what `make eval` is for — port a case there when it earns a
|
||||
permanent floor.
|
||||
- **Asking about the product is a retrieval question, not a special case.** The
|
||||
`help/` pages are indexed like anything else, so "how do I share a document?"
|
||||
is answered through the same path — and `tests/evals/test_self_knowledge_eval.py`
|
||||
measures it (10/10, and asserts the help pages do NOT outrank company documents
|
||||
for a company question, which is the failure mode that would matter).
|
||||
- **Cloud validation is still owed.** Every shipped prompt (query, refinement,
|
||||
grounding, topic summary, title) has only run against a local Gemma-class model.
|
||||
The first cloud-model eval pass is roadmap epic 15 — expect to fix behaviour a 26B
|
||||
model tolerated.
|
||||
|
||||
## Jobs & runtime
|
||||
|
||||
- **The claim transaction stays open for the whole handler.** Claiming a job
|
||||
(`FOR UPDATE SKIP LOCKED`) and completing it happen in one transaction, so a crash
|
||||
mid-handler rolls the claim back and the job is retried — at the cost of one pinned
|
||||
connection per in-flight job.
|
||||
- **Backoff** is 30s·2^n over 5 attempts; failure bookkeeping runs in its own
|
||||
transaction. Retention cleanup reschedules itself daily.
|
||||
- **The endpoint gate is per base_url, not per role** (`llm/gate.py`). chat and
|
||||
utility point at the same server in the shipped config, so a per-role limit
|
||||
would let one server take twice its slots. The waiting is bounded twice (a wait
|
||||
timeout and a queue-length cap) because an unbounded wait in front of a
|
||||
saturated endpoint is the same failure as no gate at all, just quieter. Both
|
||||
refusals reuse `llm_busy`, so the UI has one sentence rather than two. The
|
||||
background job worker is strictly sequential, so a big reindex contributes ONE
|
||||
concurrent embedding call, not a burst.
|
||||
- **Exactly one uvicorn worker.** The job loop, the metrics registry and the LLM
|
||||
client cache are per-process; a second worker would double-run the queue and split
|
||||
the metrics. The compose file pins `--workers 1`.
|
||||
|
||||
## Frontend & i18n
|
||||
|
||||
- **Citations are snapshotted into `messages.meta`.** Chunks are disposable and get
|
||||
reindexed, so an answer freezes the exact passages it cited at answer time; the
|
||||
excerpt is re-cleaned to plain prose on the way out (cleaning is idempotent).
|
||||
- **SSE via fetch + ReadableStream, not `EventSource`.** The turn is a POST that must
|
||||
be abortable (the stop button), which `EventSource` cannot do. A client abort still
|
||||
persists the partial assistant message (`asyncio.shield` on the backend).
|
||||
- **Theme per device, language per account, both applied pre-paint.** Theme lives in
|
||||
localStorage and is stamped onto `<html>` by a boot script before first paint;
|
||||
language comes from `users.locale` and is baked into the first SSR byte — otherwise
|
||||
the page flashes the wrong theme/language on every load.
|
||||
- **Paraglide pitfalls.** Source is **de** (informal "du"), en written in the same
|
||||
change; two lint rules fail the build (a missing `en` key, an em/en dash). There is
|
||||
no `url` locale strategy (locale is not in the path). `Intl` formatters must be
|
||||
passed the resolved locale explicitly or they default to the server's. ICU plurals
|
||||
have their own syntax trap. Recompiling messages under a running dev server can
|
||||
wedge it — restart after adding keys.
|
||||
- **`hooks.server.ts` needs an absolute backend URL.** A relative `event.fetch` never
|
||||
reaches the vite proxy, which silently breaks server-side auth calls.
|
||||
- **The chat renders Markdown LIVE while streaming.** Rendering plain-text tokens and
|
||||
then re-rendering to Markdown on completion read as a snap; the turn now goes through
|
||||
the sanitizing `Markdown` component every token, so the formatted output is unveiled
|
||||
progressively. Partial constructs (an unclosed `**` or `$`) render as literal text
|
||||
until closed — a small local flicker, far better than the whole-reply snap.
|
||||
- **The local model emits real LaTeX**, inline `$...$` and display `$$...$$` with
|
||||
`\frac`, `\sqrt`, superscripts. `$lib/markdown.ts` runs `marked-katex-extension` +
|
||||
KaTeX BEFORE DOMPurify, and the KaTeX CSS/fonts are bundled locally (imported in the
|
||||
root layout — no CDN, the artifacts self-host). DOMPurify keeps `<span class style>`
|
||||
by default, so the typeset math survives sanitizing; `throwOnError: false` degrades a
|
||||
malformed formula to text instead of throwing mid-answer.
|
||||
- **Prefer `SvelteSet` over a plain `Set`** for reactive membership state — eslint
|
||||
(`svelte/prefer-svelte-reactivity`) enforces it, and mutating it (`add`/`delete`)
|
||||
is reactive without reassigning.
|
||||
- **The refine suggestion is an inline CodeMirror block widget, not a side pane.**
|
||||
It appears right under the section being edited (`WritingEditor.svelte`): a
|
||||
`WidgetType` wrapping a persistent DOM element (built once, `eq()` compares by
|
||||
element identity so CodeMirror reuses it) placed by a `StateField` +
|
||||
`StateEffect`. Tokens stream by mutating that DOM directly; CodeMirror does not
|
||||
learn a widget's height changed on its own, so call `view.requestMeasure()`
|
||||
(rAF-throttled) after each update or the lines below overlap. `ignoreEvent()`
|
||||
returns true so the accept/dismiss buttons handle their own clicks, and their
|
||||
`mousedown` preventDefault keeps the editor from blurring first.
|
||||
- **The editor is not continuously autosaved** (that would empty the
|
||||
diff you are shown before saving). A `beforeNavigate` guard instead discards an
|
||||
abandoned, never-filled template draft (only headings/blank lines) and saves an
|
||||
edited-but-unsaved draft on the way out — fixing both the data-loss trap and the
|
||||
empty-draft clutter without touching the save flow. Skip it once the document
|
||||
was published from the editor, or it would re-save over the publish.
|
||||
|
||||
- **Seeded documents are built through the real process, not inserted finished**
|
||||
(`app/seed.py`). Each one gets the history it would have had: the empty draft,
|
||||
one `edited` snapshot per section as the author works down the page, the
|
||||
publish, and the questions colleagues asked afterwards — plus a few documents
|
||||
left as drafts and one archived, so every screen has something to show. Without
|
||||
it the history view, the version diffs and "recently changed" are empty or lie
|
||||
in every dev stack. Order the corpus deliberately (the overdue document has to
|
||||
be an old one, drafts have to be the newest), and give `document_events` and
|
||||
`review_requests` explicit timestamps: `server_default=now()` would stamp the
|
||||
whole life of a document at seed time.
|
||||
- **Bits UI keeps inactive tab panels mounted**, just hidden. On a page of
|
||||
independent panels (`/admin`) that means all four fetch their data on
|
||||
arrival, and controls nobody can see sit in the DOM where a click never
|
||||
reaches them (Playwright finds the element, then times out waiting for it to
|
||||
be visible). `lib/components/Tabs.svelte` renders only the open panel.
|
||||
- **A grant that ends needs its own access reason.** Answering a review request
|
||||
on someone else's draft takes the access away again — correct, and it dropped
|
||||
the reviewer on a 404 for doing what they were asked. `AccessReason.review`
|
||||
names the case ("the only thing letting you in is the request"), so the UI can
|
||||
say thank you instead. It mirrors `readable_documents_filter`, where the
|
||||
visibility rules only apply to a PUBLISHED document: on a draft, `public`
|
||||
never explains access, the request does.
|
||||
- **The classic-search fallback ORs its terms; the hybrid path ANDs them**
|
||||
(`rag/retrieval._any_term_tsquery`). `websearch_to_tsquery` builds an AND over
|
||||
every lexeme, which is fine when the vector half carries the recall — but alone
|
||||
it answers a typed-out question ("Wie läuft die Qualitätsprüfung im
|
||||
Wareneingang?") with nothing at all, because no single chunk contains every
|
||||
word. Rewriting the operators between the groups (`' & '` → `' | '` on the
|
||||
tsquery text) keeps quoted phrases and exclusions intact and leaves the ranking
|
||||
to `ts_rank_cd`. `NULLIF` the result: a query of nothing but stop words
|
||||
rewrites to an empty string, and `to_tsquery('')` is a syntax error rather than
|
||||
an empty match.
|
||||
- **The fallback needs chunks, so it needs a past embedding run.** `chunks.tsv`
|
||||
is generated from `chunks.content`, and a chunk row is only written when
|
||||
`reindex_document` succeeded — which calls the embedding endpoint. A document
|
||||
published while the embedding endpoint is down has no chunks at all, so the
|
||||
"classic search" fallback cannot find it either. It keeps an already-indexed
|
||||
knowledge base usable without a model; it does not index without one.
|
||||
- **The e2e suite pins English with Accept-Language, which an ACCOUNT language
|
||||
outranks** — that is the product rule (a language a person picked follows
|
||||
them to every device), so a dev stack where somebody switched the admin to
|
||||
German failed every spec that reads a label. The login helper hands the
|
||||
account back to "follow the browser"; specs that run as that user select
|
||||
tabs by testid rather than by label.
|
||||
- **`npm run check` compiles Paraglide with the CLI**, which used to write the
|
||||
DEFAULT strategy chain over the one `vite.config.ts` configures — leaving the
|
||||
dev server (and the e2e suite, which pins `Accept-Language`) resolving German
|
||||
for everyone until the next vite restart. The `messages` script now passes the
|
||||
same `--strategy` flags. The other half of the trap stands: recompiling while
|
||||
`vite dev` runs breaks its module graph until restart, which is why `make lint`
|
||||
skips the recompile when :5173 is listening.
|
||||
|
||||
## History
|
||||
|
||||
A dialogue-driven "interview" capture engine (a state machine that interviewed the
|
||||
employee and assembled a document from the answers) was built first, then removed
|
||||
and replaced by the writing-first editor that exists today. Nothing from it remains
|
||||
in the code path; the roadmap describes building the current system directly. The
|
||||
only ideas that carried across are generic: template import/upsert by config id, and
|
||||
the `pyyaml` dependency.
|
||||
Reference in New Issue
Block a user