ProfessorNova/pablan
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Pablan, as it stands

Browse Source 
Self-hosted knowledge management for SMEs: a split-screen Markdown editor
whose sections an LLM refines while you write, and RAG question answering
over the documents that result. FastAPI + Postgres/pgvector on the back,
SvelteKit on the front, everything OpenAI-compatible and self-hostable.

Squashed into a single commit; the development history stays local.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CA43ZJda8Rbp2hKXNy8f6b
This commit is contained in:
ProfessorNova
2026-09-04 08:36:17 +02:00
co-authored by Claude Opus 5
commit 97dbff309c
346 changed files with 43430 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/diagrams/retrieval-sequence.svg
+157
View File
@@ -0,0 +1,157 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1040 1356" width="1040" height="1356" role="img" aria-label="Retrieval: hybrid search, and the similarity path next to it">
<title>Retrieval: hybrid search, and the similarity path next to it</title>
<style>
:root {
--bg: #fbfaf9;
--surface: #ffffff;
--sunken: #f2f0ed;
--ink: #1a1815;
--muted: #63605a;
--line: #c5bfb6;
--line-soft: #e5e1db;
--accent: #b8770a;
--secondary: #b04a00;
--note: #faf3e6;
}
@media (prefers-color-scheme: dark) {
:root {
--bg: #0f0e0d;
--surface: #1a1817;
--sunken: #232120;
--ink: #efece8;
--muted: #a49d94;
--line: #4a443d;
--line-soft: #332f2b;
--accent: #e6b422;
--secondary: #e08030;
--note: #241f16;
}
}
text { font-family: ui-sans-serif, system-ui, -apple-system, 'Segoe UI', Roboto, sans-serif; fill: var(--ink); }
.t-title { font-size: 15px; font-weight: 650; }
.t-sub { font-size: 11.5px; fill: var(--muted); }
.t-node { font-size: 12.5px; font-weight: 600; }
.t-body { font-size: 11.5px; }
.t-muted { font-size: 11px; fill: var(--muted); }
.t-edge { font-size: 10.5px; fill: var(--muted); }
.t-mono { font-family: ui-monospace, SFMono-Regular, Menlo, Consolas, monospace; font-size: 11px; }
.box { fill: var(--surface); stroke: var(--line); stroke-width: 1.2; }
.box-sunken { fill: var(--sunken); stroke: var(--line-soft); stroke-width: 1; }
.box-accent { fill: var(--surface); stroke: var(--accent); stroke-width: 1.6; }
.note { fill: var(--note); stroke: var(--line-soft); stroke-width: 1; }
.group { fill: none; stroke: var(--line-soft); stroke-width: 1.2;
stroke-dasharray: 5 4; }
.lifeline { stroke: var(--line); stroke-width: 1; stroke-dasharray: 4 4; }
.edge { stroke: var(--line); stroke-width: 1.3; fill: none; }
.edge-accent { stroke: var(--secondary); stroke-width: 1.5; fill: none; }
.edge-soft { stroke: var(--line); stroke-width: 1.1; fill: none;
stroke-dasharray: 5 4; }
.edge-soft-accent { stroke: var(--secondary); stroke-width: 1.3; fill: none;
stroke-dasharray: 5 4; }
.planned { stroke-dasharray: 6 4; opacity: 0.62; }
</style>
<defs>
<marker id="arrow" viewBox="0 0 10 10" refX="9" refY="5"
markerWidth="7" markerHeight="7" orient="auto-start-reverse">
<path d="M 0 1 L 9 5 L 0 9 z" fill="var(--line)"/>
</marker>
<marker id="arrow-accent" viewBox="0 0 10 10" refX="9" refY="5"
markerWidth="7" markerHeight="7" orient="auto-start-reverse">
<path d="M 0 1 L 9 5 L 0 9 z" fill="var(--secondary)"/>
</marker>
<marker id="arrow-open" viewBox="0 0 10 10" refX="9" refY="5"
markerWidth="8" markerHeight="8" orient="auto-start-reverse">
<path d="M 0 1 L 9 5 L 0 9" fill="none" stroke="var(--line)"
stroke-width="1.3"/>
</marker>
</defs>
<rect width="1040" height="1356" fill="var(--bg)"/>
<text x="28" y="30" class="t-title" text-anchor="start">Retrieval: hybrid search, and the similarity path next to it</text>
<text x="28" y="50" class="t-sub" text-anchor="start">The permission CTE is part of the one statement. There is no search without a user.</text>
<rect x="75.3" y="74" width="151.4" height="30" rx="8" class="box" />
<text x="151" y="93" class="t-node" text-anchor="middle">Caller (mode / API)</text>
<line x1="151" y1="104" x2="151" y2="1338" class="lifeline"/>
<rect x="298.2" y="74" width="197.6" height="30" rx="8" class="box" />
<text x="397" y="93" class="t-node" text-anchor="middle">rag/retrieval + similarity</text>
<line x1="397" y1="104" x2="397" y2="1338" class="lifeline"/>
<rect x="577.2" y="74" width="131.6" height="30" rx="8" class="box" />
<text x="643" y="93" class="t-node" text-anchor="middle">llm/client.embed</text>
<line x1="643" y1="104" x2="643" y2="1338" class="lifeline"/>
<rect x="841" y="74" width="96" height="30" rx="8" class="box" />
<text x="889" y="93" class="t-node" text-anchor="middle">Postgres</text>
<line x1="889" y1="104" x2="889" y2="1338" class="lifeline"/>
<line x1="28" y1="138" x2="1012" y2="138" class="edge-soft"/>
<rect x="417.825" y="127" width="204.35" height="22" rx="11" class="box-sunken" />
<text x="520" y="142" class="t-muted" text-anchor="middle">search(): hybrid, the default</text>
<text x="274" y="170" class="t-edge" text-anchor="middle">search(db, query, user=..., top_k=5)</text>
<line x1="151" y1="187" x2="397" y2="187" class="edge" marker-end="url(#arrow)"/>
<text x="520" y="209" class="t-edge" text-anchor="middle">embed([query])</text>
<line x1="397" y1="226" x2="643" y2="226" class="edge" marker-end="url(#arrow)"/>
<text x="520" y="248" class="t-edge" text-anchor="middle">query vector (1024, bge-m3)</text>
<line x1="643" y1="265" x2="397" y2="265" class="edge-soft" marker-end="url(#arrow-open)"/>
<text x="643" y="287" class="t-edge" text-anchor="middle">ONE statement:</text>
<line x1="397" y1="304" x2="889" y2="304" class="edge-accent" marker-end="url(#arrow-accent)"/>
<rect x="423" y="320" width="419.6" height="53" rx="6" class="note" />
<text x="436" y="337" class="t-body" text-anchor="start">1. CTE &quot;allowed&quot;: the LIVE documents table. status=published AND</text>
<text x="436" y="350" class="t-body" text-anchor="start"> (public | own department | doc_permissions grant | author).</text>
<text x="436" y="363" class="t-body" text-anchor="start"> Never chunk meta, so a change applies without reindexing.</text>
<rect x="423" y="385" width="364.25" height="40" rx="6" class="note" />
<text x="436" y="402" class="t-body" text-anchor="start">2. vector candidates: top-20 by cosine distance (HNSW),</text>
<text x="436" y="415" class="t-body" text-anchor="start"> joined to allowed.</text>
<rect x="423" y="437" width="395" height="40" rx="6" class="note" />
<text x="436" y="454" class="t-body" text-anchor="start">3. FTS candidates: top-20 by ts_rank_cd over</text>
<text x="436" y="467" class="t-body" text-anchor="start"> websearch_to_tsquery(&#x27;german&#x27;, query), joined to allowed.</text>
<rect x="423" y="489" width="450.35" height="27" rx="6" class="note" />
<text x="436" y="506" class="t-body" text-anchor="start">4. RRF merge: score = sum of 1/(60+rank), ORDER BY score LIMIT top_k.</text>
<text x="643" y="534" class="t-edge" text-anchor="middle">rows: chunk, document, title, meta, score,</text>
<text x="643" y="547" class="t-edge" text-anchor="middle">vector distance, fts rank</text>
<line x1="889" y1="564" x2="397" y2="564" class="edge-soft" marker-end="url(#arrow-open)"/>
<text x="274" y="586" class="t-edge" text-anchor="middle">SearchResult[] with citation metadata</text>
<text x="274" y="599" class="t-edge" text-anchor="middle">(title, heading_path, score, vector_distance, fts_match)</text>
<line x1="397" y1="616" x2="151" y2="616" class="edge-soft" marker-end="url(#arrow-open)"/>
<rect x="304.05" y="632" width="431.9" height="53" rx="6" class="note" />
<text x="317.05" y="649" class="t-body" text-anchor="start">No-answer signal: the top result has fts_match=false AND</text>
<text x="317.05" y="662" class="t-body" text-anchor="start">vector_distance &gt;= 0.45 (calibrated for bge-m3). Callers treat the</text>
<text x="317.05" y="675" class="t-body" text-anchor="start">whole set as low confidence rather than citing it.</text>
<line x1="28" y1="709" x2="1012" y2="709" class="edge-soft"/>
<rect x="353.25" y="698" width="333.5" height="22" rx="11" class="box-sunken" />
<text x="520" y="713" class="t-muted" text-anchor="middle">text_search(): the same query with no model at all</text>
<text x="274" y="741" class="t-edge" text-anchor="middle">text_search(db, query, user=...)</text>
<line x1="151" y1="758" x2="397" y2="758" class="edge" marker-end="url(#arrow)"/>
<text x="643" y="780" class="t-edge" text-anchor="middle">SAME allowed CTE, then the german tsvector GIN index alone</text>
<line x1="397" y1="797" x2="889" y2="797" class="edge" marker-end="url(#arrow)"/>
<rect x="300.975" y="813" width="438.05" height="53" rx="6" class="note" />
<text x="313.975" y="830" class="t-body" text-anchor="start">The fallback for a dead embedding endpoint: keywords, not meaning,</text>
<text x="313.975" y="843" class="t-body" text-anchor="start">so the user is told it was a search without a model. No embed call,</text>
<text x="313.975" y="856" class="t-body" text-anchor="start">which is why it works when nothing runs behind the LLM config.</text>
<line x1="28" y1="890" x2="1012" y2="890" class="edge-soft"/>
<rect x="390.15" y="879" width="259.7" height="22" rx="11" class="box-sunken" />
<text x="520" y="894" class="t-muted" text-anchor="middle">rag/similarity: threshold, not ranking</text>
<text x="274" y="922" class="t-edge" text-anchor="middle">similar_chunks(db, text, user=..., max_distance=&lt;one of two constants&gt;)</text>
<line x1="151" y1="939" x2="397" y2="939" class="edge" marker-end="url(#arrow)"/>
<text x="520" y="961" class="t-edge" text-anchor="middle">embed([text])</text>
<line x1="397" y1="978" x2="643" y2="978" class="edge" marker-end="url(#arrow)"/>
<text x="643" y="1000" class="t-edge" text-anchor="middle">SAME allowed CTE, then pure vector:</text>
<text x="643" y="1013" class="t-edge" text-anchor="middle">ORDER BY cosine distance LIMIT top_k</text>
<line x1="397" y1="1030" x2="889" y2="1030" class="edge" marker-end="url(#arrow)"/>
<text x="643" y="1052" class="t-edge" text-anchor="middle">rows with a distance that is ALWAYS present</text>
<line x1="889" y1="1069" x2="397" y2="1069" class="edge-soft" marker-end="url(#arrow-open)"/>
<text x="274" y="1091" class="t-edge" text-anchor="middle">SimilarChunk[] filtered to distance &lt;= max_distance</text>
<text x="274" y="1104" class="t-edge" text-anchor="middle">(similar_documents groups per document, keeping the min)</text>
<line x1="397" y1="1121" x2="151" y2="1121" class="edge-soft" marker-end="url(#arrow-open)"/>
<rect x="288.675" y="1137" width="462.65" height="92" rx="6" class="note" />
<text x="301.675" y="1154" class="t-body" text-anchor="start">Why not the RRF path: its score is a fusion rank, not a similarity,</text>
<text x="301.675" y="1167" class="t-body" text-anchor="start">and vector_distance is null for FTS-only hits. A threshold needs a</text>
<text x="301.675" y="1180" class="t-body" text-anchor="start">comparable number. The permission filter is shared; only the ranking</text>
<text x="301.675" y="1193" class="t-body" text-anchor="start">differs. Two calibrated constants exist, CAPTURE_CONTEXT_MAX_DISTANCE</text>
<text x="301.675" y="1206" class="t-body" text-anchor="start">(loose) and DUPLICATE_MAX_DISTANCE (tight); exclude_builtin drops</text>
<text x="301.675" y="1219" class="t-body" text-anchor="start">Pablan&#x27;s own help pages, because a capture asks what the COMPANY knows.</text>
<rect x="294.825" y="1241" width="450.35" height="79" rx="6" class="note" />
<text x="307.825" y="1258" class="t-body" text-anchor="start">Document search (GET /api/documents/search) is the same call with a</text>
<text x="307.825" y="1271" class="t-body" text-anchor="start">larger top_k, grouped per document: one code path, so browsing and</text>
<text x="307.825" y="1284" class="t-body" text-anchor="start">chat can never disagree on access. Query mode surfaces the run as SSE</text>
<text x="307.825" y="1297" class="t-body" text-anchor="start">state events (searching, results(count) | no_answer, answering).</text>
<text x="307.825" y="1310" class="t-body" text-anchor="start">Counts only, never the query text or a passage (rule 12).</text>
</svg>
Side by Side

After

Width:  |  Height:  |  Size: 12 KiB