Pablan, as it stands
Self-hosted knowledge management for SMEs: a split-screen Markdown editor whose sections an LLM refines while you write, and RAG question answering over the documents that result. FastAPI + Postgres/pgvector on the back, SvelteKit on the front, everything OpenAI-compatible and self-hostable. Squashed into a single commit; the development history stays local. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CA43ZJda8Rbp2hKXNy8f6b
This commit is contained in:
ProfessorNova
co-authored by
Claude Opus 5
Claude Opus 5
parent
68d3a43191
commit
784b76baf7
@@ -0,0 +1,201 @@
|
||||
# CLAUDE.md — Pablan
|
||||
|
||||
Self-hosted knowledge management for SMEs. Employees write knowledge documents
|
||||
in a split-screen Markdown editor while an LLM refines the section they are
|
||||
working on; the same LLM answers questions over the resulting documents via
|
||||
RAG. Everything is stored as Markdown and runs on the customer's
|
||||
infrastructure. Team of 2 developers — bias toward simplicity, no speculative
|
||||
abstractions.
|
||||
|
||||
Design docs live in `docs/` (English). Read the relevant doc before larger changes.
|
||||
**Claude maintains the documentation**: whenever a decision, schema, API, or
|
||||
workflow changes, update the affected file in `docs/` in the same change — stale
|
||||
docs are treated as bugs. This includes the diagram sources in
|
||||
`docs/diagrams/` (hand-authored SVG, theme-aware; the SVG is the artifact,
|
||||
no diagram toolchain).
|
||||
`docs/roadmap.md` is the feature-based backlog: every feature area as an epic with
|
||||
done (`[x]`) and open (`[ ]`) stories, ordered so it reads as a rebuild manual. It
|
||||
is the only forward-looking doc — its open epics may describe not-yet-built work;
|
||||
everything else in `docs/` describes the as-is state only. `docs/notes.md` holds the
|
||||
durable implementation learnings (calibrations, gotchas, why-it-is-this-way). When a
|
||||
feature ships, tick its story in the roadmap and fold any lasting learning into
|
||||
`docs/notes.md`, in the same change. The root `README.md` gives a short high-level overview
|
||||
(what Pablan is, stack, quickstart) and links into `docs/`; it is also
|
||||
Claude-maintained and must stay short and current.
|
||||
|
||||
**`help/` is in-product documentation and follows the same rule**: those
|
||||
Markdown files are the built-in help pages users read *inside* Pablan
|
||||
(imported into `documents` on every start, `is_builtin`, not editable in
|
||||
the UI). Any change to a user-facing workflow, screen or concept updates
|
||||
the affected help page in the same change — an outdated help page is a bug
|
||||
like a stale `docs/` page, except users see this one. `docs/` explains the
|
||||
system to developers; `help/` explains the product to its users, in German
|
||||
(product content, like templates).
|
||||
|
||||
## Stack (fixed decisions — do not substitute)
|
||||
|
||||
- **Backend**: Python 3.12+, FastAPI (async), managed with `uv`. Lint+format: `ruff`.
|
||||
- **DB**: PostgreSQL + pgvector. ORM: SQLAlchemy 2.0 (async, typed). Migrations: Alembic.
|
||||
No dedicated vector DB, no Prisma, no Redis.
|
||||
- **Frontend**: SvelteKit (Svelte 5 runes) + TypeScript + Tailwind. Prettier + ESLint.
|
||||
Headless behavior via **Bits UI**; we write our own styled components on top.
|
||||
No shadcn imports, no component library dependencies beyond Bits UI.
|
||||
- **LLM**: OpenAI-compatible endpoints only (llama.cpp server locally, cloud APIs in prod).
|
||||
No LangChain / LlamaIndex — thin custom client in `backend/app/llm/client.py`.
|
||||
- **Deployment**: Docker Compose (postgres, backend, frontend, reverse proxy).
|
||||
Reverse proxy config in `deploy/` (Caddy, the only supported proxy for now).
|
||||
|
||||
## Repo layout
|
||||
|
||||
- `backend/` — FastAPI app (`app/models`, `app/auth`, `app/api`, `app/modes`,
|
||||
`app/authoring`, `app/llm`, `app/rag`, `app/ingestion`)
|
||||
- `frontend/` — SvelteKit, pure UI / API client. No DB access, no auth logic beyond
|
||||
cookie passthrough in `hooks.server.ts`.
|
||||
- `templates/` — built-in authoring templates: Markdown skeletons (YAML).
|
||||
Product content, not code.
|
||||
- `ee/` — proprietary Enterprise modules. **Core code must NEVER import from ee/.**
|
||||
ee registers itself via `app/ee_hooks.py` and the mode/frontend registries.
|
||||
- `docs/` — English design documentation, kept in sync with code by Claude.
|
||||
- License: FSL-1.1 for core, separate proprietary license in `ee/LICENSE`.
|
||||
|
||||
## Commands
|
||||
|
||||
- `make dev` — full dev stack with hot reload (postgres in docker, backend +
|
||||
frontend native)
|
||||
- `make down` — stop the dev stack
|
||||
- `make migrate` — apply Alembic migrations
|
||||
- `make seed` — seed dev data
|
||||
- `make types` — regenerate `frontend/src/lib/api/schema.d.ts` from OpenAPI.
|
||||
Run after ANY backend API change.
|
||||
- `make lint` — ruff + prettier + eslint + design-token contrast check
|
||||
(CI runs the same)
|
||||
- `make eval` — LLM eval suite in `backend/tests/evals` against the configured endpoint
|
||||
- `make e2e` — Playwright end-to-end tests against the dev stack
|
||||
|
||||
## Architecture rules (invariants — enforce in every change)
|
||||
|
||||
1. **Markdown is the source of truth.** Documents live as Markdown in Postgres.
|
||||
Chunks/embeddings are disposable derivatives; any pipeline change must allow
|
||||
full re-indexing from documents.
|
||||
2. **Permissions filter BEFORE the LLM.** All retrieval goes through
|
||||
`rag/retrieval.search(query, user=...)` — there is no search without a user.
|
||||
Never pass chunks to a prompt that the requesting user could not read.
|
||||
3. **All LLM traffic goes through `llm/client.py`** (`chat_stream`, `chat_json`,
|
||||
`embed`) with model roles `chat` / `utility` / `embedding`, each independently
|
||||
configurable (base_url, api_key, model). Never call an LLM HTTP API elsewhere.
|
||||
Its failure vocabulary lives in `llm/errors.py`: `LLMError.code` classifies
|
||||
every endpoint failure once, and the frontend phrases it.
|
||||
4. **Structured outputs use `chat_json`** with a Pydantic schema passed as
|
||||
`response_format` (JSON schema). Never parse free-form LLM text into data.
|
||||
5. **Modes implement the `Mode` protocol** (`modes/base.py`) yielding `ModeEvent`s;
|
||||
the conversations router converts events to SSE. Modes know no HTTP; routers
|
||||
know no mode logic. New modes register in `modes/registry.py`. Query (RAG Q&A)
|
||||
is the only core mode; EE adds insight. Capture is NOT a mode — see rule 6.
|
||||
6. **Capture is writing-first, not a conversation.** The user authors a
|
||||
`Document` directly (Markdown is the source of truth, rule 1); it starts in
|
||||
`draft` status — author-only (`rag/permissions.readable_documents_filter`) and
|
||||
never indexed until published (searchable requires `published`), so a draft
|
||||
never reaches another user or an LLM prompt (rule 2). Publishing is the
|
||||
author's own one-click action; the three statuses (draft, published,
|
||||
archived) say where a document stands, never whether its CONTENT is
|
||||
trusted. That is a `ReviewRequest` — "please check this", which can hang on
|
||||
a draft or on a document published months ago, grants the person asked the
|
||||
right to edit until they answer, and marks the document everywhere it
|
||||
appears including chat sources. Section refinement
|
||||
(`POST /api/documents/{id}/refine`, package `app/authoring/`) regenerates ONLY
|
||||
the section at the cursor (FIM-style: the rest of the document is prefix/suffix
|
||||
context), so large documents stay cheap and small local models (Gemma-class)
|
||||
stay reliable. The active-section boundary is computed server-side, shared with
|
||||
`rag/chunking`. No hidden engine state — the document is the state.
|
||||
7. **Prompts are rendered natural language**, never raw YAML/JSON dumps.
|
||||
8. **Auth = server-side sessions** (argon2 password hashes, `auth_sessions` table,
|
||||
httpOnly cookie). No JWT. Naming: chat threads are `conversations`,
|
||||
login sessions are `auth_sessions` — never mix these up.
|
||||
9. **Background work goes through the `jobs` table** (`ingestion/queue.py`,
|
||||
`FOR UPDATE SKIP LOCKED` loop). No new queue infrastructure.
|
||||
10. **Frontend renders LLM/document Markdown only through the sanitizing
|
||||
renderer** (DOMPurify). Treat all model output and document content as untrusted.
|
||||
11. **API contract flows one way**: FastAPI OpenAPI → `openapi-typescript` →
|
||||
typed `openapi-fetch` client. Never hand-write API response types.
|
||||
12. **NEVER log content** — no prompts, no LLM responses, no user messages,
|
||||
no document text. Log metadata only (model role, duration, token counts,
|
||||
error codes, entity IDs, correlation id). This applies to every log line,
|
||||
including exceptions (no content in error messages). Content debug
|
||||
logging only behind `PABLAN_DEBUG_LOG_PROMPTS=true`, documented as
|
||||
never-in-production.
|
||||
13. **Auth boundaries (login/logout) are full document navigations, never
|
||||
client-side.** Module-level client state (the conversation list, chat
|
||||
state, the resolved locale — all runes singletons) is guaranteed dead at
|
||||
the session boundary because the page is reloaded. Do NOT convert these
|
||||
to client navigation (`goto`/`invalidateAll`): a client nav keeps the
|
||||
previous user's singletons alive and leaks their data (conversation
|
||||
titles are an information disclosure). This is immune to stores added
|
||||
later; a per-store reset is not.
|
||||
|
||||
## Conventions
|
||||
|
||||
- Code, comments, identifiers, `docs/`, README, seed data, test strings:
|
||||
English. Template content and the test fixture corpus in
|
||||
`tests/fixtures/`: German — product content for the German market.
|
||||
- **UI copy goes through Paraglide messages** (`frontend/messages/`),
|
||||
source language **de** (informal "du"), **en** written in the same
|
||||
change. A hardcoded UI string is a bug; a missing `en` message and an
|
||||
em/en dash in any message both fail `make lint`
|
||||
(`frontend/scripts/check-messages.py`). See `docs/i18n.md` for the key
|
||||
naming convention and how the locale is resolved.
|
||||
- **The backend never renders UI-language strings.** API errors are
|
||||
`{detail, code}` and the frontend translates by `code`; SSE `state`
|
||||
events carry counts and markers, and the frontend phrases them.
|
||||
- Full type hints in Python; `ruff` rules include `I` (isort) and `B` (bugbear).
|
||||
- Svelte 5 runes only (`$state`, `$derived`, `$props`) — no legacy stores for new code.
|
||||
- Keep the base component set small; reuse `lib/components/` primitives.
|
||||
No one-off colors or spacing — see Design tokens below.
|
||||
- DB primary keys: UUID. Timestamps via the shared mixin in `models/base.py`.
|
||||
- `docs/diagrams/data-model.svg` is the ER diagram; any model or migration
|
||||
change updates the diagram and the prose in `docs/data-model.md` in the
|
||||
same change.
|
||||
- Retention/privacy defaults matter (GDPR): users can delete their own
|
||||
conversations; retention cleanup runs as a scheduled job; insights features
|
||||
must only ever see aggregated data, never per-user raw messages.
|
||||
|
||||
## Design tokens
|
||||
|
||||
- All colors are defined ONCE as semantic CSS variables in `frontend/src/app.css`
|
||||
and mapped into the Tailwind theme: `--color-primary`, `--color-secondary`,
|
||||
`--color-accent`, plus role tokens for surfaces, text, borders, and states
|
||||
(success/warning/danger), each with the shades needed for hover/muted variants.
|
||||
- Current palette values live ONLY in `frontend/src/app.css` — CLAUDE.md
|
||||
never names colors. The palette must be swappable by editing only the
|
||||
token definitions — components reference tokens exclusively (`bg-primary`,
|
||||
`text-accent`), NEVER raw hex values or Tailwind default colors.
|
||||
- The accent token is for highlights, active states, and CTAs — never for
|
||||
body text or large surfaces. Check WCAG AA contrast for any token pairing
|
||||
(`frontend/scripts/contrast-check.py`, enforced by `make lint`); every
|
||||
token needs a working value in both light and dark mode.
|
||||
- Spacing, radii, and typography sizes also come from the theme scale — no
|
||||
arbitrary values (`p-[13px]`) in components.
|
||||
|
||||
## Testing
|
||||
|
||||
- Fast unit tests colocated in `backend/tests/`; retrieval logic gets SQL-level
|
||||
tests against a real Postgres (docker).
|
||||
- `tests/evals/` holds the refinement/query eval set — extend it whenever prompt
|
||||
or engine behavior changes, and run `make eval` with both a local model and
|
||||
a cloud model before merging prompt changes.
|
||||
- E2E: Playwright in `frontend/e2e/` against the dev stack (`make e2e`).
|
||||
Cover the critical paths: login, chat streaming, the writing editor (section
|
||||
refinement + accept), publishing, review requests, permission boundaries
|
||||
(user A must not see user B's restricted docs).
|
||||
- **Visual verification**: when changing UI, use the Playwright MCP server to
|
||||
open the affected pages, take screenshots (light AND dark mode), and inspect
|
||||
the result before considering the work done. Frontend changes are not
|
||||
finished on "it compiles".
|
||||
|
||||
## Dev environment notes
|
||||
|
||||
- Local LLM: llama.cpp server (Gemma-class 12B/26B) via OpenAI-compatible API;
|
||||
quality reference: Claude Sonnet via API. Both configured purely through `.env`
|
||||
(`PABLAN_CHAT_*`, `PABLAN_UTILITY_*`, `PABLAN_EMBEDDING_*`).
|
||||
- Embeddings: multilingual model (e.g. bge-m3) — German retrieval quality is a
|
||||
first-class requirement; hybrid search (pgvector + Postgres `german` full-text
|
||||
with RRF) is the default, not an option.
|
||||
Reference in New Issue
Block a user