Knowledge bases

A knowledge base (KB) lets Pia ground assistant answers in your own documents. This page is the end-to-end setup guide for administrators running their own instance. For how it works internally, see Architecture → Knowledge base.

The KB is off until you enable it. With no vector database configured the server runs exactly as before — nothing in the chat path changes. Enabling it is additive and reversible.

The KB needs two things before it does anything: the Knowledge license feature and a configured vector database. Miss either and the chat path stays normal (ungrounded) chat and the admin KB pages stay inert — see Architecture → Knowledge base for the exact behaviour of each gate.

What you’ll need

• A running Pia Server (see Local Docker or Production deployment).
• A licence that includes the Knowledge feature (see Licensing).
• A PostgreSQL + pgvector database — the bundled pia-vectordb service provides this.
• An embedding model the server can reach (OpenAI-compatible, Azure, or a local Ollama model).
• Optionally, a chat model for contextual retrieval (improves recall; can be the same provider).

Setup

1. Bring up the vector database.

   Both Compose files already define the pia-vectordb service (image pgvector/pgvector:pg17). It is internal-only and has no public route.

   • Local: docker compose -f docker-compose.local.yml up wires and starts it automatically.
   • Production: it is included in docker-compose.prod.yml with a host-bound data volume.

   Confirm it is accepting connections:

   docker exec pia-vectordb pg_isready -U pia -d pia_knowledge   # → accepting connections

2. Point the server at the vector DB and set an ingest key.

   In your .env (production uses the raw config-key form, double underscore = config section separator):

   Knowledge__ConnectionString=Host=pia-vectordb;Database=pia_knowledge;Username=pia;Password=<your-password>
   Knowledge__IngestApiKey=<a long random secret>   # enables the REST ingest API; leave empty to disable it

   Leaving Knowledge__ConnectionString empty keeps the KB disabled.

3. Choose the embedding model.

   One embedding model per instance → one vector space and dimension. ProviderId references an entry in your Ai__Providers catalog.

   Embeddings__ProviderId=<an Ai__Providers__N__Id>
   Embeddings__Model=bge-m3
   Embeddings__Dimension=1024

   The dimension is fixed

   Embeddings__Dimension must match the compiled schema (1024). A mismatch disables the KB at startup with a clear log line. Changing it later means a migration and re-embedding every document.

4. (Optional) Enable contextual retrieval.

   On by default, this prepends a one-sentence LLM description to each chunk before embedding — but it needs its own chat provider:

   Chunking__Context__Enabled=true
   Chunking__Context__ProviderId=<an Ai__Providers__N__Id of a chat model>

   If Chunking__Context__ProviderId is left empty, the server logs a warning and auto-disables contextualization at startup — the KB still works, documents just embed the raw chunk only. To turn it off explicitly, set Chunking__Context__Enabled=false.

5. Restart and confirm the KB is enabled.

   Restart the server and check the logs for the absence of a [Knowledge] warning, then open Admin → Knowledge bases (/admin/knowledge-bases). If the page shows a “vector DB required” notice, the connection string isn’t reaching the container — re-check step 2.

6. Create a knowledge base.

   On Admin → Knowledge bases, create a KB with a name and an optional system-prompt addition (guidance the model gets whenever this KB is active, e.g. “Ground answers in the product docs.”).

7. Add documents.

   Open the KB and upload documents — plain text (.txt) and Markdown (.md / .markdown); both are ingested as raw text (no Markdown parsing). Each appears as Pending, then Ready once the worker has chunked and embedded it (or Failed with an error). You can also ingest via the REST API for CI pipelines.

8. Enable the KB for a group.

   A KB only becomes active for users in groups that allow it. On Admin → Groups, open a group and add the KB to its plugin allowlist. The change takes effect on the user’s next request.

9. Verify.

   As a user in that group, start an Assistant-mode chat and ask something answerable from your documents. You should get a grounded answer with sources. (Optimize mode and voice never engage the KB by design.)

Quotas

Per-group quotas are edited in the group editor:

Quota	Meaning
KnowledgeBases	How many KBs a group may enable
KnowledgeDocuments	Documents per KB (checked at ingestion → 409)
KnowledgeStorageBytes	Total stored bytes per KB (checked at ingestion → 409)
MonthlyEmbeddingTokens	Embedding tokens per group per calendar month

For a KB shared by several groups, the effective doc/byte/token limit is the most permissive of those groups; monthly embedding-token spend is charged to a single owner group (the lowest group id) so it is never counted twice.

Ingesting via the REST API

For automated pipelines, ingest with the instance ingest key:

curl -X POST https://your-host/api/kb/<kbId>/documents \
  -H "X-Pia-Service-Key: $KNOWLEDGE_INGEST_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"title":"Onboarding","contentType":"text/plain","content":"…"}'
# → 202 Accepted (queued); 403 if the Knowledge feature isn't licensed;
#   404 for an unknown KB; 409 on a quota breach; 503 if the KB is disabled (no vector DB)

The ingest key is instance-wide

Knowledge__IngestApiKey is a single trusted operator/CI secret. It can ingest into any KB on the instance — it is intentionally not scoped to one group. Treat it like a deploy secret; don’t hand it to mutually-distrusting teams.

Privacy

KB documents are stored unencrypted in the vector DB — the server must read them in plaintext to chunk, embed, and search. Keep this separate from users’ end-to-end-encrypted personal data. To keep all document text on your own infrastructure, point Embeddings__* and Chunking__Context__* at a local Ollama model.

Troubleshooting

Symptom	Likely cause
KB never grounds answers; ingest returns 403 feature_not_licensed	The licence doesn’t include the Knowledge feature (see Licensing)
Admin page shows “vector DB required”	Knowledge__ConnectionString empty or not reaching the container
KB disabled in logs at startup	Embeddings__Dimension ≠ 1024 (dimension mismatch)
Startup [Knowledge] warning, no contextualization	Contextual retrieval is on but Chunking__Context__ProviderId is unset (auto-disabled)
Every document → Failed	Embedding provider unreachable / wrong Embeddings__ProviderId, or returns a wrong-width vector
Ingest returns 409	A KnowledgeDocuments / KnowledgeStorageBytes / MonthlyEmbeddingTokens quota was hit
Ingest returns 503	The KB is disabled (no vector DB configured)
Chat isn’t grounded	The user’s group doesn’t allow the KB, or the mode isn’t Assistant/Research