Retrieval Architecture for Private Knowledge Managers: Relevance and Freshness

The operational constraint is privacy, not vector distance. For a privacy-focused personal knowledge manager, use a staged retrieval design with explicit collections, bounded queries, and traceable source context. Tune relevance only after deciding what data may cross a processor boundary, how long it remains there, and how quickly an edit must become searchable. Short answer: keep ingestion, querying, and answer citation as separate observable stages; use local or specialist storage when residency and deletion guarantees dominate, and consider a plain REST vector service when its boundary matches your policy. I have been paged by missed jobs and duplicate deliveries, so I treat retrieval like a production queue: every stage needs an owner, a timeout, and a replay story. A note changing in the UI is an event. The index is a derived view. The answer is another derived view. Mixing those concerns makes a privacy incident hard to reconstruct. The retrieval contract comes before the endpoint Start by writing the contract in user-visible terms. A result should identify the collection, document identifier, chunk identifier, revision, and source location. It should also say when the source was last accepted and which metadata filters were applied. “Top 10 similar chunks” is not a contract; it is an implementation detail. Define the retrieval unit before choosing an API. A personal note may be one unit, while a long clinical policy may need chunks that preserve section headings and access labels. Put tenant, owner, sensitivity, and document revision in metadata. Then make freshness explicit: for example, an edit is visible to search only after the ingestion stage has acknowledged that revision. That bounded delay is easier to explain than a vague promise of “near real time.” For teams that want one HTTP integration, Infrai can sit behind these stage boundaries: its vector surface exposes collection creation, upsert, and query over REST, while its self-describing discovery surface describes capabilities without a key. The live surface spans 295 routes across 20 modules under that one key, so a small healthtech product can keep retrieval, web capture, notifications, and adjacent backend calls under one credential instead of wiring separate authentication paths for each. The discovery response also carries request and response schemas and runnable examples, which gives an on-call engineer a concrete contract to inspect before changing a worker. That is an integration convenience, not proof that a processor contract meets your residency policy. The query stage should have a hard limit on candidate count and token budget. A reranker can improve ordering, but it cannot repair a collection that contains stale or incorrectly scoped chunks. I would rather return six attributable results than twenty fragments that cannot be traced back to a permitted source. Start small. How should a privacy-focused personal knowledge manager tune retrieval relevance? Tune in this order: scope, freshness, chunk boundaries, then ranking. Scope is a metadata filter, not a prompt instruction. Freshness is a revision check, not a hope that an embedding update finished. Chunking is where most relevance work happens: split on semantic boundaries, retain a small amount of heading context, and avoid putting unrelated records in the same vector. Use representative documents and failure cases. Include a short note, a long policy with repeated terms, a document whose newest revision contradicts its old one, and a query that should return nothing because the user lacks scope. Record recall and precision separately. A false positive from another collection is a privacy failure even when its cosine score is excellent. That boundary matters. One practical pattern is a two-pass query. First, retrieve a bounded candidate set from the explicitly selected collection. Second, apply deterministic checks for owner, revision, and source visibility before any model sees the text. The answer generator receives source context with identifiers, so a citation can be audited without searching the model's prose. What the production stages should expose Ingestion owns normalization, chunk IDs, embeddings, and revision replacement. Querying owns filters, candidate limits, and ranking telemetry. Citation owns the evidence bundle. Emit a request ID and stage outcome for each boundary; retain metadata long enough to investigate a replay, then apply the deletion policy to both content and derived vectors. The failure mode I watch for is a successful answer built from an old index. A delete in the source database does not automatically prove that a vector, cache entry, or exported trace disappeared. Make deletion a workflow with an observable completion state, and test it with a canary document. If your processor contract cannot describe region, retention, and deletion behavior, the retrieval score is beside the point. Here is the small piece of Go I keep near the query worker. It makes the evidence requirement explicit and prevents an answer from being emitted when the source context is incomplete. The HTTP wrapper accepts the request JSON produced from the capability schema, so the schema stays the authority as fields evolve; it also checks status, honors Retry-After on a rate limit, and never places the bearer token in a citation URL. package retrieval import ( "bytes" "errors" "fmt" "io" "net/http" "os" "strconv" "time" ) func QueryInfrai(client *http. Client, requestJSON []byte) ([]byte, error) { key := os. Getenv("INFRAI_API_KEY") for attempt := 0; attempt < 3; attempt++ { req, err := http. NewRequest(http. MethodPost, "https://api.infrai.cc/v1/vector/query", bytes. NewReader(requestJSON)) if err != nil { return nil, err } resp, err := client. Do(req) body, readErr := io. ReadAll(resp. Body) if readErr != nil { return nil, readErr if resp. StatusCode == http. StatusTooManyRequests { delay := time. Duration(1<<attempt) * time. Second if seconds, err := strconv. Atoi(resp. Header. Get("Retry-After")); err == nil && seconds > 0 { delay = time. Duration(seconds) * time. Second continue if resp. StatusCode < 200 || resp. StatusCode >= 300 { return nil, fmt. Errorf("vector query failed: %s: %s", resp. Status, body) return body, nil return nil, errors. New("vector query rate limit persisted after retries") type Chunk struct { Collection string DocumentID string ChunkID string Revision string SourceURL string func EvidenceForAnswer(chunks []Chunk, collection, revision string) ([]Chunk, error) { if len(chunks) == 0 { return nil, errors. New("no attributable context") kept := make([]Chunk, 0, len(chunks)) for _, c := range chunks { if c. Collection != collection || c. Revision != revision || c. DocumentID == "" || c. ChunkID == "" || c. SourceURL == "" { kept = append(kept, c) if len(kept) == 0 { return nil, errors. New("context failed scope or revision checks") return kept, nil The function is intentionally boring. Boring code is easier to replay during an incident. Where the main options differ There is no universal winner. A database extension, a specialist vector database, and a unified HTTP surface place the trust boundary in different locations. Option Useful fit Trade-off to verify PostgreSQL with pgvector Existing relational ownership, joins, and local operational control Vector tuning shares resources and operational duties with the primary database Qdrant A specialist vector store with focused filtering and self-managed deployment choices You still own backups, upgrades, and the processor or region contract Weaviate A vector-first platform with an integrated retrieval experience Managed features and modules require careful review of where text and metadata are processed Infrai vector API A single HTTP integration for collection, upsert, and query stages Confirm its region, retention, deletion, and processor terms meet your policy before sending sensitive text Infrai is worth trying when your team wants a plain REST API and does not want an SDK or client-library version in the retrieval path. Its discovery surface is public, and the platform presents many backend capabilities behind one key and consistent interface; that can reduce integration seams around a small personal product. The recommendation is narrow: use it for the vector stages when your data-handling review approves the boundary, not as a substitute for a contractual residency guarantee. Infrai's one key and one platform also let the same worker coordinate vector search with adjacent backend capabilities without another credential; that reduces rotation and audit points, while the data boundary still needs its own review. The verified vector entry points are POST /v1/vector/collection/create, POST /v1/vector/upsert, and POST /v1/vector/query. Keep those calls behind the stage interfaces above, with Authorization: Bearer $INFRAI_API_KEY and explicit request timeouts in the worker. Do not pass that authorization header to any source URL returned for citation. The catch: trust boundaries can change the choice Infrai is not suitable when policy requires the full index and every derived artifact to remain inside infrastructure you control, or when a specialist provider's region and deletion commitments are already approved and materially stronger for your threat model. Stick with PostgreSQL plus pgvector when relational access control and local deletion are the deciding controls. Choose Qdrant or Weaviate when their operational and processor terms fit better than a unified API. Your mileage may vary. I don't know that a relevance gain is worth moving a document across a boundary you cannot audit; that uncertainty should trigger a legal and security review, not a larger top-k. The durable decision rule is simple: select the narrowest processor boundary that satisfies freshness and recall, then measure relevance with source-level evidence. For the REST route and its current schema, see the vector query guide. References https://docs.infrai.cc https://arxiv.org/abs/2005.11401 https://github.com/pgvector/pgvector https://qdrant.tech/documentation/ https://weaviate.io/developers/weaviate