Corpora and the workspace graph: managed knowledge, search, tools and ownership¶
Status: proposed. Builds on RFC 0005 (federated retrieval and the catalog), which it does not change.
Goal¶
People create a corpus, a named container such as "HR policies", that holds the sources one useful body of knowledge comes from: a Drive folder, a Notion teamspace, a Snowflake schema, uploaded PDFs. A corpus decides how each source stays fresh, which ways of searching are combined, which tools it gives agents, what its entities and relations mean (its ontology), and how its quality is checked. Agents are granted corpora, not raw sources, so they search better and only where they should.
Above the corpora sits one workspace graph: people, teams, projects, systems, topics and documents, linked across corpora where allowed. It knows who wrote and owns each document, which documents are stale or contradicted, and how systems depend on each other. A built-in maintenance agent keeps it aligned with the sources, and people approve what matters.
The same machinery serves work and personal life: a team's HR policies, a warehouse of metrics, and one person's private sessions, notes and records, each searched under its own access, so an agent can help a person with the documents they are entitled to and nothing more.
Context¶
- Built.
pkg/knowledgekeeps sources as named collections of passages, ranked by BM25, with access rules per reader. RFC 0005 adds connector capabilities (live,collect,catalog), the embedder slot, hybrid fusion and the catalog. saige'sragalready has keyword and vector retrievers, fusion, a reranker, parent retrieval, HyDE, a knowledge graph with extraction, a graph retriever scoped by group, and search, lookup, update and delete tools. - Missing. A container above sources; a search recipe per container; tools scoped to a container's contents; an ontology; freshness targets; retrieval evaluation; deduplication across sources; a graph that spans sources, with ownership and staleness; an agent that maintains it.
Decision¶
1. A corpus and its members¶
| Field | Meaning |
|---|---|
name, description, tags |
Shown to people and to the router that picks a corpus |
owner, readers |
Who manages it, and who may be granted it (default deny) |
members |
Sources it draws on, each indexed or referenced (below) |
recipe |
How it is searched (section 2) |
ontology |
Types it adds to the workspace graph (section 4), optional |
graph |
shared or isolated, and whether the semantic layer is on (sections 4 and 5) |
freshness |
Staleness target per member (section 9) |
evals |
Its golden set and thresholds (section 10) |
sensitivity |
Its default label and ceiling (section 7) |
A member is a reference to a connection's resource, never a copy of the connection. One source may sit in several corpora. Every source belongs to at least one corpus, so knowledge has one concept for people: the corpus.
| Member kind | RFC 0005 capability | Sync does | Searched by |
|---|---|---|---|
| Indexed | collect |
Fetch, parse, chunk, embed, and optionally extract graph facts | The corpus's own indexes |
| Referenced | live and catalog |
Records a pointer: title, owner, schema, a change cursor. No content is copied | The source's own API, live, at query time, with the reader's credential |
Referenced is the default where the source searches well natively (a warehouse, Notion). A sync of a referenced member is a reference point: it keeps the pointer, metadata and cursor current so a live query knows where to look, and it can run as often as wanted because it copies nothing.
2. Search: a recipe of strategies¶
A recipe names the strategies a query fans out to, how results merge, and limits. Offline strategies run against the corpus's indexes; online strategies run live against referenced members.
| Strategy | Mode | Notes |
|---|---|---|
lexical |
offline | BM25 over passages (today's behavior) |
vector |
offline | The embedder slot; needs vectors for indexed members |
hybrid |
offline | Reciprocal rank fusion of lexical and vector |
rerank |
offline | A cross-encoder or model reranker over the top candidates |
parent |
offline | Expands a passage hit to its section |
hyde |
offline | Embeds a hypothetical answer; opt-in, costs a model call |
graph |
offline | Documents, facts and paths from the workspace graph, limited to this corpus or its shared links (sections 4 and 5) |
live |
online | Each referenced member's native search or query, scoped to the member |
catalog |
online | Finds which table or folder holds an answer, then live reads it |
Results from every strategy merge into one ranked list, are deduplicated by
document (section 8), and cite their member and document. A strategy that
times out or fails drops out and the result says so; it never fails the whole
search. Defaults: hybrid plus rerank for indexed members, live for
referenced ones.
3. Tools a corpus defines¶
Granting an agent a corpus grants its tools. Names are prefixed with the corpus name.
| Tool | Scope |
|---|---|
<corpus>_search |
The recipe. The default tool agents use |
<corpus>_lookup |
Fetch a cited document or passage by id |
<corpus>_graph |
Graph search: entities, relations, owners, paths, starting from this corpus |
graph_search |
Workspace-wide graph search across every corpus the agent is granted, for system-level questions |
| Source-native tools | Generated from referenced members and limited to their contents: <corpus>_sql reads only the corpus's tables, read-only; <corpus>_notion_search only its pages; <corpus>_drive_read only its folders |
Source-native tools reuse the connection's existing client or MCP server (RFC 0005 connectors) with a scope filter applied before the call, never a second client. Every tool checks the reader's own access to each hit; a corpus grant never widens what a person can read.
4. One workspace graph¶
There is one knowledge graph per workspace, and it is a top-level entity. Corpora are slices of it, not separate graphs. A corpus contributes nodes and edges, and every node and edge records which corpora it came from. Shared entities connect corpora naturally: the person who wrote the HR leave policy is the same node as the person who owns the payroll schema.
Built-in types. Every workspace graph has a core ontology that corpora extend and never replace:
| Entity | Comes from |
|---|---|
| Person | The identity directory and source authorship (author, editor, assignee) |
| Team | The identity directory and source groups |
| Project | Humagen projects and source workspaces (a repository, a teamspace) |
| System | Services, tables and schemas found in catalog metadata and code |
| Topic | Extracted from content; a subject a set of documents is about |
| Document | Every indexed or referenced file, page, table or thread |
| Relation | Meaning |
|---|---|
authored, edited |
A person wrote or changed a document, with when |
owns |
A person or team is responsible for a document, topic or system |
about, mentions |
A document covers a topic, or names an entity |
supersedes, contradicts |
A newer document replaces or disagrees with an older one |
depends_on, part_of |
Structure between systems, projects and teams |
Cross-corpus edges. A corpus chooses shared (its nodes may link to
nodes in other corpora through shared entities) or isolated (its nodes link
only within it, for sensitive material such as investigations). Built-in
Person, Team and Project nodes are always shared so people resolve once. An
edge never reveals a node the reader cannot see: traversal filters every hop
by the reader's access, so a path through a hidden corpus is cut, not shown.
Corpus ontology. A corpus extends the core with its own types:
| Mode | How it is defined | Use |
|---|---|---|
| Declared | A JSON Schema document: entity types with properties and required fields, relation types with source and target types and cardinality, and passage metadata fields | Regulated or well-understood domains: HR, finance |
| Inferred | The maintenance agent (section 5) samples the corpus and proposes types, kept proposed until an owner accepts them | New or messy corpora |
| Hybrid | A declared core plus accepted inferred extensions | The usual case |
- Versioned. The core and each corpus ontology are versioned records. A run pins the versions it read, as runs pin agent definitions.
- Validated. Extracted facts are checked against the ontology. A fact that does not fit goes to a review queue rather than into the graph.
- Re-extraction. A version change re-extracts only what it affects, incrementally and in the background.
- Metadata for vectors. Passage metadata fields (document type,
department, effective date, jurisdiction) become filters on
vector,hybridandlexical, so "the current parental leave policy in Canada" filters before it ranks.
5. Building and maintaining the graph¶
Two layers, cheapest first.
| Layer | Source | Cost | Gives |
|---|---|---|---|
| Structural | Source metadata: author, editors, timestamps, folder, links, schema, version history | No model calls. On for every corpus | Documents, people, ownership, staleness, file-level search |
| Semantic | Content extraction against the ontology | Model calls. Opt-in per corpus, estimated before enabling, metered | Topics, mentions, systems, contradictions, domain entities |
Referenced members contribute only to the structural layer; their content is never copied.
Ownership. Who owns a document or topic is inferred from authorship, edit share and recency, then shown as suggested. The person can confirm it, hand it to someone else, or decline it; a confirmed owner wins over inference. When an owner leaves the workspace, their documents become unowned and their team lead is asked to reassign. Owners receive their documents' staleness and contradiction alerts.
Staleness. Each document gets a staleness score, shown on its node and in search results:
| Signal | Effect |
|---|---|
| Age against its type's half-life | A policy ages slower than a status update; half-lives are ontology properties |
A newer document supersedes or contradicts it |
Marked stale, linked to the newer one |
| Its owner left or declined it | Marked unowned |
| Its source changed after the last sync | Marked out of date until re-synced |
Search ranks stale documents lower and labels them, never hides them.
The maintenance agent. A built-in agent keeps the graph aligned with the sources on a schedule and after each sync. It:
- merges duplicate entities (declared identity keys first, then name and properties with a confidence score);
- flags contradictions between documents and asks the owners which is current;
- proposes new types from recurring unfitted facts;
- suggests updates to owners of stale documents, with the evidence;
- drops facts whose only supporting document was deleted.
Every change it makes is a proposal with provenance (the documents and the reasoning). Low-risk changes apply on their own and stay reversible: a merge above the confidence threshold, a staleness mark, a fact retired with its source. Anything else waits for a person: low-confidence merges, new types, ownership changes, and anything touching a Person node. Proposals land as alerts in the owner's session (RFC 0009), and the agent's runs are audited like any other agent's.
Provenance. Every node and edge records the documents, members and corpora it came from, so a graph answer cites sources like any other hit, and deleting a source removes what only it supported.
Health. On the workspace graph page and each corpus page: nodes and edges by type, orphan and conflicting facts, validation failures, unowned and stale documents, open proposals, and the maintenance agent's last run.
Retrieval at three levels.
| Level | Answers | Tool |
|---|---|---|
| Passage | "What does the leave policy say about adoption?" | <corpus>_search |
| Document | "Which docs about payroll are current, and who owns them?" | <corpus>_search with document results, filtered by owner, topic, system, staleness |
| System | "What depends on the billing schema, and who should I ask?" | <corpus>_graph, or graph_search across every corpus the agent is granted |
Scoped storage. saige's group id is the workspace; corpus membership is a
property on each node and edge, and graphretriever filters by the corpora
the reader and agent are granted.
6. Files, references and records¶
A corpus holds three kinds of things, and each is handled differently.
| Kind | Examples | Stored | Extracted | Searched by |
|---|---|---|---|---|
| File | PDFs, Docs, Notion pages, slides, transcripts | Passages and vectors (indexed member) | Metadata fields, entities and facts per the ontology, tables inside documents as rows | The recipe |
| Reference | A dashboard, a URL, a Drive folder kept live, a ticket | Pointer, title, owner, change cursor | Structural graph only | live, at the source |
| Record | Snowflake and BigQuery tables, views, metrics | Schema, column descriptions, tags, row counts. Never rows | Table, column and metric nodes; joins as depends_on; owners from the warehouse |
<corpus>_sql, read-only, live, with the reader's credential |
- Rows stay in the warehouse. Answers that need data run a query at the source under the reader's identity, so warehouse row access and masking policies apply unchanged. The result is cited as the query plus the table and the time it ran, not copied into the corpus.
- Documents yield structure. A policy PDF gives effective date, jurisdiction and the roles it covers; a contract gives parties, terms and renewal date; an offer letter gives a person, role and start date. These become metadata filters and graph facts, linked back to the passage.
- Documents link to records. When a document names a metric or table ("headcount by department"), the graph links them, so an agent can read the definition in the doc and the current number from the warehouse.
7. Sensitivity, private sessions and personal context¶
Labels. Every document, column and graph node carries a sensitivity
label: public, internal, confidential, or restricted (pay, health,
performance, investigations, personal records). Labels come from the source
first (Drive labels, Snowflake and BigQuery tags and policy tags), then from
a classifier that proposes labels for owner review. A corpus has a default
label and a ceiling.
Sensitive agents. An agent declares the highest label it may read. An HR
agent declares restricted; granting it a restricted corpus needs a
workspace admin and a stated purpose. A run that reads restricted content:
- is a private session by default, visible only to the person it serves and the people they add;
- takes the highest label it touched (taint), which follows its transcript, artifacts and memory;
- cannot post its content to a Slack channel or a GitHub thread, only a pointer back to the private session;
- never writes what it read into shared agent memory.
Self access. A person may always read records about themselves, even in
a restricted corpus, through an agent that serves them. The platform passes
the person's identity to the source (Snowflake row access policies, BigQuery
row-level security) or applies a subject = reader filter on indexed
documents. So "how much parental leave do I have left?" reads their own
balance and the policy that governs it, and nothing about anyone else.
Access is visible. Reads of restricted content are audited with the agent, session and purpose. A person can see who and which agents read records about them.
Sessions in the graph. Sessions are Document nodes too (RFC 0009 message history): who took part, which documents they cited, which topics and systems they touched. A private session is visible only to its participants, so "what did I decide about the offer last week?" finds it for them and for no one else. Shared sessions surface as related context for a project or topic, under the same access rules.
Personal corpora. A person can make corpora from their own connections
(their mail, calendar, notes, their own Drive). A personal corpus is owned by
that person, is isolated, can be granted only to sessions they run, and is
never visible to admins' search. Personal and work context use the same
machinery, so one assistant can help with "prepare for my 1:1" by combining
the person's calendar, their private notes, the team's project docs and the
relevant metrics, each under its own access.
8. Consolidation¶
- One document, many origins. The same content reached through two members is stored once, keyed by a content hash, with every origin kept for citations and access checks.
- Versions, not duplicates. A re-sync of a changed document replaces its passages and graph facts and keeps the prior version for the audit window.
- Graph merges are the maintenance agent's job (section 5).
9. Freshness¶
| Field | Meaning |
|---|---|
mode |
schedule, push (the source's webhook), or on_demand |
target |
How stale a member may get, for example 24 h |
cursor |
The source's change token or ETag, so a sync reprocesses only changes |
Freshness is about syncs: is the copy current with the source? Staleness (section 5) is about content: is the document itself still true?
Each member shows Fresh, Stale (since when), Syncing or Failed; the corpus rolls them up. A referenced member is Live, with the last successful probe. A failed sync or a missed target raises an alert in the corpus owner's session (RFC 0009 alerts), with Retry.
10. Validation¶
| Offline | Online | |
|---|---|---|
| When | On a schedule, and before a recipe, member or ontology change is saved | As each search runs |
| Measures | A golden set per corpus (questions with expected documents or facts): recall at k, precision, citation accuracy, graph fact accuracy | Empty results, low top scores, strategy timeouts, thumbs down on answers that cited the corpus, citations never opened |
| Result | A score per version on the change card; a regression blocks the change without an override | Corpus health; an alert when it degrades |
Golden cases come from real use: a thumbs down with a reason on an answer that
cited the corpus becomes a candidate case. Checks are deterministic unless a
case says it is judged by a model, which is labelled. This reuses the agent
eval machinery (pkg/evals).
11. Console¶
Agents › Knowledge lists corpora as cards: name, members as tags, document count, freshness, a Shared or Isolated pill, a Semantic pill when that layer is on, and the agents granted it. ⋯ holds Edit, Sync now and Delete. New corpus asks for a name, members (from Connections › Data sources, or uploads), and which agents may use it; recipe, ontology and evals have sensible defaults and live on the corpus page.
The corpus page has tabs: Documents (browse, remove, re-sync), Members (freshness per member), Ontology (types, proposed changes to accept, JSON Schema view), Graph (this corpus's slice: health and an explorer), Evals (golden set, scores), and Access (readers and granted agents), with a Try a search box that runs the same tool an agent would. Documents show owner and staleness as pills, with Confirm owner and Hand off in ⋯.
Agents › Knowledge also has a Graph view of the whole workspace graph: health, open proposals from the maintenance agent, and an explorer that starts from a person, topic or system. Each person's profile has "What the graph says about you": documents attributed to them, topics they own, and inferred expertise, each with Correct and Remove.
Where it lands¶
| Piece | Repository |
|---|---|
Corpus, Member, Recipe, Ontology records on pkg/state |
humagen pkg/knowledge |
| Core ontology, structural layer, ownership and staleness scoring | humagen pkg/knowledge |
| Labels, taint, private sessions, self-access filter, access log | humagen internal/platform and pkg/knowledge, labels from RFC 0005 connectors |
Warehouse records: schema sync, <corpus>_sql under the reader's identity |
humagen connectors (RFC 0005), Snowflake and BigQuery first |
| Maintenance agent and its proposals | humagen built-in agent, proposals as RFC 0009 alerts |
| Sync scheduler, cursors, freshness and alerts | humagen internal/platform |
| Recipe executor and merge | humagen pkg/knowledge, composing saige rag retrievers, fusion and reranker |
Federated retriever interface for live strategies |
saige rag (new), connectors from RFC 0005 |
| Semantic extraction, entity resolution, provenance, corpus filter on traversal | saige rag/knowledge/graph (group is the workspace; corpus membership is a new node and edge property) |
Corpus tool builder and agent grant knowledge.corpora |
humagen internal/platform, spec field in pkg/spec |
| Retrieval evals | humagen pkg/evals |
API /api/corpora and the console pages |
humagen api/openapi.yaml, apps/web |
Order¶
- Corpus records, members (indexed only),
lexicalplushybrid, the<corpus>_searchand_lookuptools, the agent grant, the Knowledge cards. - Referenced members with
liveandcatalogstrategies and source-native tools (Snowflake and BigQuery first, then Notion and Drive). - Freshness: modes, cursors, targets, alerts.
- Sensitivity: labels from sources, agent ceilings, private sessions and taint, self access, the access log. This lands before any restricted corpus (HR) is offered.
- Offline and online retrieval evals and the change card gate.
- Structural graph: core ontology, Document, Person and Team nodes from
metadata, ownership suggestions and confirmation, staleness scores,
document-level search,
<corpus>_graph. No model calls. - Maintenance agent with proposals and auto-apply rules; health; the "what the graph says about you" page.
- Sessions as graph nodes, personal corpora.
- Corpus ontologies (declared JSON Schema, then inferred, then hybrid),
metadata filters on search, and the semantic layer: topics, mentions,
contradictions,
graph_searchacross corpora.
Each step ships on its own and is verified on platform.withhumagen.com.
Risks¶
- Cost. Graph extraction and HyDE spend model tokens. Opt-in per corpus, estimated before enabling, metered on the ledger.
- Leaks through metadata. Table names and ontology properties can be sensitive. Catalog visibility rules from RFC 0005 apply to referenced members and graph nodes built from metadata.
- Restricted data in the wrong place. Taint follows every output of a run that read restricted content, and only pointers leave private sessions. Warehouse rows are never copied.
- People data. Inferred ownership and expertise describe employees. Person changes always need approval, people see and correct what the graph says about them, and the workspace can turn expertise inference off.
- Agent overreach. The maintenance agent only auto-applies reversible, low-risk changes; everything else is a proposal, and its runs are audited.
- Leaks across corpora. Traversal filters every hop by the reader's access, and isolated corpora never link out.
- Ontology drift. Inferred types can sprawl. They stay proposed until an owner accepts them, and validation keeps unfit facts out of the graph.
- Live latency. Online strategies run under a per-strategy timeout and drop out rather than slowing every search.
- Stale answers. Freshness targets and alerts make staleness visible instead of silent.