stather (v.)

Architecture

Three pictures: how a document becomes a served store, how a question becomes an answer, and how many users read one store without ever sharing a file.

1. The factory: from a document to a served build

Ship

deterministic build id, provenance written, ledger: candidate

atomic swap of the serving pointer, ledger: shipped

validate the live store; failure rolls back

Gates, fail closed

reachability: every indexed node can be queried

completeness and term quality

smoke query against the built store

The store, memory-mapped

node payloads, log-structured

fixed-width node index

full-text index and a compressed edge store for relations

Ingest, one Temporal activity per shard

corpus adapter: segment each document into derivative nodes

node id = hash of its content, so the id is the dedup key

every node carries its origin document and a provenance ledger

Harvest

public datasets, arXiv e-prints, local corpora, allow-listed tiers only

hygiene gates: de-LaTeX, entropy, the prose-to-source byte invariant, zero-yield guard

raw shards, stable content ids

query service reloads a fresh view

additive run: only net-new or changed shards, from the ledger diff

A baseline build ingests every tier; an additive build ingests only what the ledger says is new or changed, then rebuilds the indexes over the cumulative store. A build that fails a gate is never registered. The build id is a canonical hash of the output, and a rebuild from the same input reproduces it byte for byte, which is how “the same dataset” is proven rather than asserted. Shipping is a pointer swap validated by re-opening the live store; if the validation fails, the previous build is re-activated before the failure is reported.1

2. The query path: from a question to a cited answer

small local modeledge store, one hopnode store, memory-mappedfull-text indexquery serviceuser or agentsmall local modeledge store, one hopnode store, memory-mappedfull-text indexquery serviceuser or agentopt[expand]a new build is a fresh re-open of the store, never an in-place reloadquestion, k, expand?tokenise the same way the index was builtranked search over node termstop k node ids with scoreshydrate the k payloads by offsetnodes with origin document and provenanceone-hop neighbours of each hitrelated nodes with edge typeranked, sourced contextreason over the retrieved context onlyanswer, with the node ids it used

The knowledge is never inside the model. The model reasons over a small, sparse context that was retrieved and can be inspected; every node carries the document it came from, so an answer is traceable to source and a wrong node can be corrected, retired or re-minted without touching a weight. Bench figures for this path are on Stages, measured.2

3. Many readers, one store: the layered stack

only ever to the top layer

a query from a principal

resolve the stack for this principal

user overlay: session context, interim results, user-derived nodes. Read-write, owner only

organisation overlay: the company's own corpus, sanitised and derivatised. Read for members, write for the ingest role

entitled data-packs: read-only, gated by entitlement

domain base stores: maths, science, ML. Read-only, shared by everyone

answer assembled nearest-layer-first: on an id collision the nearest layer wins

writes

Every layer is physically its own directory in the same on-disk format, stacked into one read view at query time. That is what makes isolation cheap: offboarding a tenant, honouring a deletion request, or backing up one organisation is “remove or snapshot the directory”, never “compact a shared store”. A domain base, a data-pack and a customer overlay are the same type of thing, a store root in the stack, which is why one mechanism serves all three. The admission-control work that keeps one slow tenant from stalling the others is measured on Stages, measured, stage 9.3

Sources

Numbers on this page are quoted from the lab's own records. The records are private; each note gives the record's date.

  1. lab record
  2. lab record, 2026-08-31
  3. lab record