Engineering write-ups, mostly about the parts that were harder than expected.
We started with a batch importer and spent six weeks regretting it. The problem was never throughput — it was that a partial failure left the graph in a state that no retry could reason about. This is the story of moving to cursor-based incremental pull, and the two invariants that made idempotent upserts actually safe.
The easiest thing to do when indexing a source system is to collapse everything into a single "can this user see it" bit. It is also the fastest way to leak. We keep the original permission shape and evaluate it at query time, which costs more and is worth it.
An answer without provenance is a rumour. Every response from our query surface carries the objects it was assembled from, and we refuse to render an answer whose citations cannot be resolved. That constraint shaped the whole retrieval design.