About the archive
How it works
Three layers, in dependency order, and nothing skips a layer.
Retrieval
One script per source, each idempotent, each writing a faithful copy and a provenance sidecar beside it. Nothing is downloaded twice and no data file is written by hand.
Integration
Derived products that fuse sources: a gazetteer of names, a concordance over the whole OCR corpus, the cartographic system the plates are drawn in.
Knowledge base
Markdown articles with structured frontmatter, validated and compiled to RDF with an OWL ontology and a SKOS concept scheme. An article without a source fails the build.
The boundary between the second and third layers is the one that matters. A machine-generated candidate — "this index card may concern this person" — is a hypothesis, and hypotheses stay in the middle layer, carrying their evidence, cited by nothing. Only a human moves a finding up into the encyclopedia, and it is re-cited to the primary record when they do. A guess in a graph is indistinguishable from a claim to everything downstream of it.
What provenance means here, concretely. Beside a retrieved file sits a .provenance.json recording the source URL, the retrieval timestamp, the SHA-256 of the bytes, the file size, the licence or rights as the holder stated them, and — the field that does the most work — the exact query that produced it: the Overpass QL, the bounding box, the chart number, the census geography clause. A result can therefore be reproduced or challenged years later.
Coverage is graded rather than binary, and the grades are published. Across the 18,709 catalogued files:
- Sidecar
- 13,780 a
.provenance.jsonsits beside the file. - Inherited
- 3,597 no sidecar of its own; traced to the scanned original it was made from. Traceable, but nothing beside the file records it.
- Inline header
- 1,323 provenance is written into the file's own leading lines, carrying the source URL.
- Unresolved
- 9 no provenance could be resolved by any route.
The last row is not rounded away: 9 files in 7 works cannot be traced, and the catalogue names them:
Daily Northern Islander · Northern Islander · cihm 07807 · indiannamesofpla0000kelt · indiannamesofpla00kelt · NOAA bathymetry · newspapers com
Reachability is surveyed too, and dated: 618 of 627 works had their source host still answering when it was last checked, across 63 hosts. A host that answers is not proof the document is still there; a silent connection and a 404 are different findings, and the survey keeps them apart. Some hosts are never contacted at all: a robots file that disallows crawling is honoured, and the material behind it is either obtained another way or recorded as not held. The bibliography's coverage note gives the per-host results and the method behind them.
Every figure on this page is computed at build time from the catalogue export.