capture-ledger

Three roles

A crawl is three jobs, and each one belongs to something different.

Role	Owner	What it does
Decide and record	capture-ledger (this repo)	decides which URLs get captured, and keeps the ledger of what came back
Submit and report	capture-scheduler (a Windmill flow)	picks a free BrowserHive, asks it to capture, and reports each level back
Capture and store	BrowserHive	opens the page in a browser, captures it, and writes the artifacts to the bucket

flowchart LR
  U((Caller)) -->|asks for a crawl| L["capture-ledger<br/>decide · record"]
  L -->|one level at a time| S["capture-scheduler<br/>submit · report"]
  S -->|Capture| B["BrowserHive<br/>capture · store"]
  B -->|artifacts| O[(bucket)]
  S -->|level report| L
  O -.->|result manifests| L

What capture-ledger is

capture-ledger answers one question: which URLs get captured. A caller asks for a crawl; capture-ledger works out the seeds — an explicit list, or the enabled rows of a Postgres capture_targets table — decides what is in scope, and hands one level at a time to a Windmill flow that does the submitting.

What is not capture-ledger’s responsibility

Driving a browser, taking screenshots, building a WACZ, writing to S3 — every one of those is BrowserHive’s responsibility. BrowserHive’s docs are not published on the web — build them from the BrowserHive checkout with pnpm run docs:local.

Submitting is not capture-ledger’s either, any more. The Windmill flow in capture-scheduler is what calls BrowserHive (POST /captures), and capture-ledger has no BrowserHive address in its configuration at all. Each level the flow reports comes back with a taskId per page, and capture-ledger registers what produced an archive in the archive ledger. Anything the level handler failed to admit is picked up later by pnpm run fga:reconcile, which reads the manifest keys those reports recorded.

Where things are

I want to…	Page
bring the stack up for the first time	Quickstart
set up the dev environment	Development environment
manage the URL list	URL source
shape how a crawl captures	Capture options
move to a newer BrowserHive	Upgrading BrowserHive
understand the moving parts	Architecture