Architecture

Stage 0 — where the project is today

flowchart LR
  U((caller)) -->|"POST /api/crawls"| W
  PG[("Postgres<br/>capture_targets / crawls / crawl_pages")] <--> W
  subgraph W["capture-ledger (Node 24 / TypeScript)"]
    direction TB
    API["api/crawls.ts<br/>rules, dedup, budget"]
    DIS["crawl/dispatch.ts<br/>one webhook POST"]
    API --> DIS
  end
  DIS -->|"webhook + token"| WM["Windmill flow<br/>(capture-scheduler)"]
  WM -->|Capture| BH1
  WM -->|Capture| BH2
  subgraph BH["BrowserHive — one process per browser"]
    BH1[browserhive-1]
    BH2[browserhive-2]
  end
  BH1 -->|CDP| C1[chromium-1]
  BH2 -->|CDP| C2[chromium-2]
  BH1 -->|PutObject| S3[(SeaweedFS<br/>bucket: browserhive)]
  BH2 -->|PutObject| S3
  WM -->|"POST /api/crawls/:id/pages"| API
  S3 -.->|".links.json / .result.json"| API

From submission to outcome

capture-ledger does not talk to BrowserHive at all. The Windmill flow calls POST /captures, waits for the answer, and reports; capture-ledger plans the next level and records what happened. There is no BrowserHive address in capture-ledger’s configuration at all. Each BrowserHive drives one browser and runs one capture at a time — a busy one refuses with 429 Busy — so picking a free endpoint is the flow’s job too.

One level is one round trip. The flow captures the URLs it was handed and posts POST /api/crawls/:id/pages with a taskId and a status per page. capture-ledger writes crawl_pages, attributes the submissions, filters the discovered links, applies the budget, and answers with the next level — or with a stopReason.

The level report does not carry artifact locations, so capture-ledger re-reads the {taskId}_{correlationId}[_{labels}].result.json manifest BrowserHive writes next to the artifacts. That file survives a BrowserHive restart, which is what makes it usable here.

Consequences worth knowing:

• A page reported failed may still have produced an archive. If the result aged out of BrowserHive’s cache while the flow was waiting, the flow sees NOT_FOUND and reports a failure — but the manifest is in the bucket, so capture-ledger admits it anyway and corrects the crawl_pages row.
• A manifest that is not there yet is skipped rather than raised. Throwing would turn a late write into a 500 on the level report, stopping the crawl; the reconciler picks it up instead.
• capture_submissions is written for every page that carried a taskId, not only the successful ones — attribution and the manifest key are what the reconciler later needs.

The ledger and authorization

The diagram above stops at submitting. Once a capture finishes, capture-ledger puts the result in the ledger and hands out signed URLs only to those allowed to read it. That is where OpenFGA comes in.

flowchart LR
  S3[(SeaweedFS<br/>bucket: browserhive)]
  subgraph W["capture-ledger"]
    direction TB
    REG["crawl/admit-level.ts<br/>admit what a level captured"]
    RC["reconcile.ts<br/>read recorded manifest keys"]
    OW["outbox-worker.ts<br/>deliver tuples"]
    API["api/routes.ts<br/>signing endpoint"]
  end
  S3 -.->|.result.json| REG
  S3 -.->|.result.json| RC
  PG -.->|pending manifest keys| RC
  REG --> PG
  RC --> PG
  PG[("Postgres<br/>archives + fga_outbox")]
  PG -->|FOR UPDATE SKIP LOCKED| OW
  OW -->|Write tuples| FGA[("OpenFGA<br/>its own Postgres")]
  U((caller)) -->|"POST /api/archives/:id/url"| API
  API -->|"Check can_view<br/>+ contextual tuple"| FGA
  FGA -->|allowed| API
  API -->|"signed URL (300s)"| U
  U -.->|GET| S3

How to read it:

• archives and fga_outbox are written in one transaction. No transaction spans Postgres and OpenFGA, so the tuple goes into the same box as an intent to send, and outbox-worker retries until OpenFGA accepts it.
• OpenFGA has a Postgres of its own. Not sharing capture-ledger’s makes the boundary visible: a tuple write cannot join the application transaction — which is exactly why the outbox exists.
• Membership is not stored. Which organizations a caller belongs to arrives from their token and is passed as a contextual tuple on every Check.
• Two paths lead into the ledger: a crawl admitting what a level captured (fast, src/crawl/admit-level.ts), and reconcile reading the manifest keys the level reports recorded (picks up what the fast path failed to admit). Both go through admitArchive, and (bucket, object_key) is unique, so reaching the same capture twice is a no-op.

See Archive ledger for the details.

Module layout

Module	Responsibility
src/api/server.ts	The only entry point. Reads the capture settings once, registers the routes, drains the outbox on a timer.
src/api/crawls.ts	Start a crawl, report a level, hand back the next one. Authorization, policy, and record-keeping.
src/crawl/	Scope, budget, level admission, and the Windmill dispatch.
src/config/capture-formats.ts	CAPTURE_LEDGER_CAPTURE_FORMATS / _SIGNING → the captureFormats every dispatch carries.
src/data/url-source.ts	The one SQL query over capture_targets. Returns { url }[] for the organization asking.
src/archive/	Manifests, admission into the ledger, and the reconciler.
src/rpc/generated/	BrowserHive’s contract (openapi.json, copied from the submodule) and the types made from it. The manifest is a CaptureResultReport, checked against it before the ledger reads it. Never edited by hand.
src/db/	Pool, Kysely wiring, migrations, seeds.

Error handling

Telling a rejection apart from an outage is the flow’s job now — it is the side that calls BrowserHive. capture-ledger’s half is what happens to the crawls row.

A dispatch that cannot be delivered closes the row as failed with the message on it. It is never left running: the partial unique index means an unclosed row blocks every later crawl, and nothing in the row distinguishes a live crawl from an abandoned one.

Unexpected failures on the API surface as a bare 500 { "error": "internal error" }. Fastify’s default would return err.message and err.code, and Postgres puts column types and mangled values in there — a client-side mistake would become a window onto the server’s internals. 4xx bodies pass through unchanged: those are written for the caller.

Network layout

The stack runs on Apple Container. The project name is the DNS domain, so every service is <service>.capture-ledger, and the containers reach each other by that name. capture-ledger itself runs on the host and uses the host ports below (published on 127.0.0.1).

Service	DNS name	Container port	Host port
postgres	postgres.capture-ledger	5432	5432
chromium-1	chromium-1.capture-ledger	9222 (CDP)	9222
chromium-2	chromium-2.capture-ledger	9222 (CDP)	9223
browserhive-1	browserhive-1.capture-ledger	50051 (HTTP)	50051
browserhive-2	browserhive-2.capture-ledger	50051 (HTTP)	50052
capture-ledger	— (runs on the host)	—	—

The artifact store (SeaweedFS) is not in this stack. It is one store shared by the three crawler repos, at seaweedfs.crawler-storage:8333 (filer on :8888). It is brought up by seaweedfs’s scripts/stack.sh — the submodule works too: .upstream/seaweedfs/scripts/stack.sh up. To empty it, pnpm run store:wipe.

There is one compose file. The one-shot migrate/seed jobs are not services: scripts/prod-smoke.sh invokes them with container run --rm, because container-compose has no run subcommand. The image’s ENTRYPOINT is node dist/api/server.js, so those jobs override it.

Bucket creation is not a separate service. BrowserHive’s SeaweedFS entrypoint spawns its init-bucket.sh alongside weed server, and its retry loop does the waiting. browserhive is given WAIT_FOR_S3; its entrypoint polls that URL until the S3 listener answers before starting the server, because the boot HeadBucket is fatal and depends_on orders starts only.

Upstream pinning

capture-ledger tracks a fixed BrowserHive version through one submodule pointer — see Upgrading BrowserHive, which reads the current value out of the repository rather than restating it.