Following links

capture-ledger’s POST /api/crawls takes seed URLs — or fromTargets, the enabled rows of capture_targets — and hands capture-scheduler one level at a time. The flow is f/waggle/crawl_level; one execution is one level.

plan_level       group by host, fetch robots.txt once per host
  ↓
compile_scripts  type-check the catalog's TypeScript and turn it into JavaScript (ts-compile-service)
  ↓
for-each         parallel across hosts (parallelism = host_parallelism)
  crawl_host       sequential within a host: finish → wait → next
  ↓
report_level     report to capture-ledger, receive the next level
  ↓
index_level      ask capture-ledger to index what landed in the ledger

compile_scripts is the one step that calls a service outside the stack’s own repositories: ts-compile-service, which runs as the ts-compile service of this repo’s compose. If a single script fails the type check, the service answers 422 and the step throws — the flow drops into fail_crawl and the crawl ends as failed, with the number of diagnostics and the first five in its reason. The reason is cut to the ledger’s limit of 2,000 characters: above it the ledger refuses the report, and the crawl would stay running and block every crawl after it. A script that does not type-check never reaches a page.

capture-ledger does the repeating. This flow ends after one level. Putting the loop in Windmill was tried: a minimal flow with stop_after_if ran 643 iterations without stopping. A mechanism for not overloading someone else’s server does not belong on top of a loop that can run away. The stopping conditions live in capture-ledger and have unit tests.

Politeness is enforced by the shape of the loop

Windmill CE’s per-key concurrency limit cannot be used — see Windmill CE for why it silently passes everything through. What does work is the for-loop’s parallelism, so politeness is expressed as loop structure:

• across hosts — the for-loop’s parallelism
• within a host — crawl_host runs sequentially and waits after each completion
• across levels — capture-ledger passes the time it last finished touching that host, and plan_level subtracts it

The gap goes after completion, not before submission

This is the part that looks like a detail and is not.

How long a capture takes is not known up front — one page is two seconds, the next is two minutes. So a gap measured from the submission says nothing about the gap the other server feels: the next request can land the moment the previous capture ends. A gap on the submitting side never reaches the other server.

The gap only becomes real when it sits between one page finishing and the next being submitted:

  // **段をまたぐぶんの待ち。** 前の段でこのホストを触っていれば、その完了からの経過を
  // 差し引いた残りをここで待つ。これが無いと段の境目だけ間隔が空かない (実測 521ms)。
  if (initialDelayMs > 0) {
    log(`前の段からの間隔を空ける: ${String(initialDelayMs)}ms`);
    await sleep(initialDelayMs);
  }

  for (const [index, url] of urls.entries()) {
    // **間隔は完了の後。** 1 件目の前は上で済ませてある。
    if (index > 0 && perHostDelayMs > 0) await sleep(perHostDelayMs);

The third rule was added later

Without it, the gap held within a level and vanished between levels. Measured against a 3000 ms setting, the boundary collapsed to 521 ms.

plan_level computes what remains, and takes robots.txt seriously in both directions — if the site asks for longer, the site wins:

    // **長いほうを採る。** 相手が言っている値をこちらの都合で縮めない。
    const crawlDelaySec = robots?.getCrawlDelay(USER_AGENT);
    const delayMs =
      typeof crawlDelaySec === "number" && Number.isFinite(crawlDelaySec)
        ? Math.max(per_host_delay_ms, Math.round(crawlDelaySec * 1000))
        : per_host_delay_ms;

    // 前の段でこのホストを触り終えてからの経過を差し引く。
    const lastFinished = list
      .map((c) => c.lastFinishedAt)
      .find((t) => t !== null && t !== undefined);
    const sinceMs =
      lastFinished === undefined || lastFinished === null
        ? Number.POSITIVE_INFINITY
        : Date.now() - new Date(lastFinished).getTime();
    const initialDelayMs = Number.isFinite(sinceMs) ? Math.max(0, delayMs - sinceMs) : 0;

Note Math.max, not Math.min. We do not shorten what the other side asked for. The two-sided test in test/plan-level.test.ts exists because an implementation that simply always used robots’ value passes a one-sided test.

One BrowserHive holds one browser

BrowserHive has no queue and no pool: POST /captures is one round trip, and a server that is already capturing refuses with 429 Busy. Picking a free server is therefore crawl_host’s job. It reads the list from u/admin/browserhive_endpoints, tries them in turn — a busy one means “next”, an unreachable one is skipped for the rest of that call — and when every one is busy it waits half a second to a second and a half and goes round again. Only when none of them is reachable does it throw ServerUnavailable, which fails the level.

Two consequences:

• host_parallelism is capped by the number of endpoints. plan_level returns parallelism = max(1, min(host_parallelism, endpoints)) and the for-loop uses that. More parallel hosts than browsers would only mean more hosts waiting on busy.
• Retries live here now. BrowserHive no longer retries on the server side — a retry there would bypass the gap measured here. A capture that fails with a transient error (connection, timeout, internal) is tried once more, after the same gap as any other request to that host. A failure to store the artifacts (artifact_sink) is not — what is broken is the store, and capturing the page again would only write to the same store. Nor is a failure to keep the URL policy (policy, from BrowserHive v18.0.0): a page that opened a WebSocket matching deny would open it again, and the handshake cannot be stopped.

Settings come from variables, not arguments

crawl_host, report_level and index_level do not take their connection settings as arguments. Windmill fills schema defaults only for UI-triggered runs, so a webhook run gets nothing — the BrowserHive target arrived undefined and the job died with “Channel target must be a string”.

What does arrive as arguments is what capture-ledger decides for that crawl and always sends: the URLs, the delay, capture_formats / signing, and scripts — the TypeScript to run inside each page. The ledger keeps it as written (its sha256 is over the TypeScript bytes); compile_scripts turns it into JavaScript and re-hashes it, and BrowserHive checks the JavaScript’s hash. The two hashes meet at that one step. report_level then carries the JavaScript hash, with the compiler version and the host-types tag the service answered with, back to capture-ledger (compiled in the level report). The ledger writes them next to the TypeScript hash on the crawl, and refuses a later level that reports a different hash (409). An empty script list (scripts: [] on the crawl) is a valid level too: the service answers 200 with no scripts, the page runs nothing, and the report still carries typescript and hostTypes. A script written on the spot (an inline element of scripts) travels the same way: capture-ledger pins it on the crawl as inline-<name>, and the flow compiles and injects it like a registered one.

scripts is required for the same reason capture_formats is. BrowserHive v11.0.0 holds no roster of its own: send nothing and nothing runs inside the page, and the capture still succeeds and still produces an archive. A forgotten argument would be a crawl that never scrolled, indistinguishable from one that did. This flow only carries them — what runs, in what order, and in which of BrowserHive’s two injection points (behavior after load, preload before navigation) is decided by capture-ledger.

pnpm run windmill:capture-ledger-token   # waggle_token / waggle_api_url /
                                 # browserhive_endpoints / browserhive_tls_ca /
                                 # ts_compile_url

u/admin/ts_compile_url is where compile_scripts sends the scripts (default http://ts-compile.capture-scheduler:8080, the ts-compile service of this repo’s compose; override with TS_COMPILE_URL). pnpm run doctor checks that the worker can reach its /healthz.

u/admin/browserhive_tls_ca is the CA certificate (PEM) for https:// endpoints of BrowserHive, and an empty value means plaintext. The variable is written even when it is empty, on purpose: if it were simply absent, getVariable would throw, and a script that reads “could not fetch it” as “no TLS wanted” turns a read failure into a plaintext connection. There is no “TLS with the system roots” mode — BrowserHive’s TLS assumes a private CA, and needing a public certificate would mean the server is on the public internet. The development stack is plaintext.

Nothing else is copied from BrowserHive. crawl_host speaks its HTTP API (POST /captures with a JSON body, since BrowserHive v12) through fetch, so there is no contract file to keep in sync — until v0.22 this repo carried a copy of BrowserHive’s gRPC .proto as a Windmill resource, with a guard to catch it rotting. The endpoints are therefore http(s):// URLs; crawl_host refuses the old host:port form and names the variable.