Skip to content
capture-ledger Docs

capture_submissions

This records who a capture was submitted for, at the moment it is submitted.

    // BrowserHive のタスク id が、結果の報告へ戻る join の鍵になる。
    .addColumn("task_id", "uuid", (col) => col.primaryKey())
    .addColumn("correlation_id", "text")
    .addColumn("org_id", "text", (col) => col.notNull())
    // それを求めた利用者。居た場合に限る。人ではなく組織に属する定期実行では NULL。
    .addColumn("submitted_by", "text")
    .addColumn("submitted_at", "timestamptz", (col) => col.notNull().defaultTo(sql`now()`))

Where each capture’s result manifest is was added in 014:

    // bucket を除いた鍵。報告が運んだ場所をそのまま —— 台帳は組まない。
    .addColumn("manifest_key", "text")
    // 書けなかった理由、または台帳には読めない場所だった理由。
    .addColumn("manifest_error", "text")

Why it is needed

Section titled “Why it is needed”

BrowserHive has no notion of an organization. capture-ledger passes it a URL and some settings, and gets back a taskId.

When the ledger is later filled from a .result.json in the bucket, that manifest carries nothing identifying an organization. Writing it down at submission time is the only source.

what capture-ledger knows          what BrowserHive knows
─────────────────          ──────────────────────
capture_targets.org_id ──┐      taskId
                         │      where the artifacts are
                         └──→   (no idea about organizations)
              capture_submissions
              maps task_id → org_id

Without this row the reconciler cannot say who an archive belongs to.

How it is written

Section titled “How it is written”

The level report handler (src/api/crawls.ts) writes a row for every reported page that carries a taskId — failures included — together with where that capture’s .result.json was written:

submitted.map((r): Insertable<CaptureSubmissionsTable> => ({
  taskId: r.taskId,
  correlationId: r.correlationId ?? crawlId,
  orgId: crawl.orgId,
  submittedBy: crawl.requestedBy,
  manifestKey: manifests.get(r.taskId)?.key ?? null,
  manifestError: manifests.get(r.taskId)?.error ?? null,
}));

capture-ledger never spells a manifest key. BrowserHive (or the sink) answers Capture with the location it wrote the manifest to, and the Windmill flow relays it as manifestLocation. The handler keeps the key part when the location is in the configured bucket. A capture that could not write its manifest is reported with manifestError instead, and that reason lands here; so does a location in another bucket, or one that is not s3:// — the reporter does not get to choose where the ledger reads.

A report that carries a taskId without exactly one of the two is refused with 400: accepting it would leave a capture with nothing to read.

A taskId is minted by BrowserHive and never reused, but a level can be reported again, hence onConflict … doNothing().

How it is read

Section titled “How it is read”

reconcile.ts reads the rows that carry a manifest_key and have no archive yet:

capture_submissions row with manifest_key, no archives row
  → GetObject(manifest_key)
  → register in archives, and queue tuples with that org_id and submitted_by

Column notes

Section titled “Column notes”
Column Note
task_id Primary key. BrowserHive’s id, the join back to the result report
org_id not null. The reason this table exists
submitted_by The user who asked, when there was one. NULL for scheduled runs that belong to an organization rather than a person
submitted_at Default now()
manifest_key Where the capture’s .result.json is, minus the bucket, exactly as reported. NULL when there is a manifest_error, and for rows older than 014
manifest_error Why there is no manifest the ledger can read: the write failed, or the reported location is not in the configured bucket