Resumable work

Named steps, cursor checkpoints, and transactionally guarded side effects.

Named steps allow a retried job to skip work that completed in a previous attempt. Headgate writes the in-progress checkpoint before side effects and rechecks the lease fence at every boundary.

Named steps

Rust

ctx.step("download", || async { download().await }).await?;
ctx.step("transform", || async { transform().await }).await?;
ctx.step("publish", || async { publish().await }).await?;

Inspect a checkpoint

Checkpoint and cursor state is intentionally absent from ordinary job/list responses. Cursor bytes can contain application data, so operators request it explicitly:

GET /api/v1/jobs/{job_id}/checkpoint

The response contains the ordered completed steps, the step checkpointed before its side effects, cursor step and base64 cursor bytes, step schema/hash, and crash counts by step. An existing job that has never used resumable steps returns an empty checkpoint; 404 means the job itself is missing.

The console loads this endpoint only when job detail is opened. Its Resumable execution section shows completed and current steps, renders JSON/text cursors when possible, keeps binary cursors in base64, and makes per-step crash attribution visible. It cannot show future handler steps because those definitions live in worker code and are not persisted with the job.

Cursor steps

Cursor steps resume a loop from durable application progress. Each cursor update is fence-verified.

Rust

let cursor_ctx = ctx.clone();
ctx.step_cursor("pages", move |raw| async move {
    let page: usize = raw.as_deref()
        .map(serde_json::from_slice).transpose()?.unwrap_or(0);
    process_page(page).await?;
    cursor_ctx.set_cursor(serde_json::to_vec(&(page + 1))?).await
}).await?;

Side effects exactly once

step_once / StepOnce claims an effect key and commits the application’s transaction with the completion checkpoint. It requires a transactional backend; Redis declines it.

Changing the step set during a deployment requires a declared mapping. An unknown step set becomes undecodable rather than silently restarting at step one.