Headgate provides at-least-once execution. A worker owns a job only while its lease ID and
fencing token remain current. Every completion, checkpoint, and transactional effect
verifies that identity so an expired worker cannot overwrite a newer attempt.
Returned errors and crashes are different
A returned error consumes attempt. A panic, process loss, or expired lease consumes
crash_attempt. Poison-pill quarantine depends on that distinction. Panic recovery and
per-task isolation are enabled by the runtime so one handler cannot take down unrelated
work.
Retries use bounded backoff and store time. rate_limited and configured non-consuming
errors requeue without consuming an attempt. snooze reschedules intentionally, while
skip and revoke are terminal decisions.
Timeouts and deadlines
A task timeout bounds one attempt. A deadline bounds the useful lifetime of the job across
attempts. The runtime cancels the handler when either expires, on shutdown, on operator
cancellation, or when renewal reports a lost lease.
Treat context cancellation as a hard stop. Fencing protects Headgate state, but code that
ignores cancellation may continue external side effects after ownership has moved.
Graceful shutdown and lease control
During graceful shutdown the worker stops admitting new jobs, lets in-flight work finish
within the configured window, and keeps renewing those leases. Handlers for genuinely long
operations can use the lease-control surface and report task tracking state. Sticky routing
can prefer a named worker while retaining a fallback policy when it is unavailable.
Terminal and suspect-job callbacks
archived is the operational dead-letter queue for jobs that exhaust retries, are
skipped, or pass their absolute deadline;- death handlers observe those jobs only after the archive transition is durable;
- stuck-job handlers receive runtime evidence about work that is no longer progressing;
- a suspect job can be moved behind healthy work without changing its attempt semantics;
- redrive returns selected archived jobs to execution through bounded operator operations;
- orphan provenance identifies jobs reclaimed after a crash without inventing a second
lifecycle state.
Use isolated execution when application code needs a process boundary. Use resumable steps
for durable intra-job progress and idempotency helpers when application effects must commit
with a fence-verified completion.
