Enqueueing

Create jobs safely with IDs, uniqueness, transactions, policy, and producer hooks.

The producer client is the validated boundary for application enqueueing. It validates the entire batch before store I/O, derives fingerprints, runs authorization and middleware, and reports typed duplicate or availability errors.

Choose the enqueue boundary

Need	Use
One or more jobs	Native client enqueue with a batch of envelopes
Same commit as application data	A PostgreSQL or MySQL transactional adapter
Service-to-service producer	POST /jobs or POST /jobs/bulk with Idempotency-Key
Wait for one result	Subscribe before enqueue, then reconcile durable state
Recurring work	A periodic definition rather than an application timer

Bulk enqueue is one atomic request. If validation or authorization rejects one member, Headgate writes none of its siblings.

Enqueue from TypeScript or Python

Headgate does not require the producer to use the Rust or Go SDK. A Node.js, NestJS, Python, or other service can call the control API. Send the task arguments as bytes in standard base64; for JSON arguments, UTF-8 encode the JSON first and then base64-encode those bytes. Headgate derives the content fingerprint on the server.

Use a stable Idempotency-Key for one logical enqueue operation. Retrying with the same key returns the original job instead of creating another one. Use a new key when you intend to create a new job, even when its payload is identical.

TypeScript

const headgateUrl = process.env.HEADGATE_URL ?? "http://127.0.0.1:8080";
const invoiceId = 9481;

const response = await fetch(`${headgateUrl}/api/v1/jobs`, {
  method: "POST",
  headers: {
    "Content-Type": "application/json",
    "Idempotency-Key": `billing-charge-${invoiceId}`,
  },
  body: JSON.stringify({
    kind: "billing:charge",
    schema_version: 1,
    payload: Buffer.from(JSON.stringify({ invoice_id: invoiceId }), "utf8").toString("base64"),
    queue: "billing",
    partition_key: "tenant-acme",
  }),
});

if (!response.ok) {
  throw new Error(`Headgate enqueue failed (${response.status}): ${await response.text()}`);
}

const job = (await response.json()) as { id: string; replayed?: boolean };

The kind must exactly match a handler registered by a Rust or Go worker, and the decoded payload bytes must follow the schema that handler expects. Authentication is supplied by the application hosting Headgate; do not expose an unauthenticated control API publicly. See the control API reference for every optional enqueue field and response status.

IDs and scheduling

Callers may provide a strict job ID for correlation and idempotency. Reusing that ID with different job content is an ID conflict, not a uniqueness match. Priority orders jobs only inside a queue; queue weight chooses among queues. Delayed jobs remain scheduled until store time reaches their timestamp.

Uniqueness, replacement, and debounce

Headgate supports lifecycle uniqueness and time-window throttling. A lifecycle key remains held while the job is live. A throttle key is released by its clock even if the original job already completed.

On conflict, a replacement mask can update only the payload bundle, scheduled time, priority, and maximum attempts. Debounce is the common replacement shape: one durable job keeps moving forward while new payloads replace its pending work. Replacement never mutates a running or terminal job.

The plaintext fingerprint participates in uniqueness and quarantine. With encrypted jobs, this preserves behavior but reveals payload equality.

Backpressure and outages

Producer admission can cap queue depth and reject enqueue before overload becomes database growth. Retryable store failures are classified separately from validation, authorization, duplicate, and ID-conflict errors. The producer circuit breaker fails fast while the store is unhealthy and probes again after its cooldown.

Applications should retry only typed unavailable errors, with their own deadline and jitter. Retrying a forbidden, malformed, or conflicting request cannot make it valid.

Resource boundaries

Every store adapter applies the same limits before starting a write: at most 1,000 jobs and 16 MiB of encoded job data per atomic enqueue, 1 MiB per payload, 128 headers totaling 64 KiB, 1 KiB per unique key, and 255 bytes for store identifiers. Timeout, deadline, and retention values cannot be negative. The control API additionally rejects request bodies larger than 2 MiB with HTTP 413.

These are safety ceilings, not recommended batch sizes. Keep ordinary producer batches small enough to fit the application’s latency budget. A workflow reserves one batch slot for its coordinator, so a graph can contain at most 999 task nodes and 10,000 dependency edges.

Authorization, middleware, and hooks

An enqueue authorizer receives application-supplied identity and every envelope. Headgate does not trust an HTTP identity header or invent application roles. A denial rejects the whole batch before store access. The raw store is intentionally a trusted low-level bypass.

Rust

let policy: Arc<dyn headgate::EnqueueAuthorizer> = Arc::new(
    |ctx: &headgate::EnqueueContext, env: &headgate::Envelope| {
        ctx.identity.as_ref().is_some_and(|identity| {
            identity.subject == "service:mailer" && env.kind == "mail.send"
        })
    },
);

let client = headgate::Client::new(store)
    .with_enqueue_authorizer(policy);

Producer middleware wraps the operation. Insert hooks run around the durable write. Plugins package middleware and hooks together with global or task-kind scope. Their order is deterministic: standalone components, global plugins in install order, then matching kind-scoped plugins in install order.

Configure exact queue-depth backpressure through the control API:

PUT /api/v1/queues/email/enqueue-limit
Idempotency-Key: email-limit-v1
Content-Type: application/json

{"max_unfinished_jobs":100000}

Transactions and ORMs

Plugins and middleware