Architecture
Recovering Background Parsing Jobs After a Worker Crash
By Vivek Jalondhara · 5 min read
Published
Separate durable job state, temporary ownership, and accepted output so interrupted parsing can recover without stale workers overwriting results.
A price-sheet parser can spend longer on a model request than an HTTP client is willing to wait. Moving that work to a background worker removes one connection dependency, but introduces another question: what happens when the worker disappears halfway through?
This educational design extends Sheet Parser’s domain with a hypothetical job table. It is a recovery exercise, not a claim about its deployed queues, provider guarantees, or processing throughput.
Save the work before reporting acceptance
An accepted job needs a durable input reference, workspace identity, current state, and a stable job ID. The source reference should identify the exact uploaded object or revision to process. If a worker later reads whichever file happens to be current, retrying the same job can produce a result from different input.
Persist the job before returning its identifier. If submitting through a separate broker, avoid committing the upload record and publishing its message as unrelated writes. A transactional outbox can record the intended message with the database change; its relay and consumers still need to tolerate duplicate delivery.
Claim briefly, then release the transaction
A database-backed worker can claim an available job in a short transaction using row locking and SKIP LOCKED. That lets another worker inspect other available rows rather than waiting for the same locked candidate. Commit the ownership change before starting the slow model call; do not keep a database lock open across that network wait.
Give each claim a new unpredictable token, an attempt number, and a lease expiry. The token identifies this specific claim, not just the worker process. Returning a job to the queue must invalidate the old token. The scheduler also needs a bounded policy for finding expired claims and making them eligible again.
Do not mistake expiry for cancellation
A lease expires because time passed, not because the old worker stopped executing. The process might be paused and then resume after another worker has taken over. Both can now hold results. An unconditional update by job ID would let the old attempt overwrite the replacement’s work.
Check ownership in the same database operation that accepts output. The example requires the current token, running state, and an unexpired lease. Zero returned rows means the worker has lost publication rights. It should discard that result or retain it only in a clearly separate diagnostic record, never treat completion as successful.
Keep output acceptance atomic
The short SQL example stores staged output directly on the guarded job row. If publishing also writes catalog rows, perform the ownership check and those writes in one transaction, holding the job-row lock until commit. Checking a token first and writing the catalog later opens another race.
Validate model output before attempting publication. A valid JSON shape is only the first check: missing currency, ambiguous units, and unsupported source references should enter an appropriate review or failure state. A retry should not turn an unresolved extraction into a catalog update merely because the second response looks more complete.
Retry only work that can use another attempt
Distinguish a transient provider failure from an invalid source or rejected output. Repeatedly submitting an unreadable sheet does not repair it. Store a bounded attempt policy, a next-eligible time, and a reason visible to operators. Backoff with jitter can spread retries, but it does not make an unsafe operation safe.
The remote call may have completed even when its response timed out. Do not assume a repeated model request has no extra cost or that the provider deduplicates it. Keep local result publication guarded and investigate the provider’s actual request contract. Cancellation likewise needs to invalidate local publication rights; aborting a request alone does not prove remote processing stopped.
Rehearse the awkward failures
Pause worker A after its model call, expire its claim, and let worker B take over. When A resumes, its completion must affect zero rows. Also crash a worker after a result is accepted but before it acknowledges delivery; the repeated message should see the completed state and avoid creating another accepted revision.
Track time spent waiting, running, and awaiting review separately. Recovery code adds complexity, so keep the job state machine small and explain every transition. Its promise is controlled re-execution and accepted output, not exactly-once execution of every external action.
Accept output only from the current claim
Educational completion operation. It assumes a unique job_id, a fresh UUID lease_token per claim, a timestamptz lease_until, and validated output. The worker checks the returned row count.
Illustrative example · SQL · Simplified for explanation
UPDATE parse_jobs
SET state = 'complete',
output = $3::jsonb,
lease_token = NULL,
lease_until = NULL
WHERE job_id = $1::uuid
AND state = 'running'
AND lease_token = $2::uuid
AND lease_until > clock_timestamp()
RETURNING job_id;
-- No returned row: this attempt must not publish further results.
-- Catalog writes need the same transaction and ownership lock.- Persist an exact input reference
- Claim with a fresh token
- Process outside the transaction
- Validate and conditionally accept output
- Retry or review unresolved work
Conceptual flow for the example above.
Tradeoff to keep in mind
The guard protects this database publication only. It cannot cancel an old remote request or fence a separate service that does not enforce the same ownership protocol.