My work on assessment platforms includes a pipeline for generating personalized answer-sheet PDFs. Schools need a usable batch, not merely a successful API response. That changes how I think about progress, retries, and operational ownership.
My contribution
I worked on backend architecture and the generation workflow, including orchestration, message boundaries, progress tracking, and scaling and failure-recovery discussions. Delivery involved backend and infrastructure engineers; the system is a team result.
Problem and constraints
A request can cover many students. Rendering is heavier than validating intent, and peak demand follows assessment schedules. Keeping a request open until every file is ready couples user-facing latency to batch size. Partial progress must survive a worker failure.
- API accepts batch intent
- Batch store records status
- Orchestrator divides work
- Queue distributes generation jobs
- Workers write PDFs to object storage
- Progress is recorded and the batch becomes available
Decisions and alternatives
I separated acceptance from execution and made individual work items recoverable. A synchronous endpoint would be simpler initially, but expose rendering failures and long execution directly to callers. A single large job reduces coordination but makes retries more expensive. Smaller units require explicit aggregation and a stable identity for each document.
Concurrency is bounded by shared dependencies, not just worker CPU. Before increasing replicas, I want to know whether database waits, object storage, or memory pressure is extending processing time.
Failure recovery
A worker can upload a file and fail before recording completion. A retry should resolve the same logical document through a stable key and guarded state transition. This is a design requirement; configuring a dead-letter queue alone does not solve duplicate side effects. Permanent failures need a visible batch outcome and a repair path.
Decision record: costs and revision criteria
This is a public reconstruction of the reasoning behind the workflow, rather than an internal architecture decision record. It separates the delivered structure from requirements that still need explicit verification.
| Choice | Alternative and cost | Revisit when |
|---|---|---|
| Accept intent, process asynchronously | A synchronous request simplifies coordination, but makes caller latency depend on rendering. Async work requires durable status, progress and recovery. | The workload is demonstrably small and predictable enough to meet the request time budget. |
| Recover individual document jobs | A whole-batch job reduces bookkeeping. Smaller jobs reduce repeated rendering but require stable identity and aggregation. | Per-document coordination dominates useful work; compare bounded chunks with the same recovery guarantees. |
| Bound work against shared dependencies | Scaling only from queue length can increase database pressure. A concurrency ceiling may leave some worker capacity idle. | Controlled tests show dependency waits stay flat while worker CPU is saturated and useful throughput improves. |
| Track artifacts separately from batch progress | Object storage and the status store do not share a transaction. Recovery must reconcile uploaded files with incomplete status. | Reconciliation cannot meet the operational recovery budget; review state transitions and completion protocol. |
A failure window worth testing
Consider a worker that writes a PDF and crashes before recording completion. A redelivery can repeat the operation. A stable document key alone is insufficient: recovery also needs a clear rule for accepting an existing artifact, updating progress once and preventing an older attempt from overwriting a newer document revision.
These are verification requirements, not a claim that every guard below is already implemented: kill the worker after upload; redeliver the same message; check that one logical document is counted; verify the artifact matches its intended revision; and confirm that a permanently failed job produces a visible, actionable batch outcome.
From operational question to public experiment
The question “should we add workers?” becomes a hypothesis about the limiting resource. The linked simulation makes one contention assumption inspectable, includes a no-contention control, and shows why an in-flight ceiling can matter. It is an educational model; it contains no company code, operational data, or measured production improvement.
Read and reproduce the concurrency experiment · Inspect the public Lab
Results and measurement
This public case describes the delivered workflow and the engineering decisions. It does not publish a verified before-and-after benchmark. Useful measures include oldest pending work, completed documents per minute, processing latency, retries, failed documents, and time until the entire batch is usable. The audience of the platform is not a document-throughput measurement.
What I would improve next
I would make partial completion and recovery actions first-class operational features, and maintain a repeatable load test with a fixed document mix. The test should identify where additional concurrency stops improving useful throughput.