Recognizing an architecture pattern is easier than applying it under load. I structure study around engineering questions: why did a queue back up, and which measurement would change my decision?

Architecture notes

An evidence-based study cycle

Researcher collects evidence, challenger tests hypotheses, an exercise requires a decision, and review informs the next exercise.
Researcher collects evidence, challenger tests hypotheses, an exercise requires a decision, and review informs the next exercise. View full-size diagram

A complete example session

In this synthetic scenario, worker count doubles from 20 to 40, throughput falls from 20 to 13.3 jobs per second, and database waits rise. CPU stays below saturation. The evidence pack includes those observations and an incident timeline. These are learning fixtures, not employer measurements.

Researcher: facts before hypotheses

List observations with evidence IDs. Shared database contention is one hypothesis; retry amplification is another. Low CPU does not prove more workers are safe. Measure database waits, connection use, retry rate, and job latency at several concurrency levels.

Challenger: missing measurements

Did document complexity change? Are retries counted as useful throughput? Do observations cover the same interval? The initial evidence suggests a bottleneck, but cannot identify an exact SQL statement or prove causality.

Exercise: a reversible decision

Would you add workers, lower concurrency, or change storage interaction? Name the first experiment, success criteria, and evidence that would reverse your decision. A defensible answer bounds concurrency while testing the bottleneck instead of optimizing replica count.

Reviewer: reasoning quality

Check whether the answer uses observations, offers an alternative hypothesis, proposes a bounded experiment, and names a rollback condition. The browser demonstration uses keyword coverage and labels its limitations. A human judges technical soundness.

Try the session

Open the interactive lab or download the Python project. The fixture provider runs offline. A model adapter can use the same narrow JSON contract; output is checked before acceptance.

python3 study_lab.py run --scenario worker-contention
python3 study_lab.py evaluate
python3 study_lab.py run --scenario worker-contention --adapter ./my_adapter.py

Close the loop

Answer before revealing the rubric. Keep the question, response, evidence IDs, and feedback. Revisit the exercise with changed constraints, such as CPU saturation and flat database waits. The goal is transfer to new decisions, not a pile of generated notes.

Related: prompt contracts

References and further reading

Primary sources for the technical concepts in this article. The examples and decisions above are my synthesis, not quotations from these sources.

  1. OpenAI documentationEvaluation best practices

    Representative tasks, explicit criteria, and regression evaluation.

  2. OpenAI documentationPrompt engineering

    Instructions, context, examples, and prompt evaluation.