Outcome
When you finish this pack, you can explain and run:
- What a golden task is (fixed scenario + expected behavior)
- Pass/fail scoring (not vibes)
- A tiny regression suite you re-run after a prompt/tool change
- One A/B (two configs, same goldens, compare scores)
- Interview one-liner: goldens ≠ unit tests ≠ asset time/token A/B
- Tag goldens with failure taxonomy IDs (spec drift, hallucinated API, loop inflation, test gaming, blast radius, silent arch)
Day map
| Day | Focus | Do | Time |
|---|---|---|---|
| 1 | Concepts | Read Day 1 + self-check | 30–40m |
| 2 | Design goldens | Read Day 2; write 5 goldens on paper/md | 30–45m |
| 3 | Run the lab | Read Day 3 + run the offline lab locally | 40–60m |
| 4 | Break & catch | Change agent config; watch suite catch it | 30–45m |
| 5 | A/B | Two system prompts; compare scorecards | 30–45m |
| 6 | Interview + close | Day 6 lines; log proof in your skill notes | 30m |
Done when
- 5+ goldens exist in the lab
- A local suite run produces a scorecard
- You deliberately made the agent worse and the suite dropped
- You can say goldens ≠ unit tests ≠ asset A/B in one breath
- Your skill notes record proof for the eval-harness axis
Anti-goals
- Framework zoo (full platform tooling optional later)
- Fake “IQ” or cognitive scores
- Only reading social threads about evals
Lab note. The runnable Python lab stays offline in
your study workspace. These pages are reading material and pointers
only—no hosted backend.