Day 1 — Concepts

Time: 30–40 min · read this whole page first · then self-check.
No code today.

Why this pack exists

You’re aiming at product agents: systems where an LLM chooses tools, touches real-ish state (orders, tickets, DB rows), and can quietly get worse after a “small” prompt or model change.

Field maps often put “verify / evals” on its own axis. This pack builds that muscle first: a behavior regression harness you can actually run.

Demos lie. A single happy chat is not evidence the agent still:

• calls the right tool
• with the right args
• refuses the dangerous thing
• doesn’t double-charge on retry

Unit tests and asset A/B don’t cover that gap. This pack builds the missing muscle: a behavior regression harness.

Three measurement layers (don’t collapse them)

All three are useful. Mixing them up is how people ship “100% tests green” agents that refund twice.

What an eval harness is

An eval harness is the boring machinery around behavior checks:

1. Fixtures — fake world (orders, users) so runs are repeatable
2. Goldens — frozen scenarios with expected behavior
3. Runner — executes the agent (or a stand-in) on each golden
4. Scorer — pass/fail (or graded rubric) from the trace
5. Scorecard — suite totals you can compare across commits/prompts
6. Gate — optional CI: don’t merge if score drops without an explicit accept

Game makers already know a cousin: quality gates. Same idea, different subject — agent behavior, not “does the scene load.”

Core vocabulary

Study these until you can teach them.

Golden task (golden)

One frozen scenario:

• Setup — world before the run (e.g. order 9 is pending)
• Input — user message (and maybe prior turns)
• Expect — what must be true after: tools called / not called, final text constraints, side effects on the world

A golden is not “the model said something smart.” It is “cancel means one cancel tool call and status=cancelled.”

Trace / run record

The transcript of what happened: messages, tool names + args, tool results, final answer. Scoring reads the trace (and world diff), not your vibe.

Score

Usually pass/fail per golden. Advanced: weighted rubrics, partial credit. Days 1–5 stay binary so the skill stays honest.

Suite + scorecard

Many goldens → one run → e.g. 8/10. That’s your regression number.

Regression (eval sense)

After you change prompt / model / tools / retrieval:

run the same suite again → compare scorecard → keep / revert / document accept

If score drops and you ship anyway without noticing, you don’t have an eval practice — you have a diary.

A/B (eval sense)

Two configs (system prompt A vs B, or model A vs B). Same goldens. Same fixtures. Compare scorecards. This is different from asset A/B (tokens/taste).

Idempotent side effect

Doing the “same” request twice doesn’t double-create, double-cancel, or double-charge. Product agents get this wrong constantly. Goldens should catch it.

Worked example

Product: Mossdesk support agent
Tools: get_order(id), cancel_order(id), refund(id, amount)
Policy: only cancel pending; never invent ids; no double cancel

Golden G-happy

• Setup: order 9 = pending
• Input: “Please cancel order 9.”
• Expect:
  • tools include cancel_order with id=9 (once)
  • order 9 status → cancelled
  • final reply confirms cancel
• Pass rule: one cancel, correct id, world updated

Golden G-refuse-missing

• Setup: no order 999
• Input: “Cancel order 999.”
• Expect:
  • must not call cancel_order
  • may call get_order
  • final reply refuses / says not found
• Pass rule: no side effect on missing id

Golden G-double

• Setup: order 9 already cancelled
• Input: “Cancel order 9 again.”
• Expect: no second destructive side effect; clear message
• Pass rule: idempotent under repeat

If you only had unit tests on cancel_order(), all three could still fail at the agent layer (wrong tool choice, invented id, double call) while pytest stays green.

What makes a good golden

1. Deterministic expectation — a stranger could grade pass/fail
2. Short — seconds, not a novella
3. One main behavior — don’t smuggle five skills into one case
4. Includes refuses — unknown, unsafe, out-of-policy
5. Stable fixtures — fake DB, not live prod
6. Tool + world checks — “sounds polite” alone is not a pass

Bad golden: “User asks for help; agent is helpful.”
Good golden: “User asks to refund shipped order 4; agent refuses refund tool; explains policy.”

Minimal engineering loop

change prompt / tool schema / model
        ↓
   run golden suite
        ↓
   read scorecard + failing traces
        ↓
   keep · revert · or accept drop with a written reason

Interview line you’ll want later:

“We freeze scenarios with expected tool calls and side effects, re-run on every prompt/model change, and treat score drops like failing tests.”

How this differs from things you already do

Common failure modes

• Demo-driven development — one happy path video, no suite
• LLM-as-judge only — fuzzy scores, no hard tool/side-effect checks
• Goldens that grade prose — “was it friendly?” while it double-refunded
• Live prod fixtures — flaky world → flaky “evals”
• Huge scenarios — slow, multi-skill, un-debuggable
• Never re-run — wrote goldens once, shipped bitrot

Self-check (only after reading above)

Answer in your own words (voice OK).

1. In one sentence: what is a golden task?
2. Why can unit tests all pass while users get a worse agent?
3. Give one concrete golden for a cancel-order support agent (setup + input + expect).
4. Asset A/B vs golden suite — one sentence each.
5. What do you do when a prompt change drops the scorecard 10 → 8?

Log: minutes · pack eval-harness-101 · did: Day 1 concepts (read + self-check) · takeaway = best one-liner you can teach someone else.