What changed, and why

Two earlier splits reshaped Everwhen's AI task set. #162 narrowed the Judge to pure writing craft (it no longer sees the goal); #163 pulled the anachronism "reach" rating out of the Timeline Engine into its own blind reach-rater. Once the tasks change shape, their per-task model settings in ai-routing.ts deserve a re-tune — that is this issue (#164).

This PR is the mechanical half of that re-tune. The routing table's settings are normally decided by a live bake-off (eval:bakeoff, judge-tune.eval.ts), which makes real model calls and needs both an Anthropic and an OpenAI key. Those keys weren't available here, so this pass lands only what is safe without a live run, and captures the one balance-sensitive change — the Judge — as a written plan for a keyed session. Net diff: one number changes (Timeline's token cap), the rest is provenance and a plan.

Timeline: trim the token cap to the new output shape

#163 removed the anachronism field from the Timeline Engine's output — it's an input now, rated upstream by the reach-rater. What's left is four small fields: a roughly nine-word headline, a one-to-two sentence detail, an integer swing, and a valence enum — under ~150 output tokens. But #163 left the maxTokens at 600, sized back when the output was larger and padded.

With thinking off (this route sets no adaptive thinking), Anthropic's max_tokens is a true output ceiling — no hidden reasoning is billed inside it — so 300 fits the real shape with better than 2x headroom. temperature and effort are untouched: the last bake-off held Timeline at 4/4 band ordering and a healthy neutral-EV corridor on those exact values, and nothing this pass moved them.

The only behavioral change in the PR: 600 -> 300, with the reasoning beside it.

server/utils/ai-routing.ts · 212 lines
server/utils/ai-routing.ts212 lines · TypeScript
⋯ 99 lines hidden (lines 1–99)
1/**
2 * The per-task model routing table — the provider seam the go-to-market plan
3 * calls for (dispatch 1's bake-off lives behind this file). Every AI task names
4 * its lane and its full parameter payload here, because valid parameter sets
5 * differ by model family:
6 *
7 * - Haiku 4.5 rejects `effort` and has no adaptive thinking — omit both.
8 * - Sonnet 4.6 defaults `effort` to HIGH — always set it explicitly (this game
9 * wants reflexes, not deliberation). `temperature` is accepted.
10 * - Opus 4.8 rejects `temperature`/`top_p`/`top_k` — steer tone via prompts.
11 * Thinking is adaptive-only; omitting `thinking` runs without thinking.
12 *
13 * Lane defaults were decided by the two-lane bake-off plus the prompt-tuning
14 * campaign (evals/bakeoff.eval.ts, evals/prompt-tune.eval.ts; snapshots +
15 * verdict in evals/results/) — quality-first, with cost a close second. All
16 * thirteen tasks run Anthropic: the prose tellings initially lost the blind
17 * pairwise 19-0 under the shared prompt, and the Claude-voice rewrite
18 * (buildChroniclePromptClaude) won them back on Sonnet. The OpenAI lane stays
19 * fully alive behind EVERWHEN_AI_LANE so a forced reverse migration is a
20 * config change, not a rewrite (the GTM's platform-risk counter).
21 */
22 
23export type AiTask =
24 | 'judge'
25 | 'character'
26 | 'timeline'
27 | 'reach-rater'
28 | 'chronicler'
29 | 'epilogue'
30 | 'archivist-study'
31 | 'archive-lookup'
32 | 'objective'
33 | 'objective-check'
34 | 'suggester'
35 | 'lifespan'
36 | 'sentinel'
37 
38/** 'fixture' is the dev-only deterministic lane (issue #105): a scripted structured
39 * output, no model call. Never selected by `routeFor` (the table is anthropic/openai
40 * only) — the gateway short-circuits to it when a dev fixture is installed, behind
41 * the dev gate. It exists in the union so telemetry can label a fixtured call. */
42export type AiLane = 'anthropic' | 'openai' | 'fixture'
43 
44export interface AnthropicParams {
45 model: string
46 maxTokens: number
47 /** Omit on Opus-tier models — the API rejects sampling params there. */
48 temperature?: number
49 /** Omit on Haiku 4.5 — the API rejects the effort parameter there.
50 * 'xhigh' exists on Opus 4.7+ and Fable 5 only. */
51 effort?: 'low' | 'medium' | 'high' | 'xhigh'
52 /** 'adaptive' turns thinking on (billed as output); omitted = no thinking. */
53 thinking?: 'adaptive'
55 
56export interface OpenAiParams {
57 model: string
58 /** gpt-5.5 bills hidden reasoning inside this cap — keep generous headroom. */
59 maxTokens: number
60 reasoningEffort?: 'low' | 'medium' | 'high'
62 
63export interface TaskRoute {
64 lane: AiLane
65 anthropic: AnthropicParams
66 openai: OpenAiParams
68 
69/** The OpenAI lane's flagship — one place, like the old exported MODEL constant. */
70const GPT = 'gpt-5.5'
71const OPENAI_DEFAULTS: OpenAiParams = { model: GPT, maxTokens: 1200, reasoningEffort: 'low' }
72 
73/**
74 * With thinking off, Anthropic's max_tokens is TRUE output budget — no hidden
75 * reasoning billed inside it — so caps are sized to real output shapes instead
76 * of the old padded 1200.
77 */
78export const AI_ROUTES: Record<AiTask, TaskRoute> = {
79 // A rubric classifier in the critical path of every dispatch (it runs before
80 // the die). Haiku for speed; the calibration eval is its gate.
81 // #164 (DEFERRED, needs keys): now that #162 narrowed this to pure writing craft,
82 // re-tune to make sharp/masterful reachable (playtests saw masterful ~1/98) and
83 // unstick Haiku's harsh middle anchor — planned change is a bump to Sonnet PLUS a
84 // rubric nudge. NOT applied here: it's a balance change to a load-bearing rater and
85 // must be confirmed against fresh judge-tune.eval.ts + eval:bakeoff data first. A
86 // Sonnet route must also pin `effort` explicitly (see ai-routing.spec.ts). Full
87 // plan + commands in evals/results/2026-06-24-retune-notes.md.
88 judge: {
89 lane: 'anthropic',
90 anthropic: { model: 'claude-haiku-4-5', maxTokens: 300, temperature: 0 },
91 openai: OPENAI_DEFAULTS
92 },
93 // The player converses with this output every turn — prose is product.
94 character: {
95 lane: 'anthropic',
96 anthropic: { model: 'claude-sonnet-4-6', maxTokens: 700, temperature: 0.9, effort: 'low' },
97 openai: OPENAI_DEFAULTS
98 },
99 // The rules-critical judgment layer; code clamps swings into the rolled band,
100 // the model places the swing within it (anachronism is rated separately, by the
101 // reach-rater, and amplifies the result deterministically). Output is four small
102 // fields now (#163 moved anachronism out): a ~9-word headline, a 1-2 sentence
103 // detail, an integer swing, and a valence enum — well under 150 output tokens.
104 // maxTokens trimmed 600 -> 300 (#164) to fit that real shape; with thinking off
105 // it's a true output ceiling, and 300 still leaves >2x headroom. temp/effort
106 // unchanged — the last bake-off held them at 4/4 band ordering and a 7.0 neutral
107 // corridor, and no fresh signal moved them.
108 timeline: {
109 lane: 'anthropic',
110 anthropic: { model: 'claude-sonnet-4-6', maxTokens: 300, temperature: 0.3, effort: 'medium' },
111 openai: OPENAI_DEFAULTS
⋯ 101 lines hidden (lines 112–212)
112 },
113 // The anachronism reach-rater (issue #163): rates how far beyond the figure's
114 // own era a message reaches — message × era only, BLIND to goal, dice, progress,
115 // and ledger, so the wager tier can't flex with the outcome it amplifies. A cheap
116 // temp-0 Haiku classifier off the critical generation path (it runs alongside the
117 // Judge, before the roll); its tier feeds the deterministic anachronism amplifier.
118 // Re-confirmed post-split (#164): Haiku/temp-0 is the cheapest tier that rates
119 // reach reliably (the canonical classifier setting), and the output is two small
120 // fields — a one-phrase reach gloss plus the tier enum, ~40 output tokens — so the
121 // 200 cap stays (a true ceiling well clear of the real shape; trimming it saves no
122 // cost, only risks clipping the tier-anchoring reach phrase). Held as-is.
123 'reach-rater': {
124 lane: 'anthropic',
125 anthropic: { model: 'claude-haiku-4-5', maxTokens: 200, temperature: 0 },
126 openai: OPENAI_DEFAULTS
127 },
128 // Re-narrates the whole timeline each turn. The original bake-off lost this
129 // slot 9-0 under the shared prompt; the Claude-voice rewrite (V2's
130 // transmission-chain bar, buildChroniclePromptClaude) won it back 8-5 in
131 // blind pairwise on Sonnet at ~1/4 the prior cost-per-telling — and the
132 // incumbent lane's quota outage made the flip availability-driven too.
133 // Higher-N confirmation pending (evals/results/2026-06-11-verdict.md).
134 chronicler: {
135 lane: 'anthropic',
136 anthropic: { model: 'claude-sonnet-4-6', maxTokens: 2000, temperature: 1, effort: 'high' },
137 openai: OPENAI_DEFAULTS
138 },
139 // The final telling — the share artifact and the thing people pay for. The
140 // Claude-voice V1 prompt on SONNET beat the incumbent 16-3 across two
141 // dual-graded rounds (7-3, then 9-0 at N=12) — decisively confirmed, and
142 // cheaper than the Opus/Fable candidates it outscored. Sonnet over Opus is
143 // what the data said, twice; Fable won too but at 4x Sonnet's price for no
144 // measured gain over it.
145 epilogue: {
146 lane: 'anthropic',
147 anthropic: { model: 'claude-sonnet-4-6', maxTokens: 2000, temperature: 1, effort: 'high' },
148 openai: OPENAI_DEFAULTS
149 },
150 // Feeds the player's wager calibration — accuracy matters.
151 'archivist-study': {
152 lane: 'anthropic',
153 anthropic: { model: 'claude-sonnet-4-6', maxTokens: 800, temperature: 0.5, effort: 'low' },
154 openai: OPENAI_DEFAULTS
155 },
156 // Factual micro-answers with a known-since stamp.
157 'archive-lookup': {
158 lane: 'anthropic',
159 anthropic: { model: 'claude-haiku-4-5', maxTokens: 500, temperature: 0.3 },
160 openai: OPENAI_DEFAULTS
161 },
162 // 0-1 calls per run; the objective steers every layer's valence.
163 objective: {
164 lane: 'anthropic',
165 anthropic: { model: 'claude-sonnet-4-6', maxTokens: 600, temperature: 1, effort: 'low' },
166 openai: OPENAI_DEFAULTS
167 },
168 // Gates a freshly composed objective on the safely-historical bounds (#71): a
169 // cheap, temp-0 classifier (the figure-floor pattern) that catches the
170 // living-people / current-events framings a year bound alone can't. Haiku,
171 // off the critical generation path — at most twice per composition.
172 'objective-check': {
173 lane: 'anthropic',
174 anthropic: { model: 'claude-haiku-4-5', maxTokens: 200, temperature: 0 },
175 openai: OPENAI_DEFAULTS
176 },
177 // Tool-using agent verified against free grounding, curated floor beneath it.
178 // maxTokens here is per ROUND of the agent loop.
179 suggester: {
180 lane: 'anthropic',
181 anthropic: { model: 'claude-haiku-4-5', maxTokens: 1500, temperature: 0.7 },
182 openai: { model: GPT, maxTokens: 2500 }
183 },
184 // A dates classifier with hard validation guards behind it (issue #31 bridge).
185 lifespan: {
186 lane: 'anthropic',
187 anthropic: { model: 'claude-haiku-4-5', maxTokens: 300, temperature: 0 },
188 openai: OPENAI_DEFAULTS
189 },
190 // The safety Sentinel — classifies a whole exchange against the verbatim
191 // Usage Policy (server/utils/usage-policy.ts). Haiku at temp 0 for a
192 // consistent, cheap classifier, exactly the tier Anthropic's content-
193 // moderation guide recommends. Runs once per user action, never on the
194 // critical generation path's tokens.
195 sentinel: {
196 lane: 'anthropic',
197 anthropic: { model: 'claude-haiku-4-5', maxTokens: 400, temperature: 0 },
198 openai: OPENAI_DEFAULTS
199 }
201 
202/**
203 * Resolves a task to its lane + parameters. EVERWHEN_AI_LANE overrides the
204 * whole table ('openai' | 'anthropic') — the emergency lever, and how the eval
205 * matrix runs identical fixtures down each lane.
206 */
207export function routeFor(task: AiTask): { lane: AiLane; route: TaskRoute } {
208 const route = AI_ROUTES[task]
209 const override = process.env.EVERWHEN_AI_LANE
210 const lane = override === 'openai' || override === 'anthropic' ? override : route.lane
211 return { lane, route }

Reach-rater: confirmed, deliberately unchanged

The issue asks for the cheapest model that rates reach reliably. #163 already pinned that — Haiku at temperature 0 is the canonical cheap-classifier tier — and the output is two small fields (a one-phrase reach gloss plus the tier enum), about 40 tokens. So the right re-tune outcome here is "hold," not "trim": the cap saves no money (you pay for real output, not the ceiling) and a lower cap would only risk clipping the reach phrase that anchors the tier.

The change is a comment that records this as a decision, so a future reader doesn't mistake the untouched route for an overlooked one.

No value change — the new lines document why holding is correct.

server/utils/ai-routing.ts · 212 lines
server/utils/ai-routing.ts212 lines · TypeScript
⋯ 112 lines hidden (lines 1–112)
1/**
2 * The per-task model routing table — the provider seam the go-to-market plan
3 * calls for (dispatch 1's bake-off lives behind this file). Every AI task names
4 * its lane and its full parameter payload here, because valid parameter sets
5 * differ by model family:
6 *
7 * - Haiku 4.5 rejects `effort` and has no adaptive thinking — omit both.
8 * - Sonnet 4.6 defaults `effort` to HIGH — always set it explicitly (this game
9 * wants reflexes, not deliberation). `temperature` is accepted.
10 * - Opus 4.8 rejects `temperature`/`top_p`/`top_k` — steer tone via prompts.
11 * Thinking is adaptive-only; omitting `thinking` runs without thinking.
12 *
13 * Lane defaults were decided by the two-lane bake-off plus the prompt-tuning
14 * campaign (evals/bakeoff.eval.ts, evals/prompt-tune.eval.ts; snapshots +
15 * verdict in evals/results/) — quality-first, with cost a close second. All
16 * thirteen tasks run Anthropic: the prose tellings initially lost the blind
17 * pairwise 19-0 under the shared prompt, and the Claude-voice rewrite
18 * (buildChroniclePromptClaude) won them back on Sonnet. The OpenAI lane stays
19 * fully alive behind EVERWHEN_AI_LANE so a forced reverse migration is a
20 * config change, not a rewrite (the GTM's platform-risk counter).
21 */
22 
23export type AiTask =
24 | 'judge'
25 | 'character'
26 | 'timeline'
27 | 'reach-rater'
28 | 'chronicler'
29 | 'epilogue'
30 | 'archivist-study'
31 | 'archive-lookup'
32 | 'objective'
33 | 'objective-check'
34 | 'suggester'
35 | 'lifespan'
36 | 'sentinel'
37 
38/** 'fixture' is the dev-only deterministic lane (issue #105): a scripted structured
39 * output, no model call. Never selected by `routeFor` (the table is anthropic/openai
40 * only) — the gateway short-circuits to it when a dev fixture is installed, behind
41 * the dev gate. It exists in the union so telemetry can label a fixtured call. */
42export type AiLane = 'anthropic' | 'openai' | 'fixture'
43 
44export interface AnthropicParams {
45 model: string
46 maxTokens: number
47 /** Omit on Opus-tier models — the API rejects sampling params there. */
48 temperature?: number
49 /** Omit on Haiku 4.5 — the API rejects the effort parameter there.
50 * 'xhigh' exists on Opus 4.7+ and Fable 5 only. */
51 effort?: 'low' | 'medium' | 'high' | 'xhigh'
52 /** 'adaptive' turns thinking on (billed as output); omitted = no thinking. */
53 thinking?: 'adaptive'
55 
56export interface OpenAiParams {
57 model: string
58 /** gpt-5.5 bills hidden reasoning inside this cap — keep generous headroom. */
59 maxTokens: number
60 reasoningEffort?: 'low' | 'medium' | 'high'
62 
63export interface TaskRoute {
64 lane: AiLane
65 anthropic: AnthropicParams
66 openai: OpenAiParams
68 
69/** The OpenAI lane's flagship — one place, like the old exported MODEL constant. */
70const GPT = 'gpt-5.5'
71const OPENAI_DEFAULTS: OpenAiParams = { model: GPT, maxTokens: 1200, reasoningEffort: 'low' }
72 
73/**
74 * With thinking off, Anthropic's max_tokens is TRUE output budget — no hidden
75 * reasoning billed inside it — so caps are sized to real output shapes instead
76 * of the old padded 1200.
77 */
78export const AI_ROUTES: Record<AiTask, TaskRoute> = {
79 // A rubric classifier in the critical path of every dispatch (it runs before
80 // the die). Haiku for speed; the calibration eval is its gate.
81 // #164 (DEFERRED, needs keys): now that #162 narrowed this to pure writing craft,
82 // re-tune to make sharp/masterful reachable (playtests saw masterful ~1/98) and
83 // unstick Haiku's harsh middle anchor — planned change is a bump to Sonnet PLUS a
84 // rubric nudge. NOT applied here: it's a balance change to a load-bearing rater and
85 // must be confirmed against fresh judge-tune.eval.ts + eval:bakeoff data first. A
86 // Sonnet route must also pin `effort` explicitly (see ai-routing.spec.ts). Full
87 // plan + commands in evals/results/2026-06-24-retune-notes.md.
88 judge: {
89 lane: 'anthropic',
90 anthropic: { model: 'claude-haiku-4-5', maxTokens: 300, temperature: 0 },
91 openai: OPENAI_DEFAULTS
92 },
93 // The player converses with this output every turn — prose is product.
94 character: {
95 lane: 'anthropic',
96 anthropic: { model: 'claude-sonnet-4-6', maxTokens: 700, temperature: 0.9, effort: 'low' },
97 openai: OPENAI_DEFAULTS
98 },
99 // The rules-critical judgment layer; code clamps swings into the rolled band,
100 // the model places the swing within it (anachronism is rated separately, by the
101 // reach-rater, and amplifies the result deterministically). Output is four small
102 // fields now (#163 moved anachronism out): a ~9-word headline, a 1-2 sentence
103 // detail, an integer swing, and a valence enum — well under 150 output tokens.
104 // maxTokens trimmed 600 -> 300 (#164) to fit that real shape; with thinking off
105 // it's a true output ceiling, and 300 still leaves >2x headroom. temp/effort
106 // unchanged — the last bake-off held them at 4/4 band ordering and a 7.0 neutral
107 // corridor, and no fresh signal moved them.
108 timeline: {
109 lane: 'anthropic',
110 anthropic: { model: 'claude-sonnet-4-6', maxTokens: 300, temperature: 0.3, effort: 'medium' },
111 openai: OPENAI_DEFAULTS
112 },
113 // The anachronism reach-rater (issue #163): rates how far beyond the figure's
114 // own era a message reaches — message × era only, BLIND to goal, dice, progress,
115 // and ledger, so the wager tier can't flex with the outcome it amplifies. A cheap
116 // temp-0 Haiku classifier off the critical generation path (it runs alongside the
117 // Judge, before the roll); its tier feeds the deterministic anachronism amplifier.
118 // Re-confirmed post-split (#164): Haiku/temp-0 is the cheapest tier that rates
119 // reach reliably (the canonical classifier setting), and the output is two small
120 // fields — a one-phrase reach gloss plus the tier enum, ~40 output tokens — so the
121 // 200 cap stays (a true ceiling well clear of the real shape; trimming it saves no
122 // cost, only risks clipping the tier-anchoring reach phrase). Held as-is.
123 'reach-rater': {
124 lane: 'anthropic',
125 anthropic: { model: 'claude-haiku-4-5', maxTokens: 200, temperature: 0 },
126 openai: OPENAI_DEFAULTS
⋯ 86 lines hidden (lines 127–212)
127 },
128 // Re-narrates the whole timeline each turn. The original bake-off lost this
129 // slot 9-0 under the shared prompt; the Claude-voice rewrite (V2's
130 // transmission-chain bar, buildChroniclePromptClaude) won it back 8-5 in
131 // blind pairwise on Sonnet at ~1/4 the prior cost-per-telling — and the
132 // incumbent lane's quota outage made the flip availability-driven too.
133 // Higher-N confirmation pending (evals/results/2026-06-11-verdict.md).
134 chronicler: {
135 lane: 'anthropic',
136 anthropic: { model: 'claude-sonnet-4-6', maxTokens: 2000, temperature: 1, effort: 'high' },
137 openai: OPENAI_DEFAULTS
138 },
139 // The final telling — the share artifact and the thing people pay for. The
140 // Claude-voice V1 prompt on SONNET beat the incumbent 16-3 across two
141 // dual-graded rounds (7-3, then 9-0 at N=12) — decisively confirmed, and
142 // cheaper than the Opus/Fable candidates it outscored. Sonnet over Opus is
143 // what the data said, twice; Fable won too but at 4x Sonnet's price for no
144 // measured gain over it.
145 epilogue: {
146 lane: 'anthropic',
147 anthropic: { model: 'claude-sonnet-4-6', maxTokens: 2000, temperature: 1, effort: 'high' },
148 openai: OPENAI_DEFAULTS
149 },
150 // Feeds the player's wager calibration — accuracy matters.
151 'archivist-study': {
152 lane: 'anthropic',
153 anthropic: { model: 'claude-sonnet-4-6', maxTokens: 800, temperature: 0.5, effort: 'low' },
154 openai: OPENAI_DEFAULTS
155 },
156 // Factual micro-answers with a known-since stamp.
157 'archive-lookup': {
158 lane: 'anthropic',
159 anthropic: { model: 'claude-haiku-4-5', maxTokens: 500, temperature: 0.3 },
160 openai: OPENAI_DEFAULTS
161 },
162 // 0-1 calls per run; the objective steers every layer's valence.
163 objective: {
164 lane: 'anthropic',
165 anthropic: { model: 'claude-sonnet-4-6', maxTokens: 600, temperature: 1, effort: 'low' },
166 openai: OPENAI_DEFAULTS
167 },
168 // Gates a freshly composed objective on the safely-historical bounds (#71): a
169 // cheap, temp-0 classifier (the figure-floor pattern) that catches the
170 // living-people / current-events framings a year bound alone can't. Haiku,
171 // off the critical generation path — at most twice per composition.
172 'objective-check': {
173 lane: 'anthropic',
174 anthropic: { model: 'claude-haiku-4-5', maxTokens: 200, temperature: 0 },
175 openai: OPENAI_DEFAULTS
176 },
177 // Tool-using agent verified against free grounding, curated floor beneath it.
178 // maxTokens here is per ROUND of the agent loop.
179 suggester: {
180 lane: 'anthropic',
181 anthropic: { model: 'claude-haiku-4-5', maxTokens: 1500, temperature: 0.7 },
182 openai: { model: GPT, maxTokens: 2500 }
183 },
184 // A dates classifier with hard validation guards behind it (issue #31 bridge).
185 lifespan: {
186 lane: 'anthropic',
187 anthropic: { model: 'claude-haiku-4-5', maxTokens: 300, temperature: 0 },
188 openai: OPENAI_DEFAULTS
189 },
190 // The safety Sentinel — classifies a whole exchange against the verbatim
191 // Usage Policy (server/utils/usage-policy.ts). Haiku at temp 0 for a
192 // consistent, cheap classifier, exactly the tier Anthropic's content-
193 // moderation guide recommends. Runs once per user action, never on the
194 // critical generation path's tokens.
195 sentinel: {
196 lane: 'anthropic',
197 anthropic: { model: 'claude-haiku-4-5', maxTokens: 400, temperature: 0 },
198 openai: OPENAI_DEFAULTS
199 }
201 
202/**
203 * Resolves a task to its lane + parameters. EVERWHEN_AI_LANE overrides the
204 * whole table ('openai' | 'anthropic') — the emergency lever, and how the eval
205 * matrix runs identical fixtures down each lane.
206 */
207export function routeFor(task: AiTask): { lane: AiLane; route: TaskRoute } {
208 const route = AI_ROUTES[task]
209 const override = process.env.EVERWHEN_AI_LANE
210 const lane = override === 'openai' || override === 'anthropic' ? override : route.lane
211 return { lane, route }

Judge: deferred, with a plan a keyed session can execute

The meatiest part of #164 is the Judge. Playtests showed "masterful" is almost unreachable (about 1 in 98 expert messages), and the bake-off flagged Haiku grading the middle rungs a notch harsh. The owner-approved direction is to bump the Judge to Sonnet and nudge the rubric so the top grades are reachable on pure writing craft.

That change is held back on purpose. The Judge sits on the dice-tilt and gains-multiplier path the run economy rides on, and the repo's rule is that the bake-off decides routes — shipping it unvalidated would break that contract. So the route is unchanged and a breadcrumb points to the plan.

The judge route is untouched; the breadcrumb explains the deferral and the Sonnet-needs-effort gotcha.

server/utils/ai-routing.ts · 212 lines
server/utils/ai-routing.ts212 lines · TypeScript
⋯ 78 lines hidden (lines 1–78)
1/**
2 * The per-task model routing table — the provider seam the go-to-market plan
3 * calls for (dispatch 1's bake-off lives behind this file). Every AI task names
4 * its lane and its full parameter payload here, because valid parameter sets
5 * differ by model family:
6 *
7 * - Haiku 4.5 rejects `effort` and has no adaptive thinking — omit both.
8 * - Sonnet 4.6 defaults `effort` to HIGH — always set it explicitly (this game
9 * wants reflexes, not deliberation). `temperature` is accepted.
10 * - Opus 4.8 rejects `temperature`/`top_p`/`top_k` — steer tone via prompts.
11 * Thinking is adaptive-only; omitting `thinking` runs without thinking.
12 *
13 * Lane defaults were decided by the two-lane bake-off plus the prompt-tuning
14 * campaign (evals/bakeoff.eval.ts, evals/prompt-tune.eval.ts; snapshots +
15 * verdict in evals/results/) — quality-first, with cost a close second. All
16 * thirteen tasks run Anthropic: the prose tellings initially lost the blind
17 * pairwise 19-0 under the shared prompt, and the Claude-voice rewrite
18 * (buildChroniclePromptClaude) won them back on Sonnet. The OpenAI lane stays
19 * fully alive behind EVERWHEN_AI_LANE so a forced reverse migration is a
20 * config change, not a rewrite (the GTM's platform-risk counter).
21 */
22 
23export type AiTask =
24 | 'judge'
25 | 'character'
26 | 'timeline'
27 | 'reach-rater'
28 | 'chronicler'
29 | 'epilogue'
30 | 'archivist-study'
31 | 'archive-lookup'
32 | 'objective'
33 | 'objective-check'
34 | 'suggester'
35 | 'lifespan'
36 | 'sentinel'
37 
38/** 'fixture' is the dev-only deterministic lane (issue #105): a scripted structured
39 * output, no model call. Never selected by `routeFor` (the table is anthropic/openai
40 * only) — the gateway short-circuits to it when a dev fixture is installed, behind
41 * the dev gate. It exists in the union so telemetry can label a fixtured call. */
42export type AiLane = 'anthropic' | 'openai' | 'fixture'
43 
44export interface AnthropicParams {
45 model: string
46 maxTokens: number
47 /** Omit on Opus-tier models — the API rejects sampling params there. */
48 temperature?: number
49 /** Omit on Haiku 4.5 — the API rejects the effort parameter there.
50 * 'xhigh' exists on Opus 4.7+ and Fable 5 only. */
51 effort?: 'low' | 'medium' | 'high' | 'xhigh'
52 /** 'adaptive' turns thinking on (billed as output); omitted = no thinking. */
53 thinking?: 'adaptive'
55 
56export interface OpenAiParams {
57 model: string
58 /** gpt-5.5 bills hidden reasoning inside this cap — keep generous headroom. */
59 maxTokens: number
60 reasoningEffort?: 'low' | 'medium' | 'high'
62 
63export interface TaskRoute {
64 lane: AiLane
65 anthropic: AnthropicParams
66 openai: OpenAiParams
68 
69/** The OpenAI lane's flagship — one place, like the old exported MODEL constant. */
70const GPT = 'gpt-5.5'
71const OPENAI_DEFAULTS: OpenAiParams = { model: GPT, maxTokens: 1200, reasoningEffort: 'low' }
72 
73/**
74 * With thinking off, Anthropic's max_tokens is TRUE output budget — no hidden
75 * reasoning billed inside it — so caps are sized to real output shapes instead
76 * of the old padded 1200.
77 */
78export const AI_ROUTES: Record<AiTask, TaskRoute> = {
79 // A rubric classifier in the critical path of every dispatch (it runs before
80 // the die). Haiku for speed; the calibration eval is its gate.
81 // #164 (DEFERRED, needs keys): now that #162 narrowed this to pure writing craft,
82 // re-tune to make sharp/masterful reachable (playtests saw masterful ~1/98) and
83 // unstick Haiku's harsh middle anchor — planned change is a bump to Sonnet PLUS a
84 // rubric nudge. NOT applied here: it's a balance change to a load-bearing rater and
85 // must be confirmed against fresh judge-tune.eval.ts + eval:bakeoff data first. A
86 // Sonnet route must also pin `effort` explicitly (see ai-routing.spec.ts). Full
87 // plan + commands in evals/results/2026-06-24-retune-notes.md.
88 judge: {
89 lane: 'anthropic',
90 anthropic: { model: 'claude-haiku-4-5', maxTokens: 300, temperature: 0 },
91 openai: OPENAI_DEFAULTS
⋯ 121 lines hidden (lines 92–212)
92 },
93 // The player converses with this output every turn — prose is product.
94 character: {
95 lane: 'anthropic',
96 anthropic: { model: 'claude-sonnet-4-6', maxTokens: 700, temperature: 0.9, effort: 'low' },
97 openai: OPENAI_DEFAULTS
98 },
99 // The rules-critical judgment layer; code clamps swings into the rolled band,
100 // the model places the swing within it (anachronism is rated separately, by the
101 // reach-rater, and amplifies the result deterministically). Output is four small
102 // fields now (#163 moved anachronism out): a ~9-word headline, a 1-2 sentence
103 // detail, an integer swing, and a valence enum — well under 150 output tokens.
104 // maxTokens trimmed 600 -> 300 (#164) to fit that real shape; with thinking off
105 // it's a true output ceiling, and 300 still leaves >2x headroom. temp/effort
106 // unchanged — the last bake-off held them at 4/4 band ordering and a 7.0 neutral
107 // corridor, and no fresh signal moved them.
108 timeline: {
109 lane: 'anthropic',
110 anthropic: { model: 'claude-sonnet-4-6', maxTokens: 300, temperature: 0.3, effort: 'medium' },
111 openai: OPENAI_DEFAULTS
112 },
113 // The anachronism reach-rater (issue #163): rates how far beyond the figure's
114 // own era a message reaches — message × era only, BLIND to goal, dice, progress,
115 // and ledger, so the wager tier can't flex with the outcome it amplifies. A cheap
116 // temp-0 Haiku classifier off the critical generation path (it runs alongside the
117 // Judge, before the roll); its tier feeds the deterministic anachronism amplifier.
118 // Re-confirmed post-split (#164): Haiku/temp-0 is the cheapest tier that rates
119 // reach reliably (the canonical classifier setting), and the output is two small
120 // fields — a one-phrase reach gloss plus the tier enum, ~40 output tokens — so the
121 // 200 cap stays (a true ceiling well clear of the real shape; trimming it saves no
122 // cost, only risks clipping the tier-anchoring reach phrase). Held as-is.
123 'reach-rater': {
124 lane: 'anthropic',
125 anthropic: { model: 'claude-haiku-4-5', maxTokens: 200, temperature: 0 },
126 openai: OPENAI_DEFAULTS
127 },
128 // Re-narrates the whole timeline each turn. The original bake-off lost this
129 // slot 9-0 under the shared prompt; the Claude-voice rewrite (V2's
130 // transmission-chain bar, buildChroniclePromptClaude) won it back 8-5 in
131 // blind pairwise on Sonnet at ~1/4 the prior cost-per-telling — and the
132 // incumbent lane's quota outage made the flip availability-driven too.
133 // Higher-N confirmation pending (evals/results/2026-06-11-verdict.md).
134 chronicler: {
135 lane: 'anthropic',
136 anthropic: { model: 'claude-sonnet-4-6', maxTokens: 2000, temperature: 1, effort: 'high' },
137 openai: OPENAI_DEFAULTS
138 },
139 // The final telling — the share artifact and the thing people pay for. The
140 // Claude-voice V1 prompt on SONNET beat the incumbent 16-3 across two
141 // dual-graded rounds (7-3, then 9-0 at N=12) — decisively confirmed, and
142 // cheaper than the Opus/Fable candidates it outscored. Sonnet over Opus is
143 // what the data said, twice; Fable won too but at 4x Sonnet's price for no
144 // measured gain over it.
145 epilogue: {
146 lane: 'anthropic',
147 anthropic: { model: 'claude-sonnet-4-6', maxTokens: 2000, temperature: 1, effort: 'high' },
148 openai: OPENAI_DEFAULTS
149 },
150 // Feeds the player's wager calibration — accuracy matters.
151 'archivist-study': {
152 lane: 'anthropic',
153 anthropic: { model: 'claude-sonnet-4-6', maxTokens: 800, temperature: 0.5, effort: 'low' },
154 openai: OPENAI_DEFAULTS
155 },
156 // Factual micro-answers with a known-since stamp.
157 'archive-lookup': {
158 lane: 'anthropic',
159 anthropic: { model: 'claude-haiku-4-5', maxTokens: 500, temperature: 0.3 },
160 openai: OPENAI_DEFAULTS
161 },
162 // 0-1 calls per run; the objective steers every layer's valence.
163 objective: {
164 lane: 'anthropic',
165 anthropic: { model: 'claude-sonnet-4-6', maxTokens: 600, temperature: 1, effort: 'low' },
166 openai: OPENAI_DEFAULTS
167 },
168 // Gates a freshly composed objective on the safely-historical bounds (#71): a
169 // cheap, temp-0 classifier (the figure-floor pattern) that catches the
170 // living-people / current-events framings a year bound alone can't. Haiku,
171 // off the critical generation path — at most twice per composition.
172 'objective-check': {
173 lane: 'anthropic',
174 anthropic: { model: 'claude-haiku-4-5', maxTokens: 200, temperature: 0 },
175 openai: OPENAI_DEFAULTS
176 },
177 // Tool-using agent verified against free grounding, curated floor beneath it.
178 // maxTokens here is per ROUND of the agent loop.
179 suggester: {
180 lane: 'anthropic',
181 anthropic: { model: 'claude-haiku-4-5', maxTokens: 1500, temperature: 0.7 },
182 openai: { model: GPT, maxTokens: 2500 }
183 },
184 // A dates classifier with hard validation guards behind it (issue #31 bridge).
185 lifespan: {
186 lane: 'anthropic',
187 anthropic: { model: 'claude-haiku-4-5', maxTokens: 300, temperature: 0 },
188 openai: OPENAI_DEFAULTS
189 },
190 // The safety Sentinel — classifies a whole exchange against the verbatim
191 // Usage Policy (server/utils/usage-policy.ts). Haiku at temp 0 for a
192 // consistent, cheap classifier, exactly the tier Anthropic's content-
193 // moderation guide recommends. Runs once per user action, never on the
194 // critical generation path's tokens.
195 sentinel: {
196 lane: 'anthropic',
197 anthropic: { model: 'claude-haiku-4-5', maxTokens: 400, temperature: 0 },
198 openai: OPENAI_DEFAULTS
199 }
201 
202/**
203 * Resolves a task to its lane + parameters. EVERWHEN_AI_LANE overrides the
204 * whole table ('openai' | 'anthropic') — the emergency lever, and how the eval
205 * matrix runs identical fixtures down each lane.
206 */
207export function routeFor(task: AiTask): { lane: AiLane; route: TaskRoute } {
208 const route = AI_ROUTES[task]
209 const override = process.env.EVERWHEN_AI_LANE
210 const lane = override === 'openai' || override === 'anthropic' ? override : route.lane
211 return { lane, route }

The full plan lives in a results note: the rationale, the exact change to make (including that a Sonnet route must pin effort explicitly, which the routing spec enforces), and the commands that gate it — judge-tune.eval.ts for absolute calibration, eval:bakeoff for the re-pin evidence, and npm run eval to keep the standing "sharp ≥ vague" behavior green.

The deferred-Judge section: decision, why, and the change to make.

evals/results/2026-06-24-retune-notes.md · 121 lines
evals/results/2026-06-24-retune-notes.md121 lines · Markdown
⋯ 49 lines hidden (lines 1–49)
1# Per-task re-tune after the Judge + reach-rater splits — 2026-06-24
2 
3Issue #164. Follow-up to the separation-of-concerns splits (#162 made the Judge
4objective-blind / pure writing craft; #163 extracted the anachronism reach-rater
5out of the Timeline Engine). Those splits reshaped the AI task set, so the
6per-task settings in `server/utils/ai-routing.ts` get a re-tune.
7 
8The bake-off harness already tracks the new shape: `bakeoff.eval.ts` and
9`behavior.eval.ts` call `callReachRater`, score anachronism under its own
10`Reach·` layer, and the Timeline block no longer rates anachronism. So this pass
11is settings, not harness.
12 
13## Scope this pass: mechanical only
14 
15No fresh bake-off was run. `eval:bakeoff` + `judge-tune.eval.ts` make real model
16calls and need `ANTHROPIC_API_KEY` + `OPENAI_API_KEY` (~$1-3), which weren't
17available here. So this pass lands only the changes that don't depend on a live
18run, and the Judge recalibration — which is a balance change to a load-bearing
19rater and must be confirmed against fresh data — is captured below and deferred
20to a keyed session. The 2026-06-11 snapshot (`bakeoff-2026-06-11.json`) still
21stands as the last live evidence.
22 
23## Applied
24 
25### Timeline — `maxTokens` 600 → 300
26 
27#163 moved the anachronism tier out of the Timeline Engine's output. The model
28now returns four small fields: a ~9-word `headline`, a 1-2 sentence `detail`, an
29integer `progressChange`, and a `valence` enum — under ~150 output tokens in the
30worst case. The 600 cap was sized for the old, padded shape (and predates the
31"caps sized to real output shapes" convention at the top of `ai-routing.ts`).
32Trimmed to 300: with thinking off it's a true output ceiling, so 300 still leaves
33better than 2x headroom over the real shape while dropping the dead padding.
34 
35`temperature` (0.3) and `effort` (medium) held. The last bake-off had Timeline at
364/4 band ordering and a 7.0 neutral-EV corridor on those settings, and nothing
37this pass moved them; re-confirm them in the next live run alongside the Judge.
38 
39### Reach-rater — confirmed `haiku` / temp-0 / `maxTokens` 200, held
40 
41The issue asks for the cheapest model that rates reach reliably. #163 already
42pinned the canonical cheap-classifier tier (Haiku, temp 0), and the output is two
43small fields — a one-phrase reach gloss plus the tier enum, ~40 output tokens. The
44200 cap is a true ceiling well clear of that, so trimming it saves no cost (you're
45billed for real output, not the cap) and only risks clipping the reach phrase that
46anchors the tier. Held as-is; this is the re-tune outcome, not a skipped task. The
47`Reach·` layer in `bakeoff.eval.ts` / `behavior.eval.ts` (≥75% era-reach accuracy)
48is its standing gate.
49 
50## Deferred — Judge recalibration (needs keys)
51 
52**Decision (owner-approved): bump the Judge to Sonnet AND nudge the rubric.** Not
53applied this pass — it changes the dice tilt + gains multiplier the run economy
54rides on, and the repo's rule is that the bake-off decides routes. Apply and
55confirm it in a keyed session.
56 
57**Why.** Two findings converge on the Judge:
58 
591. **Masterful is effectively unreachable** — ~1 in 98 expert messages in
60 playtesting. The rubric in `buildJudgePrompt` pins masterful at "the top few
61 percent" and sharp at "uncommon," so that rarity is the prompt working as
62 written. Net effect: the +2 `CRAFT_MODIFIER` and 1.45x `CRAFT_GAIN_FACTOR` (the
63 top of the craft economy) almost never pay out — dead reward surface.
642. **Haiku anchors the middle harsh** — the 2026-06-11 bake-off had Haiku
65 ordinally perfect (36/36) but grading the authored-"sound" rung a notch low
66 (read vague in 4/6 ladders). `judge-tune.eval.ts` measured Sonnet modestly
⋯ 55 lines hidden (lines 67–121)
67 better on absolute anchoring (mean deviation 0.22; middle-rung exact 21/36).
68 
69#162 narrowed the Judge from a three-axis composite (sharpness + period-fit +
70leverage) to writing craft alone. The "top few percent" language was calibrated
71for the old composite; on a single writing-quality axis the top grades should be
72reachable for genuinely excellent writing, and the harsh middle is now pure
73downward miscalibration.
74 
75**The change to make + validate:**
76 
771. **Route → Sonnet.** In `ai-routing.ts`, set the `judge` route to
78 `claude-sonnet-4-6`. A Sonnet route MUST pin `effort` explicitly — the server
79 defaults Sonnet to HIGH and `ai-routing.spec.ts` enforces it; this game wants
80 reflexes, so start at `effort: 'low'` (it's a fast pre-roll classifier). Keep
81 `temperature: 0`. Right-size `maxTokens` to the real output (craft enum +
82 <90-char reason, plus the optional event/timing/continuity fields ≈ ~90 tokens
83 worst case): ~220 is ample. Cost note: this moves a per-dispatch, pre-roll
84 critical-path call from Haiku to Sonnet — price it in the cost row.
852. **Rubric nudge** in `server/utils/prompt-builder.ts` `buildJudgePrompt`:
86 - Pull the "sound" anchor back so honest, competent asks read sound, not vague
87 (fix the harsh middle).
88 - Make sharp/masterful reachable for genuinely excellent pure-craft writing —
89 reframe "the top few percent" / "uncommon" so the top grades reward writing
90 quality on its own, without a quota mindset, while keeping masterful
91 genuinely rare (a real ceiling, not dead content).
92 - Keep the fairness floors intact: bottom rungs never read masterful, top rung
93 never reads reckless (the `hardViolations` gate in `bakeoff.eval.ts`).
94 
95**How to validate (the gate):**
96 
97```bash
98# both keys required; ~$1-3 total
99export ANTHROPIC_API_KEY=... OPENAI_API_KEY=...
100 
101# 1. Anchor sweep — Haiku vs Sonnet absolute calibration against the authored ladders
102npx vitest run --config vitest.eval.config.ts evals/judge-tune.eval.ts
103# want: Sonnet mean anchor deviation down vs the 0.22 baseline; middle-rung
104# exact up from 21/36; the authored-"sound" rung reads sound, not vague.
105 
106# 2. Full two-lane bake-off — re-pin evidence + a fresh snapshot
107npm run eval:bakeoff
108# want: Judge ladder ordinal agreement stays at/near 36/36; hard violations 0;
109# masterful now actually appears on the top rungs; cost row reflects the Sonnet bump.
110 
111# 3. Standing behavior gate must stay green
112npm run eval
113# want: "Message Judge — pairwise: sharp ≥ vague" stays ok.
114```
115 
116Iterate the rubric wording until judge-tune shows the middle anchored correctly
117and the top reachable without breaking ordering or the fairness floors, then
118re-pin the route and write a fresh `bakeoff-YYYY-MM-DD.json` + verdict. If the
119Sonnet bump's critical-path cost doesn't earn its keep over a Haiku-plus-rubric
120nudge, that comparison is the bake-off's to settle — keep both candidates in the
121sweep.

The validation gate — the exact commands a keyed session runs.

evals/results/2026-06-24-retune-notes.md · 121 lines
evals/results/2026-06-24-retune-notes.md121 lines · Markdown
⋯ 93 lines hidden (lines 1–93)
1# Per-task re-tune after the Judge + reach-rater splits — 2026-06-24
2 
3Issue #164. Follow-up to the separation-of-concerns splits (#162 made the Judge
4objective-blind / pure writing craft; #163 extracted the anachronism reach-rater
5out of the Timeline Engine). Those splits reshaped the AI task set, so the
6per-task settings in `server/utils/ai-routing.ts` get a re-tune.
7 
8The bake-off harness already tracks the new shape: `bakeoff.eval.ts` and
9`behavior.eval.ts` call `callReachRater`, score anachronism under its own
10`Reach·` layer, and the Timeline block no longer rates anachronism. So this pass
11is settings, not harness.
12 
13## Scope this pass: mechanical only
14 
15No fresh bake-off was run. `eval:bakeoff` + `judge-tune.eval.ts` make real model
16calls and need `ANTHROPIC_API_KEY` + `OPENAI_API_KEY` (~$1-3), which weren't
17available here. So this pass lands only the changes that don't depend on a live
18run, and the Judge recalibration — which is a balance change to a load-bearing
19rater and must be confirmed against fresh data — is captured below and deferred
20to a keyed session. The 2026-06-11 snapshot (`bakeoff-2026-06-11.json`) still
21stands as the last live evidence.
22 
23## Applied
24 
25### Timeline — `maxTokens` 600 → 300
26 
27#163 moved the anachronism tier out of the Timeline Engine's output. The model
28now returns four small fields: a ~9-word `headline`, a 1-2 sentence `detail`, an
29integer `progressChange`, and a `valence` enum — under ~150 output tokens in the
30worst case. The 600 cap was sized for the old, padded shape (and predates the
31"caps sized to real output shapes" convention at the top of `ai-routing.ts`).
32Trimmed to 300: with thinking off it's a true output ceiling, so 300 still leaves
33better than 2x headroom over the real shape while dropping the dead padding.
34 
35`temperature` (0.3) and `effort` (medium) held. The last bake-off had Timeline at
364/4 band ordering and a 7.0 neutral-EV corridor on those settings, and nothing
37this pass moved them; re-confirm them in the next live run alongside the Judge.
38 
39### Reach-rater — confirmed `haiku` / temp-0 / `maxTokens` 200, held
40 
41The issue asks for the cheapest model that rates reach reliably. #163 already
42pinned the canonical cheap-classifier tier (Haiku, temp 0), and the output is two
43small fields — a one-phrase reach gloss plus the tier enum, ~40 output tokens. The
44200 cap is a true ceiling well clear of that, so trimming it saves no cost (you're
45billed for real output, not the cap) and only risks clipping the reach phrase that
46anchors the tier. Held as-is; this is the re-tune outcome, not a skipped task. The
47`Reach·` layer in `bakeoff.eval.ts` / `behavior.eval.ts` (≥75% era-reach accuracy)
48is its standing gate.
49 
50## Deferred — Judge recalibration (needs keys)
51 
52**Decision (owner-approved): bump the Judge to Sonnet AND nudge the rubric.** Not
53applied this pass — it changes the dice tilt + gains multiplier the run economy
54rides on, and the repo's rule is that the bake-off decides routes. Apply and
55confirm it in a keyed session.
56 
57**Why.** Two findings converge on the Judge:
58 
591. **Masterful is effectively unreachable** — ~1 in 98 expert messages in
60 playtesting. The rubric in `buildJudgePrompt` pins masterful at "the top few
61 percent" and sharp at "uncommon," so that rarity is the prompt working as
62 written. Net effect: the +2 `CRAFT_MODIFIER` and 1.45x `CRAFT_GAIN_FACTOR` (the
63 top of the craft economy) almost never pay out — dead reward surface.
642. **Haiku anchors the middle harsh** — the 2026-06-11 bake-off had Haiku
65 ordinally perfect (36/36) but grading the authored-"sound" rung a notch low
66 (read vague in 4/6 ladders). `judge-tune.eval.ts` measured Sonnet modestly
67 better on absolute anchoring (mean deviation 0.22; middle-rung exact 21/36).
68 
69#162 narrowed the Judge from a three-axis composite (sharpness + period-fit +
70leverage) to writing craft alone. The "top few percent" language was calibrated
71for the old composite; on a single writing-quality axis the top grades should be
72reachable for genuinely excellent writing, and the harsh middle is now pure
73downward miscalibration.
74 
75**The change to make + validate:**
76 
771. **Route → Sonnet.** In `ai-routing.ts`, set the `judge` route to
78 `claude-sonnet-4-6`. A Sonnet route MUST pin `effort` explicitly — the server
79 defaults Sonnet to HIGH and `ai-routing.spec.ts` enforces it; this game wants
80 reflexes, so start at `effort: 'low'` (it's a fast pre-roll classifier). Keep
81 `temperature: 0`. Right-size `maxTokens` to the real output (craft enum +
82 <90-char reason, plus the optional event/timing/continuity fields ≈ ~90 tokens
83 worst case): ~220 is ample. Cost note: this moves a per-dispatch, pre-roll
84 critical-path call from Haiku to Sonnet — price it in the cost row.
852. **Rubric nudge** in `server/utils/prompt-builder.ts` `buildJudgePrompt`:
86 - Pull the "sound" anchor back so honest, competent asks read sound, not vague
87 (fix the harsh middle).
88 - Make sharp/masterful reachable for genuinely excellent pure-craft writing —
89 reframe "the top few percent" / "uncommon" so the top grades reward writing
90 quality on its own, without a quota mindset, while keeping masterful
91 genuinely rare (a real ceiling, not dead content).
92 - Keep the fairness floors intact: bottom rungs never read masterful, top rung
93 never reads reckless (the `hardViolations` gate in `bakeoff.eval.ts`).
94 
95**How to validate (the gate):**
96 
97```bash
98# both keys required; ~$1-3 total
99export ANTHROPIC_API_KEY=... OPENAI_API_KEY=...
100 
101# 1. Anchor sweep — Haiku vs Sonnet absolute calibration against the authored ladders
102npx vitest run --config vitest.eval.config.ts evals/judge-tune.eval.ts
103# want: Sonnet mean anchor deviation down vs the 0.22 baseline; middle-rung
104# exact up from 21/36; the authored-"sound" rung reads sound, not vague.
105 
106# 2. Full two-lane bake-off — re-pin evidence + a fresh snapshot
107npm run eval:bakeoff
108# want: Judge ladder ordinal agreement stays at/near 36/36; hard violations 0;
109# masterful now actually appears on the top rungs; cost row reflects the Sonnet bump.
110 
111# 3. Standing behavior gate must stay green
112npm run eval
113# want: "Message Judge — pairwise: sharp ≥ vague" stays ok.
⋯ 8 lines hidden (lines 114–121)
114```
115 
116Iterate the rubric wording until judge-tune shows the middle anchored correctly
117and the top reachable without breaking ordering or the fairness floors, then
118re-pin the route and write a fresh `bakeoff-YYYY-MM-DD.json` + verdict. If the
119Sonnet bump's critical-path cost doesn't earn its keep over a Haiku-plus-rubric
120nudge, that comparison is the bake-off's to settle — keep both candidates in the
121sweep.