What changed, and why
Glassbox exists to make hard ideas learnable, yet nothing measured that. The other content loops check adjacent things — content-accuracy whether the prose is true, lab-fidelity whether the labs are real, collection-coherence whether the structural beats are present — but none asks whether a lesson actually teaches. This loop does.
It is qualitative by design. The tempting version — have an agent predict a lab's output and check it against the engine — measures the agent's competence, not the teaching: a capable model computes a Bloom filter's false-positive rate from its own knowledge no matter how well or badly the lesson explains it. So instead a panel reads each lesson as a newcomer and scores it against a fixed rubric of named teaching practices.
The design's whole challenge is the one constraint that makes a quality loop usable: it must converge to a quiet PASS, not generate advice forever. Three mechanisms below do that — the fixed rubric, the majority threshold, and the per-evaluator honesty rules. Read them in that order.
The rubric is the cap
The rubric is the loop's entire universe of judgement. Evaluators may only speak to these named practices — they cannot invent new criteria, which is what stops the review from drifting into endless taste. It's a plain committed module: edit it to move the bar, deliberately.
A binary check per practice, not a vibe. Eight items; two shown.
agents/pedagogy-rubric.ts · 70 lines
agents/pedagogy-rubric.ts
⋯ 12 lines hidden (lines 1–12)
1/**
2 * The pedagogy rubric — the fixed, shared bar the `pedagogy` loop measures every
3 * lesson against. THIS FILE IS THE LOOP'S ENTIRE UNIVERSE OF JUDGEMENT: the
4 * evaluators may only speak to these named practices, never invent new ones. That
5 * bound is what keeps the loop converging to a quiet PASS instead of generating
6 * endless taste-driven advice. To raise the bar, add an item here — deliberately.
7 *
8 * Each item is a concrete, evidence-grounded teaching practice phrased as a binary
9 * check ("does the lesson do this, well?"), not a vibe. Edit freely; the loop reads
10 * this module and injects it into every evaluator's prompt.
11 */
12export interface RubricItem {
13 /** Stable id used in verdicts + the accept-list (agents/pedagogy-accept.json). */
14 id: string;
15 /** Short human name for the report. */
16 name: string;
17 /** The check, phrased for the evaluator. */
18 check: string;
19}
21export const RUBRIC: RubricItem[] = [
22 {
23 id: "motivation",
24 name: "Motivation before mechanism",
25 check:
26 'A concrete "what breaks without this" / why-it-matters lands BEFORE any machinery, so the reader wants the answer before they are handed it.',
27 },
28 {
29 id: "concrete-first",
30 name: "Concrete before abstract",
31 check:
32 "A specific instance or worked example precedes the general rule, formula, or definition — the lesson does not open cold on the abstraction.",
33 },
34 {
35 id: "scaffold-hard-step",
36 name: "The hardest step is scaffolded",
37 check:
38 'The single most load-bearing idea is built up or shown, not asserted or hand-waved (the tell: "a short argument shows…" / "it can be shown that…" with the argument omitted).',
39 },
40 {
⋯ 30 lines hidden (lines 41–70)
41 id: "name-misconception",
42 name: "Names and dispels the misconception",
43 check:
44 "The intuitive-but-wrong mental model a newcomer is likely to bring is surfaced and explicitly corrected, rather than left to silently mislead.",
45 },
46 {
47 id: "active",
48 name: "Active, not just exposition",
49 check:
50 "At least one lab makes a specific prose claim physical and load-bearing — the reader manipulates or predicts something and sees the consequence — rather than the lesson being read-only exposition.",
51 },
52 {
53 id: "paced",
54 name: "One new thing at a time",
55 check:
56 "Concepts arrive paced: terms are defined before they are used, and the reader is not asked to hold several brand-new ideas at once to follow a sentence.",
57 },
58 {
59 id: "transfer",
60 name: "Lands the transfer",
61 check:
62 "The close names the ONE transferable idea to carry to other problems, not merely a recap of what was covered.",
63 },
64 {
65 id: "signpost",
66 name: "Signposted",
67 check:
68 "The reader can always tell where they are in the arc and why this step follows the last — the lesson's shape (problem → idea → mechanism → limits → synthesis, or its own clear variant) is legible as you read.",
69 },
70];
The panel, and the majority threshold
Each lesson is reviewed by an independent panel (default 3), run in parallel. The evaluators never see each other; the only thing that makes a finding is agreement.
One pool slot per lesson; inside it, a parallel panel of read-only evaluators.
agents/pedagogy.ts · 323 lines
agents/pedagogy.ts
⋯ 281 lines hidden (lines 1–281)
1/**
2 * Pedagogy loop — Many Hands Engineering, the "does it actually teach?" pass.
3 *
4 * Every other content loop guards a different thing: `content-accuracy` guards
5 * whether the prose is TRUE, `lab-fidelity` whether the labs are REAL,
6 * `collection-coherence` whether the structural beats are PRESENT. None of them
7 * asks the question this collection exists to answer — does the lesson actually
8 * TEACH? This loop does.
9 *
10 * It is qualitative by design (testing whether an agent can compute the answer
11 * measures the agent, not the teaching). A panel of independent evaluators reads
12 * each lesson as a motivated newcomer and scores it against ONE fixed, shared
13 * rubric of named pedagogy practices (see pedagogy-rubric.ts) — and ONLY that
14 * rubric. A teaching gap is reported only when a MAJORITY of the panel
15 * independently flags the same rubric item with a cited passage. That majority +
16 * the fixed rubric + a per-lesson accept-list are the cap: the loop converges to
17 * a quiet PASS instead of producing endless advice.
18 *
19 * Honest about its reference: teaching quality is judgement, so this MAPS, it does
20 * not gate (propose-only, Read/Grep/Glob, never edits). Its rigor comes from the
21 * fixed rubric, strict cite-or-omit, the agreement threshold, and the accept-list
22 * — not from a mechanical truth.
23 *
24 * Usage:
25 * tsx pedagogy.ts # all lessons
26 * tsx pedagogy.ts paxos bloom-filters # only these lesson ids
27 * tsx pedagogy.ts --panel=3 # evaluators per lesson (default 3)
28 * tsx pedagogy.ts --concurrency=2 # lessons reviewed in parallel (default 2)
29 * tsx pedagogy.ts --budget=4 # USD cap PER EVALUATOR (default 4)
30 * tsx pedagogy.ts --dry-run # show the plan + cost ceiling, spend nothing
31 */
32import { existsSync, readFileSync } from "node:fs";
33import { resolve } from "node:path";
34import { APP_ROOT, report, runLoop } from "./lib.js";
35import {
36 LESSONS_DIR,
37 type Lesson,
38 gatherLessonFiles,
39 lessonDir,
40 lessonDirsOnDisk,
41 parseCatalog,
42 parsePerLessonArgs,
43 relList,
44 runPool,
45} from "./per-lesson.js";
46import { RUBRIC } from "./pedagogy-rubric.js";
48const ACCEPT_FILE = resolve(APP_ROOT, "agents", "pedagogy-accept.json");
49const VERDICTS = new Set(["satisfied", "weak", "missing"]);
50const RUBRIC_TEXT = RUBRIC.map((r, i) => ` ${i + 1}. id="${r.id}" — ${r.name}: ${r.check}`).join("\n");
52// ── Args ──────────────────────────────────────────────────────────────────────
53/** Read a `--flag=N` integer (the `=` form, so parsePerLessonArgs doesn't mistake
54 * a space-separated value for a lesson id). */
55function argInt(flag: string, dflt: number): number {
56 for (const a of process.argv.slice(2)) {
57 if (a.startsWith(`${flag}=`)) {
58 const n = Math.floor(Number(a.slice(flag.length + 1)));
59 if (Number.isFinite(n) && n > 0) return n;
60 }
61 }
62 return dflt;
63}
65// ── Accept-list (the intentional-deviation record) ────────────────────────────
66type Accept = Record<string, Record<string, string>>;
67function loadAccept(): Accept {
68 if (!existsSync(ACCEPT_FILE)) return {};
69 try {
70 const parsed: unknown = JSON.parse(readFileSync(ACCEPT_FILE, "utf8"));
71 return parsed && typeof parsed === "object" ? (parsed as Accept) : {};
72 } catch {
73 return {};
74 }
75}
77// ── Lesson selection (the deterministic signal) ───────────────────────────────
78function selectLessons(ids: string[], limit: number | null): { selected: Lesson[]; unknown: string[] } {
79 const onDisk = new Set(lessonDirsOnDisk());
80 const valid = parseCatalog().filter((l) => l.id !== "index" && onDisk.has(l.id));
81 const validIds = new Set(valid.map((l) => l.id));
82 const unknown = ids.filter((id) => !validIds.has(id));
83 let selected = ids.length ? valid.filter((l) => ids.includes(l.id)) : valid;
84 if (limit) selected = selected.slice(0, limit);
85 return { selected, unknown };
86}
88// ── Parse one evaluator's JSON verdict block ──────────────────────────────────
89interface Verdict {
90 verdict: string;
91 cite: string;
92 gap: string;
93}
94interface Panelist {
95 items: Map<string, Verdict>;
96 offRubric: string;
97}
99function extractJson(text: string): unknown {
100 const tryParse = (s: string): unknown => {
101 try {
102 return JSON.parse(s);
103 } catch {
104 return undefined;
105 }
106 };
107 const fence = text.match(/```(?:json)?\s*([\s\S]*?)```/i);
108 if (fence) {
109 const v = tryParse(fence[1].trim());
110 if (v !== undefined) return v;
111 }
112 const brace = text.match(/\{[\s\S]*\}/);
113 if (brace) return tryParse(brace[0]);
114 return undefined;
115}
117function parsePanelist(summary: string): Panelist | null {
118 const data = extractJson(summary);
119 if (!data || typeof data !== "object") return null;
120 const obj = data as { items?: unknown; offRubric?: unknown };
121 if (!Array.isArray(obj.items)) return null;
122 const items = new Map<string, Verdict>();
123 for (const raw of obj.items) {
124 if (!raw || typeof raw !== "object") continue;
125 const r = raw as Record<string, unknown>;
126 if (typeof r.id !== "string") continue;
127 const verdict = typeof r.verdict === "string" && VERDICTS.has(r.verdict) ? r.verdict : "satisfied";
128 items.set(r.id, {
129 verdict,
130 cite: typeof r.cite === "string" ? r.cite : "",
131 gap: typeof r.gap === "string" ? r.gap : "",
132 });
133 }
134 return { items, offRubric: typeof obj.offRubric === "string" ? obj.offRubric.trim() : "" };
135}
137// ── Prompts ───────────────────────────────────────────────────────────────────
138function systemPrompt(): string {
139 return `You are a PEDAGOGY EVALUATOR — one of an independent panel reviewing ONE Glassbox lesson for TEACHING QUALITY. Glassbox makes hard CS/systems ideas learnable; read this lesson the way a motivated newcomer to the topic would, and judge whether it actually teaches.
141You are NOT checking factual accuracy (another loop owns that), nor whether the structural beats merely exist, nor code/visual/performance. You judge the QUALITY of the teaching, and ONLY against the fixed rubric below — you may not invent new criteria. That bound is deliberate: it keeps this review from drifting into endless taste.
143Your only tools are Read / Grep / Glob. Read what you need; you cannot and must not edit anything.
145THE RUBRIC — for EACH item, return exactly one verdict:
146 - "satisfied": the lesson clearly does this, and it lands. Cite the passage that does it.
147 - "weak": the lesson attempts this but it does not land (too thin, buried, or only partial). Cite where, and name the gap in one sentence.
148 - "missing": the lesson does not do this where it should. Cite where it should have been, and name the gap.
150${RUBRIC_TEXT}
152HARD RULES (these are what make the panel trustworthy):
153 - DEFAULT TO "satisfied" WHEN UNSURE. A panel that flags everything is the failure mode. Mark "weak"/"missing" ONLY when you can point to the specific place the teaching falls short.
154 - CITE OR OMIT. Every "weak"/"missing" verdict MUST quote a real line (file:line, or an exact short phrase) from your own reading. No quote → you may not flag it; return "satisfied".
155 - Judge the lesson AS WRITTEN FOR ITS CHOSEN FORM. A declared survey, history, or essay legitimately makes different moves; do not penalize an intentional form. (The steward records those as accepted separately — you just judge honestly.)
156 - Be DIAGNOSTIC, never prescriptive: name what is missing, not how to rewrite it.
157 - The engine implementation (engine/index.js) is intentionally withheld — judge the TEACHING the reader receives, not the underlying algorithm source.
159OUTPUT: return ONLY a single fenced \`\`\`json code block — no prose before or after — of exactly this shape, with one object per rubric id (all ${RUBRIC.length} present):
161\`\`\`json
162{
163 "items": [
164 {"id": "motivation", "verdict": "satisfied", "cite": "<file:line or exact quoted phrase, or n/a>", "gap": "<one sentence, or empty if satisfied>"}
165 ],
166 "offRubric": "<at most one sentence on something important the rubric does not cover, or empty>"
167}
168\`\`\``;
169}
171function userPrompt(lesson: Lesson, files: string[]): string {
172 return `Lesson under review: \`${lesson.id}\` — "${lesson.title}". Eyebrow / claim it makes the learner: ${lesson.eyebrow || "(none stated)"}.
174Read it as a motivated newcomer to this topic. Its teaching surface (prose sections, interactive labs, lesson-local components), relative to \`${LESSONS_DIR}/${lesson.id}/\`:
175${relList(lesson.id, files)}
177Evaluate every rubric item against what THIS lesson does, and return ONLY the json verdict block.`;
178}
180// ── Aggregate one lesson's panel into a report block ──────────────────────────
181interface LessonReport {
182 lines: string[];
183 failedItems: number;
184 hasFindings: boolean;
185}
187function aggregate(
188 lesson: Lesson,
189 panelists: (Panelist | null)[],
190 panel: number,
191 threshold: number,
192 accepted: Record<string, string>,
193): LessonReport {
194 const answered = panelists.filter((p): p is Panelist => p !== null);
195 const abstained = panel - answered.length;
197 const findingBlocks: string[] = [];
198 const acceptedLines: string[] = [];
199 let clear = 0;
200 let failedItems = 0;
202 for (const item of RUBRIC) {
203 if (item.id in accepted) {
204 acceptedLines.push(` ~ ${item.name} — accepted: ${accepted[item.id]}`);
205 continue;
206 }
207 const flags = answered
208 .map((p) => p.items.get(item.id))
209 .filter((v): v is Verdict => v !== undefined && (v.verdict === "weak" || v.verdict === "missing"));
210 if (flags.length >= threshold) {
211 failedItems++;
212 findingBlocks.push(` ⚠ ${item.name} (${flags.length}/${panel} flagged)`);
213 for (const f of flags) findingBlocks.push(` · ${f.cite || "(no cite)"} — ${f.gap || "(no detail)"}`);
214 } else {
215 clear++;
216 }
217 }
219 const offNotes = answered.map((p) => p.offRubric).filter((s) => s.length > 0);
220 const lines: string[] = [
221 "",
222 "─".repeat(64),
223 `## ${lesson.id} — ${lesson.title}`,
224 `(${panel} evaluators${abstained ? `, ${abstained} unparseable` : ""} · ${clear}/${RUBRIC.length} rubric items clear${acceptedLines.length ? `, ${acceptedLines.length} accepted` : ""})`,
225 ];
226 if (findingBlocks.length) lines.push("", "Teaching gaps — majority-agreed (review & fix, or accept):", ...findingBlocks);
227 if (acceptedLines.length) lines.push("", "Accepted deviations:", ...acceptedLines);
228 if (offNotes.length) lines.push("", "Off-rubric notes (advisory, non-blocking):", ...offNotes.map((o) => ` note: ${o}`));
229 if (!findingBlocks.length) lines.push("", "✓ PASS — every rubric item clear or accepted.");
231 return { lines, failedItems, hasFindings: findingBlocks.length > 0 };
232}
234// ── Orchestrator ──────────────────────────────────────────────────────────────
235async function main(): Promise<void> {
236 const args = parsePerLessonArgs({ concurrency: 2, budget: 4 });
237 const panel = argInt("--panel", 3);
238 const threshold = Math.floor(panel / 2) + 1; // strict majority
239 const accept = loadAccept();
240 const { selected, unknown } = selectLessons(args.ids, args.limit);
242 if (unknown.length) {
243 report([
244 `RESULT: CANNOT RUN — unknown lesson id(s): ${unknown.join(", ")}`,
245 "",
246 `Known lessons: ${parseCatalog()
247 .filter((l) => l.id !== "index" && lessonDirsOnDisk().includes(l.id))
248 .map((l) => l.id)
249 .join(", ")}`,
250 ]);
251 process.exit(1);
252 }
253 if (!selected.length) {
254 report([`RESULT: CANNOT RUN — no lessons found under ${LESSONS_DIR}/.`]);
255 process.exit(1);
256 }
258 const scope = args.ids.length ? `lessons: ${selected.map((l) => l.id).join(", ")}` : "ALL lessons";
260 if (args.dryRun) {
261 report([
262 "Pedagogy — DRY RUN (no evaluators launched, nothing spent)",
263 "teaching-quality panel vs the fixed rubric, READ-ONLY",
264 `Lessons selected: ${selected.length}`,
265 `Panel size: ${panel} (≥${threshold} must agree for a finding)`,
266 `Per-evaluator budget: $${args.budget} (worst-case ceiling: $${selected.length * panel * args.budget})`,
267 `Rubric items: ${RUBRIC.length}`,
268 "",
269 "Would review:",
270 ...selected.map((l) => ` - ${l.id} (${gatherLessonFiles(lessonDir(l.id)).filter((f) => f.endsWith(".jsx")).length} teaching files)`),
271 "",
272 "RESULT: PASS — dry run only. Drop --dry-run to launch the panel.",
273 ]);
274 return;
275 }
277 console.log("Pedagogy loop — teaching-quality panel (opus · effort:max), READ-ONLY");
278 console.log(`Lessons: ${selected.length} · panel: ${panel} (≥${threshold} agree = finding) · per-evaluator budget: $${args.budget}`);
279 console.log(selected.map((l) => l.id).join(", "));
280 console.log("\n(Live `<lesson>#<n> · ToolName` markers below.)\n");
282 const reports = await runPool<Lesson, LessonReport>(selected, args.concurrency, async (lesson) => {
283 const files = gatherLessonFiles(lessonDir(lesson.id)).filter((f) => f.endsWith(".jsx"));
284 const panelists = await Promise.all(
285 Array.from({ length: panel }, (_, i) =>
286 runLoop({
287 systemPrompt: systemPrompt(),
288 allowedTools: ["Read", "Grep", "Glob"],
289 prompt: userPrompt(lesson, files),
290 model: "opus",
291 effort: "max",
292 label: `${lesson.id}#${i + 1}`,
293 maxTurns: 200,
294 maxBudgetUsd: args.budget,
295 }).then((r) => parsePanelist(r.agentSummary)),
296 ),
297 );
298 const result = aggregate(lesson, panelists, panel, threshold, accept[lesson.id] ?? {});
299 console.log(` ✓ ${lesson.id} — ${result.hasFindings ? `${result.failedItems} gap(s)` : "clear"}`);
300 return result;
301 });
303 const totalFailed = reports.reduce((s, r) => s + r.failedItems, 0);
⋯ 20 lines hidden (lines 304–323)
304 const lessonsWithFindings = reports.filter((r) => r.hasFindings).length;
306 report([
307 "Pedagogy — read-only teaching-quality map",
308 `Lessons reviewed: ${selected.length}`,
309 `Panel size: ${panel} (≥${threshold} agree = finding)`,
310 "(read-only — the working tree was not modified)",
311 ...reports.flatMap((r) => r.lines),
312 "",
313 totalFailed === 0
314 ? `RESULT: PASS — every reviewed lesson's teaching rubric is clear or accepted (${scope}).`
315 : `RESULT: FAIL — ${totalFailed} teaching gap(s) across ${lessonsWithFindings} lesson(s). Fix the prose/labs you agree with, or record an intentional deviation in agents/pedagogy-accept.json, then re-run.`,
316 ]);
317 if (totalFailed > 0) process.exitCode = 1;
318}
320main().catch((err: unknown) => {
321 report([`RESULT: ERROR — ${err instanceof Error ? err.message : String(err)}`]);
322 process.exitCode = 1;
323});
Aggregation is deterministic — no model sits in the scoring. A rubric item becomes a reported gap only when a strict majority of the panel marks it weak or missing, and only if it isn't accepted for this lesson. Everything else is either clear or a non-blocking note.
threshold = floor(panel/2)+1; a finding needs >= threshold flags AND no accept entry.
agents/pedagogy.ts · 323 lines
agents/pedagogy.ts
⋯ 198 lines hidden (lines 1–198)
1/**
2 * Pedagogy loop — Many Hands Engineering, the "does it actually teach?" pass.
3 *
4 * Every other content loop guards a different thing: `content-accuracy` guards
5 * whether the prose is TRUE, `lab-fidelity` whether the labs are REAL,
6 * `collection-coherence` whether the structural beats are PRESENT. None of them
7 * asks the question this collection exists to answer — does the lesson actually
8 * TEACH? This loop does.
9 *
10 * It is qualitative by design (testing whether an agent can compute the answer
11 * measures the agent, not the teaching). A panel of independent evaluators reads
12 * each lesson as a motivated newcomer and scores it against ONE fixed, shared
13 * rubric of named pedagogy practices (see pedagogy-rubric.ts) — and ONLY that
14 * rubric. A teaching gap is reported only when a MAJORITY of the panel
15 * independently flags the same rubric item with a cited passage. That majority +
16 * the fixed rubric + a per-lesson accept-list are the cap: the loop converges to
17 * a quiet PASS instead of producing endless advice.
18 *
19 * Honest about its reference: teaching quality is judgement, so this MAPS, it does
20 * not gate (propose-only, Read/Grep/Glob, never edits). Its rigor comes from the
21 * fixed rubric, strict cite-or-omit, the agreement threshold, and the accept-list
22 * — not from a mechanical truth.
23 *
24 * Usage:
25 * tsx pedagogy.ts # all lessons
26 * tsx pedagogy.ts paxos bloom-filters # only these lesson ids
27 * tsx pedagogy.ts --panel=3 # evaluators per lesson (default 3)
28 * tsx pedagogy.ts --concurrency=2 # lessons reviewed in parallel (default 2)
29 * tsx pedagogy.ts --budget=4 # USD cap PER EVALUATOR (default 4)
30 * tsx pedagogy.ts --dry-run # show the plan + cost ceiling, spend nothing
31 */
32import { existsSync, readFileSync } from "node:fs";
33import { resolve } from "node:path";
34import { APP_ROOT, report, runLoop } from "./lib.js";
35import {
36 LESSONS_DIR,
37 type Lesson,
38 gatherLessonFiles,
39 lessonDir,
40 lessonDirsOnDisk,
41 parseCatalog,
42 parsePerLessonArgs,
43 relList,
44 runPool,
45} from "./per-lesson.js";
46import { RUBRIC } from "./pedagogy-rubric.js";
48const ACCEPT_FILE = resolve(APP_ROOT, "agents", "pedagogy-accept.json");
49const VERDICTS = new Set(["satisfied", "weak", "missing"]);
50const RUBRIC_TEXT = RUBRIC.map((r, i) => ` ${i + 1}. id="${r.id}" — ${r.name}: ${r.check}`).join("\n");
52// ── Args ──────────────────────────────────────────────────────────────────────
53/** Read a `--flag=N` integer (the `=` form, so parsePerLessonArgs doesn't mistake
54 * a space-separated value for a lesson id). */
55function argInt(flag: string, dflt: number): number {
56 for (const a of process.argv.slice(2)) {
57 if (a.startsWith(`${flag}=`)) {
58 const n = Math.floor(Number(a.slice(flag.length + 1)));
59 if (Number.isFinite(n) && n > 0) return n;
60 }
61 }
62 return dflt;
63}
65// ── Accept-list (the intentional-deviation record) ────────────────────────────
66type Accept = Record<string, Record<string, string>>;
67function loadAccept(): Accept {
68 if (!existsSync(ACCEPT_FILE)) return {};
69 try {
70 const parsed: unknown = JSON.parse(readFileSync(ACCEPT_FILE, "utf8"));
71 return parsed && typeof parsed === "object" ? (parsed as Accept) : {};
72 } catch {
73 return {};
74 }
75}
77// ── Lesson selection (the deterministic signal) ───────────────────────────────
78function selectLessons(ids: string[], limit: number | null): { selected: Lesson[]; unknown: string[] } {
79 const onDisk = new Set(lessonDirsOnDisk());
80 const valid = parseCatalog().filter((l) => l.id !== "index" && onDisk.has(l.id));
81 const validIds = new Set(valid.map((l) => l.id));
82 const unknown = ids.filter((id) => !validIds.has(id));
83 let selected = ids.length ? valid.filter((l) => ids.includes(l.id)) : valid;
84 if (limit) selected = selected.slice(0, limit);
85 return { selected, unknown };
86}
88// ── Parse one evaluator's JSON verdict block ──────────────────────────────────
89interface Verdict {
90 verdict: string;
91 cite: string;
92 gap: string;
93}
94interface Panelist {
95 items: Map<string, Verdict>;
96 offRubric: string;
97}
99function extractJson(text: string): unknown {
100 const tryParse = (s: string): unknown => {
101 try {
102 return JSON.parse(s);
103 } catch {
104 return undefined;
105 }
106 };
107 const fence = text.match(/```(?:json)?\s*([\s\S]*?)```/i);
108 if (fence) {
109 const v = tryParse(fence[1].trim());
110 if (v !== undefined) return v;
111 }
112 const brace = text.match(/\{[\s\S]*\}/);
113 if (brace) return tryParse(brace[0]);
114 return undefined;
115}
117function parsePanelist(summary: string): Panelist | null {
118 const data = extractJson(summary);
119 if (!data || typeof data !== "object") return null;
120 const obj = data as { items?: unknown; offRubric?: unknown };
121 if (!Array.isArray(obj.items)) return null;
122 const items = new Map<string, Verdict>();
123 for (const raw of obj.items) {
124 if (!raw || typeof raw !== "object") continue;
125 const r = raw as Record<string, unknown>;
126 if (typeof r.id !== "string") continue;
127 const verdict = typeof r.verdict === "string" && VERDICTS.has(r.verdict) ? r.verdict : "satisfied";
128 items.set(r.id, {
129 verdict,
130 cite: typeof r.cite === "string" ? r.cite : "",
131 gap: typeof r.gap === "string" ? r.gap : "",
132 });
133 }
134 return { items, offRubric: typeof obj.offRubric === "string" ? obj.offRubric.trim() : "" };
135}
137// ── Prompts ───────────────────────────────────────────────────────────────────
138function systemPrompt(): string {
139 return `You are a PEDAGOGY EVALUATOR — one of an independent panel reviewing ONE Glassbox lesson for TEACHING QUALITY. Glassbox makes hard CS/systems ideas learnable; read this lesson the way a motivated newcomer to the topic would, and judge whether it actually teaches.
141You are NOT checking factual accuracy (another loop owns that), nor whether the structural beats merely exist, nor code/visual/performance. You judge the QUALITY of the teaching, and ONLY against the fixed rubric below — you may not invent new criteria. That bound is deliberate: it keeps this review from drifting into endless taste.
143Your only tools are Read / Grep / Glob. Read what you need; you cannot and must not edit anything.
145THE RUBRIC — for EACH item, return exactly one verdict:
146 - "satisfied": the lesson clearly does this, and it lands. Cite the passage that does it.
147 - "weak": the lesson attempts this but it does not land (too thin, buried, or only partial). Cite where, and name the gap in one sentence.
148 - "missing": the lesson does not do this where it should. Cite where it should have been, and name the gap.
150${RUBRIC_TEXT}
152HARD RULES (these are what make the panel trustworthy):
153 - DEFAULT TO "satisfied" WHEN UNSURE. A panel that flags everything is the failure mode. Mark "weak"/"missing" ONLY when you can point to the specific place the teaching falls short.
154 - CITE OR OMIT. Every "weak"/"missing" verdict MUST quote a real line (file:line, or an exact short phrase) from your own reading. No quote → you may not flag it; return "satisfied".
155 - Judge the lesson AS WRITTEN FOR ITS CHOSEN FORM. A declared survey, history, or essay legitimately makes different moves; do not penalize an intentional form. (The steward records those as accepted separately — you just judge honestly.)
156 - Be DIAGNOSTIC, never prescriptive: name what is missing, not how to rewrite it.
157 - The engine implementation (engine/index.js) is intentionally withheld — judge the TEACHING the reader receives, not the underlying algorithm source.
159OUTPUT: return ONLY a single fenced \`\`\`json code block — no prose before or after — of exactly this shape, with one object per rubric id (all ${RUBRIC.length} present):
161\`\`\`json
162{
163 "items": [
164 {"id": "motivation", "verdict": "satisfied", "cite": "<file:line or exact quoted phrase, or n/a>", "gap": "<one sentence, or empty if satisfied>"}
165 ],
166 "offRubric": "<at most one sentence on something important the rubric does not cover, or empty>"
167}
168\`\`\``;
169}
171function userPrompt(lesson: Lesson, files: string[]): string {
172 return `Lesson under review: \`${lesson.id}\` — "${lesson.title}". Eyebrow / claim it makes the learner: ${lesson.eyebrow || "(none stated)"}.
174Read it as a motivated newcomer to this topic. Its teaching surface (prose sections, interactive labs, lesson-local components), relative to \`${LESSONS_DIR}/${lesson.id}/\`:
175${relList(lesson.id, files)}
177Evaluate every rubric item against what THIS lesson does, and return ONLY the json verdict block.`;
178}
180// ── Aggregate one lesson's panel into a report block ──────────────────────────
181interface LessonReport {
182 lines: string[];
183 failedItems: number;
184 hasFindings: boolean;
185}
187function aggregate(
188 lesson: Lesson,
189 panelists: (Panelist | null)[],
190 panel: number,
191 threshold: number,
192 accepted: Record<string, string>,
193): LessonReport {
194 const answered = panelists.filter((p): p is Panelist => p !== null);
195 const abstained = panel - answered.length;
197 const findingBlocks: string[] = [];
198 const acceptedLines: string[] = [];
199 let clear = 0;
200 let failedItems = 0;
202 for (const item of RUBRIC) {
203 if (item.id in accepted) {
204 acceptedLines.push(` ~ ${item.name} — accepted: ${accepted[item.id]}`);
205 continue;
206 }
207 const flags = answered
208 .map((p) => p.items.get(item.id))
209 .filter((v): v is Verdict => v !== undefined && (v.verdict === "weak" || v.verdict === "missing"));
210 if (flags.length >= threshold) {
211 failedItems++;
212 findingBlocks.push(` ⚠ ${item.name} (${flags.length}/${panel} flagged)`);
213 for (const f of flags) findingBlocks.push(` · ${f.cite || "(no cite)"} — ${f.gap || "(no detail)"}`);
214 } else {
215 clear++;
216 }
217 }
219 const offNotes = answered.map((p) => p.offRubric).filter((s) => s.length > 0);
220 const lines: string[] = [
221 "",
222 "─".repeat(64),
223 `## ${lesson.id} — ${lesson.title}`,
224 `(${panel} evaluators${abstained ? `, ${abstained} unparseable` : ""} · ${clear}/${RUBRIC.length} rubric items clear${acceptedLines.length ? `, ${acceptedLines.length} accepted` : ""})`,
225 ];
226 if (findingBlocks.length) lines.push("", "Teaching gaps — majority-agreed (review & fix, or accept):", ...findingBlocks);
227 if (acceptedLines.length) lines.push("", "Accepted deviations:", ...acceptedLines);
228 if (offNotes.length) lines.push("", "Off-rubric notes (advisory, non-blocking):", ...offNotes.map((o) => ` note: ${o}`));
229 if (!findingBlocks.length) lines.push("", "✓ PASS — every rubric item clear or accepted.");
231 return { lines, failedItems, hasFindings: findingBlocks.length > 0 };
232}
⋯ 90 lines hidden (lines 234–323)
234// ── Orchestrator ──────────────────────────────────────────────────────────────
235async function main(): Promise<void> {
236 const args = parsePerLessonArgs({ concurrency: 2, budget: 4 });
237 const panel = argInt("--panel", 3);
238 const threshold = Math.floor(panel / 2) + 1; // strict majority
239 const accept = loadAccept();
240 const { selected, unknown } = selectLessons(args.ids, args.limit);
242 if (unknown.length) {
243 report([
244 `RESULT: CANNOT RUN — unknown lesson id(s): ${unknown.join(", ")}`,
245 "",
246 `Known lessons: ${parseCatalog()
247 .filter((l) => l.id !== "index" && lessonDirsOnDisk().includes(l.id))
248 .map((l) => l.id)
249 .join(", ")}`,
250 ]);
251 process.exit(1);
252 }
253 if (!selected.length) {
254 report([`RESULT: CANNOT RUN — no lessons found under ${LESSONS_DIR}/.`]);
255 process.exit(1);
256 }
258 const scope = args.ids.length ? `lessons: ${selected.map((l) => l.id).join(", ")}` : "ALL lessons";
260 if (args.dryRun) {
261 report([
262 "Pedagogy — DRY RUN (no evaluators launched, nothing spent)",
263 "teaching-quality panel vs the fixed rubric, READ-ONLY",
264 `Lessons selected: ${selected.length}`,
265 `Panel size: ${panel} (≥${threshold} must agree for a finding)`,
266 `Per-evaluator budget: $${args.budget} (worst-case ceiling: $${selected.length * panel * args.budget})`,
267 `Rubric items: ${RUBRIC.length}`,
268 "",
269 "Would review:",
270 ...selected.map((l) => ` - ${l.id} (${gatherLessonFiles(lessonDir(l.id)).filter((f) => f.endsWith(".jsx")).length} teaching files)`),
271 "",
272 "RESULT: PASS — dry run only. Drop --dry-run to launch the panel.",
273 ]);
274 return;
275 }
277 console.log("Pedagogy loop — teaching-quality panel (opus · effort:max), READ-ONLY");
278 console.log(`Lessons: ${selected.length} · panel: ${panel} (≥${threshold} agree = finding) · per-evaluator budget: $${args.budget}`);
279 console.log(selected.map((l) => l.id).join(", "));
280 console.log("\n(Live `<lesson>#<n> · ToolName` markers below.)\n");
282 const reports = await runPool<Lesson, LessonReport>(selected, args.concurrency, async (lesson) => {
283 const files = gatherLessonFiles(lessonDir(lesson.id)).filter((f) => f.endsWith(".jsx"));
284 const panelists = await Promise.all(
285 Array.from({ length: panel }, (_, i) =>
286 runLoop({
287 systemPrompt: systemPrompt(),
288 allowedTools: ["Read", "Grep", "Glob"],
289 prompt: userPrompt(lesson, files),
290 model: "opus",
291 effort: "max",
292 label: `${lesson.id}#${i + 1}`,
293 maxTurns: 200,
294 maxBudgetUsd: args.budget,
295 }).then((r) => parsePanelist(r.agentSummary)),
296 ),
297 );
298 const result = aggregate(lesson, panelists, panel, threshold, accept[lesson.id] ?? {});
299 console.log(` ✓ ${lesson.id} — ${result.hasFindings ? `${result.failedItems} gap(s)` : "clear"}`);
300 return result;
301 });
303 const totalFailed = reports.reduce((s, r) => s + r.failedItems, 0);
304 const lessonsWithFindings = reports.filter((r) => r.hasFindings).length;
306 report([
307 "Pedagogy — read-only teaching-quality map",
308 `Lessons reviewed: ${selected.length}`,
309 `Panel size: ${panel} (≥${threshold} agree = finding)`,
310 "(read-only — the working tree was not modified)",
311 ...reports.flatMap((r) => r.lines),
312 "",
313 totalFailed === 0
314 ? `RESULT: PASS — every reviewed lesson's teaching rubric is clear or accepted (${scope}).`
315 : `RESULT: FAIL — ${totalFailed} teaching gap(s) across ${lessonsWithFindings} lesson(s). Fix the prose/labs you agree with, or record an intentional deviation in agents/pedagogy-accept.json, then re-run.`,
316 ]);
317 if (totalFailed > 0) process.exitCode = 1;
318}
320main().catch((err: unknown) => {
321 report([`RESULT: ERROR — ${err instanceof Error ? err.message : String(err)}`]);
322 process.exitCode = 1;
323});
Keeping each evaluator honest
The panel only helps if each member is calibrated against false positives. The system prompt injects the rubric and three rules that bias hard toward silence: default to "satisfied" unless you can cite a specific shortfall, cite-or-omit, and judge the lesson as written for its chosen form (a declared survey legitimately makes different moves).
The rubric is injected from the module; the hard rules make a flag expensive.
agents/pedagogy.ts · 323 lines
agents/pedagogy.ts
⋯ 145 lines hidden (lines 1–145)
1/**
2 * Pedagogy loop — Many Hands Engineering, the "does it actually teach?" pass.
3 *
4 * Every other content loop guards a different thing: `content-accuracy` guards
5 * whether the prose is TRUE, `lab-fidelity` whether the labs are REAL,
6 * `collection-coherence` whether the structural beats are PRESENT. None of them
7 * asks the question this collection exists to answer — does the lesson actually
8 * TEACH? This loop does.
9 *
10 * It is qualitative by design (testing whether an agent can compute the answer
11 * measures the agent, not the teaching). A panel of independent evaluators reads
12 * each lesson as a motivated newcomer and scores it against ONE fixed, shared
13 * rubric of named pedagogy practices (see pedagogy-rubric.ts) — and ONLY that
14 * rubric. A teaching gap is reported only when a MAJORITY of the panel
15 * independently flags the same rubric item with a cited passage. That majority +
16 * the fixed rubric + a per-lesson accept-list are the cap: the loop converges to
17 * a quiet PASS instead of producing endless advice.
18 *
19 * Honest about its reference: teaching quality is judgement, so this MAPS, it does
20 * not gate (propose-only, Read/Grep/Glob, never edits). Its rigor comes from the
21 * fixed rubric, strict cite-or-omit, the agreement threshold, and the accept-list
22 * — not from a mechanical truth.
23 *
24 * Usage:
25 * tsx pedagogy.ts # all lessons
26 * tsx pedagogy.ts paxos bloom-filters # only these lesson ids
27 * tsx pedagogy.ts --panel=3 # evaluators per lesson (default 3)
28 * tsx pedagogy.ts --concurrency=2 # lessons reviewed in parallel (default 2)
29 * tsx pedagogy.ts --budget=4 # USD cap PER EVALUATOR (default 4)
30 * tsx pedagogy.ts --dry-run # show the plan + cost ceiling, spend nothing
31 */
32import { existsSync, readFileSync } from "node:fs";
33import { resolve } from "node:path";
34import { APP_ROOT, report, runLoop } from "./lib.js";
35import {
36 LESSONS_DIR,
37 type Lesson,
38 gatherLessonFiles,
39 lessonDir,
40 lessonDirsOnDisk,
41 parseCatalog,
42 parsePerLessonArgs,
43 relList,
44 runPool,
45} from "./per-lesson.js";
46import { RUBRIC } from "./pedagogy-rubric.js";
48const ACCEPT_FILE = resolve(APP_ROOT, "agents", "pedagogy-accept.json");
49const VERDICTS = new Set(["satisfied", "weak", "missing"]);
50const RUBRIC_TEXT = RUBRIC.map((r, i) => ` ${i + 1}. id="${r.id}" — ${r.name}: ${r.check}`).join("\n");
52// ── Args ──────────────────────────────────────────────────────────────────────
53/** Read a `--flag=N` integer (the `=` form, so parsePerLessonArgs doesn't mistake
54 * a space-separated value for a lesson id). */
55function argInt(flag: string, dflt: number): number {
56 for (const a of process.argv.slice(2)) {
57 if (a.startsWith(`${flag}=`)) {
58 const n = Math.floor(Number(a.slice(flag.length + 1)));
59 if (Number.isFinite(n) && n > 0) return n;
60 }
61 }
62 return dflt;
63}
65// ── Accept-list (the intentional-deviation record) ────────────────────────────
66type Accept = Record<string, Record<string, string>>;
67function loadAccept(): Accept {
68 if (!existsSync(ACCEPT_FILE)) return {};
69 try {
70 const parsed: unknown = JSON.parse(readFileSync(ACCEPT_FILE, "utf8"));
71 return parsed && typeof parsed === "object" ? (parsed as Accept) : {};
72 } catch {
73 return {};
74 }
75}
77// ── Lesson selection (the deterministic signal) ───────────────────────────────
78function selectLessons(ids: string[], limit: number | null): { selected: Lesson[]; unknown: string[] } {
79 const onDisk = new Set(lessonDirsOnDisk());
80 const valid = parseCatalog().filter((l) => l.id !== "index" && onDisk.has(l.id));
81 const validIds = new Set(valid.map((l) => l.id));
82 const unknown = ids.filter((id) => !validIds.has(id));
83 let selected = ids.length ? valid.filter((l) => ids.includes(l.id)) : valid;
84 if (limit) selected = selected.slice(0, limit);
85 return { selected, unknown };
86}
88// ── Parse one evaluator's JSON verdict block ──────────────────────────────────
89interface Verdict {
90 verdict: string;
91 cite: string;
92 gap: string;
93}
94interface Panelist {
95 items: Map<string, Verdict>;
96 offRubric: string;
97}
99function extractJson(text: string): unknown {
100 const tryParse = (s: string): unknown => {
101 try {
102 return JSON.parse(s);
103 } catch {
104 return undefined;
105 }
106 };
107 const fence = text.match(/```(?:json)?\s*([\s\S]*?)```/i);
108 if (fence) {
109 const v = tryParse(fence[1].trim());
110 if (v !== undefined) return v;
111 }
112 const brace = text.match(/\{[\s\S]*\}/);
113 if (brace) return tryParse(brace[0]);
114 return undefined;
115}
117function parsePanelist(summary: string): Panelist | null {
118 const data = extractJson(summary);
119 if (!data || typeof data !== "object") return null;
120 const obj = data as { items?: unknown; offRubric?: unknown };
121 if (!Array.isArray(obj.items)) return null;
122 const items = new Map<string, Verdict>();
123 for (const raw of obj.items) {
124 if (!raw || typeof raw !== "object") continue;
125 const r = raw as Record<string, unknown>;
126 if (typeof r.id !== "string") continue;
127 const verdict = typeof r.verdict === "string" && VERDICTS.has(r.verdict) ? r.verdict : "satisfied";
128 items.set(r.id, {
129 verdict,
130 cite: typeof r.cite === "string" ? r.cite : "",
131 gap: typeof r.gap === "string" ? r.gap : "",
132 });
133 }
134 return { items, offRubric: typeof obj.offRubric === "string" ? obj.offRubric.trim() : "" };
135}
137// ── Prompts ───────────────────────────────────────────────────────────────────
138function systemPrompt(): string {
139 return `You are a PEDAGOGY EVALUATOR — one of an independent panel reviewing ONE Glassbox lesson for TEACHING QUALITY. Glassbox makes hard CS/systems ideas learnable; read this lesson the way a motivated newcomer to the topic would, and judge whether it actually teaches.
141You are NOT checking factual accuracy (another loop owns that), nor whether the structural beats merely exist, nor code/visual/performance. You judge the QUALITY of the teaching, and ONLY against the fixed rubric below — you may not invent new criteria. That bound is deliberate: it keeps this review from drifting into endless taste.
143Your only tools are Read / Grep / Glob. Read what you need; you cannot and must not edit anything.
145THE RUBRIC — for EACH item, return exactly one verdict:
146 - "satisfied": the lesson clearly does this, and it lands. Cite the passage that does it.
147 - "weak": the lesson attempts this but it does not land (too thin, buried, or only partial). Cite where, and name the gap in one sentence.
148 - "missing": the lesson does not do this where it should. Cite where it should have been, and name the gap.
150${RUBRIC_TEXT}
152HARD RULES (these are what make the panel trustworthy):
153 - DEFAULT TO "satisfied" WHEN UNSURE. A panel that flags everything is the failure mode. Mark "weak"/"missing" ONLY when you can point to the specific place the teaching falls short.
154 - CITE OR OMIT. Every "weak"/"missing" verdict MUST quote a real line (file:line, or an exact short phrase) from your own reading. No quote → you may not flag it; return "satisfied".
155 - Judge the lesson AS WRITTEN FOR ITS CHOSEN FORM. A declared survey, history, or essay legitimately makes different moves; do not penalize an intentional form. (The steward records those as accepted separately — you just judge honestly.)
156 - Be DIAGNOSTIC, never prescriptive: name what is missing, not how to rewrite it.
157 - The engine implementation (engine/index.js) is intentionally withheld — judge the TEACHING the reader receives, not the underlying algorithm source.
159OUTPUT: return ONLY a single fenced \`\`\`json code block — no prose before or after — of exactly this shape, with one object per rubric id (all ${RUBRIC.length} present):
161\`\`\`json
162{
163 "items": [
164 {"id": "motivation", "verdict": "satisfied", "cite": "<file:line or exact quoted phrase, or n/a>", "gap": "<one sentence, or empty if satisfied>"}
165 ],
166 "offRubric": "<at most one sentence on something important the rubric does not cover, or empty>"
167}
168\`\`\``;
169}
⋯ 153 lines hidden (lines 171–323)
171function userPrompt(lesson: Lesson, files: string[]): string {
172 return `Lesson under review: \`${lesson.id}\` — "${lesson.title}". Eyebrow / claim it makes the learner: ${lesson.eyebrow || "(none stated)"}.
174Read it as a motivated newcomer to this topic. Its teaching surface (prose sections, interactive labs, lesson-local components), relative to \`${LESSONS_DIR}/${lesson.id}/\`:
175${relList(lesson.id, files)}
177Evaluate every rubric item against what THIS lesson does, and return ONLY the json verdict block.`;
178}
180// ── Aggregate one lesson's panel into a report block ──────────────────────────
181interface LessonReport {
182 lines: string[];
183 failedItems: number;
184 hasFindings: boolean;
185}
187function aggregate(
188 lesson: Lesson,
189 panelists: (Panelist | null)[],
190 panel: number,
191 threshold: number,
192 accepted: Record<string, string>,
193): LessonReport {
194 const answered = panelists.filter((p): p is Panelist => p !== null);
195 const abstained = panel - answered.length;
197 const findingBlocks: string[] = [];
198 const acceptedLines: string[] = [];
199 let clear = 0;
200 let failedItems = 0;
202 for (const item of RUBRIC) {
203 if (item.id in accepted) {
204 acceptedLines.push(` ~ ${item.name} — accepted: ${accepted[item.id]}`);
205 continue;
206 }
207 const flags = answered
208 .map((p) => p.items.get(item.id))
209 .filter((v): v is Verdict => v !== undefined && (v.verdict === "weak" || v.verdict === "missing"));
210 if (flags.length >= threshold) {
211 failedItems++;
212 findingBlocks.push(` ⚠ ${item.name} (${flags.length}/${panel} flagged)`);
213 for (const f of flags) findingBlocks.push(` · ${f.cite || "(no cite)"} — ${f.gap || "(no detail)"}`);
214 } else {
215 clear++;
216 }
217 }
219 const offNotes = answered.map((p) => p.offRubric).filter((s) => s.length > 0);
220 const lines: string[] = [
221 "",
222 "─".repeat(64),
223 `## ${lesson.id} — ${lesson.title}`,
224 `(${panel} evaluators${abstained ? `, ${abstained} unparseable` : ""} · ${clear}/${RUBRIC.length} rubric items clear${acceptedLines.length ? `, ${acceptedLines.length} accepted` : ""})`,
225 ];
226 if (findingBlocks.length) lines.push("", "Teaching gaps — majority-agreed (review & fix, or accept):", ...findingBlocks);
227 if (acceptedLines.length) lines.push("", "Accepted deviations:", ...acceptedLines);
228 if (offNotes.length) lines.push("", "Off-rubric notes (advisory, non-blocking):", ...offNotes.map((o) => ` note: ${o}`));
229 if (!findingBlocks.length) lines.push("", "✓ PASS — every rubric item clear or accepted.");
231 return { lines, failedItems, hasFindings: findingBlocks.length > 0 };
232}
234// ── Orchestrator ──────────────────────────────────────────────────────────────
235async function main(): Promise<void> {
236 const args = parsePerLessonArgs({ concurrency: 2, budget: 4 });
237 const panel = argInt("--panel", 3);
238 const threshold = Math.floor(panel / 2) + 1; // strict majority
239 const accept = loadAccept();
240 const { selected, unknown } = selectLessons(args.ids, args.limit);
242 if (unknown.length) {
243 report([
244 `RESULT: CANNOT RUN — unknown lesson id(s): ${unknown.join(", ")}`,
245 "",
246 `Known lessons: ${parseCatalog()
247 .filter((l) => l.id !== "index" && lessonDirsOnDisk().includes(l.id))
248 .map((l) => l.id)
249 .join(", ")}`,
250 ]);
251 process.exit(1);
252 }
253 if (!selected.length) {
254 report([`RESULT: CANNOT RUN — no lessons found under ${LESSONS_DIR}/.`]);
255 process.exit(1);
256 }
258 const scope = args.ids.length ? `lessons: ${selected.map((l) => l.id).join(", ")}` : "ALL lessons";
260 if (args.dryRun) {
261 report([
262 "Pedagogy — DRY RUN (no evaluators launched, nothing spent)",
263 "teaching-quality panel vs the fixed rubric, READ-ONLY",
264 `Lessons selected: ${selected.length}`,
265 `Panel size: ${panel} (≥${threshold} must agree for a finding)`,
266 `Per-evaluator budget: $${args.budget} (worst-case ceiling: $${selected.length * panel * args.budget})`,
267 `Rubric items: ${RUBRIC.length}`,
268 "",
269 "Would review:",
270 ...selected.map((l) => ` - ${l.id} (${gatherLessonFiles(lessonDir(l.id)).filter((f) => f.endsWith(".jsx")).length} teaching files)`),
271 "",
272 "RESULT: PASS — dry run only. Drop --dry-run to launch the panel.",
273 ]);
274 return;
275 }
277 console.log("Pedagogy loop — teaching-quality panel (opus · effort:max), READ-ONLY");
278 console.log(`Lessons: ${selected.length} · panel: ${panel} (≥${threshold} agree = finding) · per-evaluator budget: $${args.budget}`);
279 console.log(selected.map((l) => l.id).join(", "));
280 console.log("\n(Live `<lesson>#<n> · ToolName` markers below.)\n");
282 const reports = await runPool<Lesson, LessonReport>(selected, args.concurrency, async (lesson) => {
283 const files = gatherLessonFiles(lessonDir(lesson.id)).filter((f) => f.endsWith(".jsx"));
284 const panelists = await Promise.all(
285 Array.from({ length: panel }, (_, i) =>
286 runLoop({
287 systemPrompt: systemPrompt(),
288 allowedTools: ["Read", "Grep", "Glob"],
289 prompt: userPrompt(lesson, files),
290 model: "opus",
291 effort: "max",
292 label: `${lesson.id}#${i + 1}`,
293 maxTurns: 200,
294 maxBudgetUsd: args.budget,
295 }).then((r) => parsePanelist(r.agentSummary)),
296 ),
297 );
298 const result = aggregate(lesson, panelists, panel, threshold, accept[lesson.id] ?? {});
299 console.log(` ✓ ${lesson.id} — ${result.hasFindings ? `${result.failedItems} gap(s)` : "clear"}`);
300 return result;
301 });
303 const totalFailed = reports.reduce((s, r) => s + r.failedItems, 0);
304 const lessonsWithFindings = reports.filter((r) => r.hasFindings).length;
306 report([
307 "Pedagogy — read-only teaching-quality map",
308 `Lessons reviewed: ${selected.length}`,
309 `Panel size: ${panel} (≥${threshold} agree = finding)`,
310 "(read-only — the working tree was not modified)",
311 ...reports.flatMap((r) => r.lines),
312 "",
313 totalFailed === 0
314 ? `RESULT: PASS — every reviewed lesson's teaching rubric is clear or accepted (${scope}).`
315 : `RESULT: FAIL — ${totalFailed} teaching gap(s) across ${lessonsWithFindings} lesson(s). Fix the prose/labs you agree with, or record an intentional deviation in agents/pedagogy-accept.json, then re-run.`,
316 ]);
317 if (totalFailed > 0) process.exitCode = 1;
318}
320main().catch((err: unknown) => {
321 report([`RESULT: ERROR — ${err instanceof Error ? err.message : String(err)}`]);
322 process.exitCode = 1;
323});
Each evaluator returns a strict JSON verdict block, which the harness parses defensively — a malformed or absent block abstains rather than crashing the run, and an unknown verdict falls back to "satisfied". So a flaky evaluator can only ever lower the panel's confidence, never manufacture a finding.
Tolerant parsing: bad input → abstain or default-satisfied, never a false flag.
agents/pedagogy.ts · 323 lines
agents/pedagogy.ts
⋯ 116 lines hidden (lines 1–116)
1/**
2 * Pedagogy loop — Many Hands Engineering, the "does it actually teach?" pass.
3 *
4 * Every other content loop guards a different thing: `content-accuracy` guards
5 * whether the prose is TRUE, `lab-fidelity` whether the labs are REAL,
6 * `collection-coherence` whether the structural beats are PRESENT. None of them
7 * asks the question this collection exists to answer — does the lesson actually
8 * TEACH? This loop does.
9 *
10 * It is qualitative by design (testing whether an agent can compute the answer
11 * measures the agent, not the teaching). A panel of independent evaluators reads
12 * each lesson as a motivated newcomer and scores it against ONE fixed, shared
13 * rubric of named pedagogy practices (see pedagogy-rubric.ts) — and ONLY that
14 * rubric. A teaching gap is reported only when a MAJORITY of the panel
15 * independently flags the same rubric item with a cited passage. That majority +
16 * the fixed rubric + a per-lesson accept-list are the cap: the loop converges to
17 * a quiet PASS instead of producing endless advice.
18 *
19 * Honest about its reference: teaching quality is judgement, so this MAPS, it does
20 * not gate (propose-only, Read/Grep/Glob, never edits). Its rigor comes from the
21 * fixed rubric, strict cite-or-omit, the agreement threshold, and the accept-list
22 * — not from a mechanical truth.
23 *
24 * Usage:
25 * tsx pedagogy.ts # all lessons
26 * tsx pedagogy.ts paxos bloom-filters # only these lesson ids
27 * tsx pedagogy.ts --panel=3 # evaluators per lesson (default 3)
28 * tsx pedagogy.ts --concurrency=2 # lessons reviewed in parallel (default 2)
29 * tsx pedagogy.ts --budget=4 # USD cap PER EVALUATOR (default 4)
30 * tsx pedagogy.ts --dry-run # show the plan + cost ceiling, spend nothing
31 */
32import { existsSync, readFileSync } from "node:fs";
33import { resolve } from "node:path";
34import { APP_ROOT, report, runLoop } from "./lib.js";
35import {
36 LESSONS_DIR,
37 type Lesson,
38 gatherLessonFiles,
39 lessonDir,
40 lessonDirsOnDisk,
41 parseCatalog,
42 parsePerLessonArgs,
43 relList,
44 runPool,
45} from "./per-lesson.js";
46import { RUBRIC } from "./pedagogy-rubric.js";
48const ACCEPT_FILE = resolve(APP_ROOT, "agents", "pedagogy-accept.json");
49const VERDICTS = new Set(["satisfied", "weak", "missing"]);
50const RUBRIC_TEXT = RUBRIC.map((r, i) => ` ${i + 1}. id="${r.id}" — ${r.name}: ${r.check}`).join("\n");
52// ── Args ──────────────────────────────────────────────────────────────────────
53/** Read a `--flag=N` integer (the `=` form, so parsePerLessonArgs doesn't mistake
54 * a space-separated value for a lesson id). */
55function argInt(flag: string, dflt: number): number {
56 for (const a of process.argv.slice(2)) {
57 if (a.startsWith(`${flag}=`)) {
58 const n = Math.floor(Number(a.slice(flag.length + 1)));
59 if (Number.isFinite(n) && n > 0) return n;
60 }
61 }
62 return dflt;
63}
65// ── Accept-list (the intentional-deviation record) ────────────────────────────
66type Accept = Record<string, Record<string, string>>;
67function loadAccept(): Accept {
68 if (!existsSync(ACCEPT_FILE)) return {};
69 try {
70 const parsed: unknown = JSON.parse(readFileSync(ACCEPT_FILE, "utf8"));
71 return parsed && typeof parsed === "object" ? (parsed as Accept) : {};
72 } catch {
73 return {};
74 }
75}
77// ── Lesson selection (the deterministic signal) ───────────────────────────────
78function selectLessons(ids: string[], limit: number | null): { selected: Lesson[]; unknown: string[] } {
79 const onDisk = new Set(lessonDirsOnDisk());
80 const valid = parseCatalog().filter((l) => l.id !== "index" && onDisk.has(l.id));
81 const validIds = new Set(valid.map((l) => l.id));
82 const unknown = ids.filter((id) => !validIds.has(id));
83 let selected = ids.length ? valid.filter((l) => ids.includes(l.id)) : valid;
84 if (limit) selected = selected.slice(0, limit);
85 return { selected, unknown };
86}
88// ── Parse one evaluator's JSON verdict block ──────────────────────────────────
89interface Verdict {
90 verdict: string;
91 cite: string;
92 gap: string;
93}
94interface Panelist {
95 items: Map<string, Verdict>;
96 offRubric: string;
97}
99function extractJson(text: string): unknown {
100 const tryParse = (s: string): unknown => {
101 try {
102 return JSON.parse(s);
103 } catch {
104 return undefined;
105 }
106 };
107 const fence = text.match(/```(?:json)?\s*([\s\S]*?)```/i);
108 if (fence) {
109 const v = tryParse(fence[1].trim());
110 if (v !== undefined) return v;
111 }
112 const brace = text.match(/\{[\s\S]*\}/);
113 if (brace) return tryParse(brace[0]);
114 return undefined;
115}
117function parsePanelist(summary: string): Panelist | null {
118 const data = extractJson(summary);
119 if (!data || typeof data !== "object") return null;
120 const obj = data as { items?: unknown; offRubric?: unknown };
121 if (!Array.isArray(obj.items)) return null;
122 const items = new Map<string, Verdict>();
123 for (const raw of obj.items) {
124 if (!raw || typeof raw !== "object") continue;
125 const r = raw as Record<string, unknown>;
126 if (typeof r.id !== "string") continue;
127 const verdict = typeof r.verdict === "string" && VERDICTS.has(r.verdict) ? r.verdict : "satisfied";
128 items.set(r.id, {
129 verdict,
130 cite: typeof r.cite === "string" ? r.cite : "",
131 gap: typeof r.gap === "string" ? r.gap : "",
132 });
133 }
134 return { items, offRubric: typeof obj.offRubric === "string" ? obj.offRubric.trim() : "" };
135}
⋯ 188 lines hidden (lines 136–323)
137// ── Prompts ───────────────────────────────────────────────────────────────────
138function systemPrompt(): string {
139 return `You are a PEDAGOGY EVALUATOR — one of an independent panel reviewing ONE Glassbox lesson for TEACHING QUALITY. Glassbox makes hard CS/systems ideas learnable; read this lesson the way a motivated newcomer to the topic would, and judge whether it actually teaches.
141You are NOT checking factual accuracy (another loop owns that), nor whether the structural beats merely exist, nor code/visual/performance. You judge the QUALITY of the teaching, and ONLY against the fixed rubric below — you may not invent new criteria. That bound is deliberate: it keeps this review from drifting into endless taste.
143Your only tools are Read / Grep / Glob. Read what you need; you cannot and must not edit anything.
145THE RUBRIC — for EACH item, return exactly one verdict:
146 - "satisfied": the lesson clearly does this, and it lands. Cite the passage that does it.
147 - "weak": the lesson attempts this but it does not land (too thin, buried, or only partial). Cite where, and name the gap in one sentence.
148 - "missing": the lesson does not do this where it should. Cite where it should have been, and name the gap.
150${RUBRIC_TEXT}
152HARD RULES (these are what make the panel trustworthy):
153 - DEFAULT TO "satisfied" WHEN UNSURE. A panel that flags everything is the failure mode. Mark "weak"/"missing" ONLY when you can point to the specific place the teaching falls short.
154 - CITE OR OMIT. Every "weak"/"missing" verdict MUST quote a real line (file:line, or an exact short phrase) from your own reading. No quote → you may not flag it; return "satisfied".
155 - Judge the lesson AS WRITTEN FOR ITS CHOSEN FORM. A declared survey, history, or essay legitimately makes different moves; do not penalize an intentional form. (The steward records those as accepted separately — you just judge honestly.)
156 - Be DIAGNOSTIC, never prescriptive: name what is missing, not how to rewrite it.
157 - The engine implementation (engine/index.js) is intentionally withheld — judge the TEACHING the reader receives, not the underlying algorithm source.
159OUTPUT: return ONLY a single fenced \`\`\`json code block — no prose before or after — of exactly this shape, with one object per rubric id (all ${RUBRIC.length} present):
161\`\`\`json
162{
163 "items": [
164 {"id": "motivation", "verdict": "satisfied", "cite": "<file:line or exact quoted phrase, or n/a>", "gap": "<one sentence, or empty if satisfied>"}
165 ],
166 "offRubric": "<at most one sentence on something important the rubric does not cover, or empty>"
167}
168\`\`\``;
169}
171function userPrompt(lesson: Lesson, files: string[]): string {
172 return `Lesson under review: \`${lesson.id}\` — "${lesson.title}". Eyebrow / claim it makes the learner: ${lesson.eyebrow || "(none stated)"}.
174Read it as a motivated newcomer to this topic. Its teaching surface (prose sections, interactive labs, lesson-local components), relative to \`${LESSONS_DIR}/${lesson.id}/\`:
175${relList(lesson.id, files)}
177Evaluate every rubric item against what THIS lesson does, and return ONLY the json verdict block.`;
178}
180// ── Aggregate one lesson's panel into a report block ──────────────────────────
181interface LessonReport {
182 lines: string[];
183 failedItems: number;
184 hasFindings: boolean;
185}
187function aggregate(
188 lesson: Lesson,
189 panelists: (Panelist | null)[],
190 panel: number,
191 threshold: number,
192 accepted: Record<string, string>,
193): LessonReport {
194 const answered = panelists.filter((p): p is Panelist => p !== null);
195 const abstained = panel - answered.length;
197 const findingBlocks: string[] = [];
198 const acceptedLines: string[] = [];
199 let clear = 0;
200 let failedItems = 0;
202 for (const item of RUBRIC) {
203 if (item.id in accepted) {
204 acceptedLines.push(` ~ ${item.name} — accepted: ${accepted[item.id]}`);
205 continue;
206 }
207 const flags = answered
208 .map((p) => p.items.get(item.id))
209 .filter((v): v is Verdict => v !== undefined && (v.verdict === "weak" || v.verdict === "missing"));
210 if (flags.length >= threshold) {
211 failedItems++;
212 findingBlocks.push(` ⚠ ${item.name} (${flags.length}/${panel} flagged)`);
213 for (const f of flags) findingBlocks.push(` · ${f.cite || "(no cite)"} — ${f.gap || "(no detail)"}`);
214 } else {
215 clear++;
216 }
217 }
219 const offNotes = answered.map((p) => p.offRubric).filter((s) => s.length > 0);
220 const lines: string[] = [
221 "",
222 "─".repeat(64),
223 `## ${lesson.id} — ${lesson.title}`,
224 `(${panel} evaluators${abstained ? `, ${abstained} unparseable` : ""} · ${clear}/${RUBRIC.length} rubric items clear${acceptedLines.length ? `, ${acceptedLines.length} accepted` : ""})`,
225 ];
226 if (findingBlocks.length) lines.push("", "Teaching gaps — majority-agreed (review & fix, or accept):", ...findingBlocks);
227 if (acceptedLines.length) lines.push("", "Accepted deviations:", ...acceptedLines);
228 if (offNotes.length) lines.push("", "Off-rubric notes (advisory, non-blocking):", ...offNotes.map((o) => ` note: ${o}`));
229 if (!findingBlocks.length) lines.push("", "✓ PASS — every rubric item clear or accepted.");
231 return { lines, failedItems, hasFindings: findingBlocks.length > 0 };
232}
234// ── Orchestrator ──────────────────────────────────────────────────────────────
235async function main(): Promise<void> {
236 const args = parsePerLessonArgs({ concurrency: 2, budget: 4 });
237 const panel = argInt("--panel", 3);
238 const threshold = Math.floor(panel / 2) + 1; // strict majority
239 const accept = loadAccept();
240 const { selected, unknown } = selectLessons(args.ids, args.limit);
242 if (unknown.length) {
243 report([
244 `RESULT: CANNOT RUN — unknown lesson id(s): ${unknown.join(", ")}`,
245 "",
246 `Known lessons: ${parseCatalog()
247 .filter((l) => l.id !== "index" && lessonDirsOnDisk().includes(l.id))
248 .map((l) => l.id)
249 .join(", ")}`,
250 ]);
251 process.exit(1);
252 }
253 if (!selected.length) {
254 report([`RESULT: CANNOT RUN — no lessons found under ${LESSONS_DIR}/.`]);
255 process.exit(1);
256 }
258 const scope = args.ids.length ? `lessons: ${selected.map((l) => l.id).join(", ")}` : "ALL lessons";
260 if (args.dryRun) {
261 report([
262 "Pedagogy — DRY RUN (no evaluators launched, nothing spent)",
263 "teaching-quality panel vs the fixed rubric, READ-ONLY",
264 `Lessons selected: ${selected.length}`,
265 `Panel size: ${panel} (≥${threshold} must agree for a finding)`,
266 `Per-evaluator budget: $${args.budget} (worst-case ceiling: $${selected.length * panel * args.budget})`,
267 `Rubric items: ${RUBRIC.length}`,
268 "",
269 "Would review:",
270 ...selected.map((l) => ` - ${l.id} (${gatherLessonFiles(lessonDir(l.id)).filter((f) => f.endsWith(".jsx")).length} teaching files)`),
271 "",
272 "RESULT: PASS — dry run only. Drop --dry-run to launch the panel.",
273 ]);
274 return;
275 }
277 console.log("Pedagogy loop — teaching-quality panel (opus · effort:max), READ-ONLY");
278 console.log(`Lessons: ${selected.length} · panel: ${panel} (≥${threshold} agree = finding) · per-evaluator budget: $${args.budget}`);
279 console.log(selected.map((l) => l.id).join(", "));
280 console.log("\n(Live `<lesson>#<n> · ToolName` markers below.)\n");
282 const reports = await runPool<Lesson, LessonReport>(selected, args.concurrency, async (lesson) => {
283 const files = gatherLessonFiles(lessonDir(lesson.id)).filter((f) => f.endsWith(".jsx"));
284 const panelists = await Promise.all(
285 Array.from({ length: panel }, (_, i) =>
286 runLoop({
287 systemPrompt: systemPrompt(),
288 allowedTools: ["Read", "Grep", "Glob"],
289 prompt: userPrompt(lesson, files),
290 model: "opus",
291 effort: "max",
292 label: `${lesson.id}#${i + 1}`,
293 maxTurns: 200,
294 maxBudgetUsd: args.budget,
295 }).then((r) => parsePanelist(r.agentSummary)),
296 ),
297 );
298 const result = aggregate(lesson, panelists, panel, threshold, accept[lesson.id] ?? {});
299 console.log(` ✓ ${lesson.id} — ${result.hasFindings ? `${result.failedItems} gap(s)` : "clear"}`);
300 return result;
301 });
303 const totalFailed = reports.reduce((s, r) => s + r.failedItems, 0);
304 const lessonsWithFindings = reports.filter((r) => r.hasFindings).length;
306 report([
307 "Pedagogy — read-only teaching-quality map",
308 `Lessons reviewed: ${selected.length}`,
309 `Panel size: ${panel} (≥${threshold} agree = finding)`,
310 "(read-only — the working tree was not modified)",
311 ...reports.flatMap((r) => r.lines),
312 "",
313 totalFailed === 0
314 ? `RESULT: PASS — every reviewed lesson's teaching rubric is clear or accepted (${scope}).`
315 : `RESULT: FAIL — ${totalFailed} teaching gap(s) across ${lessonsWithFindings} lesson(s). Fix the prose/labs you agree with, or record an intentional deviation in agents/pedagogy-accept.json, then re-run.`,
316 ]);
317 if (totalFailed > 0) process.exitCode = 1;
318}
320main().catch((err: unknown) => {
321 report([`RESULT: ERROR — ${err instanceof Error ? err.message : String(err)}`]);
322 process.exitCode = 1;
323});
What it found, and how it stays quiet
Calibrated live on the two gold-standard lessons. bloom-filters → PASS, 8/8 items clear; its survey-dense back third surfaced only as a non-blocking note. paxos → one finding: the hardest step is scaffolded, flagged by 2 of 3 evaluators at Binding.jsx:46-49 — the protocol's crux ("why the highest ballot, not the most common?") asserted as "a short argument shows…" rather than shown. One high-confidence, cited finding — the exact cliff a by-hand analysis predicted, caught mechanically.
Convergence is the accept-list's job. A gap the steward judges intentional (an essay that's passive by design) is recorded in agents/pedagogy-accept.json, keyed lesson → rubric-item → reason, and the loop falls silent on it. RESULT: PASS when every item is clear or accepted; otherwise it exits non-zero with the cited gaps. It maps — it never edits a lesson.
Starts empty; each entry is a deliberate, diffable 'this is intentional'.