What this PR is
The last consequence of retiring ClickStack/Temporal/froot (#22). model-metrics was a dashboard over ClickHouse data that no longer exists; the Tailscale stack tunneled cluster workloads to Ollama on the Mac Studio, and froot was its last caller. Before teardown, a sweep for ollama.llm confirmed no remaining consumer. Both stacks were then shut down live — namespaces model-metrics, llm, and tailscale, the tailscale-operator helm release, the proxyclasses.tailscale.com CRD, and the models.mseeks.me DNS record — and this PR deletes their code and scrubs the references.
The one wrinkle in the live teardown: the CRD delete hung on the lean ProxyClass, whose tailscale.com/finalizer could never clear once the operator was gone. Its finalizer was patched away by hand.
| Removed directory | What it was |
|---|---|
infra/k8s/model-metrics/ | Read-only model-traffic dashboard reading ClickStack's otel_logs |
infra/k8s/tailscale/ | Tailscale operator + lean ProxyClass + egress Service + nginx Host-rewrite proxy (http://ollama.llm:11434) |
dns.tf — the models record goes the way of froot's
Mirror image of #22's DNS change: the mseeks_me_models A record is deleted (destroyed live with a targeted apply; state and zone verified clean), and the LOAD-BEARING host-list comment on the ynab record shrinks again, now to (playeverwhen.com, static).
The trimmed host list, and the everwhen wildcard now running straight into the msull92.com zone where the models record used to sit.
infra/dns.tf · 378 lines
infra/dns.tf
⋯ 68 lines hidden (lines 1–68)
1# DNS — reverse-engineered from the live DigitalOcean zones and imported into
2# state (target: `terraform plan` reports no changes).
3#
4# SOA and the default DigitalOcean NS records (ns1/2/3.digitalocean.com) are
5# managed automatically by DO and are intentionally NOT declared here.
6#
7# Every zone here is an iCloud Custom Email Domain (Apple) setup: MX -> icloud
8# mail, an SPF TXT, an apple-domain verification TXT, and a DKIM CNAME.
9# (mseeks.me additionally carries the zo-k8s load-balancer A records below.)
10#
11# SPF quoting gotcha: write SPF TXT values WITHOUT surrounding quotes (e.g.
12# `v=spf1 include:icloud.com ~all`). DigitalOcean adds the DNS wire quotes on
13# its own, so escaping quotes into the value makes `terraform apply` CREATE a
14# doubly-quoted record (`""v=spf1 ...""`) that SPF verifiers reject. `plan`
15# won't warn you: the provider compares TXT values quote-insensitively, so even
16# a broken live record shows no drift. The mseeks.me / msull92.com SPF entries
17# still carry escaped quotes but resolve clean (created pre-terraform); left
18# as-is because recreating them from the quoted value would re-break them.
19# playeverwhen.com and 2ls.tech below use the correct unquoted form.
21# ============================== mseeks.me ==============================
23resource "digitalocean_domain" "mseeks_me" {
24 name = "mseeks.me"
25}
27resource "digitalocean_record" "mseeks_me_mx_01" {
28 domain = digitalocean_domain.mseeks_me.name
29 type = "MX"
30 name = "@"
31 value = "mx01.mail.icloud.com."
32 priority = 10
33 ttl = 14400
34}
36resource "digitalocean_record" "mseeks_me_mx_02" {
37 domain = digitalocean_domain.mseeks_me.name
38 type = "MX"
39 name = "@"
40 value = "mx02.mail.icloud.com."
41 priority = 10
42 ttl = 14400
43}
45resource "digitalocean_record" "mseeks_me_apple_domain" {
46 domain = digitalocean_domain.mseeks_me.name
47 type = "TXT"
48 name = "@"
49 value = "apple-domain=Y94Tzo0dJVhRhQxf"
50 ttl = 3600
51}
53resource "digitalocean_record" "mseeks_me_spf" {
54 domain = digitalocean_domain.mseeks_me.name
55 type = "TXT"
56 name = "@"
57 value = "\"v=spf1 include:icloud.com ~all\""
58 ttl = 3600
59}
61resource "digitalocean_record" "mseeks_me_dkim" {
62 domain = digitalocean_domain.mseeks_me.name
63 type = "CNAME"
64 name = "sig1._domainkey"
65 value = "sig1.dkim.mseeks.me.at.icloudmailadmin.com."
66 ttl = 43200
67}
69# ynab.mseeks.me -> the zo-k8s ingress Load Balancer. The LB itself is owned by
70# the DigitalOcean CCM (via the ingress-nginx Service, see infra/k8s/ingress-nginx);
71# Terraform only READS it through data.digitalocean_loadbalancer.ingress
72# (loadbalancer.tf) to keep this record's IP in sync automatically.
73#
74# LOAD-BEARING — keep it. This is the LB's advertised hostname
75# (do-loadbalancer-hostname in infra/k8s/ingress-nginx/values.yaml). DO LBs don't
76# hairpin, so cert-manager's HTTP-01 self-check for EVERY host on the LB
77# (playeverwhen.com, static) resolves this name out via DNS
78# and back in. The ynab-agent app it was originally named for has been retired,
79# but the record stays as pure LB infra — deleting it breaks TLS renewal
80# cluster-wide.
81resource "digitalocean_record" "mseeks_me_ynab" {
82 domain = digitalocean_domain.mseeks_me.name
83 type = "A"
84 name = "ynab"
85 # Sourced from the live LB; if the LB is recreated (new IP), a plain
86 # `terraform apply` re-points this record — no hardcoded address to chase.
87 value = data.digitalocean_loadbalancer.ingress.ip
88 # Short TTL: this points at a DO LB whose IP can change if the LB is recreated,
89 # so keep resolver caches from pinning a stale address for long.
90 ttl = 300
91}
⋯ 37 lines hidden (lines 92–128)
93# glassbox.mseeks.me -> Vercel. Glassbox is hosted on Vercel, not the cluster:
94# the project + custom domain are managed in the Vercel dashboard and deploys
95# ride its GitHub integration (push to main in mseeks/glassbox), so the only
96# infra Terraform owns is this record. Vercel terminates TLS at its edge — no
97# cert-manager involvement. The value is this project's own CNAME target, the
98# one the dashboard issued when glassbox.mseeks.me was added to the project
99# (Vercel mints per-project targets; the old universal cname.vercel-dns.com is
100# legacy). If the Vercel project is ever deleted and recreated, it gets a new
101# target — read the fresh value off the project's Domains tab, and only
102# re-point this record once the domain is attached there (flipped early, the
103# host serves Vercel's DEPLOYMENT_NOT_FOUND page).
104resource "digitalocean_record" "mseeks_me_glassbox" {
105 domain = digitalocean_domain.mseeks_me.name
106 type = "CNAME"
107 name = "glassbox"
108 value = "e24be5a7fd9d016f.vercel-dns-017.com."
109 # Unlike the LB A records above, the target here is stable for the life of
110 # the Vercel project, so no short-TTL churn guard is needed.
111 ttl = 3600
112}
114# static.mseeks.me -> the same zo-k8s ingress Load Balancer. Public entrypoint
115# for the `static` drop-and-share site (https://static.mseeks.me; see
116# infra/k8s/static) and the host cert-manager gets a Let's Encrypt cert for.
117# Same read-only LB-IP sourcing as ynab above — no hardcoded address to
118# chase.
119resource "digitalocean_record" "mseeks_me_static" {
120 domain = digitalocean_domain.mseeks_me.name
121 type = "A"
122 name = "static"
123 value = data.digitalocean_loadbalancer.ingress.ip
124 # Short TTL for the same reason as ynab: the target is a DO LB whose
125 # IP can change if the LB is recreated.
126 ttl = 300
127}
129# *.everwhen.mseeks.me -> the same zo-k8s ingress Load Balancer. Wildcard record
130# so ephemeral per-PR preview hosts (pr-<N>.everwhen.mseeks.me; see
131# infra/k8s/everwhen-preview) resolve to the LB without a per-PR DNS change. The
132# bare everwhen.mseeks.me A record is gone — the Everwhen app moved to
133# playeverwhen.com — so this wildcard now serves per-PR previews only (it never
134# matched the bare apex anyway; a wildcard covers deeper labels, not the name
135# itself). cert-manager issues a per-host Let's Encrypt cert on first request
136# (HTTP-01), so each preview gets its own TLS with no wildcard cert needed. Same
137# read-only LB-IP sourcing as the records above — no hardcoded address to chase.
138resource "digitalocean_record" "mseeks_me_everwhen_wildcard" {
139 domain = digitalocean_domain.mseeks_me.name
140 type = "A"
141 name = "*.everwhen"
142 value = data.digitalocean_loadbalancer.ingress.ip
143 # Short TTL for the same reason as the others: the target is a DO LB whose IP
144 # can change if the LB is recreated.
145 ttl = 300
146}
148# ============================== msull92.com ==============================
⋯ 230 lines hidden (lines 149–378)
150resource "digitalocean_domain" "msull92_com" {
151 name = "msull92.com"
152}
154resource "digitalocean_record" "msull92_com_mx_01" {
155 domain = digitalocean_domain.msull92_com.name
156 type = "MX"
157 name = "@"
158 value = "mx01.mail.icloud.com."
159 priority = 10
160 ttl = 14400
161}
163resource "digitalocean_record" "msull92_com_mx_02" {
164 domain = digitalocean_domain.msull92_com.name
165 type = "MX"
166 name = "@"
167 value = "mx02.mail.icloud.com."
168 priority = 10
169 ttl = 14400
170}
172resource "digitalocean_record" "msull92_com_apple_domain" {
173 domain = digitalocean_domain.msull92_com.name
174 type = "TXT"
175 name = "@"
176 value = "apple-domain=SLKQpEYsouHgb3mI"
177 ttl = 3600
178}
180resource "digitalocean_record" "msull92_com_spf" {
181 domain = digitalocean_domain.msull92_com.name
182 type = "TXT"
183 name = "@"
184 value = "\"v=spf1 include:icloud.com ~all\""
185 ttl = 3600
186}
188resource "digitalocean_record" "msull92_com_dkim" {
189 domain = digitalocean_domain.msull92_com.name
190 type = "CNAME"
191 name = "sig1._domainkey"
192 value = "sig1.dkim.msull92.com.at.icloudmailadmin.com."
193 ttl = 43200
194}
196# ============================== 2ls.tech ==============================
197# HCL resource names can't start with a digit, so these use a `_2ls_tech`
198# prefix (a leading underscore is valid) rather than the plain `2ls_tech`
199# the mseeks_me/msull92_com convention would suggest.
201resource "digitalocean_domain" "_2ls_tech" {
202 name = "2ls.tech"
203}
205resource "digitalocean_record" "_2ls_tech_mx_01" {
206 domain = digitalocean_domain._2ls_tech.name
207 type = "MX"
208 name = "@"
209 value = "mx01.mail.icloud.com."
210 priority = 10
211 ttl = 14400
212}
214resource "digitalocean_record" "_2ls_tech_mx_02" {
215 domain = digitalocean_domain._2ls_tech.name
216 type = "MX"
217 name = "@"
218 value = "mx02.mail.icloud.com."
219 priority = 10
220 ttl = 14400
221}
223resource "digitalocean_record" "_2ls_tech_apple_domain" {
224 domain = digitalocean_domain._2ls_tech.name
225 type = "TXT"
226 name = "@"
227 value = "apple-domain=KAYDndJUw5TrEwCe"
228 ttl = 3600
229}
231# Google Search Console domain verification (TXT at the apex). Value UNQUOTED
232# like the apple-domain record above — DigitalOcean adds the DNS wire quotes
233# itself (see the SPF quoting gotcha at the top of this file).
234resource "digitalocean_record" "_2ls_tech_google_site_verification" {
235 domain = digitalocean_domain._2ls_tech.name
236 type = "TXT"
237 name = "@"
238 value = "google-site-verification=9e1Dm56-fdkP2jTv1pGNI5II2ol0pR6Xtw8RtS1gXwM"
239 ttl = 3600
240}
242# SPF value UNQUOTED on purpose — see the SPF quoting gotcha at the top of this
243# file. Recreated once (terraform apply -replace) to strip the doubly-quoted
244# `""v=spf1 ...""` the original escaped-quote value produced on create.
245resource "digitalocean_record" "_2ls_tech_spf" {
246 domain = digitalocean_domain._2ls_tech.name
247 type = "TXT"
248 name = "@"
249 value = "v=spf1 include:icloud.com ~all"
250 ttl = 3600
251}
253resource "digitalocean_record" "_2ls_tech_dkim" {
254 domain = digitalocean_domain._2ls_tech.name
255 type = "CNAME"
256 name = "sig1._domainkey"
257 value = "sig1.dkim.2ls.tech.at.icloudmailadmin.com."
258 ttl = 43200
259}
261# 2ls.tech (+ www) -> Vercel. The Two Ls company site is hosted on Vercel, not
262# the cluster: the project + both domains are managed in the Vercel dashboard
263# and deploys ride its GitHub integration (push to main in mseeks/2ls.tech).
264# Vercel terminates TLS at its edge (no cert-manager), serves the apex, and
265# redirects www to it. The mail/verification records above are unaffected.
266#
267# Values: both are this project's own targets, issued by the dashboard when
268# the domains were added (Vercel mints per-project values now; the universal
269# 76.76.21.21 / cname.vercel-dns.com are legacy). If the Vercel project is
270# ever deleted and recreated it gets new targets — read them off the
271# project's Domains tab, and only re-point these records once the domains
272# are attached there.
273resource "digitalocean_record" "_2ls_tech_apex" {
274 domain = digitalocean_domain._2ls_tech.name
275 type = "A"
276 name = "@"
277 value = "216.198.79.1"
278 # Stable Vercel target (unlike the LB IP this used to track), so no
279 # short-TTL churn guard is needed.
280 ttl = 3600
281}
283resource "digitalocean_record" "_2ls_tech_www" {
284 domain = digitalocean_domain._2ls_tech.name
285 type = "CNAME"
286 name = "www"
287 value = "bae588b1e31b34e9.vercel-dns-017.com."
288 ttl = 3600
289}
291# ============================== playeverwhen.com ==============================
293resource "digitalocean_domain" "playeverwhen_com" {
294 name = "playeverwhen.com"
295}
297resource "digitalocean_record" "playeverwhen_com_mx_01" {
298 domain = digitalocean_domain.playeverwhen_com.name
299 type = "MX"
300 name = "@"
301 value = "mx01.mail.icloud.com."
302 priority = 10
303 ttl = 14400
304}
306resource "digitalocean_record" "playeverwhen_com_mx_02" {
307 domain = digitalocean_domain.playeverwhen_com.name
308 type = "MX"
309 name = "@"
310 value = "mx02.mail.icloud.com."
311 priority = 10
312 ttl = 14400
313}
315resource "digitalocean_record" "playeverwhen_com_apple_domain" {
316 domain = digitalocean_domain.playeverwhen_com.name
317 type = "TXT"
318 name = "@"
319 value = "apple-domain=DEESFXkRhJ4wE6Lm"
320 ttl = 3600
321}
323# Google Search Console domain verification (TXT at the apex). Value UNQUOTED
324# like the apple-domain record above — DigitalOcean adds the DNS wire quotes
325# itself (see the SPF quoting gotcha at the top of this file).
326resource "digitalocean_record" "playeverwhen_com_google_site_verification" {
327 domain = digitalocean_domain.playeverwhen_com.name
328 type = "TXT"
329 name = "@"
330 value = "google-site-verification=1FSmsnv2MPG_mKoCrdlSABXfyXFTaeVBETNRmLNi85o"
331 ttl = 3600
332}
334# SPF value UNQUOTED on purpose — see the SPF quoting gotcha at the top of this
335# file. DigitalOcean adds the DNS wire quotes itself.
336resource "digitalocean_record" "playeverwhen_com_spf" {
337 domain = digitalocean_domain.playeverwhen_com.name
338 type = "TXT"
339 name = "@"
340 value = "v=spf1 include:icloud.com ~all"
341 ttl = 3600
342}
344resource "digitalocean_record" "playeverwhen_com_dkim" {
345 domain = digitalocean_domain.playeverwhen_com.name
346 type = "CNAME"
347 name = "sig1._domainkey"
348 value = "sig1.dkim.playeverwhen.com.at.icloudmailadmin.com."
349 ttl = 43200
350}
352# playeverwhen.com (apex) -> the zo-k8s ingress Load Balancer. Public entrypoint
353# for the Everwhen SSR app (https://playeverwhen.com; see infra/k8s/everwhen),
354# migrated here from everwhen.mseeks.me. cert-manager gets a Let's Encrypt cert
355# for it (HTTP-01, so the apex must resolve to the LB). An apex A record (a CNAME
356# is illegal at a zone apex) works because the value is the LB IP, sourced
357# read-only from data.digitalocean_loadbalancer.ingress — no hardcoded address to
358# chase. Coexists fine with the iCloud MX/TXT records above.
359resource "digitalocean_record" "playeverwhen_com_apex" {
360 domain = digitalocean_domain.playeverwhen_com.name
361 type = "A"
362 name = "@"
363 value = data.digitalocean_loadbalancer.ingress.ip
364 # Short TTL: the target is a DO LB whose IP can change if the LB is recreated.
365 ttl = 300
366}
368# www.playeverwhen.com -> the same LB. Serves only a 301 redirect to the apex
369# (see the everwhen-www-redirect Ingress in infra/k8s/everwhen); cert-manager
370# issues its own cert so the redirect is reachable over HTTPS. Same read-only
371# LB-IP sourcing as the apex.
372resource "digitalocean_record" "playeverwhen_com_www" {
373 domain = digitalocean_domain.playeverwhen_com.name
374 type = "A"
375 name = "www"
376 value = data.digitalocean_loadbalancer.ingress.ip
377 ttl = 300
378}
CLAUDE.md — the stack list is gone
The tailscale entry was the stack list's last survivor, so the list itself is retired: the retirement parenthetical now names all five stacks, and a one-line inventory says what infra/k8s/ still holds (the sites, everwhen and its previews, and cluster plumbing). One fact is deliberately preserved: Ollama still runs on the Mac Studio via the com.zo.ollama LaunchAgent — only the cluster's tunnel to it was removed. That sentence is now the only place the LaunchAgent is documented, since the tailscale README that covered it is deleted.
The Infrastructure section, now describing what runs rather than what used to.
CLAUDE.md · 90 lines
CLAUDE.md
⋯ 78 lines hidden (lines 1–78)
1# Zo
3You are **Zo**, the AI assistant for this personal workspace, home to the user's projects, code, research, and writing.
5## Identity
7You are precise, direct, and low-ceremony, a craftsperson about correctness. You do exactly what's asked and say what you think plainly. Your core domains are engineering, research, and writing.
9## Many Hands Engineering (MHE)
11MHE is the framework underpinning the agentic and code-maintenance work across this workspace. Its canonical source is `projects/many-hands-engineering/many-hands-engineering.typ` (plain-text Typst source; `many-hands-engineering.pdf` is the rendered human version). Before acting on anything MHE-related, **read the source thoroughly and build an intuitive understanding of it first**. Don't pattern-match from fragments or secondhand summaries.
13## Guidelines
15- When the user expresses a lasting preference ("from now on"), update this file to reflect that preference.
16- Do exactly what's asked, and don't expand scope. Surface adjacent improvements as suggestions, not actions. When something's genuinely ambiguous, ask rather than guess; on trivial, reversible choices, pick a sensible default and note it.
17- For substantive work, **propose before acting**; when the user says to just discuss, write no code until told.
18- Bias hard to **KISS**: the simplest design that works, accreted slowly. Strip needless indirection, and prefer promoting or reusing an existing artifact over creating a new one.
19- **Chat style:** terse and neutral. Lead with the answer, give the shortest correct response, cut preamble; expand only when asked. Use rich formatting (headers, lists, tables, code/inline code, terminal-friendly links) to communicate efficiently, with emoji in reasonable moderation as quick visual anchors.
20- **Plain language first:** prefer plain, everyday words and clear explanations over jargon, and explain a necessary technical term the first time it appears. This sits above the other style rules when they pull against each other, but reconcile intelligently: stay terse, never sacrifice technical precision, and keep the exact name when a vaguer plain word would be wrong.
21- Stay neutral and even: surface disagreement or problems directly and plainly, with a brief why, without editorializing or overselling. Say it once, then stay open to well-supported counterpoints.
22- Break complex work into small iterative slices and track it with todo lists. Narrate sparingly: a line at meaningful milestones, not step-by-step.
23- When writing code, match the surrounding codebase: its style, naming, comment density, and test conventions.
24- Keep personal-workspace plumbing (static/k8s deploy, private-repo references, local paths) out of reusable or public artifacts. It belongs only in zo's root CLAUDE.md. Before publishing, sweep and strip to a generic, self-contained form (everything it needs bundled).
25- Connected tools: read freely, but confirm before any write or send. Query judiciously. Don't over-fetch and pollute context.
26- Reads run free. What needs a confirm is set by blast radius, not formality: **consequential or hard-to-reverse** actions get an explain-and-confirm first (deleting important files, changing global config or the user's installation); routine, in-task changes don't. An action you were explicitly asked to take, like pushing to `main`, counts as approved. If a suggested fix looks wrong for the setup, say so instead of running it.
27- Never commit secrets. Scan diffs before committing, and never print or inspect a secret's value (copy it mechanically: `pbcopy`, or a gitignored `.env` between repos). Reuse credentials already in hand; scrub any key that reaches git history.
28- In personal repos, commit and push to `main` when asked. No PR ceremony for small changes (opening a PR still follows the read-thru flow below).
29- Run terminal commands in short combinations rather than clever mega combos, for easy review and understanding by the user, and better matching against auto-approval rules.
30- Always check if you're working in a git worktree.
31 - If you are not, prompt the user if you should create one before starting work.
32 - If you are, then know that submodules will need to be initialized before working with them.
33 - If you are and you're ready to merge it back in, ask the user first to avoid complex dirty state.
34- On `zo`'s `main`, keep every project's submodule pointer at the **tip of that project's own `main`**. Periodically fetch upstream and advance each pointer (committing the bumps) so they stay aligned. A current, uniform starting point keeps worktree checkouts consistent and avoids confusion over where each one begins.
36## Writing
38How to write across everything you produce: chat replies, prose and docs, code comments (not code itself), and prompts you author for models. Apply with judgment for the context; these are defaults, not absolute bans.
40- **Rhythm:** mix short and long sentences; don't let them settle into a uniform length. Break up long sentences. Vary word choice, and prefer plain, short words over long or inflated ones.
41- **Avoid these words and phrases** (prefer concrete, plain alternatives): leverage, robust, scalable, innovative, cutting-edge, state-of-the-art, seamless, synergy, synergistic, passionate, thrilled, honored, delighted, spearhead, pivotal, transformative, transformational, paradigm, unlock, unleash, empower, deep dive, delve, navigate the landscape, ever-evolving, fast-paced, dynamic environment, moreover, furthermore, additionally, in essence, in today's, in the realm of, at the intersection of, look forward to, would welcome the opportunity, excited to discuss, I'd love to discuss, tapestry, underscore, testament to, speaks volumes.
42- **Avoid these constructions:** "not just X but Y"; reflexive "X, Y, and Z" triads; "it is important / worth noting / essential / crucial that…" hedging (state it plainly); restating the same point ("in other words", "to put it simply", "put simply", "more specifically", "that is to say"), say it once.
43- **Punctuation:** go light on em-dashes (use commas, periods, or parentheses); use plain quotes, not curly ones; and split overly long, comma-stacked sentences.
45## Code
47**Stack defaults**
49| Concern | Standard |
50|---|---|
51| UI language | TypeScript |
52| Python deps | `uv` |
53| Python format + lint | `ruff` |
54| Python types | `mypy` |
55| TypeScript format / lint | `prettier` / `eslint` |
56| Config (& secrets) | Pydantic Settings (`SecretStr` for secrets) |
57| System-wide installs | `brew` |
58| Local model calls | **Gemma 4 26B MoE** via Ollama (all model calls, going forward) |
60- **Python quality bar:** strict DDD with MyPy-enforced typing so invalid states are unrepresentable; functional style where it fits, without ceremony; Google style guide (80-char lines); tests valued for meaning, not coverage.
61- **Docs:** stale docs/comments are bugs; fix them when a change invalidates one. Respect the user's hand-edits and regenerate derived artifacts *from* them; never overwrite. Write for humans and future agents alike; comments justify *why*.
63## Verification
65- **Scale rigor to the change.** Small, low-risk changes get a light touch: a quick check and a plain note of what you did. Save the full bar below for substantial or outward-facing work; don't put full-suite runs, boot-and-verify, or subagent review on a one-liner.
66- **Full suite green (for substantive changes):** nothing is done until the project's full check suite passes: tests, formatters, linters (and the project's loops), run in sensible order and re-run after each fix to confirm it resolved. Show the evidence when asked or when the result is non-obvious. Add the *discriminating* test that only the new layer could catch, not coverage padding.
67- **Boot it and lay eyes on it:** passing tests ≠ working UX. For runtime/UI work, boot the server and hand over the URL, *and* self-verify (screenshots, click controls, hunt runtime errors). For telemetry/infra, give a concrete way to observe each signal flowing: a ready query, API call, or endpoint.
68- **Independent adversarial review:** delegate serious analysis/verification to independent subagents; don't bias them with the answer you want. Canonical form: **adversarial review** (e.g. alpha-player playtests for UI, fact-checking every claim for prose).
70## Pull requests
72- Whenever you open a PR, use the **`make-read-thru`** skill to generate a code-level reading guide *of that PR's content*: a walkthrough a reviewer can read top to bottom and vouch for (size it to the change; the skill's `references/depth-and-scope.md` has the tiers).
73- **Deploy** the guide to the static site: write it to `projects/static/public/read-thru/<repo>/pr-<n>-<slug>.html` (slug = short kebab summary of the PR), push the static repo's `main`, then run `infra/k8s/static/install.sh`.
74- In the PR body, add a **bare link, no preamble** to the page's clean URL (no `.html` suffix): `https://static.mseeks.me/read-thru/<repo>/pr-<n>-<slug>`.
75- Deploy/namespacing above is **personal-workspace-specific and lives only here**. The `make-read-thru` skill knows nothing about the static site or k8s; it just produces the HTML.
76- **Title & body:** the title is human, **terse, and specific** to the change, never generic. The body is concise, well-structured, and comprehensible, focused on *what* the change is and *why* the PR exists at all. Avoid "meta talk": don't leak conversation or process concerns into the PR, and don't over-explain things that aren't immediately relevant.
77- **Chaining dependent PRs:** when changes build on each other, chain the PRs by targeting each other in order so each reviews cleanly.
79## Infrastructure
81- `infra/` holds Terraform for personal DigitalOcean infrastructure (single source of truth for the DO account). Currently DNS plus a two-node DOKS cluster (`infra/kubernetes.tf`: `zo-k8s`, 2x `s-4vcpu-8gb` nodes in nyc3, ~$96/mo); App Platform static sites/apps later. (The ClickStack/HyperDX observability stack, the Temporal cluster, the froot worker, the model-metrics dashboard, and the Tailscale/Ollama egress that ran here were retired in July 2026. Ollama itself still runs on the Mac Studio via the `com.zo.ollama` LaunchAgent — only the cluster's tunnel to it was removed.)
82- On-cluster workloads are deployed with **kubectl/Helm + install scripts, NOT Terraform** (Terraform owns only the cluster). What remains in `infra/k8s/` is the sites (`static`, `2ls`), everwhen (+ `everwhen-preview`), and cluster plumbing (`ingress-nginx`, `cert-manager`).
83- **Headroom:** since the 2026 stack retirements the request budget has real slack. Requests are reservations, not usage, and there's **no metrics-server**, so `kubectl top` won't work in-cluster (DO's dashboard is the source of truth for real load). Still pin **small requests** when adding pods — a pod goes `Pending` on request budget, not actual load.
84- Secrets never go in the repo: the DO token is read from `DIGITALOCEAN_ACCESS_TOKEN` via gitignored `infra/.env` (used by both terraform and doctl).
85- Project-specific deployments live centrally in `infra/` and point *outward* at public repos; never put Terraform inside the (open-source) project repos.
86- **Everwhen PR previews (ephemeral deploys).** (Prod itself now lives at `https://playeverwhen.com`; previews still live on `*.everwhen.mseeks.me`, which is why that wildcard DNS record stays.) When asked to deploy / preview / "spin up an ephemeral" for everwhen PR `<N>`, stand up a throwaway deploy of that PR at `https://pr-<N>.everwhen.mseeks.me`, then hand back the URL:
87 - Ensure the PR has the `preview` label (`gh pr edit <N> --repo mseeks/everwhen --add-label preview`) — that label is what makes the everwhen CI build `ghcr.io/mseeks/everwhen:pr-<N>`. **Wait for that CI build to finish** before deploying, or the pod sits in `ImagePullBackOff`.
88 - With kubectl on zo-k8s: `infra/k8s/everwhen-preview/deploy-pr.sh <N>` (idempotent — re-run after new commits land a fresh `:pr-<N>`). For a one-off without a PR build, set `EW_PREVIEW_IMAGE` (e.g. `:latest` or a `:<sha>`).
89 - Tear down when merged/done: `infra/k8s/everwhen-preview/teardown-pr.sh <N>`, or `… --merged` to reap every closed PR's preview.
90 - Previews are **public** (no auth, like prod), reuse prod's `everwhen-secrets` with the canonical origin pointed at the preview host, and reserve ~256Mi each — tear them down promptly when done. Full docs: `infra/k8s/everwhen-preview/README.md`.
kubernetes.tf, and what a reviewer should double-check
kubernetes.tf gets a two-word demotion: the cluster's job description drops "the Ollama egress". Everything else in the file is untouched.
What's not in the diff, for the record: the Mac-side tailscale serve --tcp 11434 still exposes Ollama to the tailnet (stop it there if it should go dark), the zo-k8s-operator device may linger in the Tailscale admin console, and infra/.env still holds TS_OAUTH_* and model-metrics basic-auth secrets. All three are hand-cleanup items outside the repo.
The header after the demotion; the resource block is unchanged.
infra/kubernetes.tf · 99 lines
infra/kubernetes.tf
1# DigitalOcean Kubernetes (DOKS) — a small, persistent cluster. Its job is to
2# run our managed-k8s workloads (the k8s/ stacks: the sites and everwhen) and
3# give us a place to learn/experiment without standing anything up by hand.
4# Modest, but a real running cluster.
5#
6# Cost (on top of the ~$5/mo Spaces state bucket):
7# - Control plane .......... $0/mo (standard, non-HA — see `ha` below)
8# - 2x s-4vcpu-8gb nodes ... $96/mo (~$0.0714/hr each, billed hourly)
9# - LoadBalancer (if any) .. $12/mo (~$0.0179/hr) — only while a
10# type=LoadBalancer Service exists; the
11# echo smoke test (k8s/echo-demo.yaml)
12# creates one, then we delete it.
13# Everything is billed hourly, so the per-node cost prorates if the pool is ever
14# scaled down or the cluster is torn down with `terraform destroy`.
⋯ 85 lines hidden (lines 15–99)
16# Latest patch release within a pinned MINOR series. A prefix (not a hard pin)
17# means we automatically track security patches inside 1.36.x, while moving to a
18# new minor stays a deliberate, reviewed edit to this file. When DO eventually
19# retires 1.36, `terraform plan` will fail with "no versions match" — that error
20# is the signal to bump the prefix to a currently-supported minor:
21# doctl kubernetes options versions
22data "digitalocean_kubernetes_versions" "current" {
23 version_prefix = "1.36."
24}
26resource "digitalocean_kubernetes_cluster" "zo" {
27 name = "zo-k8s"
28 region = "nyc3" # same region as the Spaces state bucket — keep it all local
29 version = data.digitalocean_kubernetes_versions.current.latest_version
31 # Single, non-HA control plane. The standard control plane is FREE; an HA
32 # control plane adds ~$40/mo and isn't worth it for this small personal cluster.
33 ha = false
35 # Terraform owns upgrades. When the data source above surfaces a newer 1.36
36 # patch, a normal `apply` rolls it out under our review. DO's automatic
37 # maintenance-window upgrades are left OFF so the version never changes behind
38 # our back. (surge_upgrade adds a temporary extra node during an upgrade so
39 # the pool is replaced without going fully offline.)
40 auto_upgrade = false
41 surge_upgrade = true
43 # IMPORTANT for clean teardown and no surprise bills: on `terraform destroy`,
44 # also delete resources the cluster provisioned out-of-band — most notably the
45 # DO Load Balancer that a type=LoadBalancer Service creates, plus any block
46 # storage volumes (PVCs). Without this, a destroy can leave an orphaned LB
47 # quietly billing $12/mo.
48 destroy_all_associated_resources = true
50 tags = ["zo", "terraform", "sandbox"]
52 # 4 vCPU / 8 GB per node (originally sized for the since-retired ClickStack
53 # observability stack; the remaining workloads fit with plenty of headroom).
54 # NOTE: a node pool's `size` slug is immutable on the DO API, so changing it
55 # REPLACES the pool — the old node is drained/destroyed and a new one
56 # provisioned. Anything NOT on a do-block-storage PVC is lost (PVCs detach and
57 # re-attach to the new node, so their data survives). `node_count`, unlike
58 # `size`, is mutable — scaling it adds/removes nodes in place (no replacement);
59 # DOKS cordons/drains a removed node and its PVC-backed pods re-attach to the
60 # survivor. No autoscaling.
61 node_pool {
62 name = "default"
63 size = "s-4vcpu-8gb"
64 node_count = 1
65 auto_scale = false
66 tags = ["zo", "sandbox"]
67 }
68}
70# --- Outputs -----------------------------------------------------------------
71# Non-sensitive metadata + the one command you actually need to start using the
72# cluster. We intentionally do NOT output the raw kubeconfig: the provider's
73# kubeconfig carries a short-lived token and would land in remote state. Instead
74# fetch creds on demand with doctl (which refreshes the token automatically).
76output "kubernetes_cluster_id" {
77 value = digitalocean_kubernetes_cluster.zo.id
78 description = "DOKS cluster UUID."
79}
81output "kubernetes_cluster_name" {
82 value = digitalocean_kubernetes_cluster.zo.name
83 description = "DOKS cluster name."
84}
86output "kubernetes_endpoint" {
87 value = digitalocean_kubernetes_cluster.zo.endpoint
88 description = "Kubernetes API server endpoint (not a secret)."
89}
91output "kubernetes_version" {
92 value = digitalocean_kubernetes_cluster.zo.version
93 description = "Resolved Kubernetes version running on the cluster."
94}
96output "kubeconfig_command" {
97 value = "doctl kubernetes cluster kubeconfig save ${digitalocean_kubernetes_cluster.zo.name}"
98 description = "With DIGITALOCEAN_ACCESS_TOKEN set, run this to merge cluster credentials into ~/.kube/config."
99}