What this PR is
The observability stack (ClickStack/HyperDX), the Temporal cluster, and the froot worker have been shut down on zo-k8s. The live teardown already happened: the three namespaces, four helm releases, the ClickHouse/MongoDB CRDs, ~43Gi of block-storage PVCs, and the froot.mseeks.me DNS record (destroyed with a targeted terraform apply) are gone. This PR is the code half: it deletes the three infra/k8s/ stack directories (~2.9k lines) and scrubs every doc and comment the shutdown made stale.
Almost all of the diff is deletion. The review question is not "is this code right" but "did anything get left pointing at a stack that no longer exists" — the sections below walk the places that changed rather than vanished.
| Removed directory | What it was |
|---|---|
infra/k8s/observability/ | ClickStack/HyperDX: ClickHouse + Keeper + MongoDB + OTel collectors (Helm) |
infra/k8s/temporal/ | Lean Temporal cluster: auto-setup server + Postgres + UI + its OTel collector |
infra/k8s/froot/ | The froot worker and its code-maintenance loops, dashboard, and cronjobs |
dns.tf — the one live Terraform change
Two edits. The mseeks_me_froot A record (froot's dashboard entrypoint) is deleted — this is the only resource Terraform destroyed, and both state and the live zone were verified clean afterwards. And the LOAD-BEARING comment on the ynab record, which enumerates every host riding the ingress LB for cert-manager's HTTP-01 self-check, drops froot from that list.
The corrected host list on the load-bearing ynab record, and the models record that now directly follows the everwhen wildcard where the froot record used to sit.
infra/dns.tf · 383 lines
infra/dns.tf
⋯ 68 lines hidden (lines 1–68)
1# DNS — reverse-engineered from the live DigitalOcean zones and imported into
2# state (target: `terraform plan` reports no changes).
3#
4# SOA and the default DigitalOcean NS records (ns1/2/3.digitalocean.com) are
5# managed automatically by DO and are intentionally NOT declared here.
6#
7# Every zone here is an iCloud Custom Email Domain (Apple) setup: MX -> icloud
8# mail, an SPF TXT, an apple-domain verification TXT, and a DKIM CNAME.
9# (mseeks.me additionally carries the zo-k8s load-balancer A records below.)
10#
11# SPF quoting gotcha: write SPF TXT values WITHOUT surrounding quotes (e.g.
12# `v=spf1 include:icloud.com ~all`). DigitalOcean adds the DNS wire quotes on
13# its own, so escaping quotes into the value makes `terraform apply` CREATE a
14# doubly-quoted record (`""v=spf1 ...""`) that SPF verifiers reject. `plan`
15# won't warn you: the provider compares TXT values quote-insensitively, so even
16# a broken live record shows no drift. The mseeks.me / msull92.com SPF entries
17# still carry escaped quotes but resolve clean (created pre-terraform); left
18# as-is because recreating them from the quoted value would re-break them.
19# playeverwhen.com and 2ls.tech below use the correct unquoted form.
21# ============================== mseeks.me ==============================
23resource "digitalocean_domain" "mseeks_me" {
24 name = "mseeks.me"
25}
27resource "digitalocean_record" "mseeks_me_mx_01" {
28 domain = digitalocean_domain.mseeks_me.name
29 type = "MX"
30 name = "@"
31 value = "mx01.mail.icloud.com."
32 priority = 10
33 ttl = 14400
34}
36resource "digitalocean_record" "mseeks_me_mx_02" {
37 domain = digitalocean_domain.mseeks_me.name
38 type = "MX"
39 name = "@"
40 value = "mx02.mail.icloud.com."
41 priority = 10
42 ttl = 14400
43}
45resource "digitalocean_record" "mseeks_me_apple_domain" {
46 domain = digitalocean_domain.mseeks_me.name
47 type = "TXT"
48 name = "@"
49 value = "apple-domain=Y94Tzo0dJVhRhQxf"
50 ttl = 3600
51}
53resource "digitalocean_record" "mseeks_me_spf" {
54 domain = digitalocean_domain.mseeks_me.name
55 type = "TXT"
56 name = "@"
57 value = "\"v=spf1 include:icloud.com ~all\""
58 ttl = 3600
59}
61resource "digitalocean_record" "mseeks_me_dkim" {
62 domain = digitalocean_domain.mseeks_me.name
63 type = "CNAME"
64 name = "sig1._domainkey"
65 value = "sig1.dkim.mseeks.me.at.icloudmailadmin.com."
66 ttl = 43200
67}
69# ynab.mseeks.me -> the zo-k8s ingress Load Balancer. The LB itself is owned by
70# the DigitalOcean CCM (via the ingress-nginx Service, see infra/k8s/ingress-nginx);
71# Terraform only READS it through data.digitalocean_loadbalancer.ingress
72# (loadbalancer.tf) to keep this record's IP in sync automatically.
73#
74# LOAD-BEARING — keep it. This is the LB's advertised hostname
75# (do-loadbalancer-hostname in infra/k8s/ingress-nginx/values.yaml). DO LBs don't
76# hairpin, so cert-manager's HTTP-01 self-check for EVERY host on the LB
77# (playeverwhen.com, static, models) resolves this name out via DNS
78# and back in. The ynab-agent app it was originally named for has been retired,
79# but the record stays as pure LB infra — deleting it breaks TLS renewal
80# cluster-wide.
81resource "digitalocean_record" "mseeks_me_ynab" {
82 domain = digitalocean_domain.mseeks_me.name
83 type = "A"
84 name = "ynab"
85 # Sourced from the live LB; if the LB is recreated (new IP), a plain
86 # `terraform apply` re-points this record — no hardcoded address to chase.
87 value = data.digitalocean_loadbalancer.ingress.ip
88 # Short TTL: this points at a DO LB whose IP can change if the LB is recreated,
89 # so keep resolver caches from pinning a stale address for long.
90 ttl = 300
91}
⋯ 56 lines hidden (lines 92–147)
93# glassbox.mseeks.me -> Vercel. Glassbox is hosted on Vercel, not the cluster:
94# the project + custom domain are managed in the Vercel dashboard and deploys
95# ride its GitHub integration (push to main in mseeks/glassbox), so the only
96# infra Terraform owns is this record. Vercel terminates TLS at its edge — no
97# cert-manager involvement. The value is this project's own CNAME target, the
98# one the dashboard issued when glassbox.mseeks.me was added to the project
99# (Vercel mints per-project targets; the old universal cname.vercel-dns.com is
100# legacy). If the Vercel project is ever deleted and recreated, it gets a new
101# target — read the fresh value off the project's Domains tab, and only
102# re-point this record once the domain is attached there (flipped early, the
103# host serves Vercel's DEPLOYMENT_NOT_FOUND page).
104resource "digitalocean_record" "mseeks_me_glassbox" {
105 domain = digitalocean_domain.mseeks_me.name
106 type = "CNAME"
107 name = "glassbox"
108 value = "e24be5a7fd9d016f.vercel-dns-017.com."
109 # Unlike the LB A records above, the target here is stable for the life of
110 # the Vercel project, so no short-TTL churn guard is needed.
111 ttl = 3600
112}
114# static.mseeks.me -> the same zo-k8s ingress Load Balancer. Public entrypoint
115# for the `static` drop-and-share site (https://static.mseeks.me; see
116# infra/k8s/static) and the host cert-manager gets a Let's Encrypt cert for.
117# Same read-only LB-IP sourcing as ynab above — no hardcoded address to
118# chase.
119resource "digitalocean_record" "mseeks_me_static" {
120 domain = digitalocean_domain.mseeks_me.name
121 type = "A"
122 name = "static"
123 value = data.digitalocean_loadbalancer.ingress.ip
124 # Short TTL for the same reason as ynab: the target is a DO LB whose
125 # IP can change if the LB is recreated.
126 ttl = 300
127}
129# *.everwhen.mseeks.me -> the same zo-k8s ingress Load Balancer. Wildcard record
130# so ephemeral per-PR preview hosts (pr-<N>.everwhen.mseeks.me; see
131# infra/k8s/everwhen-preview) resolve to the LB without a per-PR DNS change. The
132# bare everwhen.mseeks.me A record is gone — the Everwhen app moved to
133# playeverwhen.com — so this wildcard now serves per-PR previews only (it never
134# matched the bare apex anyway; a wildcard covers deeper labels, not the name
135# itself). cert-manager issues a per-host Let's Encrypt cert on first request
136# (HTTP-01), so each preview gets its own TLS with no wildcard cert needed. Same
137# read-only LB-IP sourcing as the records above — no hardcoded address to chase.
138resource "digitalocean_record" "mseeks_me_everwhen_wildcard" {
139 domain = digitalocean_domain.mseeks_me.name
140 type = "A"
141 name = "*.everwhen"
142 value = data.digitalocean_loadbalancer.ingress.ip
143 # Short TTL for the same reason as the others: the target is a DO LB whose IP
144 # can change if the LB is recreated.
145 ttl = 300
146}
148# models.mseeks.me -> the same zo-k8s ingress Load Balancer. Public entrypoint
149# for the read-only model-traffic dashboard (https://models.mseeks.me; see
150# infra/k8s/model-metrics), gated by HTTP basic auth, and the host cert-manager
151# gets a Let's Encrypt cert for. Same read-only LB-IP sourcing as the others.
152resource "digitalocean_record" "mseeks_me_models" {
153 domain = digitalocean_domain.mseeks_me.name
154 type = "A"
155 name = "models"
156 value = data.digitalocean_loadbalancer.ingress.ip
157 ttl = 300
158}
160# ============================== msull92.com ==============================
⋯ 222 lines hidden (lines 162–383)
162resource "digitalocean_domain" "msull92_com" {
163 name = "msull92.com"
164}
166resource "digitalocean_record" "msull92_com_mx_01" {
167 domain = digitalocean_domain.msull92_com.name
168 type = "MX"
169 name = "@"
170 value = "mx01.mail.icloud.com."
171 priority = 10
172 ttl = 14400
173}
175resource "digitalocean_record" "msull92_com_mx_02" {
176 domain = digitalocean_domain.msull92_com.name
177 type = "MX"
178 name = "@"
179 value = "mx02.mail.icloud.com."
180 priority = 10
181 ttl = 14400
182}
184resource "digitalocean_record" "msull92_com_apple_domain" {
185 domain = digitalocean_domain.msull92_com.name
186 type = "TXT"
187 name = "@"
188 value = "apple-domain=SLKQpEYsouHgb3mI"
189 ttl = 3600
190}
192resource "digitalocean_record" "msull92_com_spf" {
193 domain = digitalocean_domain.msull92_com.name
194 type = "TXT"
195 name = "@"
196 value = "\"v=spf1 include:icloud.com ~all\""
197 ttl = 3600
198}
200resource "digitalocean_record" "msull92_com_dkim" {
201 domain = digitalocean_domain.msull92_com.name
202 type = "CNAME"
203 name = "sig1._domainkey"
204 value = "sig1.dkim.msull92.com.at.icloudmailadmin.com."
205 ttl = 43200
206}
208# ============================== 2ls.tech ==============================
209# HCL resource names can't start with a digit, so these use a `_2ls_tech`
210# prefix (a leading underscore is valid) rather than the plain `2ls_tech`
211# the mseeks_me/msull92_com convention would suggest.
213resource "digitalocean_domain" "_2ls_tech" {
214 name = "2ls.tech"
215}
217resource "digitalocean_record" "_2ls_tech_mx_01" {
218 domain = digitalocean_domain._2ls_tech.name
219 type = "MX"
220 name = "@"
221 value = "mx01.mail.icloud.com."
222 priority = 10
223 ttl = 14400
224}
226resource "digitalocean_record" "_2ls_tech_mx_02" {
227 domain = digitalocean_domain._2ls_tech.name
228 type = "MX"
229 name = "@"
230 value = "mx02.mail.icloud.com."
231 priority = 10
232 ttl = 14400
233}
235resource "digitalocean_record" "_2ls_tech_apple_domain" {
236 domain = digitalocean_domain._2ls_tech.name
237 type = "TXT"
238 name = "@"
239 value = "apple-domain=KAYDndJUw5TrEwCe"
240 ttl = 3600
241}
243# Google Search Console domain verification (TXT at the apex). Value UNQUOTED
244# like the apple-domain record above — DigitalOcean adds the DNS wire quotes
245# itself (see the SPF quoting gotcha at the top of this file).
246resource "digitalocean_record" "_2ls_tech_google_site_verification" {
247 domain = digitalocean_domain._2ls_tech.name
248 type = "TXT"
249 name = "@"
250 value = "google-site-verification=9e1Dm56-fdkP2jTv1pGNI5II2ol0pR6Xtw8RtS1gXwM"
251 ttl = 3600
252}
254# SPF value UNQUOTED on purpose — see the SPF quoting gotcha at the top of this
255# file. Recreated once (terraform apply -replace) to strip the doubly-quoted
256# `""v=spf1 ...""` the original escaped-quote value produced on create.
257resource "digitalocean_record" "_2ls_tech_spf" {
258 domain = digitalocean_domain._2ls_tech.name
259 type = "TXT"
260 name = "@"
261 value = "v=spf1 include:icloud.com ~all"
262 ttl = 3600
263}
265resource "digitalocean_record" "_2ls_tech_dkim" {
266 domain = digitalocean_domain._2ls_tech.name
267 type = "CNAME"
268 name = "sig1._domainkey"
269 value = "sig1.dkim.2ls.tech.at.icloudmailadmin.com."
270 ttl = 43200
271}
273# 2ls.tech (+ www) -> the zo-k8s ingress Load Balancer. Public entrypoint for
274# the Two Ls company site (https://2ls.tech; see infra/k8s/2ls) and the hosts
275# cert-manager gets a Let's Encrypt cert for. Same read-only LB-IP sourcing as
276# the mseeks.me records above — no hardcoded address to chase. APPLIED — these
277# records are live.
278resource "digitalocean_record" "_2ls_tech_apex" {
279 domain = digitalocean_domain._2ls_tech.name
280 type = "A"
281 name = "@"
282 value = data.digitalocean_loadbalancer.ingress.ip
283 # Short TTL: the target is a DO LB whose IP can change if the LB is
284 # recreated.
285 ttl = 300
286}
288resource "digitalocean_record" "_2ls_tech_www" {
289 domain = digitalocean_domain._2ls_tech.name
290 type = "A"
291 name = "www"
292 value = data.digitalocean_loadbalancer.ingress.ip
293 ttl = 300
294}
296# ============================== playeverwhen.com ==============================
298resource "digitalocean_domain" "playeverwhen_com" {
299 name = "playeverwhen.com"
300}
302resource "digitalocean_record" "playeverwhen_com_mx_01" {
303 domain = digitalocean_domain.playeverwhen_com.name
304 type = "MX"
305 name = "@"
306 value = "mx01.mail.icloud.com."
307 priority = 10
308 ttl = 14400
309}
311resource "digitalocean_record" "playeverwhen_com_mx_02" {
312 domain = digitalocean_domain.playeverwhen_com.name
313 type = "MX"
314 name = "@"
315 value = "mx02.mail.icloud.com."
316 priority = 10
317 ttl = 14400
318}
320resource "digitalocean_record" "playeverwhen_com_apple_domain" {
321 domain = digitalocean_domain.playeverwhen_com.name
322 type = "TXT"
323 name = "@"
324 value = "apple-domain=DEESFXkRhJ4wE6Lm"
325 ttl = 3600
326}
328# Google Search Console domain verification (TXT at the apex). Value UNQUOTED
329# like the apple-domain record above — DigitalOcean adds the DNS wire quotes
330# itself (see the SPF quoting gotcha at the top of this file).
331resource "digitalocean_record" "playeverwhen_com_google_site_verification" {
332 domain = digitalocean_domain.playeverwhen_com.name
333 type = "TXT"
334 name = "@"
335 value = "google-site-verification=1FSmsnv2MPG_mKoCrdlSABXfyXFTaeVBETNRmLNi85o"
336 ttl = 3600
337}
339# SPF value UNQUOTED on purpose — see the SPF quoting gotcha at the top of this
340# file. DigitalOcean adds the DNS wire quotes itself.
341resource "digitalocean_record" "playeverwhen_com_spf" {
342 domain = digitalocean_domain.playeverwhen_com.name
343 type = "TXT"
344 name = "@"
345 value = "v=spf1 include:icloud.com ~all"
346 ttl = 3600
347}
349resource "digitalocean_record" "playeverwhen_com_dkim" {
350 domain = digitalocean_domain.playeverwhen_com.name
351 type = "CNAME"
352 name = "sig1._domainkey"
353 value = "sig1.dkim.playeverwhen.com.at.icloudmailadmin.com."
354 ttl = 43200
355}
357# playeverwhen.com (apex) -> the zo-k8s ingress Load Balancer. Public entrypoint
358# for the Everwhen SSR app (https://playeverwhen.com; see infra/k8s/everwhen),
359# migrated here from everwhen.mseeks.me. cert-manager gets a Let's Encrypt cert
360# for it (HTTP-01, so the apex must resolve to the LB). An apex A record (a CNAME
361# is illegal at a zone apex) works because the value is the LB IP, sourced
362# read-only from data.digitalocean_loadbalancer.ingress — no hardcoded address to
363# chase. Coexists fine with the iCloud MX/TXT records above.
364resource "digitalocean_record" "playeverwhen_com_apex" {
365 domain = digitalocean_domain.playeverwhen_com.name
366 type = "A"
367 name = "@"
368 value = data.digitalocean_loadbalancer.ingress.ip
369 # Short TTL: the target is a DO LB whose IP can change if the LB is recreated.
370 ttl = 300
371}
373# www.playeverwhen.com -> the same LB. Serves only a 301 redirect to the apex
374# (see the everwhen-www-redirect Ingress in infra/k8s/everwhen); cert-manager
375# issues its own cert so the redirect is reachable over HTTPS. Same read-only
376# LB-IP sourcing as the apex.
377resource "digitalocean_record" "playeverwhen_com_www" {
378 domain = digitalocean_domain.playeverwhen_com.name
379 type = "A"
380 name = "www"
381 value = data.digitalocean_loadbalancer.ingress.ip
382 ttl = 300
383}
kubernetes.tf — the cluster no longer exists for ClickStack
The cluster resource itself is untouched; only its story changes. The header and the node-pool comment both justified the s-4vcpu-8gb sizing by ClickHouse's appetite. The rewrite keeps the two operational facts that outlive the stack — a pool's size slug is immutable (changing it replaces the nodes), while node_count scales in place — and marks the sizing rationale as historical.
Rewritten header and sizing comment; the resource block itself is unchanged.
infra/kubernetes.tf · 99 lines
infra/kubernetes.tf
1# DigitalOcean Kubernetes (DOKS) — a small, persistent cluster. Its job is to
2# run our managed-k8s workloads (the k8s/ stacks: sites, everwhen, the Ollama
3# egress) and give us a place to learn/experiment without standing anything up
4# by hand. Modest, but a real running cluster.
5#
6# Cost (on top of the ~$5/mo Spaces state bucket):
7# - Control plane .......... $0/mo (standard, non-HA — see `ha` below)
8# - 2x s-4vcpu-8gb nodes ... $96/mo (~$0.0714/hr each, billed hourly)
9# - LoadBalancer (if any) .. $12/mo (~$0.0179/hr) — only while a
10# type=LoadBalancer Service exists; the
11# echo smoke test (k8s/echo-demo.yaml)
12# creates one, then we delete it.
13# Everything is billed hourly, so the per-node cost prorates if the pool is ever
14# scaled down or the cluster is torn down with `terraform destroy`.
⋯ 37 lines hidden (lines 15–51)
16# Latest patch release within a pinned MINOR series. A prefix (not a hard pin)
17# means we automatically track security patches inside 1.36.x, while moving to a
18# new minor stays a deliberate, reviewed edit to this file. When DO eventually
19# retires 1.36, `terraform plan` will fail with "no versions match" — that error
20# is the signal to bump the prefix to a currently-supported minor:
21# doctl kubernetes options versions
22data "digitalocean_kubernetes_versions" "current" {
23 version_prefix = "1.36."
24}
26resource "digitalocean_kubernetes_cluster" "zo" {
27 name = "zo-k8s"
28 region = "nyc3" # same region as the Spaces state bucket — keep it all local
29 version = data.digitalocean_kubernetes_versions.current.latest_version
31 # Single, non-HA control plane. The standard control plane is FREE; an HA
32 # control plane adds ~$40/mo and isn't worth it for this small personal cluster.
33 ha = false
35 # Terraform owns upgrades. When the data source above surfaces a newer 1.36
36 # patch, a normal `apply` rolls it out under our review. DO's automatic
37 # maintenance-window upgrades are left OFF so the version never changes behind
38 # our back. (surge_upgrade adds a temporary extra node during an upgrade so
39 # the pool is replaced without going fully offline.)
40 auto_upgrade = false
41 surge_upgrade = true
43 # IMPORTANT for clean teardown and no surprise bills: on `terraform destroy`,
44 # also delete resources the cluster provisioned out-of-band — most notably the
45 # DO Load Balancer that a type=LoadBalancer Service creates, plus any block
46 # storage volumes (PVCs). Without this, a destroy can leave an orphaned LB
47 # quietly billing $12/mo.
48 destroy_all_associated_resources = true
50 tags = ["zo", "terraform", "sandbox"]
52 # 4 vCPU / 8 GB per node (originally sized for the since-retired ClickStack
53 # observability stack; the remaining workloads fit with plenty of headroom).
54 # NOTE: a node pool's `size` slug is immutable on the DO API, so changing it
55 # REPLACES the pool — the old node is drained/destroyed and a new one
56 # provisioned. Anything NOT on a do-block-storage PVC is lost (PVCs detach and
57 # re-attach to the new node, so their data survives). `node_count`, unlike
58 # `size`, is mutable — scaling it adds/removes nodes in place (no replacement);
59 # DOKS cordons/drains a removed node and its PVC-backed pods re-attach to the
60 # survivor. No autoscaling.
61 node_pool {
62 name = "default"
63 size = "s-4vcpu-8gb"
64 node_count = 1
65 auto_scale = false
66 tags = ["zo", "sandbox"]
67 }
⋯ 32 lines hidden (lines 68–99)
68}
70# --- Outputs -----------------------------------------------------------------
71# Non-sensitive metadata + the one command you actually need to start using the
72# cluster. We intentionally do NOT output the raw kubeconfig: the provider's
73# kubeconfig carries a short-lived token and would land in remote state. Instead
74# fetch creds on demand with doctl (which refreshes the token automatically).
76output "kubernetes_cluster_id" {
77 value = digitalocean_kubernetes_cluster.zo.id
78 description = "DOKS cluster UUID."
79}
81output "kubernetes_cluster_name" {
82 value = digitalocean_kubernetes_cluster.zo.name
83 description = "DOKS cluster name."
84}
86output "kubernetes_endpoint" {
87 value = digitalocean_kubernetes_cluster.zo.endpoint
88 description = "Kubernetes API server endpoint (not a secret)."
89}
91output "kubernetes_version" {
92 value = digitalocean_kubernetes_cluster.zo.version
93 description = "Resolved Kubernetes version running on the cluster."
94}
96output "kubeconfig_command" {
97 value = "doctl kubernetes cluster kubeconfig save ${digitalocean_kubernetes_cluster.zo.name}"
98 description = "With DIGITALOCEAN_ACCESS_TOKEN set, run this to merge cluster credentials into ~/.kube/config."
99}
The doc sweep: CLAUDE.md and infra/README.md
infra/README.md loses its whole Observability (ClickStack) and Temporal sections plus the two k8s/ tree entries, and the DOKS intro no longer claims the first node is saturated. CLAUDE.md gets the bigger rewrite: the stack list shrinks to tailscale, and the long "headroom nuance" paragraph — built around a node-1 request budget at 98% — is replaced with the post-teardown reality.
The Infrastructure section after the rewrite: retirement noted up top, headroom guidance kept only where it still teaches something.
CLAUDE.md · 92 lines
CLAUDE.md
⋯ 78 lines hidden (lines 1–78)
1# Zo
3You are **Zo**, the AI assistant for this personal workspace, home to the user's projects, code, research, and writing.
5## Identity
7You are precise, direct, and low-ceremony, a craftsperson about correctness. You do exactly what's asked and say what you think plainly. Your core domains are engineering, research, and writing.
9## Many Hands Engineering (MHE)
11MHE is the framework underpinning the agentic and code-maintenance work across this workspace. Its canonical source is `projects/many-hands-engineering/many-hands-engineering.typ` (plain-text Typst source; `many-hands-engineering.pdf` is the rendered human version). Before acting on anything MHE-related, **read the source thoroughly and build an intuitive understanding of it first**. Don't pattern-match from fragments or secondhand summaries.
13## Guidelines
15- When the user expresses a lasting preference ("from now on"), update this file to reflect that preference.
16- Do exactly what's asked, and don't expand scope. Surface adjacent improvements as suggestions, not actions. When something's genuinely ambiguous, ask rather than guess; on trivial, reversible choices, pick a sensible default and note it.
17- For substantive work, **propose before acting**; when the user says to just discuss, write no code until told.
18- Bias hard to **KISS**: the simplest design that works, accreted slowly. Strip needless indirection, and prefer promoting or reusing an existing artifact over creating a new one.
19- **Chat style:** terse and neutral. Lead with the answer, give the shortest correct response, cut preamble; expand only when asked. Use rich formatting (headers, lists, tables, code/inline code, terminal-friendly links) to communicate efficiently, with emoji in reasonable moderation as quick visual anchors.
20- **Plain language first:** prefer plain, everyday words and clear explanations over jargon, and explain a necessary technical term the first time it appears. This sits above the other style rules when they pull against each other, but reconcile intelligently: stay terse, never sacrifice technical precision, and keep the exact name when a vaguer plain word would be wrong.
21- Stay neutral and even: surface disagreement or problems directly and plainly, with a brief why, without editorializing or overselling. Say it once, then stay open to well-supported counterpoints.
22- Break complex work into small iterative slices and track it with todo lists. Narrate sparingly: a line at meaningful milestones, not step-by-step.
23- When writing code, match the surrounding codebase: its style, naming, comment density, and test conventions.
24- Keep personal-workspace plumbing (static/k8s deploy, private-repo references, local paths) out of reusable or public artifacts. It belongs only in zo's root CLAUDE.md. Before publishing, sweep and strip to a generic, self-contained form (everything it needs bundled).
25- Connected tools: read freely, but confirm before any write or send. Query judiciously. Don't over-fetch and pollute context.
26- Reads run free. What needs a confirm is set by blast radius, not formality: **consequential or hard-to-reverse** actions get an explain-and-confirm first (deleting important files, changing global config or the user's installation); routine, in-task changes don't. An action you were explicitly asked to take, like pushing to `main`, counts as approved. If a suggested fix looks wrong for the setup, say so instead of running it.
27- Never commit secrets. Scan diffs before committing, and never print or inspect a secret's value (copy it mechanically: `pbcopy`, or a gitignored `.env` between repos). Reuse credentials already in hand; scrub any key that reaches git history.
28- In personal repos, commit and push to `main` when asked. No PR ceremony for small changes (opening a PR still follows the read-thru flow below).
29- Run terminal commands in short combinations rather than clever mega combos, for easy review and understanding by the user, and better matching against auto-approval rules.
30- Always check if you're working in a git worktree.
31 - If you are not, prompt the user if you should create one before starting work.
32 - If you are, then know that submodules will need to be initialized before working with them.
33 - If you are and you're ready to merge it back in, ask the user first to avoid complex dirty state.
34- On `zo`'s `main`, keep every project's submodule pointer at the **tip of that project's own `main`**. Periodically fetch upstream and advance each pointer (committing the bumps) so they stay aligned. A current, uniform starting point keeps worktree checkouts consistent and avoids confusion over where each one begins.
36## Writing
38How to write across everything you produce: chat replies, prose and docs, code comments (not code itself), and prompts you author for models. Apply with judgment for the context; these are defaults, not absolute bans.
40- **Rhythm:** mix short and long sentences; don't let them settle into a uniform length. Break up long sentences. Vary word choice, and prefer plain, short words over long or inflated ones.
41- **Avoid these words and phrases** (prefer concrete, plain alternatives): leverage, robust, scalable, innovative, cutting-edge, state-of-the-art, seamless, synergy, synergistic, passionate, thrilled, honored, delighted, spearhead, pivotal, transformative, transformational, paradigm, unlock, unleash, empower, deep dive, delve, navigate the landscape, ever-evolving, fast-paced, dynamic environment, moreover, furthermore, additionally, in essence, in today's, in the realm of, at the intersection of, look forward to, would welcome the opportunity, excited to discuss, I'd love to discuss, tapestry, underscore, testament to, speaks volumes.
42- **Avoid these constructions:** "not just X but Y"; reflexive "X, Y, and Z" triads; "it is important / worth noting / essential / crucial that…" hedging (state it plainly); restating the same point ("in other words", "to put it simply", "put simply", "more specifically", "that is to say"), say it once.
43- **Punctuation:** go light on em-dashes (use commas, periods, or parentheses); use plain quotes, not curly ones; and split overly long, comma-stacked sentences.
45## Code
47**Stack defaults**
49| Concern | Standard |
50|---|---|
51| UI language | TypeScript |
52| Python deps | `uv` |
53| Python format + lint | `ruff` |
54| Python types | `mypy` |
55| TypeScript format / lint | `prettier` / `eslint` |
56| Config (& secrets) | Pydantic Settings (`SecretStr` for secrets) |
57| System-wide installs | `brew` |
58| Local model calls | **Gemma 4 26B MoE** via Ollama (all model calls, going forward) |
60- **Python quality bar:** strict DDD with MyPy-enforced typing so invalid states are unrepresentable; functional style where it fits, without ceremony; Google style guide (80-char lines); tests valued for meaning, not coverage.
61- **Docs:** stale docs/comments are bugs; fix them when a change invalidates one. Respect the user's hand-edits and regenerate derived artifacts *from* them; never overwrite. Write for humans and future agents alike; comments justify *why*.
63## Verification
65- **Scale rigor to the change.** Small, low-risk changes get a light touch: a quick check and a plain note of what you did. Save the full bar below for substantial or outward-facing work; don't put full-suite runs, boot-and-verify, or subagent review on a one-liner.
66- **Full suite green (for substantive changes):** nothing is done until the project's full check suite passes: tests, formatters, linters (and the project's loops), run in sensible order and re-run after each fix to confirm it resolved. Show the evidence when asked or when the result is non-obvious. Add the *discriminating* test that only the new layer could catch, not coverage padding.
67- **Boot it and lay eyes on it:** passing tests ≠ working UX. For runtime/UI work, boot the server and hand over the URL, *and* self-verify (screenshots, click controls, hunt runtime errors). For telemetry/infra, give a concrete way to observe each signal flowing: a ready query, API call, or endpoint.
68- **Independent adversarial review:** delegate serious analysis/verification to independent subagents; don't bias them with the answer you want. Canonical form: **adversarial review** (e.g. alpha-player playtests for UI, fact-checking every claim for prose).
70## Pull requests
72- Whenever you open a PR, use the **`make-read-thru`** skill to generate a code-level reading guide *of that PR's content*: a walkthrough a reviewer can read top to bottom and vouch for (size it to the change; the skill's `references/depth-and-scope.md` has the tiers).
73- **Deploy** the guide to the static site: write it to `projects/static/public/read-thru/<repo>/pr-<n>-<slug>.html` (slug = short kebab summary of the PR), push the static repo's `main`, then run `infra/k8s/static/install.sh`.
74- In the PR body, add a **bare link, no preamble** to the page's clean URL (no `.html` suffix): `https://static.mseeks.me/read-thru/<repo>/pr-<n>-<slug>`.
75- Deploy/namespacing above is **personal-workspace-specific and lives only here**. The `make-read-thru` skill knows nothing about the static site or k8s; it just produces the HTML.
76- **Title & body:** the title is human, **terse, and specific** to the change, never generic. The body is concise, well-structured, and comprehensible, focused on *what* the change is and *why* the PR exists at all. Avoid "meta talk": don't leak conversation or process concerns into the PR, and don't over-explain things that aren't immediately relevant.
77- **Chaining dependent PRs:** when changes build on each other, chain the PRs by targeting each other in order so each reviews cleanly.
79## Infrastructure
81- `infra/` holds Terraform for personal DigitalOcean infrastructure (single source of truth for the DO account). Currently DNS plus a two-node DOKS cluster (`infra/kubernetes.tf`: `zo-k8s`, 2x `s-4vcpu-8gb` nodes in nyc3, ~$96/mo); App Platform static sites/apps later. (The ClickStack/HyperDX observability stack, the Temporal cluster, and the froot worker that ran here were retired in July 2026.)
82- On-cluster workloads are deployed with **kubectl/Helm + install scripts, NOT Terraform** (Terraform owns only the cluster).
83- **Headroom:** since the observability/Temporal/froot retirement the request budget has real slack. Requests are reservations, not usage, and there's **no metrics-server**, so `kubectl top` won't work in-cluster (DO's dashboard is the source of truth for real load). Still pin **small requests** when adding pods — a pod goes `Pending` on request budget, not actual load.
84- **Stacks:**
85 - `infra/k8s/tailscale/` (Tailscale operator giving the cluster **egress** to Ollama on the Mac Studio: Ollama runs via a `com.zo.ollama` LaunchAgent on the Mac, exposed to the tailnet with `tailscale serve --tcp`, and fronted in-cluster by an nginx Host-rewrite proxy reachable at `http://ollama.llm:11434`; see its README for the macOS-firewall and Ollama DNS-rebind/Host-check nuances)
86- Secrets never go in the repo: the DO token is read from `DIGITALOCEAN_ACCESS_TOKEN` via gitignored `infra/.env` (used by both terraform and doctl).
87- Project-specific deployments live centrally in `infra/` and point *outward* at public repos; never put Terraform inside the (open-source) project repos.
88- **Everwhen PR previews (ephemeral deploys).** (Prod itself now lives at `https://playeverwhen.com`; previews still live on `*.everwhen.mseeks.me`, which is why that wildcard DNS record stays.) When asked to deploy / preview / "spin up an ephemeral" for everwhen PR `<N>`, stand up a throwaway deploy of that PR at `https://pr-<N>.everwhen.mseeks.me`, then hand back the URL:
89 - Ensure the PR has the `preview` label (`gh pr edit <N> --repo mseeks/everwhen --add-label preview`) — that label is what makes the everwhen CI build `ghcr.io/mseeks/everwhen:pr-<N>`. **Wait for that CI build to finish** before deploying, or the pod sits in `ImagePullBackOff`.
90 - With kubectl on zo-k8s: `infra/k8s/everwhen-preview/deploy-pr.sh <N>` (idempotent — re-run after new commits land a fresh `:pr-<N>`). For a one-off without a PR build, set `EW_PREVIEW_IMAGE` (e.g. `:latest` or a `:<sha>`).
91 - Tear down when merged/done: `infra/k8s/everwhen-preview/teardown-pr.sh <N>`, or `… --merged` to reap every closed PR's preview.
92 - Previews are **public** (no auth, like prod), reuse prod's `everwhen-secrets` with the canonical origin pointed at the preview host, and reserve ~256Mi each — tear them down promptly when done. Full docs: `infra/k8s/everwhen-preview/README.md`.
Peripheral comment fixes
Five smaller files referenced the stacks in comments and get one-line to one-paragraph corrections, no behavior change anywhere:
- infra/k8s/tailscale/README.md — the resource-footprint section drops the per-node saturation numbers and the "right-size ClickStack" lever. - infra/k8s/tailscale/values-operator.yaml — the "single, tight node" comment becomes a plain keep-it-small note. - infra/k8s/ingress-nginx/values.yaml — metrics stay enabled, with the comment now honest that nothing scrapes them today. - infra/k8s/everwhen-preview/README.md and the CLAUDE.md preview bullet — the cluster is no longer described as request-tight; the advice becomes "tear previews down promptly". - infra/k8s/tailscale/manifests/30-ollama-proxy.yaml and infra/k8s/static/manifests/00-namespace.yaml — log-shipping and "no Temporal" phrasing updated.
The requests-vs-usage explainer, now stack-agnostic.
infra/k8s/tailscale/README.md · 232 lines
infra/k8s/tailscale/README.md
⋯ 161 lines hidden (lines 1–161)
1# Tailscale egress: Ollama on the Mac Studio → zo-k8s
3Gives workloads on `zo-k8s` (a 2-node DOKS cluster in nyc3) a plain in-cluster
4endpoint that tunnels over your existing tailnet to **Ollama on the Mac Studio**,
5so the cluster can do LLM work off local hardware (Gemma 4, Qwen, …) with no
6public exposure.
8```
9 your tailnet (WireGuard, e2e encrypted)
10 ┌──────────────────────── Mac Studio ─────────────────────────┐
11 │ ollama serve launchd: com.zo.ollama — survives │
12 │ 127.0.0.1:11434 ◀───── reboots, Metal GPU, flash-attn + │
13 │ ▲ (gemma4:*, qwen) q8 KV cache │
14 │ │ loopback (firewall-exempt) │
15 │ tailscaled: tailscale serve --tcp 11434 ◀── tailnet :11434│
16 └─────────────────────────────────▲───────────────────────────┘
17 │ matthews-mac-studio.tail834adf.ts.net:11434
18 ┌────────────────────── zo-k8s (DOKS) ──┼──────────────────────┐
19 │ ns tailscale: operator → egress proxy (tag:k8s) ───────────┘
20 │ ns llm: ollama-tailnet (ExternalName ─▶ egress proxy) │
21 │ ▲ │
22 │ nginx "ollama-proxy" (rewrites Host:localhost)│
23 │ ▲ │
24 │ workloads ── curl http://ollama.llm:11434/... ───────────────┘
25 └────────────────────────────────────────────────────────────────┘
26```
28**Direction is egress** (cluster → tailnet device). The Mac is not, and need not
29be, a Kubernetes node — it's just another device on the tailnet that the cluster
30reaches through a Tailscale operator proxy.
32## Two non-obvious things this design works around
341. **macOS won't let Ollama be reached on the tailnet by a plain bind.** The
35 Application Firewall blocks inbound non-loopback connections to the Homebrew
36 `ollama` binary (it's not in the allow list). `tailscale serve --tcp` sidesteps
37 it: `tailscaled` accepts the tailnet connection and dials Ollama over
38 **loopback** (firewall-exempt, upgrade-proof, Ollama stays localhost-bound).
392. **Ollama 403s requests whose `Host` header isn't localhost** (DNS-rebind
40 protection). In-cluster clients connect to `ollama.llm`, so they send
41 `Host: ollama…` → 403. Binding Ollama to `0.0.0.0` would relax the check but
42 **collides with `tailscale serve`** on `:11434` (and is pointless behind the
43 firewall). So a tiny in-cluster nginx rewrites `Host: localhost` before the
44 egress. Ollama stays on `127.0.0.1:11434`, and your local `ollama` CLI keeps
45 working on the default port.
47## Mac Studio setup (Ollama + serve)
49Both pieces below are already configured on the Mac. Reference artifacts live in
50`macstudio/`.
52**Ollama as a reboot-surviving LaunchAgent** (`macstudio/com.zo.ollama.plist` →
53`~/Library/LaunchAgents/`). We use a dedicated agent instead of `brew services`
54because the Homebrew formula hardcodes its service env block, so brew regenerates
55the plist on every upgrade and would wipe added vars. This agent has
56`RunAtLoad`+`KeepAlive` (starts at login, auto-restarts), runs in the `Aqua`
57session (Metal/GPU), and pins `OLLAMA_HOST=127.0.0.1:11434` plus the formula's
58`OLLAMA_FLASH_ATTENTION=1` / `OLLAMA_KV_CACHE_TYPE=q8_0`.
60```bash
61cp macstudio/com.zo.ollama.plist ~/Library/LaunchAgents/
62brew services stop ollama # don't let brew's agent fight ours
63launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/com.zo.ollama.plist
64```
66> Reboot survival caveat: a LaunchAgent (and Metal) needs the user session, so
67> Ollama returns after **login**. For unattended boot, enable auto-login — though
68> with FileVault on, a cold boot always needs one manual unlock.
70**Expose it to the tailnet** — raw TCP, plain HTTP (the "TLS over TCP" label in
71`serve status` is cosmetic):
73```bash
74tailscale serve --bg --tcp 11434 tcp://127.0.0.1:11434
75tailscale serve status
76# verify from any tailnet device:
77curl -s http://matthews-mac-studio.tail834adf.ts.net:11434/api/tags
78```
80`--bg` persists across reboots (stored in the daemon's serve config). To stop:
81`tailscale serve --tcp=11434 off`.
83## Tailscale admin setup (one-time, admin console — do this first)
85At <https://login.tailscale.com/admin>.
87### 1. ACL — declare the tags (required) + least-privilege egress grant (recommended)
89The operator owns `tag:k8s-operator` (its own identity) and `tag:k8s` (every
90proxy it creates); these `tagOwners` are **mandatory**. The `hosts`+`acls` grant
91restricts the proxy to just Ollama's port:
93```jsonc
94{
95 "tagOwners": {
96 "tag:k8s-operator": [],
97 "tag:k8s": ["tag:k8s-operator"]
98 },
99 "hosts": { "mac-studio": "100.83.52.30" },
100 "acls": [
101 // ...your existing rules...
102 { "action": "accept", "src": ["tag:k8s"], "dst": ["mac-studio:11434"] }
103 ]
104}
105```
107> On the default allow-all ACL, only the `tagOwners` block is strictly required.
109### 2. OAuth client (required) → `infra/.env`
111**Settings → OAuth clients → Generate**. Grant **write** to **Devices › Core**
112and **Auth Keys**; attach tag **`tag:k8s-operator`**. Then:
114```bash
115echo 'TS_OAUTH_CLIENT_ID=k123abc...' >> infra/.env
116echo 'TS_OAUTH_CLIENT_SECRET=tskey-client-k123abc-...' >> infra/.env
117```
119## Install
121```bash
122doctl kubernetes cluster kubeconfig save zo-k8s # if not already current
123set -a; source infra/.env; set +a # exports TS_OAUTH_CLIENT_*
124infra/k8s/tailscale/install.sh
125```
127Idempotent: installs/upgrades the operator (pinned chart **1.98.4**) with
128`values-operator.yaml`, applies the `llm` namespace + `lean` ProxyClass + egress
129Service + nginx Host-rewrite proxy, waits for the operator to wire up the egress
130and for nginx to roll out, then runs the smoke test.
132## Using it from the cluster
134Any pod, any namespace — it's a normal Service:
136```
137http://ollama.llm:11434 # the nginx proxy; works cluster-wide
138```
140```bash
141# list models
142curl -s http://ollama.llm:11434/api/tags | jq '.models[].name'
144# generate with Gemma 4
145curl -s http://ollama.llm:11434/api/generate \
146 -d '{"model":"gemma4:e4b","prompt":"Explain WireGuard in one sentence.","stream":false}'
148# OpenAI-compatible (base_url=http://ollama.llm:11434/v1 for most SDKs)
149curl -s http://ollama.llm:11434/v1/chat/completions \
150 -d '{"model":"gemma4:26b","messages":[{"role":"user","content":"hello"}]}'
151```
153Models on the Mac: `gemma4:31b`, `gemma4:26b`, `gemma4:e4b`, `gemma4:e2b`,
154`qwen3.5:0.8b` (`ollama list` on the Mac for the live set).
156> **Cold start:** the first ~30-45s after the egress proxy is *created or
157> recreated* (fresh install, node reboot, operator upgrade), connections fail
158> fast until its tailnet route + Service endpoints are programmed. Steady-state
159> is immediate. Give client apps normal retry/backoff and this is invisible; the
160> smoke test retries internally.
162## Resource footprint — requests vs. actual usage (important nuance)
164Two different numbers, easy to confuse:
166- **`kubectl describe node` → "Allocated resources".** This is the **sum of pod
167 *requests*** (reservations the scheduler hands out) per node, as a fraction of
168 allocatable. It only governs whether a *new* pod can be scheduled on that node —
169 it is **not** how busy the CPU is. Since the ClickStack/Temporal/froot stacks
170 were retired, the request budget has real slack cluster-wide.
171- **DigitalOcean dashboard CPU.** This is **actual measured utilization** —
172 usually far below the request numbers, because workloads tend to reserve more
173 than they use.
175Both are correct. Practical consequence: a *new* pod that declares CPU/memory
176requests can still land `Pending` on a node whose request budget is full even
177though that node is mostly idle. (`kubectl top` shows nothing here —
178metrics-server isn't installed — so DO's hypervisor metrics are the source of
179truth for real load.)
181This stack is pinned tiny so it stays a negligible reservation:
183| Pod | CPU req | Mem req | Mem limit |
184|-----|--------:|--------:|----------:|
185| operator (`ns tailscale`) | 50m | 96Mi | 192Mi |
186| egress proxy (`lean` ProxyClass) | 25m | 64Mi | 128Mi |
⋯ 46 lines hidden (lines 187–232)
187| nginx `ollama-proxy` (`ns llm`) | 10m | 16Mi | 64Mi |
189~85m CPU / ~176Mi of *requests* (~1% of the cluster). To schedule more later
190you have two levers before paying for bigger nodes: (1) **scale out** — bump
191`node_pool.node_count` in `infra/kubernetes.tf` (adds a node in place, no
192restarts); or (2) resize the DOKS node pool in `infra/kubernetes.tf` —
193**changing the node size recreates the nodes**, so every workload restarts.
195## Operations
197- **Update operator / config:** edit `values-operator.yaml` (or bump
198 `CHART_VERSION` in `install.sh`) and re-run `install.sh`.
199- **Change the Host-rewrite / nginx config:** edit `manifests/30-ollama-proxy.yaml`
200 and re-run `install.sh` (it rolls the nginx deploy to pick up the ConfigMap).
201- **Re-run the smoke test:**
202 `kubectl delete job ollama-smoketest -n llm --ignore-not-found && kubectl apply -f smoketest.yaml && kubectl logs -f job/ollama-smoketest -n llm`
203- **Remove everything:** `infra/k8s/tailscale/uninstall.sh` (then delete the
204 tailnet devices + OAuth client in the console).
206### Troubleshooting
208- **`403` from Ollama** → the request reached Ollama but its Host check rejected
209 it: the nginx Host-rewrite isn't in the path (client hit `ollama-tailnet`
210 directly instead of `ollama`), or nginx lost its `proxy_set_header Host
211 localhost`. Confirm via the Mac's `/opt/homebrew/var/log/ollama.log` GIN lines.
212- **`502`/`504` from nginx** → upstream (egress) not reachable yet (cold start),
213 or the kube-dns resolver IP in `manifests/30-ollama-proxy.yaml` is wrong for
214 this cluster (`kubectl get svc kube-dns -n kube-system` → it's `10.108.64.10`).
215- **`externalName` stays `placeholder`** → operator couldn't provision the proxy.
216 `kubectl logs -n tailscale deploy/operator`. Usual cause: missing `tagOwners`
217 or wrong OAuth scopes/tag.
218- **Connects but times out at the Mac** → on the Mac confirm
219 `tailscale serve status` shows the `:11434` forward, `launchctl print
220 gui/$(id -u)/com.zo.ollama` shows the agent active, and `curl
221 http://127.0.0.1:11434/api/tags` works.
222- **MagicDNS disabled** → swap the `tailscale.com/tailnet-fqdn` annotation in
223 `manifests/20-egress-ollama.yaml` for `tailscale.com/tailnet-ip: "100.83.52.30"`.
225## Security model
227The tailnet is the boundary: Ollama (no app-level auth) is reachable only by
228tailnet devices the ACL allows — here, only `tag:k8s` to `:11434`. Ollama binds
229loopback; the only inbound path is `serve`→loopback. Never expose it via
230Tailscale **Funnel** or a public LoadBalancer. OAuth creds live only in
231`infra/.env` (gitignored) and the in-cluster `operator-oauth` Secret.
232```
Deliberately left alone
Three loose ends are named in the PR body rather than fixed here:
1. model-metrics still runs and models.mseeks.me still resolves, but its only data source (ClickStack's ClickHouse) and the log pipeline that fed it are gone — the dashboard will render its "ClickHouse unreachable" state. It needs its own retire-or-repoint decision, so this PR doesn't touch it. 2. The pending 1.36.0-do.3 patch upgrade in terraform plan (see the dns.tf section above). 3. The gitignored infra/.env still carries CLICKSTACK_* / TEMPORAL_* / FROOT_* secrets; safe to prune by hand.