What changed, and why
Glassbox leaves the zo-k8s cluster for Vercel. After this PR, Terraform's whole footprint for the site is the one thing it already owned: a DNS record. The glassbox.mseeks.me A record, which pointed at the ingress load balancer, becomes a CNAME at Vercel's edge. And the infra/k8s/glassbox stack goes away entirely, 219 lines of namespace, Deployment and Service, Ingress, install script, and README.
The deliberate non-change deserves a sentence. There is no vercel Terraform provider here. Managing the project declaratively would put a long-lived API token in infra/.env and add provider state to reconcile, and the payoff is thin for one static site whose GitHub-app authorization happens in a browser regardless. The dashboard owns the project. Terraform keeps the zone.
The app-repo half is mseeks/glassbox#20: Dockerfile and nginx out, vercel.json in.
dns.tf: the A record becomes a CNAME
The old record had a moving part. Its value came off data.digitalocean_loadbalancer.ingress, the live lookup that keeps every cluster host pointed at whatever IP the load balancer currently holds. The new record is plain data. Vercel's CNAME target is a fixed name, so nothing needs tracking. That is also why the TTL rises from 300 to 3600: the short TTL existed to ride out LB recreation, and no LB sits in this path anymore. TLS changes hands too. Vercel terminates at its own edge, which takes cert-manager out of the loop for this host.
Comment lines move too. The load-bearing ynab record cites hosts that lean on it for cert-manager's HTTP-01 self-check, glassbox among them until now, and the static record's two "same as ynab/glassbox" asides slim down the same way.
The hairpin host list (glassbox removed) and the new CNAME record with its cutover-ordering comment.
infra/dns.tf · 398 lines
infra/dns.tf
⋯ 73 lines hidden (lines 1–73)
1# DNS — reverse-engineered from the live DigitalOcean zones and imported into
2# state (target: `terraform plan` reports no changes).
3#
4# SOA and the default DigitalOcean NS records (ns1/2/3.digitalocean.com) are
5# managed automatically by DO and are intentionally NOT declared here.
6#
7# Every zone here is an iCloud Custom Email Domain (Apple) setup: MX -> icloud
8# mail, an SPF TXT, an apple-domain verification TXT, and a DKIM CNAME.
9# (mseeks.me additionally carries the zo-k8s load-balancer A records below.)
10#
11# SPF quoting gotcha: write SPF TXT values WITHOUT surrounding quotes (e.g.
12# `v=spf1 include:icloud.com ~all`). DigitalOcean adds the DNS wire quotes on
13# its own, so escaping quotes into the value makes `terraform apply` CREATE a
14# doubly-quoted record (`""v=spf1 ...""`) that SPF verifiers reject. `plan`
15# won't warn you: the provider compares TXT values quote-insensitively, so even
16# a broken live record shows no drift. The mseeks.me / msull92.com SPF entries
17# still carry escaped quotes but resolve clean (created pre-terraform); left
18# as-is because recreating them from the quoted value would re-break them.
19# playeverwhen.com and 2ls.tech below use the correct unquoted form.
21# ============================== mseeks.me ==============================
23resource "digitalocean_domain" "mseeks_me" {
24 name = "mseeks.me"
25}
27resource "digitalocean_record" "mseeks_me_mx_01" {
28 domain = digitalocean_domain.mseeks_me.name
29 type = "MX"
30 name = "@"
31 value = "mx01.mail.icloud.com."
32 priority = 10
33 ttl = 14400
34}
36resource "digitalocean_record" "mseeks_me_mx_02" {
37 domain = digitalocean_domain.mseeks_me.name
38 type = "MX"
39 name = "@"
40 value = "mx02.mail.icloud.com."
41 priority = 10
42 ttl = 14400
43}
45resource "digitalocean_record" "mseeks_me_apple_domain" {
46 domain = digitalocean_domain.mseeks_me.name
47 type = "TXT"
48 name = "@"
49 value = "apple-domain=Y94Tzo0dJVhRhQxf"
50 ttl = 3600
51}
53resource "digitalocean_record" "mseeks_me_spf" {
54 domain = digitalocean_domain.mseeks_me.name
55 type = "TXT"
56 name = "@"
57 value = "\"v=spf1 include:icloud.com ~all\""
58 ttl = 3600
59}
61resource "digitalocean_record" "mseeks_me_dkim" {
62 domain = digitalocean_domain.mseeks_me.name
63 type = "CNAME"
64 name = "sig1._domainkey"
65 value = "sig1.dkim.mseeks.me.at.icloudmailadmin.com."
66 ttl = 43200
67}
69# ynab.mseeks.me -> the zo-k8s ingress Load Balancer. The LB itself is owned by
70# the DigitalOcean CCM (via the ingress-nginx Service, see infra/k8s/ingress-nginx);
71# Terraform only READS it through data.digitalocean_loadbalancer.ingress
72# (loadbalancer.tf) to keep this record's IP in sync automatically.
73#
74# LOAD-BEARING — keep it. This is the LB's advertised hostname
75# (do-loadbalancer-hostname in infra/k8s/ingress-nginx/values.yaml). DO LBs don't
76# hairpin, so cert-manager's HTTP-01 self-check for EVERY host on the LB
77# (playeverwhen.com, static, froot, models) resolves this name out via DNS
78# and back in. The ynab-agent app it was originally named for has been retired,
79# but the record stays as pure LB infra — deleting it breaks TLS renewal
80# cluster-wide.
⋯ 12 lines hidden (lines 81–92)
81resource "digitalocean_record" "mseeks_me_ynab" {
82 domain = digitalocean_domain.mseeks_me.name
83 type = "A"
84 name = "ynab"
85 # Sourced from the live LB; if the LB is recreated (new IP), a plain
86 # `terraform apply` re-points this record — no hardcoded address to chase.
87 value = data.digitalocean_loadbalancer.ingress.ip
88 # Short TTL: this points at a DO LB whose IP can change if the LB is recreated,
89 # so keep resolver caches from pinning a stale address for long.
90 ttl = 300
91}
93# glassbox.mseeks.me -> Vercel. Glassbox is hosted on Vercel, not the cluster:
94# the project + custom domain are managed in the Vercel dashboard and deploys
95# ride its GitHub integration (push to main in mseeks/glassbox), so the only
96# infra Terraform owns is this record. Vercel terminates TLS at its edge — no
97# cert-manager involvement. The value is this project's own CNAME target, the
98# one the dashboard issued when glassbox.mseeks.me was added to the project
99# (Vercel mints per-project targets; the old universal cname.vercel-dns.com is
100# legacy). If the Vercel project is ever deleted and recreated, it gets a new
101# target — read the fresh value off the project's Domains tab, and only
102# re-point this record once the domain is attached there (flipped early, the
103# host serves Vercel's DEPLOYMENT_NOT_FOUND page).
104resource "digitalocean_record" "mseeks_me_glassbox" {
105 domain = digitalocean_domain.mseeks_me.name
106 type = "CNAME"
107 name = "glassbox"
108 value = "e24be5a7fd9d016f.vercel-dns-017.com."
109 # Unlike the LB A records above, the target here is stable for the life of
110 # the Vercel project, so no short-TTL churn guard is needed.
111 ttl = 3600
⋯ 287 lines hidden (lines 112–398)
112}
114# static.mseeks.me -> the same zo-k8s ingress Load Balancer. Public entrypoint
115# for the `static` drop-and-share site (https://static.mseeks.me; see
116# infra/k8s/static) and the host cert-manager gets a Let's Encrypt cert for.
117# Same read-only LB-IP sourcing as ynab above — no hardcoded address to
118# chase.
119resource "digitalocean_record" "mseeks_me_static" {
120 domain = digitalocean_domain.mseeks_me.name
121 type = "A"
122 name = "static"
123 value = data.digitalocean_loadbalancer.ingress.ip
124 # Short TTL for the same reason as ynab: the target is a DO LB whose
125 # IP can change if the LB is recreated.
126 ttl = 300
127}
129# *.everwhen.mseeks.me -> the same zo-k8s ingress Load Balancer. Wildcard record
130# so ephemeral per-PR preview hosts (pr-<N>.everwhen.mseeks.me; see
131# infra/k8s/everwhen-preview) resolve to the LB without a per-PR DNS change. The
132# bare everwhen.mseeks.me A record is gone — the Everwhen app moved to
133# playeverwhen.com — so this wildcard now serves per-PR previews only (it never
134# matched the bare apex anyway; a wildcard covers deeper labels, not the name
135# itself). cert-manager issues a per-host Let's Encrypt cert on first request
136# (HTTP-01), so each preview gets its own TLS with no wildcard cert needed. Same
137# read-only LB-IP sourcing as the records above — no hardcoded address to chase.
138resource "digitalocean_record" "mseeks_me_everwhen_wildcard" {
139 domain = digitalocean_domain.mseeks_me.name
140 type = "A"
141 name = "*.everwhen"
142 value = data.digitalocean_loadbalancer.ingress.ip
143 # Short TTL for the same reason as the others: the target is a DO LB whose IP
144 # can change if the LB is recreated.
145 ttl = 300
146}
148# froot.mseeks.me -> the same zo-k8s ingress Load Balancer. Public entrypoint
149# for froot's read-only read-model dashboard (https://froot.mseeks.me; see
150# infra/k8s/froot), gated by HTTP basic auth, and the host cert-manager gets a
151# Let's Encrypt cert for. Same read-only LB-IP sourcing as the records above —
152# no hardcoded address to chase.
153resource "digitalocean_record" "mseeks_me_froot" {
154 domain = digitalocean_domain.mseeks_me.name
155 type = "A"
156 name = "froot"
157 value = data.digitalocean_loadbalancer.ingress.ip
158 # Short TTL for the same reason as the others: the target is a DO LB whose IP
159 # can change if the LB is recreated.
160 ttl = 300
161}
163# models.mseeks.me -> the same zo-k8s ingress Load Balancer. Public entrypoint
164# for the read-only model-traffic dashboard (https://models.mseeks.me; see
165# infra/k8s/model-metrics), gated by HTTP basic auth, and the host cert-manager
166# gets a Let's Encrypt cert for. Same read-only LB-IP sourcing as the others.
167resource "digitalocean_record" "mseeks_me_models" {
168 domain = digitalocean_domain.mseeks_me.name
169 type = "A"
170 name = "models"
171 value = data.digitalocean_loadbalancer.ingress.ip
172 ttl = 300
173}
175# ============================== msull92.com ==============================
177resource "digitalocean_domain" "msull92_com" {
178 name = "msull92.com"
179}
181resource "digitalocean_record" "msull92_com_mx_01" {
182 domain = digitalocean_domain.msull92_com.name
183 type = "MX"
184 name = "@"
185 value = "mx01.mail.icloud.com."
186 priority = 10
187 ttl = 14400
188}
190resource "digitalocean_record" "msull92_com_mx_02" {
191 domain = digitalocean_domain.msull92_com.name
192 type = "MX"
193 name = "@"
194 value = "mx02.mail.icloud.com."
195 priority = 10
196 ttl = 14400
197}
199resource "digitalocean_record" "msull92_com_apple_domain" {
200 domain = digitalocean_domain.msull92_com.name
201 type = "TXT"
202 name = "@"
203 value = "apple-domain=SLKQpEYsouHgb3mI"
204 ttl = 3600
205}
207resource "digitalocean_record" "msull92_com_spf" {
208 domain = digitalocean_domain.msull92_com.name
209 type = "TXT"
210 name = "@"
211 value = "\"v=spf1 include:icloud.com ~all\""
212 ttl = 3600
213}
215resource "digitalocean_record" "msull92_com_dkim" {
216 domain = digitalocean_domain.msull92_com.name
217 type = "CNAME"
218 name = "sig1._domainkey"
219 value = "sig1.dkim.msull92.com.at.icloudmailadmin.com."
220 ttl = 43200
221}
223# ============================== 2ls.tech ==============================
224# HCL resource names can't start with a digit, so these use a `_2ls_tech`
225# prefix (a leading underscore is valid) rather than the plain `2ls_tech`
226# the mseeks_me/msull92_com convention would suggest.
228resource "digitalocean_domain" "_2ls_tech" {
229 name = "2ls.tech"
230}
232resource "digitalocean_record" "_2ls_tech_mx_01" {
233 domain = digitalocean_domain._2ls_tech.name
234 type = "MX"
235 name = "@"
236 value = "mx01.mail.icloud.com."
237 priority = 10
238 ttl = 14400
239}
241resource "digitalocean_record" "_2ls_tech_mx_02" {
242 domain = digitalocean_domain._2ls_tech.name
243 type = "MX"
244 name = "@"
245 value = "mx02.mail.icloud.com."
246 priority = 10
247 ttl = 14400
248}
250resource "digitalocean_record" "_2ls_tech_apple_domain" {
251 domain = digitalocean_domain._2ls_tech.name
252 type = "TXT"
253 name = "@"
254 value = "apple-domain=KAYDndJUw5TrEwCe"
255 ttl = 3600
256}
258# Google Search Console domain verification (TXT at the apex). Value UNQUOTED
259# like the apple-domain record above — DigitalOcean adds the DNS wire quotes
260# itself (see the SPF quoting gotcha at the top of this file).
261resource "digitalocean_record" "_2ls_tech_google_site_verification" {
262 domain = digitalocean_domain._2ls_tech.name
263 type = "TXT"
264 name = "@"
265 value = "google-site-verification=9e1Dm56-fdkP2jTv1pGNI5II2ol0pR6Xtw8RtS1gXwM"
266 ttl = 3600
267}
269# SPF value UNQUOTED on purpose — see the SPF quoting gotcha at the top of this
270# file. Recreated once (terraform apply -replace) to strip the doubly-quoted
271# `""v=spf1 ...""` the original escaped-quote value produced on create.
272resource "digitalocean_record" "_2ls_tech_spf" {
273 domain = digitalocean_domain._2ls_tech.name
274 type = "TXT"
275 name = "@"
276 value = "v=spf1 include:icloud.com ~all"
277 ttl = 3600
278}
280resource "digitalocean_record" "_2ls_tech_dkim" {
281 domain = digitalocean_domain._2ls_tech.name
282 type = "CNAME"
283 name = "sig1._domainkey"
284 value = "sig1.dkim.2ls.tech.at.icloudmailadmin.com."
285 ttl = 43200
286}
288# 2ls.tech (+ www) -> the zo-k8s ingress Load Balancer. Public entrypoint for
289# the Two Ls company site (https://2ls.tech; see infra/k8s/2ls) and the hosts
290# cert-manager gets a Let's Encrypt cert for. Same read-only LB-IP sourcing as
291# the mseeks.me records above — no hardcoded address to chase. APPLIED — these
292# records are live.
293resource "digitalocean_record" "_2ls_tech_apex" {
294 domain = digitalocean_domain._2ls_tech.name
295 type = "A"
296 name = "@"
297 value = data.digitalocean_loadbalancer.ingress.ip
298 # Short TTL: the target is a DO LB whose IP can change if the LB is
299 # recreated.
300 ttl = 300
301}
303resource "digitalocean_record" "_2ls_tech_www" {
304 domain = digitalocean_domain._2ls_tech.name
305 type = "A"
306 name = "www"
307 value = data.digitalocean_loadbalancer.ingress.ip
308 ttl = 300
309}
311# ============================== playeverwhen.com ==============================
313resource "digitalocean_domain" "playeverwhen_com" {
314 name = "playeverwhen.com"
315}
317resource "digitalocean_record" "playeverwhen_com_mx_01" {
318 domain = digitalocean_domain.playeverwhen_com.name
319 type = "MX"
320 name = "@"
321 value = "mx01.mail.icloud.com."
322 priority = 10
323 ttl = 14400
324}
326resource "digitalocean_record" "playeverwhen_com_mx_02" {
327 domain = digitalocean_domain.playeverwhen_com.name
328 type = "MX"
329 name = "@"
330 value = "mx02.mail.icloud.com."
331 priority = 10
332 ttl = 14400
333}
335resource "digitalocean_record" "playeverwhen_com_apple_domain" {
336 domain = digitalocean_domain.playeverwhen_com.name
337 type = "TXT"
338 name = "@"
339 value = "apple-domain=DEESFXkRhJ4wE6Lm"
340 ttl = 3600
341}
343# Google Search Console domain verification (TXT at the apex). Value UNQUOTED
344# like the apple-domain record above — DigitalOcean adds the DNS wire quotes
345# itself (see the SPF quoting gotcha at the top of this file).
346resource "digitalocean_record" "playeverwhen_com_google_site_verification" {
347 domain = digitalocean_domain.playeverwhen_com.name
348 type = "TXT"
349 name = "@"
350 value = "google-site-verification=1FSmsnv2MPG_mKoCrdlSABXfyXFTaeVBETNRmLNi85o"
351 ttl = 3600
352}
354# SPF value UNQUOTED on purpose — see the SPF quoting gotcha at the top of this
355# file. DigitalOcean adds the DNS wire quotes itself.
356resource "digitalocean_record" "playeverwhen_com_spf" {
357 domain = digitalocean_domain.playeverwhen_com.name
358 type = "TXT"
359 name = "@"
360 value = "v=spf1 include:icloud.com ~all"
361 ttl = 3600
362}
364resource "digitalocean_record" "playeverwhen_com_dkim" {
365 domain = digitalocean_domain.playeverwhen_com.name
366 type = "CNAME"
367 name = "sig1._domainkey"
368 value = "sig1.dkim.playeverwhen.com.at.icloudmailadmin.com."
369 ttl = 43200
370}
372# playeverwhen.com (apex) -> the zo-k8s ingress Load Balancer. Public entrypoint
373# for the Everwhen SSR app (https://playeverwhen.com; see infra/k8s/everwhen),
374# migrated here from everwhen.mseeks.me. cert-manager gets a Let's Encrypt cert
375# for it (HTTP-01, so the apex must resolve to the LB). An apex A record (a CNAME
376# is illegal at a zone apex) works because the value is the LB IP, sourced
377# read-only from data.digitalocean_loadbalancer.ingress — no hardcoded address to
378# chase. Coexists fine with the iCloud MX/TXT records above.
379resource "digitalocean_record" "playeverwhen_com_apex" {
380 domain = digitalocean_domain.playeverwhen_com.name
381 type = "A"
382 name = "@"
383 value = data.digitalocean_loadbalancer.ingress.ip
384 # Short TTL: the target is a DO LB whose IP can change if the LB is recreated.
385 ttl = 300
386}
388# www.playeverwhen.com -> the same LB. Serves only a 301 redirect to the apex
389# (see the everwhen-www-redirect Ingress in infra/k8s/everwhen); cert-manager
390# issues its own cert so the redirect is reachable over HTTPS. Same read-only
391# LB-IP sourcing as the apex.
392resource "digitalocean_record" "playeverwhen_com_www" {
393 domain = digitalocean_domain.playeverwhen_com.name
394 type = "A"
395 name = "www"
396 value = data.digitalocean_loadbalancer.ingress.ip
397 ttl = 300
398}
The k8s stack, retired
Five files under infra/k8s/glassbox/ disappear, 219 lines in all. There is no one-for-one replacement. Vercel absorbs most of the work, while a couple of jobs (probes, the namespace itself) end with the pod they served:
| file | what it did | replaced by |
|---|---|---|
manifests/00-namespace.yaml | the glassbox namespace | nothing to isolate |
manifests/10-deployment.yaml | one nginx pod serving ghcr.io/mseeks/glassbox:latest (Recreate, /healthz probes, 16Mi request) and its Service | Vercel's edge serves the build output |
manifests/20-ingress.yaml | routed glassbox.mseeks.me through the DO LB; cert-manager filled glassbox-mseeks-me-tls via letsencrypt-prod | Vercel terminates TLS itself |
install.sh | context preflight, kubectl apply, rollout restart + status | git push — the integration deploys |
README.md | how the stack was wired | the app repo's new Deploy section |
Doc sweep: glassbox was everyone's example
Glassbox was the example everything else pointed at. The cert-manager README used its Ingress as the live sample, the sibling stacks compared themselves to it, and dns.tf counted it among the hairpin hosts. Eight spots get small swaps, five READMEs and three dns.tf comments, so no doc references a deleted directory:
| file | swap |
|---|---|
cert-manager/README.md | live example is now static's Ingress (static.mseeks.me → static-mseeks-me-tls) |
static/README.md | drops the "Like glassbox" comparison; describes itself standalone |
2ls/README.md | "same shape as static/glassbox" → "same shape as static" |
everwhen/README.md | "Unlike glassbox/static" → "Unlike static/2ls" |
ingress-nginx/README.md | example Ingress host is now static.mseeks.me |
dns.tf (comments) | out of the hairpin host list, and out of the static record's two "same as ynab/glassbox" asides |
The requirements stay generic; only the live example moves to a host that still terminates TLS in-cluster.
infra/k8s/cert-manager/README.md · 52 lines
infra/k8s/cert-manager/README.md
⋯ 26 lines hidden (lines 1–26)
1# cert-manager (automatic TLS)
3[cert-manager](https://cert-manager.io/) on zo-k8s, plus two Let's Encrypt
4`ClusterIssuer`s. It watches `Ingress` objects annotated with
5`cert-manager.io/cluster-issuer` and provisions + auto-renews their TLS certs.
7```bash
8doctl kubernetes cluster kubeconfig save zo-k8s
9./install.sh # helm upgrade --install (with CRDs) + applies the issuers
10kubectl get clusterissuer # letsencrypt-prod / letsencrypt-staging -> Ready
11```
13`values.yaml` pins small resource requests (three low-traffic controllers sharing
14the cluster) and installs the CRDs with the chart.
16## Issuers
18| Issuer | ACME directory | Use |
19|---|---|---|
20| `letsencrypt-prod` | production | real, trusted certs (the default) |
21| `letsencrypt-staging` | staging | dry-run issuance without burning the prod rate limit (5 duplicate certs/week); certs are untrusted |
23Both solve **HTTP-01 through ingress-nginx**: cert-manager stands up a temporary
24solver Ingress for `/.well-known/acme-challenge/<token>`, and Let's Encrypt
25reaches it over `http://<host>` → the DO LB → nginx → the solver pod.
27## Requirements for a cert to issue
291. The host's DNS resolves to the ingress LB (see [`../../dns.tf`](../../dns.tf)).
30 ([`../ingress-nginx`](../ingress-nginx) must be up first.)
312. Port 80 reaches nginx through the LB (the solver serves the challenge on HTTP).
323. The app `Ingress` carries `cert-manager.io/cluster-issuer: letsencrypt-prod`
33 and a `tls:` block naming the secret to populate.
35Then it's hands-off: see [`../static/manifests/20-ingress.yaml`](../static/manifests/20-ingress.yaml)
36for a live example (`static.mseeks.me` → `static-mseeks-me-tls`).
⋯ 16 lines hidden (lines 37–52)
38## Note: validation needs *fresh* DNS
40Let's Encrypt's external validators resolve the host fresh, but cert-manager's
41in-cluster HTTP-01 self-check resolves via CoreDNS. If you re-pointed the host's
42A record (e.g. after recreating the LB), an upstream resolver may briefly serve
43the **old** IP and the self-check logs "context deadline exceeded" until that
44cache expires. Issuance still completes once the cache clears (the short TTL on
45the record keeps this brief); it does not require touching cluster DNS.
47## Inspecting a stuck issuance
49```bash
50kubectl get certificate,order,challenge -n <ns>
51kubectl describe challenge -n <ns> # the 'Reason' explains a pending self-check
52```
Verification, and the cutover runbook
terraform fmt -check, init, and validate pass on the branch, and terraform plan against live state shows the intended replacement, with one unrelated change beside it:
digitalocean_record.mseeks_me_glassboxmust be replaced —type: "A" -> "CNAME"(forces replacement),value: "174.138.116.75" -> "e24be5a7fd9d016f.vercel-dns-017.com.",ttl: 300 -> 3600.
That second change is drift: the DOKS version data source now returns 1.36.0-do.2, so a plain apply would also patch-bump the cluster. That upgrade is not this PR's business — the cutover apply targets the record (terraform apply -target=digitalocean_record.mseeks_me_glassbox), leaving the cluster bump as its own deliberate decision.
The cutover, in order:
1. Create the Vercel project (import mseeks/glassbox; the Vite preset covers build and output), let it deploy, and attach the glassbox.mseeks.me domain. The dashboard mints the project's CNAME target at that moment; the committed value is the one it issued. 2. Merge both PRs; apply the targeted record flip. 3. Verify: a hashed asset serves immutable, a nonsense path serves the app shell, and the response says Vercel, not nginx. 4. kubectl delete namespace glassbox — the pod, Service, Ingress, and cert secret go with it, and the cluster frees the pod's small reservation.