Filter figure autocomplete to people (#74)

What changed, and why

The contact step has a name box with autocomplete. It was deliberately unfiltered — a raw Wikipedia title search — so it surfaced films, places and concepts ("Cleopatra (1963 film)", "Death of Cleopatra", "Mongolia") right alongside people. But the contact gate (#72 / #73) only reaches a real, deceased person, so previewing a non-person invited a pick that gets refused at send.

This PR adds a people filter. After the title search returns, one batched Wikidata wbgetentities call classifies the matches and keeps a result only when Wikidata calls it a person: **instance-of human (P31 → Q5), or it has a recorded death (P570)**. Everything else is dropped.

Blast radius is small and server-side: the logic lives in one util (server/utils/figure-search.ts); the API route and the picker component change only comments. The FigureSearchResult shape the client renders is untouched, so the combobox needs no changes. Crucially, the #58 rate-limit hardening on the search path is preserved — the filter is layered on top and degrades by falling open, never by blanking a working search.

The search body: filter, fall open, cache only when settled

searchFigures keeps its #58 shape exactly: a too-short query short-circuits, the cache is checked, the title search runs with one spaced retry, and a search that never answers returns { results: [], transient: true } so the combobox retries rather than reading weather as "no matches".

The new block is the people filter (lines 224–239). It only runs when the search returned results — an empty answer skips the lookup, so there's no wasted request. The interesting decision is the failure handling: if the filter lookup blips (people === null), the code does not blank the search. It sets filterTransient and falls open, returning the unfiltered matches with transient: false, because there genuinely are results to show. The cache write at the end fires only when neither stage was transient, so a fall-open is never cached and the next keystroke re-attempts the filter.

searchFigures: the search path is unchanged; the filter is the 224–239 block.

server/utils/figure-search.ts · 255 lines

server/utils/figure-search.ts255 lines · TypeScript

⋯ 206 lines hidden (lines 1–206)1/**
* Name autocomplete for the contact step — Wikipedia REST title search (the
* prefix-matching endpoint built for exactly this), key-less like the grounding
* pipeline. Never throws.
*
* People only (issue #74): raw title search surfaces films, places and concepts
* ("Cleopatra (1963 film)", "Death of Cleopatra", "Mongolia"), and the contact
* gate (#72/#73) only reaches a real, deceased person — so previewing non-people
* invites a pick that's refused at send. After the search returns, ONE batched
* Wikidata call (`wbgetentities` over the result titles) keeps a result when it
* looks like a *person*: instance-of human (P31 → Q5), OR it carries a recorded
* death (P570).
*
* Why the death clause, and why death and not birth:
*  - It catches genuinely contactable figures that are NOT tagged Q5 — Moses,
*    Abraham, King Arthur and other biblical / legendary / mythological figures are
*    typed "biblical figure" or "legendary figure", yet the gate accepts them (they
*    resolve and carry a death date) and the lifespan estimator (#31) is explicitly
*    built to date the likes of "King Arthur, Laozi". A bare Q5 test wrongly hides
*    them; keying on death keeps them.
*  - Death, not birth, because the gate's floor IS death (#72). A non-human entity
*    that has only a birth date is exactly a fictional character with an in-universe
*    birthday — Sherlock Holmes, Harry Potter — whom the gate refuses as "not
*    deceased". Previewing them would just reproduce the confusing pick. (A birth
*    clause would also never WIDEN coverage: the gate dates the death-less only via
*    the #31 bridge, which fires only when there's NO birth year — so a birth date
*    can't be the thing that makes a figure contactable.) Real LIVING people still
*    appear, on the Q5 clause, and the dossier explains the deceased-only floor.
*
* Relationship to the gate (stated honestly — it is NOT a perfect superset): the
* gate is `resolved && died`, with no instance-of check, where `died` can come from
* Wikidata P570 OR the #31 AI bridge. This filter keeps everyone the gate can reach
* via a Wikidata death (they have P570), plus all humans (living ones included —
* looser than the gate there, by design). The one under-coverage gap is a figure
* datable ONLY by the AI bridge AND not tagged Q5 — a rare, obscure case; it won't
* be suggested, but the gate stays the authoritative backstop and the full name can
* still be typed and sent. A famous dated animal (a racehorse, say) is kept — it's
* datable and the gate would accept it too. We do NOT filter on the death's *value*
* (the safely-historical floor): that's the gate's job, and judging it here would
* need the per-result grounding the batched call is built to avoid.
*
* Failure honesty (the lesson of issue #31, applied to search): a rate-limit 429
* or timeout is weather, not "no matches". A failed search gets one spaced retry,
* and if that fails too the outcome says `transient: true` instead of posing as
* an empty result — so the combobox can quietly try again rather than silently
* never opening. The people-filter is layered ON TOP without regressing this
* (#58): a search that succeeded but whose filter lookup blips falls OPEN — the
* unfiltered matches are shown (the description line still tells the pharaoh from
* the film) rather than blanking a working autocomplete. Only definitive answers
* — search AND filter both settled — are cached.
*/
52import { pickClaimYear, safeUrl, WIKI_HEADERS } from './figure-grounding'
53 
54export interface FigureSearchResult {
  name: string
  description?: string
  thumbnail?: string
58}
59 
60export interface FigureSearchOutcome {
  results: FigureSearchResult[]
  /** True when Wikipedia couldn't be reached (429, timeout, network): the empty
   *  answer is weather, and the caller may retry soon. Never set on a real
   *  "no matches" answer. */
  transient: boolean
66}
67 
68/** Cache: queries repeat heavily while typing (and across players). Capped so a
*  crawl of unique queries can't grow it without bound (the grounding cache's
*  known gap — not repeated here). */
71const CACHE_TTL_MS = 60 * 60 * 1000
72const CACHE_MAX_ENTRIES = 500
73const cache = new Map<string, { value: FigureSearchResult[]; expires: number }>()
74 
75const RESULT_LIMIT = 5
76const SEARCH_RETRY_DELAY_MS = 250
77 
78/** Wikidata properties the people test reads: "instance of" (P31, → human Q5) and
*  date of death (P570). Q5 OR a recorded death marks a person (see the file header
*  for why death — not birth — is the second clause). */
81const WIKIDATA_INSTANCE_OF = 'P31'
82const WIKIDATA_HUMAN = 'Q5'
83const WIKIDATA_DEATH = 'P570'
84 
85/** The slice of the Wikipedia REST v1 title-search response we read. */
86interface WikiTitleSearch {
  pages?: Array<{
      title?: string
      description?: string
      thumbnail?: { url?: string } | null
  }>
92}
93 
94/** Wikipedia thumbnails arrive protocol-relative ("//upload.wikimedia.org/…"). */
95function absoluteThumb(url?: string): string | undefined {
  if (!url) return undefined
  return safeUrl(url.startsWith('//') ? `https:${url}` : url)
98}
99 
100/** One attempt against the title-search endpoint; null means it didn't answer. */
101async function fetchSearch(query: string): Promise<FigureSearchResult[] | null> {
  try {
      const res = await fetch(
          `https://en.wikipedia.org/w/rest.php/v1/search/title?q=${encodeURIComponent(query)}&limit=${RESULT_LIMIT}`,
          { headers: WIKI_HEADERS, signal: AbortSignal.timeout(4000) }
      )
      if (!res.ok) return null
      const data = (await res.json()) as WikiTitleSearch
      return (data.pages ?? [])
          .filter(p => typeof p.title === 'string' && p.title.length > 0)
          .slice(0, RESULT_LIMIT)
          .map(p => ({
              name: p.title as string,
              description: p.description || undefined,
              thumbnail: absoluteThumb(p.thumbnail?.url)
          }))
  } catch {
      return null
  }
120}
121 
122/** The slice of one Wikidata claim the people-filter reads: an item value (P31's
*  `id`) or a time value (P570's `time`). Shaped to be accepted by the grounding
*  module's `pickClaimYear` (which reads `rank` / `snaktype` / `time`), so the
*  "usable date" rule stays single-sourced. */
126interface WikidataClaimSlice {
  rank?: string
  mainsnak?: {
      snaktype?: string
      datavalue?: { value?: { id?: string; time?: string; precision?: number } }
  }
132}
133 
134/** The slice of the Wikidata `wbgetentities` response the people-filter reads:
*  per entity, its claims (P31 / P570) and its enwiki sitelink title (which maps
*  the entity back to the search result it came from). */
137interface WikidataPeopleEntities {
  entities?: Record<
      string,
      {
          claims?: Record<string, WikidataClaimSlice[]>
          sitelinks?: { enwiki?: { title?: string } }
      }
  >
145}
146 
147/** Match a REST search title to a Wikidata sitelink title. Both are canonical
*  enwiki titles, so this is belt-and-suspenders against underscore/space/case
*  drift rather than a real transform. */
150function titleKey(title: string): string {
  return title.replace(/_/g, ' ').trim().toLowerCase()
152}
153 
154/** One attempt at the batched Wikidata lookup; null means it didn't answer (so the
*  caller can fall open rather than blank a working search). `props=claims` carries
*  P31 / P570; `sitelinks` + `sitefilter=enwiki` carry just the enwiki title back
*  per entity so each result can be matched to its claims. (No `normalize=1`:
*  it's only valid for a single title, and this call batches several.) */
159async function fetchInstanceClaims(titles: string[]): Promise<WikidataPeopleEntities['entities'] | null> {
  try {
      const url =
          'https://www.wikidata.org/w/api.php?action=wbgetentities' +
          `&sites=enwiki&titles=${encodeURIComponent(titles.join('|'))}` +
          '&props=claims%7Csitelinks&sitefilter=enwiki&languages=en&format=json&origin=*'
      const res = await fetch(url, { headers: WIKI_HEADERS, signal: AbortSignal.timeout(4000) })
      if (!res.ok) return null
      const data = (await res.json()) as WikidataPeopleEntities
      return data.entities ?? null
  } catch {
      return null
  }
172}
173 
174/** Does this entity look like a PERSON? Instance-of human (P31 → Q5), OR it has a
*  recorded death (P570) — the clause that catches biblical / legendary / mythological
*  figures that aren't tagged Q5 (Moses, King Arthur) but are contactable. (Death,
*  not birth: a birth-only non-human is a fictional character the gate would refuse;
*  see the file header.) `pickClaimYear` (reused from grounding) applies the same
*  usable-claim rule as the dossier: non-deprecated, real `value` snak, has a time. */
180function isPerson(claims?: Record<string, WikidataClaimSlice[]>): boolean {
  const human = (claims?.[WIKIDATA_INSTANCE_OF] ?? []).some(
      c => c.mainsnak?.snaktype === 'value' && c.mainsnak.datavalue?.value?.id === WIKIDATA_HUMAN
  )
  return human || !!pickClaimYear(claims?.[WIKIDATA_DEATH])
185}
186 
187/** Which of these article titles are PEOPLE? Returns a set of the person titles
*  (normalized via `titleKey`), or null when Wikidata never answered — same
*  one-spaced-retry-on-a-blip pattern the grounding lookup uses, then fall open.
*  A title with no Wikidata item (or one that isn't a person) isn't in the set. */
191async function fetchPeopleTitles(titles: string[]): Promise<Set<string> | null> {
  let entities = await fetchInstanceClaims(titles)
  if (!entities) {
      await new Promise(resolve => setTimeout(resolve, SEARCH_RETRY_DELAY_MS))
      entities = await fetchInstanceClaims(titles)
  }
  if (!entities) return null
198 
  const people = new Set<string>()
  for (const entity of Object.values(entities)) {
      const title = entity.sitelinks?.enwiki?.title
      if (title && isPerson(entity.claims)) people.add(titleKey(title))
  }
  return people
205}
206 
207export async function searchFigures(rawQuery: string): Promise<FigureSearchOutcome> {
  const query = (rawQuery || '').trim()
  if (query.length < 2) return { results: [], transient: false }
210 
  const key = query.toLowerCase()
  const hit = cache.get(key)
  if (hit && hit.expires > Date.now()) return { results: hit.value, transient: false }
214 
  let results = await fetchSearch(query)
  if (results === null) {
      // Rate-limit blips are the dominant real-world failure here (shared egress
      // IP, plus the suggester's own wiki bursts) — one spaced retry rescues most.
      await new Promise(resolve => setTimeout(resolve, SEARCH_RETRY_DELAY_MS))
      results = await fetchSearch(query)
  }
  if (results === null) return { results: [], transient: true }
223 
  // People filter (#74): keep only results Wikidata calls a person (P31 → Q5, or a
  // recorded death). An empty search answer skips the lookup (nothing to classify,
  // no wasted call).
  let filterTransient = false
  if (results.length > 0) {
      const people = await fetchPeopleTitles(results.map(r => r.name))
      if (people === null) {
          // The search SUCCEEDED; only the filter hit weather. Fall open with the
          // unfiltered matches rather than blanking a working autocomplete (#58) —
          // the description line still tells the pharaoh from the film, and the
          // contact gate (#72/#73) is the authoritative backstop. Don't cache it.
          filterTransient = true
      } else {
          results = results.filter(r => people.has(titleKey(r.name)))
      }
  }
240 
  // Cache DEFINITIVE ANSWERS only — search AND filter both settled. A timeout or a
  // Wikipedia 5xx (search) must not blank this query for an hour, and neither must a
  // filter blip we fell open on (the next keystroke should get a clean filter). A
  // legitimate "no people match" empty result IS an answer and caches normally.
  if (!filterTransient) {
      if (cache.size >= CACHE_MAX_ENTRIES) {
          // Drop the oldest entry (Map preserves insertion order) — a tiny, boring
          // eviction that keeps the cap honest without LRU machinery.
          const oldest = cache.keys().next().value
          if (oldest !== undefined) cache.delete(oldest)
      }
      cache.set(key, { value: results, expires: Date.now() + CACHE_TTL_MS })
  }
  return { results, transient: false }
255}

The predicate: human, or has a recorded death

isPerson is the heart of the change, and it's deliberately two clauses. A result is kept if its Wikidata entity is instance-of human (P31 → Q5), or it has a usable death date (P570). The death check reuses pickClaimYear from the grounding module, so "usable" means exactly what the dossier means: a non-deprecated claim with a real value snak that carries a time.

Q5 OR a usable P570 death — the people test.

server/utils/figure-search.ts · 255 lines

server/utils/figure-search.ts255 lines · TypeScript

⋯ 179 lines hidden (lines 1–179)1/**
* Name autocomplete for the contact step — Wikipedia REST title search (the
* prefix-matching endpoint built for exactly this), key-less like the grounding
* pipeline. Never throws.
*
* People only (issue #74): raw title search surfaces films, places and concepts
* ("Cleopatra (1963 film)", "Death of Cleopatra", "Mongolia"), and the contact
* gate (#72/#73) only reaches a real, deceased person — so previewing non-people
* invites a pick that's refused at send. After the search returns, ONE batched
* Wikidata call (`wbgetentities` over the result titles) keeps a result when it
* looks like a *person*: instance-of human (P31 → Q5), OR it carries a recorded
* death (P570).
*
* Why the death clause, and why death and not birth:
*  - It catches genuinely contactable figures that are NOT tagged Q5 — Moses,
*    Abraham, King Arthur and other biblical / legendary / mythological figures are
*    typed "biblical figure" or "legendary figure", yet the gate accepts them (they
*    resolve and carry a death date) and the lifespan estimator (#31) is explicitly
*    built to date the likes of "King Arthur, Laozi". A bare Q5 test wrongly hides
*    them; keying on death keeps them.
*  - Death, not birth, because the gate's floor IS death (#72). A non-human entity
*    that has only a birth date is exactly a fictional character with an in-universe
*    birthday — Sherlock Holmes, Harry Potter — whom the gate refuses as "not
*    deceased". Previewing them would just reproduce the confusing pick. (A birth
*    clause would also never WIDEN coverage: the gate dates the death-less only via
*    the #31 bridge, which fires only when there's NO birth year — so a birth date
*    can't be the thing that makes a figure contactable.) Real LIVING people still
*    appear, on the Q5 clause, and the dossier explains the deceased-only floor.
*
* Relationship to the gate (stated honestly — it is NOT a perfect superset): the
* gate is `resolved && died`, with no instance-of check, where `died` can come from
* Wikidata P570 OR the #31 AI bridge. This filter keeps everyone the gate can reach
* via a Wikidata death (they have P570), plus all humans (living ones included —
* looser than the gate there, by design). The one under-coverage gap is a figure
* datable ONLY by the AI bridge AND not tagged Q5 — a rare, obscure case; it won't
* be suggested, but the gate stays the authoritative backstop and the full name can
* still be typed and sent. A famous dated animal (a racehorse, say) is kept — it's
* datable and the gate would accept it too. We do NOT filter on the death's *value*
* (the safely-historical floor): that's the gate's job, and judging it here would
* need the per-result grounding the batched call is built to avoid.
*
* Failure honesty (the lesson of issue #31, applied to search): a rate-limit 429
* or timeout is weather, not "no matches". A failed search gets one spaced retry,
* and if that fails too the outcome says `transient: true` instead of posing as
* an empty result — so the combobox can quietly try again rather than silently
* never opening. The people-filter is layered ON TOP without regressing this
* (#58): a search that succeeded but whose filter lookup blips falls OPEN — the
* unfiltered matches are shown (the description line still tells the pharaoh from
* the film) rather than blanking a working autocomplete. Only definitive answers
* — search AND filter both settled — are cached.
*/
52import { pickClaimYear, safeUrl, WIKI_HEADERS } from './figure-grounding'
53 
54export interface FigureSearchResult {
  name: string
  description?: string
  thumbnail?: string
58}
59 
60export interface FigureSearchOutcome {
  results: FigureSearchResult[]
  /** True when Wikipedia couldn't be reached (429, timeout, network): the empty
   *  answer is weather, and the caller may retry soon. Never set on a real
   *  "no matches" answer. */
  transient: boolean
66}
67 
68/** Cache: queries repeat heavily while typing (and across players). Capped so a
*  crawl of unique queries can't grow it without bound (the grounding cache's
*  known gap — not repeated here). */
71const CACHE_TTL_MS = 60 * 60 * 1000
72const CACHE_MAX_ENTRIES = 500
73const cache = new Map<string, { value: FigureSearchResult[]; expires: number }>()
74 
75const RESULT_LIMIT = 5
76const SEARCH_RETRY_DELAY_MS = 250
77 
78/** Wikidata properties the people test reads: "instance of" (P31, → human Q5) and
*  date of death (P570). Q5 OR a recorded death marks a person (see the file header
*  for why death — not birth — is the second clause). */
81const WIKIDATA_INSTANCE_OF = 'P31'
82const WIKIDATA_HUMAN = 'Q5'
83const WIKIDATA_DEATH = 'P570'
84 
85/** The slice of the Wikipedia REST v1 title-search response we read. */
86interface WikiTitleSearch {
  pages?: Array<{
      title?: string
      description?: string
      thumbnail?: { url?: string } | null
  }>
92}
93 
94/** Wikipedia thumbnails arrive protocol-relative ("//upload.wikimedia.org/…"). */
95function absoluteThumb(url?: string): string | undefined {
  if (!url) return undefined
  return safeUrl(url.startsWith('//') ? `https:${url}` : url)
98}
99 
100/** One attempt against the title-search endpoint; null means it didn't answer. */
101async function fetchSearch(query: string): Promise<FigureSearchResult[] | null> {
  try {
      const res = await fetch(
          `https://en.wikipedia.org/w/rest.php/v1/search/title?q=${encodeURIComponent(query)}&limit=${RESULT_LIMIT}`,
          { headers: WIKI_HEADERS, signal: AbortSignal.timeout(4000) }
      )
      if (!res.ok) return null
      const data = (await res.json()) as WikiTitleSearch
      return (data.pages ?? [])
          .filter(p => typeof p.title === 'string' && p.title.length > 0)
          .slice(0, RESULT_LIMIT)
          .map(p => ({
              name: p.title as string,
              description: p.description || undefined,
              thumbnail: absoluteThumb(p.thumbnail?.url)
          }))
  } catch {
      return null
  }
120}
121 
122/** The slice of one Wikidata claim the people-filter reads: an item value (P31's
*  `id`) or a time value (P570's `time`). Shaped to be accepted by the grounding
*  module's `pickClaimYear` (which reads `rank` / `snaktype` / `time`), so the
*  "usable date" rule stays single-sourced. */
126interface WikidataClaimSlice {
  rank?: string
  mainsnak?: {
      snaktype?: string
      datavalue?: { value?: { id?: string; time?: string; precision?: number } }
  }
132}
133 
134/** The slice of the Wikidata `wbgetentities` response the people-filter reads:
*  per entity, its claims (P31 / P570) and its enwiki sitelink title (which maps
*  the entity back to the search result it came from). */
137interface WikidataPeopleEntities {
  entities?: Record<
      string,
      {
          claims?: Record<string, WikidataClaimSlice[]>
          sitelinks?: { enwiki?: { title?: string } }
      }
  >
145}
146 
147/** Match a REST search title to a Wikidata sitelink title. Both are canonical
*  enwiki titles, so this is belt-and-suspenders against underscore/space/case
*  drift rather than a real transform. */
150function titleKey(title: string): string {
  return title.replace(/_/g, ' ').trim().toLowerCase()
152}
153 
154/** One attempt at the batched Wikidata lookup; null means it didn't answer (so the
*  caller can fall open rather than blank a working search). `props=claims` carries
*  P31 / P570; `sitelinks` + `sitefilter=enwiki` carry just the enwiki title back
*  per entity so each result can be matched to its claims. (No `normalize=1`:
*  it's only valid for a single title, and this call batches several.) */
159async function fetchInstanceClaims(titles: string[]): Promise<WikidataPeopleEntities['entities'] | null> {
  try {
      const url =
          'https://www.wikidata.org/w/api.php?action=wbgetentities' +
          `&sites=enwiki&titles=${encodeURIComponent(titles.join('|'))}` +
          '&props=claims%7Csitelinks&sitefilter=enwiki&languages=en&format=json&origin=*'
      const res = await fetch(url, { headers: WIKI_HEADERS, signal: AbortSignal.timeout(4000) })
      if (!res.ok) return null
      const data = (await res.json()) as WikidataPeopleEntities
      return data.entities ?? null
  } catch {
      return null
  }
172}
173 
174/** Does this entity look like a PERSON? Instance-of human (P31 → Q5), OR it has a
*  recorded death (P570) — the clause that catches biblical / legendary / mythological
*  figures that aren't tagged Q5 (Moses, King Arthur) but are contactable. (Death,
*  not birth: a birth-only non-human is a fictional character the gate would refuse;
*  see the file header.) `pickClaimYear` (reused from grounding) applies the same
*  usable-claim rule as the dossier: non-deprecated, real `value` snak, has a time. */
180function isPerson(claims?: Record<string, WikidataClaimSlice[]>): boolean {
  const human = (claims?.[WIKIDATA_INSTANCE_OF] ?? []).some(
      c => c.mainsnak?.snaktype === 'value' && c.mainsnak.datavalue?.value?.id === WIKIDATA_HUMAN
  )
  return human || !!pickClaimYear(claims?.[WIKIDATA_DEATH])
185}
⋯ 70 lines hidden (lines 186–255)186 
187/** Which of these article titles are PEOPLE? Returns a set of the person titles
*  (normalized via `titleKey`), or null when Wikidata never answered — same
*  one-spaced-retry-on-a-blip pattern the grounding lookup uses, then fall open.
*  A title with no Wikidata item (or one that isn't a person) isn't in the set. */
191async function fetchPeopleTitles(titles: string[]): Promise<Set<string> | null> {
  let entities = await fetchInstanceClaims(titles)
  if (!entities) {
      await new Promise(resolve => setTimeout(resolve, SEARCH_RETRY_DELAY_MS))
      entities = await fetchInstanceClaims(titles)
  }
  if (!entities) return null
198 
  const people = new Set<string>()
  for (const entity of Object.values(entities)) {
      const title = entity.sitelinks?.enwiki?.title
      if (title && isPerson(entity.claims)) people.add(titleKey(title))
  }
  return people
205}
206 
207export async function searchFigures(rawQuery: string): Promise<FigureSearchOutcome> {
  const query = (rawQuery || '').trim()
  if (query.length < 2) return { results: [], transient: false }
210 
  const key = query.toLowerCase()
  const hit = cache.get(key)
  if (hit && hit.expires > Date.now()) return { results: hit.value, transient: false }
214 
  let results = await fetchSearch(query)
  if (results === null) {
      // Rate-limit blips are the dominant real-world failure here (shared egress
      // IP, plus the suggester's own wiki bursts) — one spaced retry rescues most.
      await new Promise(resolve => setTimeout(resolve, SEARCH_RETRY_DELAY_MS))
      results = await fetchSearch(query)
  }
  if (results === null) return { results: [], transient: true }
223 
  // People filter (#74): keep only results Wikidata calls a person (P31 → Q5, or a
  // recorded death). An empty search answer skips the lookup (nothing to classify,
  // no wasted call).
  let filterTransient = false
  if (results.length > 0) {
      const people = await fetchPeopleTitles(results.map(r => r.name))
      if (people === null) {
          // The search SUCCEEDED; only the filter hit weather. Fall open with the
          // unfiltered matches rather than blanking a working autocomplete (#58) —
          // the description line still tells the pharaoh from the film, and the
          // contact gate (#72/#73) is the authoritative backstop. Don't cache it.
          filterTransient = true
      } else {
          results = results.filter(r => people.has(titleKey(r.name)))
      }
  }
240 
  // Cache DEFINITIVE ANSWERS only — search AND filter both settled. A timeout or a
  // Wikipedia 5xx (search) must not blank this query for an hour, and neither must a
  // filter blip we fell open on (the next keystroke should get a clean filter). A
  // legitimate "no people match" empty result IS an answer and caches normally.
  if (!filterTransient) {
      if (cache.size >= CACHE_MAX_ENTRIES) {
          // Drop the oldest entry (Map preserves insertion order) — a tiny, boring
          // eviction that keeps the cap honest without LRU machinery.
          const oldest = cache.keys().next().value
          if (oldest !== undefined) cache.delete(oldest)
      }
      cache.set(key, { value: results, expires: Date.now() + CACHE_TTL_MS })
  }
  return { results, transient: false }
255}

🧭Why it's built this way

Why not just Q5? Because a whole class of contactable figures isn't tagged human. Moses, Abraham and King Arthur are typed "biblical figure" / "legendary figure" on Wikidata, never Q5 — yet they resolve with a death date and the gate accepts them (the lifespan estimator #31 is built to date the likes of "King Arthur, Laozi"). A bare-Q5 filter would hide them from the very game that supports them.

Why death (P570), not birth (P569)? A non-human entity that has only an in-universe birthday is a fictional character — Sherlock Holmes, Harry Potter — whom the gate refuses as "not deceased". Previewing them would just reproduce the confusing pick. And a birth date can never widen coverage: the gate dates the death-less only via the #31 AI bridge, which runs only when there's no birth year. Real living people still appear (on the Q5 clause), and the dossier explains the deceased-only floor.

The full rationale — including that this is an honest near-superset of the gate, not a perfect one (a figure datable only by the AI bridge and not tagged Q5 won't be suggested) — lives in the module header, so the next reader inherits the reasoning rather than re-deriving it.

The module header: the people-filter rationale and its honest relationship to the gate.

server/utils/figure-search.ts · 255 lines

server/utils/figure-search.ts255 lines · TypeScript

1/**
* Name autocomplete for the contact step — Wikipedia REST title search (the
* prefix-matching endpoint built for exactly this), key-less like the grounding
* pipeline. Never throws.
*
* People only (issue #74): raw title search surfaces films, places and concepts
* ("Cleopatra (1963 film)", "Death of Cleopatra", "Mongolia"), and the contact
* gate (#72/#73) only reaches a real, deceased person — so previewing non-people
* invites a pick that's refused at send. After the search returns, ONE batched
* Wikidata call (`wbgetentities` over the result titles) keeps a result when it
* looks like a *person*: instance-of human (P31 → Q5), OR it carries a recorded
* death (P570).
*
* Why the death clause, and why death and not birth:
*  - It catches genuinely contactable figures that are NOT tagged Q5 — Moses,
*    Abraham, King Arthur and other biblical / legendary / mythological figures are
*    typed "biblical figure" or "legendary figure", yet the gate accepts them (they
*    resolve and carry a death date) and the lifespan estimator (#31) is explicitly
*    built to date the likes of "King Arthur, Laozi". A bare Q5 test wrongly hides
*    them; keying on death keeps them.
*  - Death, not birth, because the gate's floor IS death (#72). A non-human entity
*    that has only a birth date is exactly a fictional character with an in-universe
*    birthday — Sherlock Holmes, Harry Potter — whom the gate refuses as "not
*    deceased". Previewing them would just reproduce the confusing pick. (A birth
*    clause would also never WIDEN coverage: the gate dates the death-less only via
*    the #31 bridge, which fires only when there's NO birth year — so a birth date
*    can't be the thing that makes a figure contactable.) Real LIVING people still
*    appear, on the Q5 clause, and the dossier explains the deceased-only floor.
*
* Relationship to the gate (stated honestly — it is NOT a perfect superset): the
⋯ 225 lines hidden (lines 31–255)31 * gate is `resolved && died`, with no instance-of check, where `died` can come from
* Wikidata P570 OR the #31 AI bridge. This filter keeps everyone the gate can reach
* via a Wikidata death (they have P570), plus all humans (living ones included —
* looser than the gate there, by design). The one under-coverage gap is a figure
* datable ONLY by the AI bridge AND not tagged Q5 — a rare, obscure case; it won't
* be suggested, but the gate stays the authoritative backstop and the full name can
* still be typed and sent. A famous dated animal (a racehorse, say) is kept — it's
* datable and the gate would accept it too. We do NOT filter on the death's *value*
* (the safely-historical floor): that's the gate's job, and judging it here would
* need the per-result grounding the batched call is built to avoid.
*
* Failure honesty (the lesson of issue #31, applied to search): a rate-limit 429
* or timeout is weather, not "no matches". A failed search gets one spaced retry,
* and if that fails too the outcome says `transient: true` instead of posing as
* an empty result — so the combobox can quietly try again rather than silently
* never opening. The people-filter is layered ON TOP without regressing this
* (#58): a search that succeeded but whose filter lookup blips falls OPEN — the
* unfiltered matches are shown (the description line still tells the pharaoh from
* the film) rather than blanking a working autocomplete. Only definitive answers
* — search AND filter both settled — are cached.
*/
52import { pickClaimYear, safeUrl, WIKI_HEADERS } from './figure-grounding'
53 
54export interface FigureSearchResult {
  name: string
  description?: string
  thumbnail?: string
58}
59 
60export interface FigureSearchOutcome {
  results: FigureSearchResult[]
  /** True when Wikipedia couldn't be reached (429, timeout, network): the empty
   *  answer is weather, and the caller may retry soon. Never set on a real
   *  "no matches" answer. */
  transient: boolean
66}
67 
68/** Cache: queries repeat heavily while typing (and across players). Capped so a
*  crawl of unique queries can't grow it without bound (the grounding cache's
*  known gap — not repeated here). */
71const CACHE_TTL_MS = 60 * 60 * 1000
72const CACHE_MAX_ENTRIES = 500
73const cache = new Map<string, { value: FigureSearchResult[]; expires: number }>()
74 
75const RESULT_LIMIT = 5
76const SEARCH_RETRY_DELAY_MS = 250
77 
78/** Wikidata properties the people test reads: "instance of" (P31, → human Q5) and
*  date of death (P570). Q5 OR a recorded death marks a person (see the file header
*  for why death — not birth — is the second clause). */
81const WIKIDATA_INSTANCE_OF = 'P31'
82const WIKIDATA_HUMAN = 'Q5'
83const WIKIDATA_DEATH = 'P570'
84 
85/** The slice of the Wikipedia REST v1 title-search response we read. */
86interface WikiTitleSearch {
  pages?: Array<{
      title?: string
      description?: string
      thumbnail?: { url?: string } | null
  }>
92}
93 
94/** Wikipedia thumbnails arrive protocol-relative ("//upload.wikimedia.org/…"). */
95function absoluteThumb(url?: string): string | undefined {
  if (!url) return undefined
  return safeUrl(url.startsWith('//') ? `https:${url}` : url)
98}
99 
100/** One attempt against the title-search endpoint; null means it didn't answer. */
101async function fetchSearch(query: string): Promise<FigureSearchResult[] | null> {
  try {
      const res = await fetch(
          `https://en.wikipedia.org/w/rest.php/v1/search/title?q=${encodeURIComponent(query)}&limit=${RESULT_LIMIT}`,
          { headers: WIKI_HEADERS, signal: AbortSignal.timeout(4000) }
      )
      if (!res.ok) return null
      const data = (await res.json()) as WikiTitleSearch
      return (data.pages ?? [])
          .filter(p => typeof p.title === 'string' && p.title.length > 0)
          .slice(0, RESULT_LIMIT)
          .map(p => ({
              name: p.title as string,
              description: p.description || undefined,
              thumbnail: absoluteThumb(p.thumbnail?.url)
          }))
  } catch {
      return null
  }
120}
121 
122/** The slice of one Wikidata claim the people-filter reads: an item value (P31's
*  `id`) or a time value (P570's `time`). Shaped to be accepted by the grounding
*  module's `pickClaimYear` (which reads `rank` / `snaktype` / `time`), so the
*  "usable date" rule stays single-sourced. */
126interface WikidataClaimSlice {
  rank?: string
  mainsnak?: {
      snaktype?: string
      datavalue?: { value?: { id?: string; time?: string; precision?: number } }
  }
132}
133 
134/** The slice of the Wikidata `wbgetentities` response the people-filter reads:
*  per entity, its claims (P31 / P570) and its enwiki sitelink title (which maps
*  the entity back to the search result it came from). */
137interface WikidataPeopleEntities {
  entities?: Record<
      string,
      {
          claims?: Record<string, WikidataClaimSlice[]>
          sitelinks?: { enwiki?: { title?: string } }
      }
  >
145}
146 
147/** Match a REST search title to a Wikidata sitelink title. Both are canonical
*  enwiki titles, so this is belt-and-suspenders against underscore/space/case
*  drift rather than a real transform. */
150function titleKey(title: string): string {
  return title.replace(/_/g, ' ').trim().toLowerCase()
152}
153 
154/** One attempt at the batched Wikidata lookup; null means it didn't answer (so the
*  caller can fall open rather than blank a working search). `props=claims` carries
*  P31 / P570; `sitelinks` + `sitefilter=enwiki` carry just the enwiki title back
*  per entity so each result can be matched to its claims. (No `normalize=1`:
*  it's only valid for a single title, and this call batches several.) */
159async function fetchInstanceClaims(titles: string[]): Promise<WikidataPeopleEntities['entities'] | null> {
  try {
      const url =
          'https://www.wikidata.org/w/api.php?action=wbgetentities' +
          `&sites=enwiki&titles=${encodeURIComponent(titles.join('|'))}` +
          '&props=claims%7Csitelinks&sitefilter=enwiki&languages=en&format=json&origin=*'
      const res = await fetch(url, { headers: WIKI_HEADERS, signal: AbortSignal.timeout(4000) })
      if (!res.ok) return null
      const data = (await res.json()) as WikidataPeopleEntities
      return data.entities ?? null
  } catch {
      return null
  }
172}
173 
174/** Does this entity look like a PERSON? Instance-of human (P31 → Q5), OR it has a
*  recorded death (P570) — the clause that catches biblical / legendary / mythological
*  figures that aren't tagged Q5 (Moses, King Arthur) but are contactable. (Death,
*  not birth: a birth-only non-human is a fictional character the gate would refuse;
*  see the file header.) `pickClaimYear` (reused from grounding) applies the same
*  usable-claim rule as the dossier: non-deprecated, real `value` snak, has a time. */
180function isPerson(claims?: Record<string, WikidataClaimSlice[]>): boolean {
  const human = (claims?.[WIKIDATA_INSTANCE_OF] ?? []).some(
      c => c.mainsnak?.snaktype === 'value' && c.mainsnak.datavalue?.value?.id === WIKIDATA_HUMAN
  )
  return human || !!pickClaimYear(claims?.[WIKIDATA_DEATH])
185}
186 
187/** Which of these article titles are PEOPLE? Returns a set of the person titles
*  (normalized via `titleKey`), or null when Wikidata never answered — same
*  one-spaced-retry-on-a-blip pattern the grounding lookup uses, then fall open.
*  A title with no Wikidata item (or one that isn't a person) isn't in the set. */
191async function fetchPeopleTitles(titles: string[]): Promise<Set<string> | null> {
  let entities = await fetchInstanceClaims(titles)
  if (!entities) {
      await new Promise(resolve => setTimeout(resolve, SEARCH_RETRY_DELAY_MS))
      entities = await fetchInstanceClaims(titles)
  }
  if (!entities) return null
198 
  const people = new Set<string>()
  for (const entity of Object.values(entities)) {
      const title = entity.sitelinks?.enwiki?.title
      if (title && isPerson(entity.claims)) people.add(titleKey(title))
  }
  return people
205}
206 
207export async function searchFigures(rawQuery: string): Promise<FigureSearchOutcome> {
  const query = (rawQuery || '').trim()
  if (query.length < 2) return { results: [], transient: false }
210 
  const key = query.toLowerCase()
  const hit = cache.get(key)
  if (hit && hit.expires > Date.now()) return { results: hit.value, transient: false }
214 
  let results = await fetchSearch(query)
  if (results === null) {
      // Rate-limit blips are the dominant real-world failure here (shared egress
      // IP, plus the suggester's own wiki bursts) — one spaced retry rescues most.
      await new Promise(resolve => setTimeout(resolve, SEARCH_RETRY_DELAY_MS))
      results = await fetchSearch(query)
  }
  if (results === null) return { results: [], transient: true }
223 
  // People filter (#74): keep only results Wikidata calls a person (P31 → Q5, or a
  // recorded death). An empty search answer skips the lookup (nothing to classify,
  // no wasted call).
  let filterTransient = false
  if (results.length > 0) {
      const people = await fetchPeopleTitles(results.map(r => r.name))
      if (people === null) {
          // The search SUCCEEDED; only the filter hit weather. Fall open with the
          // unfiltered matches rather than blanking a working autocomplete (#58) —
          // the description line still tells the pharaoh from the film, and the
          // contact gate (#72/#73) is the authoritative backstop. Don't cache it.
          filterTransient = true
      } else {
          results = results.filter(r => people.has(titleKey(r.name)))
      }
  }
240 
  // Cache DEFINITIVE ANSWERS only — search AND filter both settled. A timeout or a
  // Wikipedia 5xx (search) must not blank this query for an hour, and neither must a
  // filter blip we fell open on (the next keystroke should get a clean filter). A
  // legitimate "no people match" empty result IS an answer and caches normally.
  if (!filterTransient) {
      if (cache.size >= CACHE_MAX_ENTRIES) {
          // Drop the oldest entry (Map preserves insertion order) — a tiny, boring
          // eviction that keeps the cap honest without LRU machinery.
          const oldest = cache.keys().next().value
          if (oldest !== undefined) cache.delete(oldest)
      }
      cache.set(key, { value: results, expires: Date.now() + CACHE_TTL_MS })
  }
  return { results, transient: false }
255}

One batched Wikidata call, matched back by title

fetchInstanceClaims is one attempt at the batched lookup. It asks wbgetentities for the result titles in a single request: props=claims carries P31/P570, and sitelinks + sitefilter=enwiki carry each entity's English title back so it can be matched to the search result it came from. normalize=1 is omitted on purpose — it's only valid for a single title, and this call batches several. A non-OK response or a throw returns null (the signal to fall open), never an exception.

The batched request. WIKI_HEADERS is the shared, policy-compliant User-Agent (#58).

server/utils/figure-search.ts · 255 lines

server/utils/figure-search.ts255 lines · TypeScript

⋯ 158 lines hidden (lines 1–158)1/**
* Name autocomplete for the contact step — Wikipedia REST title search (the
* prefix-matching endpoint built for exactly this), key-less like the grounding
* pipeline. Never throws.
*
* People only (issue #74): raw title search surfaces films, places and concepts
* ("Cleopatra (1963 film)", "Death of Cleopatra", "Mongolia"), and the contact
* gate (#72/#73) only reaches a real, deceased person — so previewing non-people
* invites a pick that's refused at send. After the search returns, ONE batched
* Wikidata call (`wbgetentities` over the result titles) keeps a result when it
* looks like a *person*: instance-of human (P31 → Q5), OR it carries a recorded
* death (P570).
*
* Why the death clause, and why death and not birth:
*  - It catches genuinely contactable figures that are NOT tagged Q5 — Moses,
*    Abraham, King Arthur and other biblical / legendary / mythological figures are
*    typed "biblical figure" or "legendary figure", yet the gate accepts them (they
*    resolve and carry a death date) and the lifespan estimator (#31) is explicitly
*    built to date the likes of "King Arthur, Laozi". A bare Q5 test wrongly hides
*    them; keying on death keeps them.
*  - Death, not birth, because the gate's floor IS death (#72). A non-human entity
*    that has only a birth date is exactly a fictional character with an in-universe
*    birthday — Sherlock Holmes, Harry Potter — whom the gate refuses as "not
*    deceased". Previewing them would just reproduce the confusing pick. (A birth
*    clause would also never WIDEN coverage: the gate dates the death-less only via
*    the #31 bridge, which fires only when there's NO birth year — so a birth date
*    can't be the thing that makes a figure contactable.) Real LIVING people still
*    appear, on the Q5 clause, and the dossier explains the deceased-only floor.
*
* Relationship to the gate (stated honestly — it is NOT a perfect superset): the
* gate is `resolved && died`, with no instance-of check, where `died` can come from
* Wikidata P570 OR the #31 AI bridge. This filter keeps everyone the gate can reach
* via a Wikidata death (they have P570), plus all humans (living ones included —
* looser than the gate there, by design). The one under-coverage gap is a figure
* datable ONLY by the AI bridge AND not tagged Q5 — a rare, obscure case; it won't
* be suggested, but the gate stays the authoritative backstop and the full name can
* still be typed and sent. A famous dated animal (a racehorse, say) is kept — it's
* datable and the gate would accept it too. We do NOT filter on the death's *value*
* (the safely-historical floor): that's the gate's job, and judging it here would
* need the per-result grounding the batched call is built to avoid.
*
* Failure honesty (the lesson of issue #31, applied to search): a rate-limit 429
* or timeout is weather, not "no matches". A failed search gets one spaced retry,
* and if that fails too the outcome says `transient: true` instead of posing as
* an empty result — so the combobox can quietly try again rather than silently
* never opening. The people-filter is layered ON TOP without regressing this
* (#58): a search that succeeded but whose filter lookup blips falls OPEN — the
* unfiltered matches are shown (the description line still tells the pharaoh from
* the film) rather than blanking a working autocomplete. Only definitive answers
* — search AND filter both settled — are cached.
*/
52import { pickClaimYear, safeUrl, WIKI_HEADERS } from './figure-grounding'
53 
54export interface FigureSearchResult {
  name: string
  description?: string
  thumbnail?: string
58}
59 
60export interface FigureSearchOutcome {
  results: FigureSearchResult[]
  /** True when Wikipedia couldn't be reached (429, timeout, network): the empty
   *  answer is weather, and the caller may retry soon. Never set on a real
   *  "no matches" answer. */
  transient: boolean
66}
67 
68/** Cache: queries repeat heavily while typing (and across players). Capped so a
*  crawl of unique queries can't grow it without bound (the grounding cache's
*  known gap — not repeated here). */
71const CACHE_TTL_MS = 60 * 60 * 1000
72const CACHE_MAX_ENTRIES = 500
73const cache = new Map<string, { value: FigureSearchResult[]; expires: number }>()
74 
75const RESULT_LIMIT = 5
76const SEARCH_RETRY_DELAY_MS = 250
77 
78/** Wikidata properties the people test reads: "instance of" (P31, → human Q5) and
*  date of death (P570). Q5 OR a recorded death marks a person (see the file header
*  for why death — not birth — is the second clause). */
81const WIKIDATA_INSTANCE_OF = 'P31'
82const WIKIDATA_HUMAN = 'Q5'
83const WIKIDATA_DEATH = 'P570'
84 
85/** The slice of the Wikipedia REST v1 title-search response we read. */
86interface WikiTitleSearch {
  pages?: Array<{
      title?: string
      description?: string
      thumbnail?: { url?: string } | null
  }>
92}
93 
94/** Wikipedia thumbnails arrive protocol-relative ("//upload.wikimedia.org/…"). */
95function absoluteThumb(url?: string): string | undefined {
  if (!url) return undefined
  return safeUrl(url.startsWith('//') ? `https:${url}` : url)
98}
99 
100/** One attempt against the title-search endpoint; null means it didn't answer. */
101async function fetchSearch(query: string): Promise<FigureSearchResult[] | null> {
  try {
      const res = await fetch(
          `https://en.wikipedia.org/w/rest.php/v1/search/title?q=${encodeURIComponent(query)}&limit=${RESULT_LIMIT}`,
          { headers: WIKI_HEADERS, signal: AbortSignal.timeout(4000) }
      )
      if (!res.ok) return null
      const data = (await res.json()) as WikiTitleSearch
      return (data.pages ?? [])
          .filter(p => typeof p.title === 'string' && p.title.length > 0)
          .slice(0, RESULT_LIMIT)
          .map(p => ({
              name: p.title as string,
              description: p.description || undefined,
              thumbnail: absoluteThumb(p.thumbnail?.url)
          }))
  } catch {
      return null
  }
120}
121 
122/** The slice of one Wikidata claim the people-filter reads: an item value (P31's
*  `id`) or a time value (P570's `time`). Shaped to be accepted by the grounding
*  module's `pickClaimYear` (which reads `rank` / `snaktype` / `time`), so the
*  "usable date" rule stays single-sourced. */
126interface WikidataClaimSlice {
  rank?: string
  mainsnak?: {
      snaktype?: string
      datavalue?: { value?: { id?: string; time?: string; precision?: number } }
  }
132}
133 
134/** The slice of the Wikidata `wbgetentities` response the people-filter reads:
*  per entity, its claims (P31 / P570) and its enwiki sitelink title (which maps
*  the entity back to the search result it came from). */
137interface WikidataPeopleEntities {
  entities?: Record<
      string,
      {
          claims?: Record<string, WikidataClaimSlice[]>
          sitelinks?: { enwiki?: { title?: string } }
      }
  >
145}
146 
147/** Match a REST search title to a Wikidata sitelink title. Both are canonical
*  enwiki titles, so this is belt-and-suspenders against underscore/space/case
*  drift rather than a real transform. */
150function titleKey(title: string): string {
  return title.replace(/_/g, ' ').trim().toLowerCase()
152}
153 
154/** One attempt at the batched Wikidata lookup; null means it didn't answer (so the
*  caller can fall open rather than blank a working search). `props=claims` carries
*  P31 / P570; `sitelinks` + `sitefilter=enwiki` carry just the enwiki title back
*  per entity so each result can be matched to its claims. (No `normalize=1`:
*  it's only valid for a single title, and this call batches several.) */
159async function fetchInstanceClaims(titles: string[]): Promise<WikidataPeopleEntities['entities'] | null> {
  try {
      const url =
          'https://www.wikidata.org/w/api.php?action=wbgetentities' +
          `&sites=enwiki&titles=${encodeURIComponent(titles.join('|'))}` +
          '&props=claims%7Csitelinks&sitefilter=enwiki&languages=en&format=json&origin=*'
      const res = await fetch(url, { headers: WIKI_HEADERS, signal: AbortSignal.timeout(4000) })
      if (!res.ok) return null
      const data = (await res.json()) as WikidataPeopleEntities
      return data.entities ?? null
  } catch {
      return null
  }
172}
173 
174/** Does this entity look like a PERSON? Instance-of human (P31 → Q5), OR it has a
*  recorded death (P570) — the clause that catches biblical / legendary / mythological
*  figures that aren't tagged Q5 (Moses, King Arthur) but are contactable. (Death,
*  not birth: a birth-only non-human is a fictional character the gate would refuse;
⋯ 78 lines hidden (lines 178–255)178 *  see the file header.) `pickClaimYear` (reused from grounding) applies the same
*  usable-claim rule as the dossier: non-deprecated, real `value` snak, has a time. */
180function isPerson(claims?: Record<string, WikidataClaimSlice[]>): boolean {
  const human = (claims?.[WIKIDATA_INSTANCE_OF] ?? []).some(
      c => c.mainsnak?.snaktype === 'value' && c.mainsnak.datavalue?.value?.id === WIKIDATA_HUMAN
  )
  return human || !!pickClaimYear(claims?.[WIKIDATA_DEATH])
185}
186 
187/** Which of these article titles are PEOPLE? Returns a set of the person titles
*  (normalized via `titleKey`), or null when Wikidata never answered — same
*  one-spaced-retry-on-a-blip pattern the grounding lookup uses, then fall open.
*  A title with no Wikidata item (or one that isn't a person) isn't in the set. */
191async function fetchPeopleTitles(titles: string[]): Promise<Set<string> | null> {
  let entities = await fetchInstanceClaims(titles)
  if (!entities) {
      await new Promise(resolve => setTimeout(resolve, SEARCH_RETRY_DELAY_MS))
      entities = await fetchInstanceClaims(titles)
  }
  if (!entities) return null
198 
  const people = new Set<string>()
  for (const entity of Object.values(entities)) {
      const title = entity.sitelinks?.enwiki?.title
      if (title && isPerson(entity.claims)) people.add(titleKey(title))
  }
  return people
205}
206 
207export async function searchFigures(rawQuery: string): Promise<FigureSearchOutcome> {
  const query = (rawQuery || '').trim()
  if (query.length < 2) return { results: [], transient: false }
210 
  const key = query.toLowerCase()
  const hit = cache.get(key)
  if (hit && hit.expires > Date.now()) return { results: hit.value, transient: false }
214 
  let results = await fetchSearch(query)
  if (results === null) {
      // Rate-limit blips are the dominant real-world failure here (shared egress
      // IP, plus the suggester's own wiki bursts) — one spaced retry rescues most.
      await new Promise(resolve => setTimeout(resolve, SEARCH_RETRY_DELAY_MS))
      results = await fetchSearch(query)
  }
  if (results === null) return { results: [], transient: true }
223 
  // People filter (#74): keep only results Wikidata calls a person (P31 → Q5, or a
  // recorded death). An empty search answer skips the lookup (nothing to classify,
  // no wasted call).
  let filterTransient = false
  if (results.length > 0) {
      const people = await fetchPeopleTitles(results.map(r => r.name))
      if (people === null) {
          // The search SUCCEEDED; only the filter hit weather. Fall open with the
          // unfiltered matches rather than blanking a working autocomplete (#58) —
          // the description line still tells the pharaoh from the film, and the
          // contact gate (#72/#73) is the authoritative backstop. Don't cache it.
          filterTransient = true
      } else {
          results = results.filter(r => people.has(titleKey(r.name)))
      }
  }
240 
  // Cache DEFINITIVE ANSWERS only — search AND filter both settled. A timeout or a
  // Wikipedia 5xx (search) must not blank this query for an hour, and neither must a
  // filter blip we fell open on (the next keystroke should get a clean filter). A
  // legitimate "no people match" empty result IS an answer and caches normally.
  if (!filterTransient) {
      if (cache.size >= CACHE_MAX_ENTRIES) {
          // Drop the oldest entry (Map preserves insertion order) — a tiny, boring
          // eviction that keeps the cap honest without LRU machinery.
          const oldest = cache.keys().next().value
          if (oldest !== undefined) cache.delete(oldest)
      }
      cache.set(key, { value: results, expires: Date.now() + CACHE_TTL_MS })
  }
  return { results, transient: false }
255}

fetchPeopleTitles wraps that attempt in the same one-spaced-retry-then-give-up pattern the grounding lookup uses, then builds the set of person titles. Each entity is matched to its result through titleKey, which normalizes underscores, spacing and case on both sides — both are canonical enwiki titles, so this is belt-and-suspenders against cosmetic drift. A title with no Wikidata item, or one that isn't a person, simply never enters the set and is dropped.

titleKey (the matching key) and fetchPeopleTitles (retry, then collect person titles).

server/utils/figure-search.ts · 255 lines

server/utils/figure-search.ts255 lines · TypeScript

⋯ 145 lines hidden (lines 1–145)1/**
* Name autocomplete for the contact step — Wikipedia REST title search (the
* prefix-matching endpoint built for exactly this), key-less like the grounding
* pipeline. Never throws.
*
* People only (issue #74): raw title search surfaces films, places and concepts
* ("Cleopatra (1963 film)", "Death of Cleopatra", "Mongolia"), and the contact
* gate (#72/#73) only reaches a real, deceased person — so previewing non-people
* invites a pick that's refused at send. After the search returns, ONE batched
* Wikidata call (`wbgetentities` over the result titles) keeps a result when it
* looks like a *person*: instance-of human (P31 → Q5), OR it carries a recorded
* death (P570).
*
* Why the death clause, and why death and not birth:
*  - It catches genuinely contactable figures that are NOT tagged Q5 — Moses,
*    Abraham, King Arthur and other biblical / legendary / mythological figures are
*    typed "biblical figure" or "legendary figure", yet the gate accepts them (they
*    resolve and carry a death date) and the lifespan estimator (#31) is explicitly
*    built to date the likes of "King Arthur, Laozi". A bare Q5 test wrongly hides
*    them; keying on death keeps them.
*  - Death, not birth, because the gate's floor IS death (#72). A non-human entity
*    that has only a birth date is exactly a fictional character with an in-universe
*    birthday — Sherlock Holmes, Harry Potter — whom the gate refuses as "not
*    deceased". Previewing them would just reproduce the confusing pick. (A birth
*    clause would also never WIDEN coverage: the gate dates the death-less only via
*    the #31 bridge, which fires only when there's NO birth year — so a birth date
*    can't be the thing that makes a figure contactable.) Real LIVING people still
*    appear, on the Q5 clause, and the dossier explains the deceased-only floor.
*
* Relationship to the gate (stated honestly — it is NOT a perfect superset): the
* gate is `resolved && died`, with no instance-of check, where `died` can come from
* Wikidata P570 OR the #31 AI bridge. This filter keeps everyone the gate can reach
* via a Wikidata death (they have P570), plus all humans (living ones included —
* looser than the gate there, by design). The one under-coverage gap is a figure
* datable ONLY by the AI bridge AND not tagged Q5 — a rare, obscure case; it won't
* be suggested, but the gate stays the authoritative backstop and the full name can
* still be typed and sent. A famous dated animal (a racehorse, say) is kept — it's
* datable and the gate would accept it too. We do NOT filter on the death's *value*
* (the safely-historical floor): that's the gate's job, and judging it here would
* need the per-result grounding the batched call is built to avoid.
*
* Failure honesty (the lesson of issue #31, applied to search): a rate-limit 429
* or timeout is weather, not "no matches". A failed search gets one spaced retry,
* and if that fails too the outcome says `transient: true` instead of posing as
* an empty result — so the combobox can quietly try again rather than silently
* never opening. The people-filter is layered ON TOP without regressing this
* (#58): a search that succeeded but whose filter lookup blips falls OPEN — the
* unfiltered matches are shown (the description line still tells the pharaoh from
* the film) rather than blanking a working autocomplete. Only definitive answers
* — search AND filter both settled — are cached.
*/
52import { pickClaimYear, safeUrl, WIKI_HEADERS } from './figure-grounding'
53 
54export interface FigureSearchResult {
  name: string
  description?: string
  thumbnail?: string
58}
59 
60export interface FigureSearchOutcome {
  results: FigureSearchResult[]
  /** True when Wikipedia couldn't be reached (429, timeout, network): the empty
   *  answer is weather, and the caller may retry soon. Never set on a real
   *  "no matches" answer. */
  transient: boolean
66}
67 
68/** Cache: queries repeat heavily while typing (and across players). Capped so a
*  crawl of unique queries can't grow it without bound (the grounding cache's
*  known gap — not repeated here). */
71const CACHE_TTL_MS = 60 * 60 * 1000
72const CACHE_MAX_ENTRIES = 500
73const cache = new Map<string, { value: FigureSearchResult[]; expires: number }>()
74 
75const RESULT_LIMIT = 5
76const SEARCH_RETRY_DELAY_MS = 250
77 
78/** Wikidata properties the people test reads: "instance of" (P31, → human Q5) and
*  date of death (P570). Q5 OR a recorded death marks a person (see the file header
*  for why death — not birth — is the second clause). */
81const WIKIDATA_INSTANCE_OF = 'P31'
82const WIKIDATA_HUMAN = 'Q5'
83const WIKIDATA_DEATH = 'P570'
84 
85/** The slice of the Wikipedia REST v1 title-search response we read. */
86interface WikiTitleSearch {
  pages?: Array<{
      title?: string
      description?: string
      thumbnail?: { url?: string } | null
  }>
92}
93 
94/** Wikipedia thumbnails arrive protocol-relative ("//upload.wikimedia.org/…"). */
95function absoluteThumb(url?: string): string | undefined {
  if (!url) return undefined
  return safeUrl(url.startsWith('//') ? `https:${url}` : url)
98}
99 
100/** One attempt against the title-search endpoint; null means it didn't answer. */
101async function fetchSearch(query: string): Promise<FigureSearchResult[] | null> {
  try {
      const res = await fetch(
          `https://en.wikipedia.org/w/rest.php/v1/search/title?q=${encodeURIComponent(query)}&limit=${RESULT_LIMIT}`,
          { headers: WIKI_HEADERS, signal: AbortSignal.timeout(4000) }
      )
      if (!res.ok) return null
      const data = (await res.json()) as WikiTitleSearch
      return (data.pages ?? [])
          .filter(p => typeof p.title === 'string' && p.title.length > 0)
          .slice(0, RESULT_LIMIT)
          .map(p => ({
              name: p.title as string,
              description: p.description || undefined,
              thumbnail: absoluteThumb(p.thumbnail?.url)
          }))
  } catch {
      return null
  }
120}
121 
122/** The slice of one Wikidata claim the people-filter reads: an item value (P31's
*  `id`) or a time value (P570's `time`). Shaped to be accepted by the grounding
*  module's `pickClaimYear` (which reads `rank` / `snaktype` / `time`), so the
*  "usable date" rule stays single-sourced. */
126interface WikidataClaimSlice {
  rank?: string
  mainsnak?: {
      snaktype?: string
      datavalue?: { value?: { id?: string; time?: string; precision?: number } }
  }
132}
133 
134/** The slice of the Wikidata `wbgetentities` response the people-filter reads:
*  per entity, its claims (P31 / P570) and its enwiki sitelink title (which maps
*  the entity back to the search result it came from). */
137interface WikidataPeopleEntities {
  entities?: Record<
      string,
      {
          claims?: Record<string, WikidataClaimSlice[]>
          sitelinks?: { enwiki?: { title?: string } }
      }
  >
145}
146 
147/** Match a REST search title to a Wikidata sitelink title. Both are canonical
*  enwiki titles, so this is belt-and-suspenders against underscore/space/case
*  drift rather than a real transform. */
150function titleKey(title: string): string {
  return title.replace(/_/g, ' ').trim().toLowerCase()
152}
⋯ 38 lines hidden (lines 153–190)153 
154/** One attempt at the batched Wikidata lookup; null means it didn't answer (so the
*  caller can fall open rather than blank a working search). `props=claims` carries
*  P31 / P570; `sitelinks` + `sitefilter=enwiki` carry just the enwiki title back
*  per entity so each result can be matched to its claims. (No `normalize=1`:
*  it's only valid for a single title, and this call batches several.) */
159async function fetchInstanceClaims(titles: string[]): Promise<WikidataPeopleEntities['entities'] | null> {
  try {
      const url =
          'https://www.wikidata.org/w/api.php?action=wbgetentities' +
          `&sites=enwiki&titles=${encodeURIComponent(titles.join('|'))}` +
          '&props=claims%7Csitelinks&sitefilter=enwiki&languages=en&format=json&origin=*'
      const res = await fetch(url, { headers: WIKI_HEADERS, signal: AbortSignal.timeout(4000) })
      if (!res.ok) return null
      const data = (await res.json()) as WikidataPeopleEntities
      return data.entities ?? null
  } catch {
      return null
  }
172}
173 
174/** Does this entity look like a PERSON? Instance-of human (P31 → Q5), OR it has a
*  recorded death (P570) — the clause that catches biblical / legendary / mythological
*  figures that aren't tagged Q5 (Moses, King Arthur) but are contactable. (Death,
*  not birth: a birth-only non-human is a fictional character the gate would refuse;
*  see the file header.) `pickClaimYear` (reused from grounding) applies the same
*  usable-claim rule as the dossier: non-deprecated, real `value` snak, has a time. */
180function isPerson(claims?: Record<string, WikidataClaimSlice[]>): boolean {
  const human = (claims?.[WIKIDATA_INSTANCE_OF] ?? []).some(
      c => c.mainsnak?.snaktype === 'value' && c.mainsnak.datavalue?.value?.id === WIKIDATA_HUMAN
  )
  return human || !!pickClaimYear(claims?.[WIKIDATA_DEATH])
185}
186 
187/** Which of these article titles are PEOPLE? Returns a set of the person titles
*  (normalized via `titleKey`), or null when Wikidata never answered — same
*  one-spaced-retry-on-a-blip pattern the grounding lookup uses, then fall open.
*  A title with no Wikidata item (or one that isn't a person) isn't in the set. */
191async function fetchPeopleTitles(titles: string[]): Promise<Set<string> | null> {
  let entities = await fetchInstanceClaims(titles)
  if (!entities) {
      await new Promise(resolve => setTimeout(resolve, SEARCH_RETRY_DELAY_MS))
      entities = await fetchInstanceClaims(titles)
  }
  if (!entities) return null
198 
  const people = new Set<string>()
  for (const entity of Object.values(entities)) {
      const title = entity.sitelinks?.enwiki?.title
      if (title && isPerson(entity.claims)) people.add(titleKey(title))
  }
  return people
205}
⋯ 50 lines hidden (lines 206–255)206 
207export async function searchFigures(rawQuery: string): Promise<FigureSearchOutcome> {
  const query = (rawQuery || '').trim()
  if (query.length < 2) return { results: [], transient: false }
210 
  const key = query.toLowerCase()
  const hit = cache.get(key)
  if (hit && hit.expires > Date.now()) return { results: hit.value, transient: false }
214 
  let results = await fetchSearch(query)
  if (results === null) {
      // Rate-limit blips are the dominant real-world failure here (shared egress
      // IP, plus the suggester's own wiki bursts) — one spaced retry rescues most.
      await new Promise(resolve => setTimeout(resolve, SEARCH_RETRY_DELAY_MS))
      results = await fetchSearch(query)
  }
  if (results === null) return { results: [], transient: true }
223 
  // People filter (#74): keep only results Wikidata calls a person (P31 → Q5, or a
  // recorded death). An empty search answer skips the lookup (nothing to classify,
  // no wasted call).
  let filterTransient = false
  if (results.length > 0) {
      const people = await fetchPeopleTitles(results.map(r => r.name))
      if (people === null) {
          // The search SUCCEEDED; only the filter hit weather. Fall open with the
          // unfiltered matches rather than blanking a working autocomplete (#58) —
          // the description line still tells the pharaoh from the film, and the
          // contact gate (#72/#73) is the authoritative backstop. Don't cache it.
          filterTransient = true
      } else {
          results = results.filter(r => people.has(titleKey(r.name)))
      }
  }
240 
  // Cache DEFINITIVE ANSWERS only — search AND filter both settled. A timeout or a
  // Wikipedia 5xx (search) must not blank this query for an hour, and neither must a
  // filter blip we fell open on (the next keystroke should get a clean filter). A
  // legitimate "no people match" empty result IS an answer and caches normally.
  if (!filterTransient) {
      if (cache.size >= CACHE_MAX_ENTRIES) {
          // Drop the oldest entry (Map preserves insertion order) — a tiny, boring
          // eviction that keeps the cap honest without LRU machinery.
          const oldest = cache.keys().next().value
          if (oldest !== undefined) cache.delete(oldest)
      }
      cache.set(key, { value: results, expires: Date.now() + CACHE_TTL_MS })
  }
  return { results, transient: false }
255}

Tests: the cases that would catch a regression

The spec mocks fetch and routes by URL, so the search and the Wikidata filter are driven independently. Beyond the core "non-people are filtered" case, several tests exist specifically to pin behaviour a naive change would silently break.

The two predicate-defining cases: a non-Q5 figure with a death is KEPT; a fictional birth-only is DROPPED.

tests/unit/server/utils/figure-search.spec.ts · 449 lines

tests/unit/server/utils/figure-search.spec.ts449 lines · TypeScript

⋯ 161 lines hidden (lines 1–161)1import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest'
2import { searchFigures } from '../../../../server/utils/figure-search'
3 
4/** Wikipedia REST v1 title-search response, shaped like the real thing. */
5function searchResp(pages: unknown[]) {
  return { ok: true, json: async () => ({ pages }) } as Response
7}
8 
9/** Wikidata `wbgetentities` response, shaped like the real thing (entities keyed
*  by qid, each carrying its P31 "instance of" claims + its enwiki sitelink). */
11function wikidataResp(entities: Record<string, unknown>) {
  return { ok: true, json: async () => ({ entities }) } as Response
13}
14 
15/** Build a Wikidata entities map from per-title specs, mirroring the real shape the
*  people-filter reads. A title is a PERSON when P31 = Q5 (human) OR it carries a
*  usable birth/death date — so the specs can model each independently:
*   - `instanceOf`: the P31 item value (Q5 = human; anything else = a non-human type)
*   - `p31Snaktype`: 'somevalue'|'novalue' to model an "unknown instance-of" snak
*     (no real value) — exercises the snaktype guard
*   - `born` / `died`: a Wikidata time literal (e.g. '-1500-01-01T00:00:00Z') to
*     give the entity a P569 / P570 date (the clause that keeps dated non-Q5 figures)
*  Omit everything to model an article with no usable claim (or a missing item) —
*  that title won't survive the filter. */
25function entitiesFor(
  specs: Array<{ title: string; instanceOf?: string; p31Snaktype?: 'somevalue' | 'novalue'; born?: string; died?: string }>
27) {
  const dateClaim = (time: string) => [
      { mainsnak: { snaktype: 'value', datavalue: { value: { time, precision: 9 } } } }
  ]
  const out: Record<string, unknown> = {}
  specs.forEach((s, i) => {
      const claims: Record<string, unknown> = {}
      if (s.p31Snaktype) claims.P31 = [{ mainsnak: { snaktype: s.p31Snaktype } }]
      else if (s.instanceOf) claims.P31 = [{ mainsnak: { snaktype: 'value', datavalue: { value: { id: s.instanceOf } } } }]
      if (s.born) claims.P569 = dateClaim(s.born)
      if (s.died) claims.P570 = dateClaim(s.died)
      out[`Q${100 + i}`] = { claims, sitelinks: { enwiki: { title: s.title } } }
  })
  return out
41}
42 
43const isWikidata = (url: unknown) => String(url).includes('wikidata.org')
44 
45/** Route the two endpoints search now hits: the Wikipedia title search and the
*  batched Wikidata people-filter. Each handler gets the count of prior calls to
*  ITS endpoint, so a test can model a first-attempt blip then a recovery. */
48function route(
  fetchMock: ReturnType<typeof vi.fn>,
  handlers: {
      search: (n: number) => Response
      wikidata?: (n: number) => Response
  }
54) {
  let s = 0
  let w = 0
  fetchMock.mockImplementation((url: unknown) =>
      isWikidata(url)
          ? Promise.resolve(handlers.wikidata ? handlers.wikidata(w++) : wikidataResp({}))
          : Promise.resolve(handlers.search(s++))
  )
62}
63 
64describe('figure search (name autocomplete, people-filtered)', () => {
  // Each test uses a fresh query string: searchFigures' cache is module-level and
  // shared per run (the same convention as the grounding spec), so a reused query
  // would hit a stale cache and the fetch-count assertions would mislead.
  let fetchMock: ReturnType<typeof vi.fn>
69 
  beforeEach(() => {
      fetchMock = vi.fn()
      vi.stubGlobal('fetch', fetchMock)
  })
74 
  afterEach(() => {
      vi.useRealTimers()
      vi.unstubAllGlobals()
  })
79 
  it('filters out non-people (films, places, concepts) — the #74 core', async () => {
      route(fetchMock, {
          search: () =>
              searchResp([
                  { title: 'Cleopatra', description: 'Pharaoh of Egypt from 51 to 30 BC' },
                  { title: 'Cleopatra (1963 film)', description: '1963 film' },
                  { title: 'Death of Cleopatra', description: '' }
              ]),
          wikidata: () =>
              wikidataResp(
                  entitiesFor([
                      { title: 'Cleopatra', instanceOf: 'Q5' }, // human
                      { title: 'Cleopatra (1963 film)', instanceOf: 'Q11424' }, // film
                      { title: 'Death of Cleopatra', instanceOf: 'Q1190554' } // event
                  ])
              )
      })
97 
      const { results, transient } = await searchFigures('cleopat filter')
99 
      expect(transient).toBe(false)
      expect(results.map(r => r.name)).toEqual(['Cleopatra']) // film + event dropped
  })
103 
  it('maps titles, descriptions, and protocol-relative thumbnails for the people kept', async () => {
      route(fetchMock, {
          search: () =>
              searchResp([
                  {
                      title: 'Cleopatra',
                      description: 'Pharaoh of Egypt from 51 to 30 BC',
                      thumbnail: { url: '//upload.wikimedia.org/kleo.jpg' }
                  },
                  { title: 'Nikola Tesla', description: 'Inventor', thumbnail: null }
              ]),
          wikidata: () =>
              wikidataResp(
                  entitiesFor([
                      { title: 'Cleopatra', instanceOf: 'Q5' },
                      { title: 'Nikola Tesla', instanceOf: 'Q5' }
                  ])
              )
      })
123 
      const outcome = await searchFigures('two people probe')
125 
      expect(outcome.transient).toBe(false)
      expect(outcome.results).toEqual([
          {
              name: 'Cleopatra',
              description: 'Pharaoh of Egypt from 51 to 30 BC',
              thumbnail: 'https://upload.wikimedia.org/kleo.jpg'
          },
          { name: 'Nikola Tesla', description: 'Inventor', thumbnail: undefined }
      ])
  })
136 
  it('drops unsafe thumbnail schemes (the unsafe-render rule)', async () => {
      route(fetchMock, {
          search: () =>
              searchResp([{ title: 'Real Person', description: 'x', thumbnail: { url: 'javascript:alert(1)' } }]),
          wikidata: () => wikidataResp(entitiesFor([{ title: 'Real Person', instanceOf: 'Q5' }]))
      })
      const { results } = await searchFigures('spoofed person')
      expect(results[0].name).toBe('Real Person')
      expect(results[0].thumbnail).toBeUndefined()
  })
147 
  it('drops a result with no Wikidata item (unclassifiable → not a confirmed person)', async () => {
      route(fetchMock, {
          search: () =>
              searchResp([
                  { title: 'Marie Curie', description: 'Physicist' },
                  { title: 'Some Obscure Page', description: 'no wikidata item' }
              ]),
          // Only Marie Curie comes back from Wikidata; the obscure page is absent.
          wikidata: () => wikidataResp(entitiesFor([{ title: 'Marie Curie', instanceOf: 'Q5' }]))
      })
      const { results } = await searchFigures('partial item probe')
      expect(results.map(r => r.name)).toEqual(['Marie Curie'])
  })
161 
  it('keeps a contactable non-Q5 figure that has a Wikidata death date (Moses / King Arthur class)', async () => {
      // The bug a bare-Q5 filter would have: biblical / legendary figures are typed
      // "biblical figure" / "legendary figure", NOT Q5, yet the contact gate accepts
      // them (they resolve and carry a death date). The date clause keeps them.
      route(fetchMock, {
          search: () =>
              searchResp([
                  { title: 'Moses', description: 'Biblical figure' },
                  { title: 'The Ten Commandments (1956 film)', description: '1956 film' }
              ]),
          wikidata: () =>
              wikidataResp(
                  entitiesFor([
                      { title: 'Moses', instanceOf: 'Q20643955', died: '-1500-01-01T00:00:00Z' }, // not Q5, but dated
                      { title: 'The Ten Commandments (1956 film)', instanceOf: 'Q11424' } // film, no date
                  ])
              )
      })
      const { results } = await searchFigures('moses class probe')
      expect(results.map(r => r.name)).toEqual(['Moses']) // kept via P570; film dropped
  })
183 
  it('drops a fictional character with only an in-universe birth date — death, not birth', async () => {
      // Sherlock Holmes / Harry Potter carry a P569 birthday but no P570 death; the
      // gate refuses them as "not deceased", so search must not preview them. The
      // people test keys on death (P570), not birth — a birth date alone is not a
      // person the gate can reach.
      route(fetchMock, {
          search: () =>
              searchResp([
                  { title: 'Sherlock Holmes', description: 'fictional detective' },
                  { title: 'Arthur Conan Doyle', description: 'author' }
              ]),
          wikidata: () =>
              wikidataResp(
                  entitiesFor([
                      { title: 'Sherlock Holmes', instanceOf: 'Q15632617', born: '1854-01-06T00:00:00Z' }, // birth only
                      { title: 'Arthur Conan Doyle', instanceOf: 'Q5', died: '1930-07-07T00:00:00Z' } // real, deceased
                  ])
              )
      })
      const { results } = await searchFigures('sherlock vs doyle probe')
      expect(results.map(r => r.name)).toEqual(['Arthur Conan Doyle'])
  })
⋯ 244 lines hidden (lines 206–449)206 
  it('drops an entity whose only P31 is a somevalue snak with no dates (the snaktype guard does work)', async () => {
      route(fetchMock, {
          search: () =>
              searchResp([
                  { title: 'Vague Concept', description: 'instance-of: unknown value' },
                  { title: 'Marie Curie', description: 'Physicist' }
              ]),
          wikidata: () =>
              wikidataResp(
                  entitiesFor([
                      { title: 'Vague Concept', p31Snaktype: 'somevalue' }, // no real value, no dates
                      { title: 'Marie Curie', instanceOf: 'Q5' }
                  ])
              )
      })
      const { results } = await searchFigures('somevalue snak probe')
      expect(results.map(r => r.name)).toEqual(['Marie Curie'])
  })
225 
  it('matches a result to its entity across underscore/case drift (titleKey normalization)', async () => {
      route(fetchMock, {
          search: () => searchResp([{ title: 'Marie Curie', description: 'Physicist' }]),
          // The Wikidata sitelink comes back with an underscore and different case;
          // titleKey must normalize both sides or this person would be wrongly dropped.
          wikidata: () => wikidataResp(entitiesFor([{ title: 'marie_curie', instanceOf: 'Q5' }]))
      })
      const { results } = await searchFigures('normalize title probe')
      expect(results.map(r => r.name)).toEqual(['Marie Curie'])
  })
236 
  it('builds the Wikidata request the filter depends on (props, sites, sitefilter, all titles)', async () => {
      // The mock fabricates the response, so without this the request contract is
      // unchecked: if props dropped `sitelinks`, real Wikidata returns no title to
      // match and EVERY result would be silently filtered out.
      route(fetchMock, {
          search: () =>
              searchResp([
                  { title: 'Ada Lovelace', description: 'Mathematician' },
                  { title: 'Nikola Tesla', description: 'Inventor' }
              ]),
          wikidata: () =>
              wikidataResp(
                  entitiesFor([
                      { title: 'Ada Lovelace', instanceOf: 'Q5' },
                      { title: 'Nikola Tesla', instanceOf: 'Q5' }
                  ])
              )
      })
      await searchFigures('request contract probe')
256 
      const wdCall = fetchMock.mock.calls.find(c => isWikidata(c[0]))
      expect(wdCall).toBeTruthy()
      const url = decodeURIComponent(String(wdCall![0]))
      expect(url).toContain('action=wbgetentities')
      expect(url).toContain('sites=enwiki')
      expect(url).toContain('sitefilter=enwiki')
      expect(url).toContain('props=claims|sitelinks')
      expect(url).toContain('Ada Lovelace') // both result titles carried in titles=
      expect(url).toContain('Nikola Tesla')
  })
267 
  it('caches a settled "no people match" — all results filtered out, then served from cache', async () => {
      route(fetchMock, {
          search: () =>
              searchResp([
                  { title: 'Some Film', description: 'a film' },
                  { title: 'Some Place', description: 'a city' }
              ]),
          wikidata: () =>
              wikidataResp(
                  entitiesFor([
                      { title: 'Some Film', instanceOf: 'Q11424' },
                      { title: 'Some Place', instanceOf: 'Q515' }
                  ])
              )
      })
      const first = await searchFigures('all nonpeople probe')
      expect(first).toEqual({ results: [], transient: false }) // a definitive answer, not weather
      await searchFigures('all nonpeople probe')
      expect(fetchMock).toHaveBeenCalledTimes(2) // search + filter once; the empty answer cached
  })
288 
  it('returns [] without fetching for queries under two characters', async () => {
      expect(await searchFigures('a')).toEqual({ results: [], transient: false })
      expect(await searchFigures('  ')).toEqual({ results: [], transient: false })
      expect(fetchMock).not.toHaveBeenCalled()
  })
294 
  it('caches a query — the second call refetches neither endpoint', async () => {
      route(fetchMock, {
          search: () => searchResp([{ title: 'Nikola Tesla', description: 'Inventor' }]),
          wikidata: () => wikidataResp(entitiesFor([{ title: 'Nikola Tesla', instanceOf: 'Q5' }]))
      })
      await searchFigures('tesla cache probe')
      await searchFigures('Tesla Cache Probe') // case-insensitive key
      expect(fetchMock).toHaveBeenCalledTimes(2) // one search + one filter, then served from cache
  })
304 
  it('sends a Wikimedia-policy-compliant User-Agent on BOTH the search and the filter call', async () => {
      // A non-compliant agent (no contact) gets throttled harder — the dominant
      // cause of the autocomplete silently failing (issue #58).
      route(fetchMock, {
          search: () => searchResp([{ title: 'Leonardo da Vinci', description: 'Polymath' }]),
          wikidata: () => wikidataResp(entitiesFor([{ title: 'Leonardo da Vinci', instanceOf: 'Q5' }]))
      })
      await searchFigures('leonardo ua probe')
313 
      for (const call of fetchMock.mock.calls) {
          const init = call[1] as RequestInit
          const ua = String((init.headers as Record<string, string>)['User-Agent'])
          expect(ua).toContain('Revisionist')
          expect(ua).toMatch(/mailto:\S+@\S+|https?:\/\/\S+/) // a reachable contact, per the UA policy
      }
      expect(fetchMock).toHaveBeenCalledTimes(2)
  })
322 
  it('retries a rate-limited search once and recovers the matches', async () => {
      vi.useFakeTimers()
      route(fetchMock, {
          search: n =>
              n === 0
                  ? ({ ok: false, status: 429 } as Response)
                  : searchResp([{ title: 'Alfred Nobel', description: 'Inventor of dynamite' }]),
          wikidata: () => wikidataResp(entitiesFor([{ title: 'Alfred Nobel', instanceOf: 'Q5' }]))
      })
332 
      const pending = searchFigures('alfred retry probe')
      await vi.advanceTimersByTimeAsync(250)
      const outcome = await pending
336 
      expect(outcome.transient).toBe(false)
      expect(outcome.results[0].name).toBe('Alfred Nobel')
  })
340 
  it('a search failure that survives the retry is flagged transient (filter never runs)', async () => {
      vi.useFakeTimers()
      route(fetchMock, {
          search: () => {
              throw new Error('offline')
          }
      })
348 
      const pending = searchFigures('failing query one')
      await vi.advanceTimersByTimeAsync(250)
      expect(await pending).toEqual({ results: [], transient: true })
      // The filter is never reached when the search itself never answered.
      expect(fetchMock.mock.calls.every(c => !isWikidata(c[0]))).toBe(true)
  })
355 
  it('a real empty search answer is NOT transient, caches, and skips the filter call', async () => {
      route(fetchMock, {
          search: () => searchResp([]),
          wikidata: () => wikidataResp({})
      })
      expect(await searchFigures('xqzv nobody')).toEqual({ results: [], transient: false })
      await searchFigures('xqzv nobody')
      expect(fetchMock).toHaveBeenCalledTimes(1) // one search; no filter (nothing to classify), then cached
      expect(fetchMock.mock.calls.every(c => !isWikidata(c[0]))).toBe(true)
  })
366 
  it('never caches a search failure — the next call for the same query refetches', async () => {
      vi.useFakeTimers()
      let searchCalls = 0
      route(fetchMock, {
          search: () => {
              searchCalls++
              if (searchCalls <= 2) throw new Error('wikipedia blip')
              return searchResp([{ title: 'Recovered', description: 'after the blip' }])
          },
          wikidata: () => wikidataResp(entitiesFor([{ title: 'Recovered', instanceOf: 'Q5' }]))
      })
378 
      const pending = searchFigures('transient blip probe')
      await vi.advanceTimersByTimeAsync(250)
      expect((await pending).transient).toBe(true)
382 
      const second = await searchFigures('transient blip probe')
      expect(second.results).toEqual([{ name: 'Recovered', description: 'after the blip', thumbnail: undefined }])
      expect(searchCalls).toBe(3) // two failed attempts (not cached) + one fresh success
  })
387 
  it('FALLS OPEN when the people-filter blips: a working search is not blanked (no #58 regression)', async () => {
      vi.useFakeTimers()
      route(fetchMock, {
          search: () =>
              searchResp([
                  { title: 'Cleopatra', description: 'Pharaoh' },
                  { title: 'Cleopatra (1963 film)', description: '1963 film' }
              ]),
          // Wikidata never answers — a network error, then (on retry) a 5xx. Both
          // failure modes are exercised; the filter gives up and falls open.
          wikidata: n => {
              if (n === 0) throw new Error('network down')
              return { ok: false, status: 503 } as Response
          }
      })
403 
      const pending = searchFigures('fallopen probe')
      await vi.advanceTimersByTimeAsync(250) // the filter's one spaced retry
      const outcome = await pending
407 
      // Unfiltered matches are returned (the search worked); not flagged transient,
      // because we DO have results to show.
      expect(outcome.transient).toBe(false)
      expect(outcome.results.map(r => r.name)).toEqual(['Cleopatra', 'Cleopatra (1963 film)'])
412 
      // A fall-open is NOT cached: the next call re-attempts the filter.
      expect(fetchMock.mock.calls.filter(c => isWikidata(c[0])).length).toBe(2) // attempt + retry
415 
      const pending2 = searchFigures('fallopen probe')
      await vi.advanceTimersByTimeAsync(250)
      await pending2
      expect(fetchMock.mock.calls.filter(c => isWikidata(c[0])).length).toBeGreaterThan(2)
  })
421 
  it('retries the people-filter once on a no-entities blip, then filters cleanly', async () => {
      vi.useFakeTimers()
      route(fetchMock, {
          search: () =>
              searchResp([
                  { title: 'Genghis Khan', description: 'Mongol emperor' },
                  { title: 'Genghis Khan (restaurant)', description: 'restaurant' }
              ]),
          wikidata: n =>
              n === 0
                  ? ({ ok: false, status: 429 } as Response) // first filter attempt blips
                  : wikidataResp(
                        entitiesFor([
                            { title: 'Genghis Khan', instanceOf: 'Q5' },
                            { title: 'Genghis Khan (restaurant)', instanceOf: 'Q11707' }
                        ])
                    )
      })
440 
      const pending = searchFigures('genghis retry filter')
      await vi.advanceTimersByTimeAsync(250) // the filter retry delay
      const outcome = await pending
444 
      expect(outcome.transient).toBe(false)
      expect(outcome.results.map(r => r.name)).toEqual(['Genghis Khan']) // restaurant dropped
      expect(fetchMock.mock.calls.filter(c => isWikidata(c[0])).length).toBe(2)
  })
449})

Two more guard the request contract and the resilience story. The contract test asserts the actual Wikidata URL carries props=claims|sitelinks, sitefilter and every result title — without it, dropping sitelinks would silently filter everything out yet still pass. The fall-open test drives a Wikidata failure and asserts the unfiltered matches come back, transient: false, and are not cached.

The request-contract assertion and the fall-open (no #58 regression) test.

tests/unit/server/utils/figure-search.spec.ts · 449 lines

tests/unit/server/utils/figure-search.spec.ts449 lines · TypeScript

⋯ 236 lines hidden (lines 1–236)1import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest'
2import { searchFigures } from '../../../../server/utils/figure-search'
3 
4/** Wikipedia REST v1 title-search response, shaped like the real thing. */
5function searchResp(pages: unknown[]) {
  return { ok: true, json: async () => ({ pages }) } as Response
7}
8 
9/** Wikidata `wbgetentities` response, shaped like the real thing (entities keyed
*  by qid, each carrying its P31 "instance of" claims + its enwiki sitelink). */
11function wikidataResp(entities: Record<string, unknown>) {
  return { ok: true, json: async () => ({ entities }) } as Response
13}
14 
15/** Build a Wikidata entities map from per-title specs, mirroring the real shape the
*  people-filter reads. A title is a PERSON when P31 = Q5 (human) OR it carries a
*  usable birth/death date — so the specs can model each independently:
*   - `instanceOf`: the P31 item value (Q5 = human; anything else = a non-human type)
*   - `p31Snaktype`: 'somevalue'|'novalue' to model an "unknown instance-of" snak
*     (no real value) — exercises the snaktype guard
*   - `born` / `died`: a Wikidata time literal (e.g. '-1500-01-01T00:00:00Z') to
*     give the entity a P569 / P570 date (the clause that keeps dated non-Q5 figures)
*  Omit everything to model an article with no usable claim (or a missing item) —
*  that title won't survive the filter. */
25function entitiesFor(
  specs: Array<{ title: string; instanceOf?: string; p31Snaktype?: 'somevalue' | 'novalue'; born?: string; died?: string }>
27) {
  const dateClaim = (time: string) => [
      { mainsnak: { snaktype: 'value', datavalue: { value: { time, precision: 9 } } } }
  ]
  const out: Record<string, unknown> = {}
  specs.forEach((s, i) => {
      const claims: Record<string, unknown> = {}
      if (s.p31Snaktype) claims.P31 = [{ mainsnak: { snaktype: s.p31Snaktype } }]
      else if (s.instanceOf) claims.P31 = [{ mainsnak: { snaktype: 'value', datavalue: { value: { id: s.instanceOf } } } }]
      if (s.born) claims.P569 = dateClaim(s.born)
      if (s.died) claims.P570 = dateClaim(s.died)
      out[`Q${100 + i}`] = { claims, sitelinks: { enwiki: { title: s.title } } }
  })
  return out
41}
42 
43const isWikidata = (url: unknown) => String(url).includes('wikidata.org')
44 
45/** Route the two endpoints search now hits: the Wikipedia title search and the
*  batched Wikidata people-filter. Each handler gets the count of prior calls to
*  ITS endpoint, so a test can model a first-attempt blip then a recovery. */
48function route(
  fetchMock: ReturnType<typeof vi.fn>,
  handlers: {
      search: (n: number) => Response
      wikidata?: (n: number) => Response
  }
54) {
  let s = 0
  let w = 0
  fetchMock.mockImplementation((url: unknown) =>
      isWikidata(url)
          ? Promise.resolve(handlers.wikidata ? handlers.wikidata(w++) : wikidataResp({}))
          : Promise.resolve(handlers.search(s++))
  )
62}
63 
64describe('figure search (name autocomplete, people-filtered)', () => {
  // Each test uses a fresh query string: searchFigures' cache is module-level and
  // shared per run (the same convention as the grounding spec), so a reused query
  // would hit a stale cache and the fetch-count assertions would mislead.
  let fetchMock: ReturnType<typeof vi.fn>
69 
  beforeEach(() => {
      fetchMock = vi.fn()
      vi.stubGlobal('fetch', fetchMock)
  })
74 
  afterEach(() => {
      vi.useRealTimers()
      vi.unstubAllGlobals()
  })
79 
  it('filters out non-people (films, places, concepts) — the #74 core', async () => {
      route(fetchMock, {
          search: () =>
              searchResp([
                  { title: 'Cleopatra', description: 'Pharaoh of Egypt from 51 to 30 BC' },
                  { title: 'Cleopatra (1963 film)', description: '1963 film' },
                  { title: 'Death of Cleopatra', description: '' }
              ]),
          wikidata: () =>
              wikidataResp(
                  entitiesFor([
                      { title: 'Cleopatra', instanceOf: 'Q5' }, // human
                      { title: 'Cleopatra (1963 film)', instanceOf: 'Q11424' }, // film
                      { title: 'Death of Cleopatra', instanceOf: 'Q1190554' } // event
                  ])
              )
      })
97 
      const { results, transient } = await searchFigures('cleopat filter')
99 
      expect(transient).toBe(false)
      expect(results.map(r => r.name)).toEqual(['Cleopatra']) // film + event dropped
  })
103 
  it('maps titles, descriptions, and protocol-relative thumbnails for the people kept', async () => {
      route(fetchMock, {
          search: () =>
              searchResp([
                  {
                      title: 'Cleopatra',
                      description: 'Pharaoh of Egypt from 51 to 30 BC',
                      thumbnail: { url: '//upload.wikimedia.org/kleo.jpg' }
                  },
                  { title: 'Nikola Tesla', description: 'Inventor', thumbnail: null }
              ]),
          wikidata: () =>
              wikidataResp(
                  entitiesFor([
                      { title: 'Cleopatra', instanceOf: 'Q5' },
                      { title: 'Nikola Tesla', instanceOf: 'Q5' }
                  ])
              )
      })
123 
      const outcome = await searchFigures('two people probe')
125 
      expect(outcome.transient).toBe(false)
      expect(outcome.results).toEqual([
          {
              name: 'Cleopatra',
              description: 'Pharaoh of Egypt from 51 to 30 BC',
              thumbnail: 'https://upload.wikimedia.org/kleo.jpg'
          },
          { name: 'Nikola Tesla', description: 'Inventor', thumbnail: undefined }
      ])
  })
136 
  it('drops unsafe thumbnail schemes (the unsafe-render rule)', async () => {
      route(fetchMock, {
          search: () =>
              searchResp([{ title: 'Real Person', description: 'x', thumbnail: { url: 'javascript:alert(1)' } }]),
          wikidata: () => wikidataResp(entitiesFor([{ title: 'Real Person', instanceOf: 'Q5' }]))
      })
      const { results } = await searchFigures('spoofed person')
      expect(results[0].name).toBe('Real Person')
      expect(results[0].thumbnail).toBeUndefined()
  })
147 
  it('drops a result with no Wikidata item (unclassifiable → not a confirmed person)', async () => {
      route(fetchMock, {
          search: () =>
              searchResp([
                  { title: 'Marie Curie', description: 'Physicist' },
                  { title: 'Some Obscure Page', description: 'no wikidata item' }
              ]),
          // Only Marie Curie comes back from Wikidata; the obscure page is absent.
          wikidata: () => wikidataResp(entitiesFor([{ title: 'Marie Curie', instanceOf: 'Q5' }]))
      })
      const { results } = await searchFigures('partial item probe')
      expect(results.map(r => r.name)).toEqual(['Marie Curie'])
  })
161 
  it('keeps a contactable non-Q5 figure that has a Wikidata death date (Moses / King Arthur class)', async () => {
      // The bug a bare-Q5 filter would have: biblical / legendary figures are typed
      // "biblical figure" / "legendary figure", NOT Q5, yet the contact gate accepts
      // them (they resolve and carry a death date). The date clause keeps them.
      route(fetchMock, {
          search: () =>
              searchResp([
                  { title: 'Moses', description: 'Biblical figure' },
                  { title: 'The Ten Commandments (1956 film)', description: '1956 film' }
              ]),
          wikidata: () =>
              wikidataResp(
                  entitiesFor([
                      { title: 'Moses', instanceOf: 'Q20643955', died: '-1500-01-01T00:00:00Z' }, // not Q5, but dated
                      { title: 'The Ten Commandments (1956 film)', instanceOf: 'Q11424' } // film, no date
                  ])
              )
      })
      const { results } = await searchFigures('moses class probe')
      expect(results.map(r => r.name)).toEqual(['Moses']) // kept via P570; film dropped
  })
183 
  it('drops a fictional character with only an in-universe birth date — death, not birth', async () => {
      // Sherlock Holmes / Harry Potter carry a P569 birthday but no P570 death; the
      // gate refuses them as "not deceased", so search must not preview them. The
      // people test keys on death (P570), not birth — a birth date alone is not a
      // person the gate can reach.
      route(fetchMock, {
          search: () =>
              searchResp([
                  { title: 'Sherlock Holmes', description: 'fictional detective' },
                  { title: 'Arthur Conan Doyle', description: 'author' }
              ]),
          wikidata: () =>
              wikidataResp(
                  entitiesFor([
                      { title: 'Sherlock Holmes', instanceOf: 'Q15632617', born: '1854-01-06T00:00:00Z' }, // birth only
                      { title: 'Arthur Conan Doyle', instanceOf: 'Q5', died: '1930-07-07T00:00:00Z' } // real, deceased
                  ])
              )
      })
      const { results } = await searchFigures('sherlock vs doyle probe')
      expect(results.map(r => r.name)).toEqual(['Arthur Conan Doyle'])
  })
206 
  it('drops an entity whose only P31 is a somevalue snak with no dates (the snaktype guard does work)', async () => {
      route(fetchMock, {
          search: () =>
              searchResp([
                  { title: 'Vague Concept', description: 'instance-of: unknown value' },
                  { title: 'Marie Curie', description: 'Physicist' }
              ]),
          wikidata: () =>
              wikidataResp(
                  entitiesFor([
                      { title: 'Vague Concept', p31Snaktype: 'somevalue' }, // no real value, no dates
                      { title: 'Marie Curie', instanceOf: 'Q5' }
                  ])
              )
      })
      const { results } = await searchFigures('somevalue snak probe')
      expect(results.map(r => r.name)).toEqual(['Marie Curie'])
  })
225 
  it('matches a result to its entity across underscore/case drift (titleKey normalization)', async () => {
      route(fetchMock, {
          search: () => searchResp([{ title: 'Marie Curie', description: 'Physicist' }]),
          // The Wikidata sitelink comes back with an underscore and different case;
          // titleKey must normalize both sides or this person would be wrongly dropped.
          wikidata: () => wikidataResp(entitiesFor([{ title: 'marie_curie', instanceOf: 'Q5' }]))
      })
      const { results } = await searchFigures('normalize title probe')
      expect(results.map(r => r.name)).toEqual(['Marie Curie'])
  })
236 
  it('builds the Wikidata request the filter depends on (props, sites, sitefilter, all titles)', async () => {
      // The mock fabricates the response, so without this the request contract is
      // unchecked: if props dropped `sitelinks`, real Wikidata returns no title to
      // match and EVERY result would be silently filtered out.
      route(fetchMock, {
          search: () =>
              searchResp([
                  { title: 'Ada Lovelace', description: 'Mathematician' },
                  { title: 'Nikola Tesla', description: 'Inventor' }
              ]),
          wikidata: () =>
              wikidataResp(
                  entitiesFor([
                      { title: 'Ada Lovelace', instanceOf: 'Q5' },
                      { title: 'Nikola Tesla', instanceOf: 'Q5' }
                  ])
              )
      })
      await searchFigures('request contract probe')
256 
      const wdCall = fetchMock.mock.calls.find(c => isWikidata(c[0]))
      expect(wdCall).toBeTruthy()
      const url = decodeURIComponent(String(wdCall![0]))
      expect(url).toContain('action=wbgetentities')
      expect(url).toContain('sites=enwiki')
      expect(url).toContain('sitefilter=enwiki')
      expect(url).toContain('props=claims|sitelinks')
      expect(url).toContain('Ada Lovelace') // both result titles carried in titles=
      expect(url).toContain('Nikola Tesla')
  })
⋯ 121 lines hidden (lines 267–387)267 
  it('caches a settled "no people match" — all results filtered out, then served from cache', async () => {
      route(fetchMock, {
          search: () =>
              searchResp([
                  { title: 'Some Film', description: 'a film' },
                  { title: 'Some Place', description: 'a city' }
              ]),
          wikidata: () =>
              wikidataResp(
                  entitiesFor([
                      { title: 'Some Film', instanceOf: 'Q11424' },
                      { title: 'Some Place', instanceOf: 'Q515' }
                  ])
              )
      })
      const first = await searchFigures('all nonpeople probe')
      expect(first).toEqual({ results: [], transient: false }) // a definitive answer, not weather
      await searchFigures('all nonpeople probe')
      expect(fetchMock).toHaveBeenCalledTimes(2) // search + filter once; the empty answer cached
  })
288 
  it('returns [] without fetching for queries under two characters', async () => {
      expect(await searchFigures('a')).toEqual({ results: [], transient: false })
      expect(await searchFigures('  ')).toEqual({ results: [], transient: false })
      expect(fetchMock).not.toHaveBeenCalled()
  })
294 
  it('caches a query — the second call refetches neither endpoint', async () => {
      route(fetchMock, {
          search: () => searchResp([{ title: 'Nikola Tesla', description: 'Inventor' }]),
          wikidata: () => wikidataResp(entitiesFor([{ title: 'Nikola Tesla', instanceOf: 'Q5' }]))
      })
      await searchFigures('tesla cache probe')
      await searchFigures('Tesla Cache Probe') // case-insensitive key
      expect(fetchMock).toHaveBeenCalledTimes(2) // one search + one filter, then served from cache
  })
304 
  it('sends a Wikimedia-policy-compliant User-Agent on BOTH the search and the filter call', async () => {
      // A non-compliant agent (no contact) gets throttled harder — the dominant
      // cause of the autocomplete silently failing (issue #58).
      route(fetchMock, {
          search: () => searchResp([{ title: 'Leonardo da Vinci', description: 'Polymath' }]),
          wikidata: () => wikidataResp(entitiesFor([{ title: 'Leonardo da Vinci', instanceOf: 'Q5' }]))
      })
      await searchFigures('leonardo ua probe')
313 
      for (const call of fetchMock.mock.calls) {
          const init = call[1] as RequestInit
          const ua = String((init.headers as Record<string, string>)['User-Agent'])
          expect(ua).toContain('Revisionist')
          expect(ua).toMatch(/mailto:\S+@\S+|https?:\/\/\S+/) // a reachable contact, per the UA policy
      }
      expect(fetchMock).toHaveBeenCalledTimes(2)
  })
322 
  it('retries a rate-limited search once and recovers the matches', async () => {
      vi.useFakeTimers()
      route(fetchMock, {
          search: n =>
              n === 0
                  ? ({ ok: false, status: 429 } as Response)
                  : searchResp([{ title: 'Alfred Nobel', description: 'Inventor of dynamite' }]),
          wikidata: () => wikidataResp(entitiesFor([{ title: 'Alfred Nobel', instanceOf: 'Q5' }]))
      })
332 
      const pending = searchFigures('alfred retry probe')
      await vi.advanceTimersByTimeAsync(250)
      const outcome = await pending
336 
      expect(outcome.transient).toBe(false)
      expect(outcome.results[0].name).toBe('Alfred Nobel')
  })
340 
  it('a search failure that survives the retry is flagged transient (filter never runs)', async () => {
      vi.useFakeTimers()
      route(fetchMock, {
          search: () => {
              throw new Error('offline')
          }
      })
348 
      const pending = searchFigures('failing query one')
      await vi.advanceTimersByTimeAsync(250)
      expect(await pending).toEqual({ results: [], transient: true })
      // The filter is never reached when the search itself never answered.
      expect(fetchMock.mock.calls.every(c => !isWikidata(c[0]))).toBe(true)
  })
355 
  it('a real empty search answer is NOT transient, caches, and skips the filter call', async () => {
      route(fetchMock, {
          search: () => searchResp([]),
          wikidata: () => wikidataResp({})
      })
      expect(await searchFigures('xqzv nobody')).toEqual({ results: [], transient: false })
      await searchFigures('xqzv nobody')
      expect(fetchMock).toHaveBeenCalledTimes(1) // one search; no filter (nothing to classify), then cached
      expect(fetchMock.mock.calls.every(c => !isWikidata(c[0]))).toBe(true)
  })
366 
  it('never caches a search failure — the next call for the same query refetches', async () => {
      vi.useFakeTimers()
      let searchCalls = 0
      route(fetchMock, {
          search: () => {
              searchCalls++
              if (searchCalls <= 2) throw new Error('wikipedia blip')
              return searchResp([{ title: 'Recovered', description: 'after the blip' }])
          },
          wikidata: () => wikidataResp(entitiesFor([{ title: 'Recovered', instanceOf: 'Q5' }]))
      })
378 
      const pending = searchFigures('transient blip probe')
      await vi.advanceTimersByTimeAsync(250)
      expect((await pending).transient).toBe(true)
382 
      const second = await searchFigures('transient blip probe')
      expect(second.results).toEqual([{ name: 'Recovered', description: 'after the blip', thumbnail: undefined }])
      expect(searchCalls).toBe(3) // two failed attempts (not cached) + one fresh success
  })
387 
  it('FALLS OPEN when the people-filter blips: a working search is not blanked (no #58 regression)', async () => {
      vi.useFakeTimers()
      route(fetchMock, {
          search: () =>
              searchResp([
                  { title: 'Cleopatra', description: 'Pharaoh' },
                  { title: 'Cleopatra (1963 film)', description: '1963 film' }
              ]),
          // Wikidata never answers — a network error, then (on retry) a 5xx. Both
          // failure modes are exercised; the filter gives up and falls open.
          wikidata: n => {
              if (n === 0) throw new Error('network down')
              return { ok: false, status: 503 } as Response
          }
      })
403 
      const pending = searchFigures('fallopen probe')
      await vi.advanceTimersByTimeAsync(250) // the filter's one spaced retry
      const outcome = await pending
407 
      // Unfiltered matches are returned (the search worked); not flagged transient,
      // because we DO have results to show.
      expect(outcome.transient).toBe(false)
      expect(outcome.results.map(r => r.name)).toEqual(['Cleopatra', 'Cleopatra (1963 film)'])
412 
      // A fall-open is NOT cached: the next call re-attempts the filter.
      expect(fetchMock.mock.calls.filter(c => isWikidata(c[0])).length).toBe(2) // attempt + retry
415 
      const pending2 = searchFigures('fallopen probe')
      await vi.advanceTimersByTimeAsync(250)
      await pending2
      expect(fetchMock.mock.calls.filter(c => isWikidata(c[0])).length).toBeGreaterThan(2)
  })
⋯ 29 lines hidden (lines 421–449)421 
  it('retries the people-filter once on a no-entities blip, then filters cleanly', async () => {
      vi.useFakeTimers()
      route(fetchMock, {
          search: () =>
              searchResp([
                  { title: 'Genghis Khan', description: 'Mongol emperor' },
                  { title: 'Genghis Khan (restaurant)', description: 'restaurant' }
              ]),
          wikidata: n =>
              n === 0
                  ? ({ ok: false, status: 429 } as Response) // first filter attempt blips
                  : wikidataResp(
                        entitiesFor([
                            { title: 'Genghis Khan', instanceOf: 'Q5' },
                            { title: 'Genghis Khan (restaurant)', instanceOf: 'Q11707' }
                        ])
                    )
      })
440 
      const pending = searchFigures('genghis retry filter')
      await vi.advanceTimersByTimeAsync(250) // the filter retry delay
      const outcome = await pending
444 
      expect(outcome.transient).toBe(false)
      expect(outcome.results.map(r => r.name)).toEqual(['Genghis Khan']) // restaurant dropped
      expect(fetchMock.mock.calls.filter(c => isWikidata(c[0])).length).toBe(2)
  })
449})

How it lines up with the contact gate

The gate this preview defers to is isDeceased in the grounding module: resolved && !!died, with no instance-of check. died can come from Wikidata P570 or the #31 AI bridge. So the filter keeps everyone the gate reaches via a Wikidata death (they have P570), plus all humans — looser than the gate on living people (by design; the dossier explains the floor), and with one narrow under-coverage gap (AI-bridge-only, non-Q5) the header calls out honestly.

The gate the search preview is reasoning about — death is its floor.

server/utils/figure-grounding.ts · 323 lines

server/utils/figure-grounding.ts323 lines · TypeScript

⋯ 59 lines hidden (lines 1–59)1/**
* Real-world grounding for historical figures — the backbone of the "timeline RPG"
* direction (see docs/ROADMAP.md). Resolves a free-form name to solid facts so the
* game can show who someone was, validate *when* you may contact them, and feed the
* role-play real dates.
*
* Pipeline (the wiki APIs are open + key-less):
*   1. Wikipedia REST summary — the reliable resolver. It follows redirects and
*      picks the notable article, giving a canonical title, a one-line description,
*      a blurb, a portrait, and the Wikidata item id. (Wikidata search alone is NOT
*      reliable: "Cleopatra" resolves to "female given name", "Genghis Khan" to an
*      Iron Maiden song.)
*   2. Wikidata, looked up by that item id (title as fallback) — structured birth
*      (P569) / death (P570), taking the best-ranked claim that actually carries a
*      date, with sub-year precision surfaced honestly as `circa`.
*   3. If the record still has no birth year, a small AI estimator bridges the gap
*      with circa dates (issue #31) — so a real, resolvable figure keeps their
*      "when" slider even when Wikidata can't date them.
*
* Anything unresolved returns `{ resolved: false }` (with `transient` set when the
* lookup couldn't complete). Contact callers now BLOCK an ungrounded name (#73) —
* only a real, documented figure can be reached — while a transient miss is a retry.
*
* Cache honesty: only definitive answers — a real Wikidata reply (dated or not,
* with the estimator genuinely consulted), a genuine no-such-article miss, a
* disambiguation page — keep the full TTL. A transient failure anywhere on the
* path (a rate-limit 429, a timeout, an estimator outage) gets a short TTL, so
* one blip never poisons a figure for a day.
*/
30import { estimateLifespan } from './lifespan-estimator'
31 
32/** A year on the timeline. `signed` is AD-positive / BCE-negative for comparison. */
33export interface FigureYear {
  year: number
  bce: boolean
  signed: number
  display: string
  /** Approximate: sub-year Wikidata precision (decade/century/…) or an AI estimate. */
  circa?: boolean
40}
41 
42export interface GroundedFigure {
  name: string
  resolved: boolean
  description?: string
  extract?: string
  born?: FigureYear
  died?: FigureYear
  living?: boolean
  thumbnail?: string
  wikiUrl?: string
  /** The lookup couldn't be completed (rate-limit / timeout / outage), so an
   *  unresolved result is "we don't know yet", not "no such person". The contact
   *  gate treats this as a retry rather than a definitive miss — and never as a
   *  free-form pass (#72 / #73). */
  transient?: boolean
57}
58 
59/**
* The deceased-only contact floor (#72): a figure may be reached only once we can
* confirm they have died. A resolved figure with no death year is living (or not
* datable enough to prove otherwise) and is never contactable — fail closed. An
* unresolved name is not deceased here either (the free-form path is removed in #73).
* The canonical predicate, shared by the suggester and the send-message gate; the
* client store inlines the same rule (it can't import this server-only module).
*/
67export function isDeceased(figure: GroundedFigure): boolean {
  return figure.resolved && !!figure.died
69}
⋯ 254 lines hidden (lines 70–323)70 
71/**
* The one header for every Wikipedia / Wikidata call the server makes — grounding,
* the name autocomplete, and the suggester's lookups all share it (issue #58).
* Wikimedia's User-Agent policy wants the client identified plus a reachable CONTACT
* so an operator can be reached about problematic traffic; an agent without one gets
* throttled harder, and rate-limit blips are the dominant real-world failure for the
* autocomplete. A mailto is that contact — the deployed site is auth-gated and the
* repo private, so neither would reach a human. Exported so every caller sends the
* identical compliant string instead of drifting per-feature copies.
*/
81export const WIKI_HEADERS = {
  accept: 'application/json',
  'User-Agent': 'Revisionist/1.0 (timeline RPG; mailto:matthew@mseeks.me)'
84}
85 
86const FULL_TTL_MS = 24 * 60 * 60 * 1000
87const TRANSIENT_TTL_MS = 10 * 60 * 1000
88const WIKIDATA_RETRY_DELAY_MS = 400
89const cache = new Map<string, { value: GroundedFigure; expires: number }>()
90 
91/** Builds a `FigureYear`; the display string owns the "c." honesty marker. */
92export function makeFigureYear(year: number, bce: boolean, circa = false): FigureYear {
  const base = bce ? `${year} BC` : `${year}`
  return {
      year,
      bce,
      signed: bce ? -year : year,
      display: circa ? `c. ${base}` : base,
      ...(circa ? { circa: true } : {})
  }
101}
102 
103/**
* Parses a Wikidata time literal (e.g. "+1856-07-10T..", "-0069-01-13T..") to a year.
* Precision 9 means year-level; anything below (8 decade, 7 century, 6 millennium)
* still carries a representative year in the literal, but only approximately —
* those parse as `circa` rather than being dropped or displayed as exact.
*/
109export function parseWikidataYear(time?: string, precision?: number): FigureYear | undefined {
  if (!time) return undefined
  const m = /^([+-])0*(\d+)-/.exec(time)
  if (!m) return undefined
  const year = parseInt(m[2], 10)
  if (!year) return undefined
  const bce = m[1] === '-'
  return makeFigureYear(year, bce, precision != null && precision < 9)
117}
118 
119/** The slice of the Wikipedia REST summary we actually read. */
120interface WikiSummary {
  type?: string
  title?: string
  description?: string
  extract?: string
  thumbnail?: { source?: string }
  content_urls?: { desktop?: { page?: string } }
  wikibase_item?: string
128}
129 
130/** The slice of a Wikidata date claim we actually read. */
131interface WikidataClaim {
  rank?: string
  mainsnak?: {
      snaktype?: string
      datavalue?: { value?: { time?: string; precision?: number } }
  }
137}
138 
139/** The slice of the Wikidata `wbgetentities` response we actually read. */
140interface WikidataEntities {
  entities?: Record<string, { claims?: Record<string, WikidataClaim[]> }>
142}
143 
144/**
* The best date a claim list actually carries. Wikidata routinely stacks several
* birth/death claims (scholarly guesses, deprecated values, "unknown value"
* placeholders) and the community-preferred one is often NOT first — Moses, Jesus
* and Confucius all keep theirs at index 2. So: only claims whose snak carries a
* real time (somevalue/novalue snaks don't), never deprecated, preferred rank first.
*/
151export function pickClaimYear(claims?: WikidataClaim[]): FigureYear | undefined {
  const usable = (claims ?? []).filter(c =>
      c.rank !== 'deprecated' &&
      c.mainsnak?.snaktype === 'value' &&
      c.mainsnak.datavalue?.value?.time
  )
  const best = usable.find(c => c.rank === 'preferred') ?? usable[0]
  const value = best?.mainsnak?.datavalue?.value
  return parseWikidataYear(value?.time, value?.precision)
160}
161 
162/**
* A fetch that tells a definitive miss from a transient one: 404 means the thing
* genuinely isn't there (cacheable at full TTL), anything else — 429, 5xx, timeout,
* network error — is weather to retry soon, not remember for a day.
*/
167interface FetchOutcome<T> {
  data: T | null
  transient: boolean
170}
171 
172async function fetchJson<T>(url: string): Promise<FetchOutcome<T>> {
  try {
      const res = await fetch(url, { headers: WIKI_HEADERS, signal: AbortSignal.timeout(6000) })
      if (!res.ok) return { data: null, transient: res.status !== 404 }
      return { data: (await res.json()) as T, transient: false }
  } catch {
      return { data: null, transient: true }
  }
180}
181 
182interface WikidataDates {
  born?: FigureYear
  died?: FigureYear
  /** True when we never got a real answer (so the result must not cache long). */
  transient: boolean
187}
188 
189/**
* Birth/death from Wikidata — by item id when the summary supplied one (immune to
* title-matching quirks), by sitelink title otherwise. Rate-limit blips are the
* dominant way dates go missing in practice (a 429 body carries no `entities`), so
* a no-entities response gets one spaced retry before being declared transient.
*/
195async function wikidataDates(qid: string | undefined, title: string): Promise<WikidataDates> {
  const subject = qid
      ? `ids=${encodeURIComponent(qid)}`
      : `sites=enwiki&titles=${encodeURIComponent(title)}&normalize=1`
  const url = `https://www.wikidata.org/w/api.php?action=wbgetentities&${subject}&props=claims&languages=en&format=json&origin=*`
  let outcome = await fetchJson<WikidataEntities>(url)
  if (!outcome.data?.entities) {
      await new Promise(resolve => setTimeout(resolve, WIKIDATA_RETRY_DELAY_MS))
      outcome = await fetchJson<WikidataEntities>(url)
  }
  const entities = outcome.data?.entities
  if (!entities) return { transient: true }
  const id = Object.keys(entities)[0]
  const claims = id ? entities[id]?.claims : undefined
  return { born: pickClaimYear(claims?.P569), died: pickClaimYear(claims?.P570), transient: false }
210}
211 
212/**
* Resolves a name to grounded facts (cached). Never throws; returns
* `{ resolved: false }` when the figure can't be confidently identified.
*/
216export async function groundFigure(rawName: string): Promise<GroundedFigure> {
  const name = (rawName || '').trim()
  if (!name) return { name: '', resolved: false }
219 
  const key = name.toLowerCase()
  const hit = cache.get(key)
  if (hit && hit.expires > Date.now()) return hit.value
223 
  const { value, ttlMs } = await resolve(name)
  cache.set(key, { value, expires: Date.now() + ttlMs })
  return value
227}
228 
229/**
* Keep a URL only if it's a plain http(s) link — drop `javascript:` / `data:` /
* relative values. These come from a remote Wikipedia response and end up bound to
* `:href` / `:src` in the figure dossier; Vue escapes the value but NOT the scheme,
* so a spoofed `javascript:` URL would execute on click. (Surfaced by the
* unsafe-render loop.) Exported for the figure-search route, which binds Wikipedia
* thumbnails the same way.
*/
237export function safeUrl(url?: string): string | undefined {
  if (!url) return undefined
  try {
      const { protocol } = new URL(url)
      return protocol === 'https:' || protocol === 'http:' ? url : undefined
  } catch {
      return undefined
  }
245}
246 
247interface Resolution {
  value: GroundedFigure
  ttlMs: number
250}
251 
252/** A purely AI-estimated death is only trusted as "historical" (and the figure
*  contactable, #72) when it predates this year — a recent estimated death is the
*  least trustworthy signal and could mark a living person dead. Documented deaths
*  are exempt (they're facts). */
256const ESTIMATED_DEATH_HISTORICAL_BEFORE = new Date().getFullYear() - 100
257 
258async function resolve(name: string): Promise<Resolution> {
  const summary = await fetchJson<WikiSummary>(
      `https://en.wikipedia.org/api/rest_v1/page/summary/${encodeURIComponent(name)}`
  )
262 
  // No article, or an ambiguous disambiguation page → can't pin a person.
  const article = summary.data
  if (!article || article.type === 'disambiguation' || !article.title) {
      return {
          value: { name, resolved: false, transient: summary.transient },
          ttlMs: summary.transient ? TRANSIENT_TTL_MS : FULL_TTL_MS
      }
  }
271 
  const title: string = article.title
273 
  const dates = await wikidataDates(article.wikibase_item, title)
  let born = dates.born
  let died = dates.died
  let bridgeTransient = false
278 
  // The AI bridge (issue #31): the record knows WHO they are but not WHEN — either
  // Wikidata genuinely has no usable birth claim, or the lookup is down right now.
  // Estimated years arrive marked circa, so the dossier and prompts stay honest.
  // A recorded death is law: the estimate only fills what the record lacks, and an
  // estimate that contradicts the record (born after the documented death) is
  // dropped whole rather than shipped as an incoherent lifespan.
  if (!born) {
      const bridge = await estimateLifespan(title, article.description, article.extract, died?.display)
      bridgeTransient = bridge.transient
      if (bridge.estimate) {
          const estBorn = makeFigureYear(bridge.estimate.born.year, bridge.estimate.born.bce, true)
          if (!died || estBorn.signed <= died.signed) {
              born = estBorn
              // A purely AI-ESTIMATED death is trusted as "deceased" (the floor in
              // #72) only when it's clearly historical. An LLM guess of a RECENT
              // death is exactly what could mark a living person contactable, so we
              // don't promote it — the figure stays `living` and is blocked. A
              // DOCUMENTED death (already in `died`) is a fact and is kept as-is.
              const estDied = bridge.estimate.died
                  ? makeFigureYear(bridge.estimate.died.year, bridge.estimate.died.bce, true)
                  : undefined
              died = died ?? (estDied && estDied.signed <= ESTIMATED_DEATH_HISTORICAL_BEFORE ? estDied : undefined)
          }
      }
  }
304 
  return {
      value: {
          name: title,
          resolved: true,
          description: article.description || undefined,
          extract: article.extract || undefined,
          born,
          died,
          living: !!born && !died,
          thumbnail: safeUrl(article.thumbnail?.source),
          wikiUrl: safeUrl(article.content_urls?.desktop?.page),
          transient: dates.transient || bridgeTransient
      },
      // Anything short of a definitive answer keeps the short TTL: a transient
      // Wikidata miss (even when the estimator papered over it — the next lookup
      // should get a shot at the real dates) and an estimator outage alike.
      ttlMs: dates.transient || bridgeTransient ? TRANSIENT_TTL_MS : FULL_TTL_MS
  }
323}