Friday: game awareness spec

Ground truth for the plan: how the setup should behave. Updated 2026-10-19. Status: building; steps 1 and 3 done (detector numbers below), step 2 next. Replaces the architecture in research/07.

Pipeline

codeLunaSol

Code does all the routine seeing. Luna reads words and thinks; it sees an image only when code is unsure or Friday asks.

Ledger and World

Ledger: the recordEvery fact anyone reports, whether seen, read, heard or guessed, plus every chat and voice line, with its game time, source and confidence. Only ever added to.

World: the pictureOne entry per champion, objective and camp, plus the last few minutes of talk, rebuilt from the ledger on every new fact: latest values, duplicates merged, conflicts settled, old guesses dropped. Friday, triggers and the Analyst read only this.

Like a bank statement and its balance. Example, 9:38 to 10:00:

Ledger, by game time

9:38	minimap	Nocturne seen, their raptors	.97
9:41	voice	John: "Cho flashed"	line
9:41	analyst	Cho'Gath Flash used, from John	.7
9:43	friday	Cho'Gath Flash used, you told it	.9
9:44	chat	Cho'Gath Flash back 14:41	1
9:46	minimap	Nocturne seen, top river	.95
9:52	api	Nocturne level 6	1
9:58	analyst	Nocturne likely in our top jungle, until 10:18	.6

→

World at 10:00

Nocturnelevel 6, seen top river 14 s ago
trail: their raptors 9:38, top river 9:46
likely our top jungle, guess, until 10:18

Cho'GathFlash down until 14:41, 3 sources, sure

What Friday reads

Nocturne L6: top river, 14 s ago
  from their raptors 9:38
  likely our top jungle (guess)
Cho'Gath: no Flash until 14:41

Writers only add entries, so code readers, the Analyst and Friday can't overwrite each other. One set of rules builds the World.
Facts arrive late and out of order: John's 9:41 line becomes a fact when the Analyst runs. Ordering by game time puts it in place.
History comes free: trails, "unseen for" and "Flash used at" are read off the ledger.
Every World value knows the entries behind it, so a wrong call can be traced. Replays rebuild the World from a logged ledger, so rules can be tuned offline.

One minute of activity

How fresh the World is

From a minimap blip to Friday speaking. Targets, to measure once built.

0.5 s

Sol + voice ~3.5 s

Positions reach the World within half a second of showing on the minimap.
The Analyst's guesses about unseen enemies lag ~10 s. They're guesses, so that's fine.

Who writes what

Store rules

The ledger is append-only. Nothing is edited or deleted.
One writer process, the coach. Readers and agents return facts; the coach appends them.
Order and staleness use t, the game time of the evidence, not arrival.
Same fact from several sources merges: sources unioned, confidence up.
Conflicts follow the ladder. A fresh fact beats an expired one. Unresolved conflicts are shown to Friday.
Corrections reference what they fix (corrects). Dedupe by key.

Precedence, by evidence

api

screen

chat

voice

inferred

// ledger entry
{ seq, t, source, kind, subject, data, conf, evidence, key?, corrects?, expires? }
// subject  "enemy:Chogath" | "ally:Brand" | "obj:dragon" | "camp:their-raptors"

// World champion (all 10)
{ team, role, level, alive, respawnAt?, vitals?{hp,mana,t}, ult,
  spells{name: ready | {backAt,sources} | unknown},
  seen?{zone,x,y,t}, trail[{t,zone}], likely?{zone,conf,why,t} }

Minimap detector

Code, every frame (2/s). No model.

Native crop: 320 px; icons 26 px, a 2 px ring over ~24 px of portrait. No upscaling.
Templates: each champion's iconCircle from its skin's data (CommunityDragon skinN.bin.json). Many skins reuse the base circle.
Match the 10 portraits: correlation over the inner disc, coarse at half size, refined in colour. Each champion placed once. ~40 ms a frame.
Against false hits: the team's ring colour (blue allies, red enemies) unless the portrait match is strong; the patch must not look like the empty map (median of a game's frames); a weak match must beat the other champions clearly.
Tracking: each champion is searched near its spot 0.5 s ago with a lower bar (allies only; enemies vanish into fog, so they keep the ring rule). One that vanishes under another icon stays there as stacked (up to 8 s; Yuumi follows her ally).
Capture: League's own window, so another window over the corner can't hide the minimap. Frames that still aren't the minimap are dropped by comparing with a reference map.
Places: map units from the pixel, then lane, river, jungle quarter or base, and the camp or pit within ~15 px, said from our side ("their raptors", "dragon pit").
Pass bar, ~50 labelled frames: ≥95% of icons right, no invented enemies. Met on the first game: 52 frames reviewed by eye, 268 visible icons, 97.4% right, 0 invented enemies, 2 wrong ally placements. The labels started from the detector's output, so a second game should confirm.
Same game against the API (allies, 333 frames): found 91%, found or stacked 95%.
Next: camp and epic-monster state from the icons at fixed spots.

Friday

The only Sol agent. Proactive on triggers, reactive on questions.

Per-call part of the prompt, target tokens

World render 1.8k

talk 0.5k

events

Talk: last 2 min. Events: last 3 min. Last segment: the trigger.

World render: ages on everything, everyone's trails for the last 3 min.
Writes with record({kind, subject, data, say?}); the turn ends after the call. say only when addressed directly.
Images only through Look. If Sol is unavailable, triggers speak plain lines.

Analyst

Luna, text only. The only routine model call besides Friday.

Runs every ~10 s if the World changed, and within 3 s of a new chat or voice line.
In: World render, new lines, its last guesses. No images.
Out: facts from words ("Cho flashed", "going drake"), where unseen enemies probably are, the plan, worth a call.

Look

Luna with a small image. Rare.

The detector's unsure spot: a ~64 px crop with the candidate icons. Answer: which champions, how many.
Friday's question about the screen: one downscaled frame.

Triggers

Code, on every World change. The periodic check stays as a backstop.

enemy jungler on our side, near you, near our objective objective ≤60 s + lane priority + who's alive you in their jungle, ≥2 enemies missing fight startedfight ended kill window: lane enemy's Flash down, jungler near summoner or ult back soon analyst: worth a call

Model calls per 30-minute game

Code: capture, minimap detector, ally frames, chat OCR, speech-to-text, game API, World, triggers.
Luna: Analyst and Look. Medium effort, standard tier, never Fast.
Sol: Friday only, for both proactive lines and answers.
Images per game: ~30 small crops.
Left out: your own HUD. You know your cooldowns.
No rationing: run until the 5-hour window is empty, then triggers speak plain lines. Cost per call from token counts (codex.call).

Build order

1Record frames ✓

2Ledger, World, record

3Minimap detector, labelled test ✓

4Triggers, Friday on the World

5Analyst

6Ally frames, chat OCR

7Look

Open

Stacked icons: ~4% of ally sightings in the first game are only "stacked". Is "group of 3 near drake" enough without Look?
Briar-style red halos (and big pings) hide an enemy's ring; the detector then needs a strong portrait match. Watch how often enemies are missed in fights.
Fast for Friday's direct answers: keep or drop?
When the 5-hour window is empty: spend credits or stop?

Game settings

chat timestamps on 2048×1152 fullscreen minimap 50% (MinimapScale 1.5), not flipped: still ~320 px, rect in src/capture/minimap.json Tab spell posts read "16:49 Name (Aphelios): Warwick Flash": no timer, code adds the cooldown to the chat time

Source: research/09-awareness-agents.html in the repo. Publish with research/deploy-spec.sh.