rspektrum/AGENTS.md

Working on a raylib app as an agent: headless run, see, and drive it

This is a portable playbook for an AI agent doing code-health / refactor work on a raylib (or any GLFW/OpenGL) desktop app. The problem it solves: you can edit and compile the code, but you can't see the window or click it — so how do you prove a refactor didn't change behavior, and how do you exercise a UI path?

The answer is a tight loop: run under a virtual display → capture a frame → diff it against a baseline → (optionally) inject keyboard/mouse → look at the PNG. Everything below is the accumulated know-how for doing that efficiently, plus the traps.

This repo's working reference implementation is shot_input.sh. Read it alongside this doc; the "Porting" checklist at the end says what to change.

---

0. The loop in one breath

Xvfb :99 -screen 0 1280x800x24 &              # 1. a fake screen
DISPLAY=:99 ./bin/Debug/game &                # 2. run the app on it
sleep 2                                        # 3. let it reach a steady frame
DISPLAY=:99 import -window root /tmp/a.png     # 4. screenshot the root
compare -metric AE /tmp/base.png /tmp/a.png /dev/null   # 5. 0 == pixel-identical

If step 5 prints 0, the render is byte-for-byte what it was before your change. That single number is the workhorse of refactor verification.

---

1. Prerequisites (Debian/Ubuntu)

sudo apt-get install xvfb imagemagick xdotool
# If OpenGL fails under Xvfb (black frames / GLX errors), add software rendering:
#   sudo apt-get install libgl1-mesa-dri
#   and run the app with  LIBGL_ALWAYS_SOFTWARE=1

• Xvfb — an X server that renders to memory, no monitor. raylib's GL runs on it via Mesa's software rasterizer; you do not need a GPU.
• ImageMagick — import (grab) + compare (diff). On ImageMagick v7 the commands are magick import / magick compare; v6 uses import / compare directly.
• xdotool — synthesizes keyboard and mouse events into the X server.

---

2. Run headless

Xvfb :99 -screen 0 1280x800x24 >/tmp/xvfb.log 2>&1 &
DISPLAY=:99 ./bin/Debug/game [args] >/tmp/app.log 2>&1 &

• Match the Xvfb screen size to the app's window so a root grab == the app's frame.
• There is no window manager, so the window sits at 0,0 and fills the screen, and X input focus is unmanaged (see §4).
• Capture stdout/stderr to a log — it's your only window into TraceLog/crashes.

---

3. Capture a frame — two ways

A. Grab the X root (zero code change):

DISPLAY=:99 import -window root /tmp/shot.png

If your ImageMagick policy blocks import ("not authorized"), use option B.

B. raylib's own framebuffer dump (one line of code, more robust):

TakeScreenshot("/tmp/shot.png");   // writes exactly what GL rendered, no chrome/cursor

Trigger it from a temp keybinding or a --shoot-after N frames flag. This bypasses ImageMagick entirely and is the exact framebuffer — preferable when you can edit code (which, in a refactor task, you already are).

Then look at it: open the PNG with your image-reading tool. Vision catches "the HUD vanished" or "text is now black-on-black" that a pixel count won't explain.

---

4. Inject input (keyboard + mouse)

WID=$(DISPLAY=:99 xdotool search --name "MyGameWindowTitle" | head -1)
DISPLAY=:99 xdotool windowactivate "$WID"   # give it focus first; keys need this
DISPLAY=:99 xdotool key space               # keys: by keysym (Return, Escape, F1, a, ...)
DISPLAY=:99 xdotool mousemove 640 360
DISPLAY=:99 xdotool mousedown 1 ; sleep 0.2 ; xdotool mouseup 1

Both keys and mouse buttons reach GLFW under Xvfb. (If a guide tells you synthetic clicks are "dropped," it's wrong — see the next bullet for what actually bites you.)

The one mouse gotcha that wastes an hour

A button's press and release must fall on different frames of the app's loop. raylib detects a click as current && !previous (a rising edge). A bare xdotool click 1 presses and releases inside ~1ms — if both land in the same PollInputEvents() tick, the app sees the button go down and up in one frame and the rising edge never registers. Always do mousedown 1 → sleep ≥0.15s → mouseup 1. Same for drags: mousemove A → mousedown → mousemove B → sleep → mouseup.

Lock/modifier keys

Scroll_Lock, Num_Lock, Caps_Lock get eaten by X as LED/lock toggles and do not arrive as normal key presses. For temp test hooks use ordinary keys (letters, F-keys); F1–F12 are reliable.

Hit the right rectangle

A click only does something if the pointer is over a live hit-target. Buttons and draggable regions are rectangles computed in code — read the layout code and aim at a rectangle's center, don't eyeball the screenshot. Many "the click didn't work" dead-ends are really "I clicked 10px into the sidebar / a dead margin / below a minimum-drag threshold." (This app, for instance, ignores selection drags under 5px.)

---

5. Determinism — the part that decides whether pixel-diff even works

compare -metric AE is only a proof if the same inputs produce the same pixels. Two cases:

Mostly-static apps (viewers, editors, tools)

Once loaded/settled the frame is constant, so two runs of the same binary give AE 0. Beware async/background work: if the app keeps computing after the window appears (streaming load, background workers, lazy caches), an early grab is nondeterministic. Capture after it settles — find the quiet point and add margin (this app needs ~12s for a background high-res pass; a 6s grab varied run-to-run and produced scary false diffs). Confirm your settle time by grabbing the same binary twice and checking the diff is 0.

Games and anything continuously animating

Frames differ every run even with zero code change, so AE 0 is unattainable and the naive gate is useless. Make the test deterministic instead, roughly in this order:

1. Pin the randomness and the clock. SetRandomSeed(N) (raylib) / srand(N); don't drive simulation off wall-clock GetTime() in test builds.
2. Fixed timestep. Don't advance physics by GetFrameTime() (which jitters under software GL). In a test mode, step a constant dt a fixed number of frames — or add a "single-step frame" debug key and advance exactly N steps before the grab. Now "frame N" is reproducible → its screenshot is a golden image.
3. Freeze and pose. Add a pause/step debug key. Drive the game with a scripted input sequence to a known state, pause, then grab — a paused frame is static and comparable.
4. Pick naturally-static screens. Menus, the level-1 spawn, pause overlays, and game-over screens are often deterministic even when gameplay isn't. Script the input to land on one of those and diff that.
5. When you can't fully pin it, measure the noise floor. Run the same binary twice, compare the two shots — that AE is the inherent run-to-run variance. A refactor is "preserving" if pre-vs-post AE stays within that floor. Use compare -metric AE -fuzz 2% to ignore sub-threshold per-pixel wobble, and/or crop to a stable region.

Even when pixel-diff isn't reliable, the screenshot is still worth grabbing: eyeball it to confirm the refactor didn't break rendering wholesale.

---

6. Reaching states normal input can't

To test a code path you can't drive from the keyboard/mouse (an internal reset, a state deep in a menu, a value only mouse-wheel-settable), add a temporary keybinding that forces the state, verify, then remove it. Example pattern:

// TEMP TEST KEYS (remove before commit)
if (IsKeyPressed(KEY_F2)) { /* force the app into the dirty/edge state */ }
if (IsKeyPressed(KEY_F3)) { TheFunctionUnderTest(); }

Drive F2 then F3 via xdotool, screenshot between, diff. This is how you exercise a function end-to-end (e.g. "does load-new-file reset the view?") without the real trigger. Build with the temp keys, verify, delete them, rebuild, and re-confirm the baseline is unchanged before committing.

---

7. A reusable harness

Wrap §2–§4 in one script so a whole scenario is a single command. This repo's shot_input.sh takes an output path plus a list of actions and runs them in order, with sleep N, key NAME, click X Y, drag X1 Y1 X2 Y2, and rdrag (modifier+drag) helpers that already bake in the frame-gap from §4:

./shot_input.sh /tmp/out.png "sleep 12" "drag 450 120 1000 400" "click 150 200"

The win for an agent: one Bash call launches, drives, screenshots, and tears down — no multi-turn babysitting of background processes, and the action list reads like a test case.

Porting it to another raylib project

Change these and nothing else:

• REPO and the run command (binary path + how it takes/needs a data file).
• The window title in xdotool search --name (must match your InitWindow title).
• The settle sleep (§5) — or drop it to a small value if the app is static at once.
• The coordinate cheat-sheet in the header comment — recompute your own hit-rects from the layout code at your window size, or just pass raw mousemove/mousedown actions.

---

8. Refactor verification philosophy

• Pure mechanical renames are compiler-proven. When you rename a field/function and the old name ceases to exist, a clean compile means every reference was updated — the compiler found any you missed. For these, pixel-diff is a belt-and-suspenders check, not the primary safety.
• Behavior-changing edits need the visual gate. Anything that alters control flow, values, or layout: establish a baseline screenshot before the change, then prove the intended pixels changed and nothing else did.
• Keep a stable baseline (/tmp/base.png) for the duration of a refactor arc so every step diffs against the same known-good frame.
• Commit at green points. Build clean + diff clean = a safe checkpoint.

---

9. Quick-reference: traps that cost time

Symptom	Cause	Fix
Black/empty frames	No GL under Xvfb	LIBGL_ALWAYS_SOFTWARE=1, install mesa-dri
import "not authorized"	ImageMagick policy	Use raylib TakeScreenshot() instead
Keys do nothing	Window not focused	xdotool windowactivate $WID first
Click does nothing	Press+release same frame	mousedown → sleep → mouseup
Scroll_Lock/Caps ignored	X swallows lock keys	Use letters / F-keys for temp hooks
Click "misses"	Wrong rectangle / dead margin / min-drag threshold	Aim at a hit-rect center from layout code
Diff nonzero with no code change	Async load or animation not settled	Settle longer; or determinize (§5)
Diff flaky run-to-run	RNG / wall-clock / variable timestep	Seed RNG, fixed dt, measure noise floor