tyler
f833ed17a1
Portable guide for an agent doing refactor/code-health work on a raylib (or any GLFW/OpenGL) app: Xvfb + screenshot + pixel-diff loop, xdotool key/mouse injection (incl. the press-release-must-span-frames gotcha), determinism strategies for static apps vs continuously-animating games, temp-keybinding state forcing, and a porting checklist for shot_input.sh. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
230 lines
11 KiB
Markdown
230 lines
11 KiB
Markdown
# Working on a raylib app as an agent: headless run, see, and drive it
|
||
|
||
This is a portable playbook for an AI agent doing code-health / refactor work on a
|
||
**raylib** (or any GLFW/OpenGL) desktop app. The problem it solves: you can edit and
|
||
compile the code, but you can't *see* the window or click it — so how do you prove a
|
||
refactor didn't change behavior, and how do you exercise a UI path?
|
||
|
||
The answer is a tight loop: **run under a virtual display → capture a frame → diff it
|
||
against a baseline → (optionally) inject keyboard/mouse → look at the PNG.** Everything
|
||
below is the accumulated know-how for doing that *efficiently*, plus the traps.
|
||
|
||
> This repo's working reference implementation is [`shot_input.sh`](shot_input.sh).
|
||
> Read it alongside this doc; the "Porting" checklist at the end says what to change.
|
||
|
||
---
|
||
|
||
## 0. The loop in one breath
|
||
|
||
```bash
|
||
Xvfb :99 -screen 0 1280x800x24 & # 1. a fake screen
|
||
DISPLAY=:99 ./bin/Debug/game & # 2. run the app on it
|
||
sleep 2 # 3. let it reach a steady frame
|
||
DISPLAY=:99 import -window root /tmp/a.png # 4. screenshot the root
|
||
compare -metric AE /tmp/base.png /tmp/a.png /dev/null # 5. 0 == pixel-identical
|
||
```
|
||
|
||
If step 5 prints `0`, the render is byte-for-byte what it was before your change. That
|
||
single number is the workhorse of refactor verification.
|
||
|
||
---
|
||
|
||
## 1. Prerequisites (Debian/Ubuntu)
|
||
|
||
```bash
|
||
sudo apt-get install xvfb imagemagick xdotool
|
||
# If OpenGL fails under Xvfb (black frames / GLX errors), add software rendering:
|
||
# sudo apt-get install libgl1-mesa-dri
|
||
# and run the app with LIBGL_ALWAYS_SOFTWARE=1
|
||
```
|
||
|
||
- **Xvfb** — an X server that renders to memory, no monitor. raylib's GL runs on it via
|
||
Mesa's software rasterizer; you do not need a GPU.
|
||
- **ImageMagick** — `import` (grab) + `compare` (diff). On ImageMagick v7 the commands
|
||
are `magick import` / `magick compare`; v6 uses `import` / `compare` directly.
|
||
- **xdotool** — synthesizes keyboard and mouse events into the X server.
|
||
|
||
---
|
||
|
||
## 2. Run headless
|
||
|
||
```bash
|
||
Xvfb :99 -screen 0 1280x800x24 >/tmp/xvfb.log 2>&1 &
|
||
DISPLAY=:99 ./bin/Debug/game [args] >/tmp/app.log 2>&1 &
|
||
```
|
||
|
||
- Match the Xvfb screen size to the app's window so a root grab == the app's frame.
|
||
- There is **no window manager**, so the window sits at `0,0` and fills the screen, and
|
||
X input focus is unmanaged (see §4).
|
||
- Capture stdout/stderr to a log — it's your only window into `TraceLog`/crashes.
|
||
|
||
---
|
||
|
||
## 3. Capture a frame — two ways
|
||
|
||
**A. Grab the X root (zero code change):**
|
||
```bash
|
||
DISPLAY=:99 import -window root /tmp/shot.png
|
||
```
|
||
If your ImageMagick policy blocks `import` ("not authorized"), use option B.
|
||
|
||
**B. raylib's own framebuffer dump (one line of code, more robust):**
|
||
```c
|
||
TakeScreenshot("/tmp/shot.png"); // writes exactly what GL rendered, no chrome/cursor
|
||
```
|
||
Trigger it from a temp keybinding or a `--shoot-after N frames` flag. This bypasses
|
||
ImageMagick entirely and is the exact framebuffer — preferable when you can edit code
|
||
(which, in a refactor task, you already are).
|
||
|
||
Then **look at it**: open the PNG with your image-reading tool. Vision catches "the HUD
|
||
vanished" or "text is now black-on-black" that a pixel count won't explain.
|
||
|
||
---
|
||
|
||
## 4. Inject input (keyboard + mouse)
|
||
|
||
```bash
|
||
WID=$(DISPLAY=:99 xdotool search --name "MyGameWindowTitle" | head -1)
|
||
DISPLAY=:99 xdotool windowactivate "$WID" # give it focus first; keys need this
|
||
DISPLAY=:99 xdotool key space # keys: by keysym (Return, Escape, F1, a, ...)
|
||
DISPLAY=:99 xdotool mousemove 640 360
|
||
DISPLAY=:99 xdotool mousedown 1 ; sleep 0.2 ; xdotool mouseup 1
|
||
```
|
||
|
||
**Both keys and mouse buttons reach GLFW under Xvfb.** (If a guide tells you synthetic
|
||
clicks are "dropped," it's wrong — see the next bullet for what actually bites you.)
|
||
|
||
### The one mouse gotcha that wastes an hour
|
||
A button's **press and release must fall on different frames** of the app's loop.
|
||
raylib detects a click as `current && !previous` (a rising edge). A bare
|
||
`xdotool click 1` presses and releases inside ~1ms — if both land in the same
|
||
`PollInputEvents()` tick, the app sees the button go down *and* up in one frame and the
|
||
rising edge never registers. **Always** do `mousedown 1` → `sleep ≥0.15s` → `mouseup 1`.
|
||
Same for drags: `mousemove A` → `mousedown` → `mousemove B` → `sleep` → `mouseup`.
|
||
|
||
### Lock/modifier keys
|
||
`Scroll_Lock`, `Num_Lock`, `Caps_Lock` get eaten by X as LED/lock toggles and do **not**
|
||
arrive as normal key presses. For temp test hooks use ordinary keys (letters, F-keys);
|
||
`F1`–`F12` are reliable.
|
||
|
||
### Hit the right rectangle
|
||
A click only does something if the pointer is over a live hit-target. Buttons and
|
||
draggable regions are rectangles computed in code — read the layout code and aim at a
|
||
rectangle's center, don't eyeball the screenshot. Many "the click didn't work" dead-ends
|
||
are really "I clicked 10px into the sidebar / a dead margin / below a minimum-drag
|
||
threshold." (This app, for instance, ignores selection drags under 5px.)
|
||
|
||
---
|
||
|
||
## 5. Determinism — the part that decides whether pixel-diff even works
|
||
|
||
`compare -metric AE` is only a *proof* if the same inputs produce the same pixels. Two
|
||
cases:
|
||
|
||
### Mostly-static apps (viewers, editors, tools)
|
||
Once loaded/settled the frame is constant, so two runs of the same binary give `AE 0`.
|
||
Beware **async/background work**: if the app keeps computing after the window appears
|
||
(streaming load, background workers, lazy caches), an early grab is nondeterministic.
|
||
Capture *after it settles* — find the quiet point and add margin (this app needs ~12s
|
||
for a background high-res pass; a 6s grab varied run-to-run and produced scary false
|
||
diffs). Confirm your settle time by grabbing the **same binary twice** and checking the
|
||
diff is 0.
|
||
|
||
### Games and anything continuously animating
|
||
Frames differ every run even with zero code change, so `AE 0` is unattainable and the
|
||
naive gate is useless. Make the test deterministic instead, roughly in this order:
|
||
|
||
1. **Pin the randomness and the clock.** `SetRandomSeed(N)` (raylib) / `srand(N)`; don't
|
||
drive simulation off wall-clock `GetTime()` in test builds.
|
||
2. **Fixed timestep.** Don't advance physics by `GetFrameTime()` (which jitters under
|
||
software GL). In a test mode, step a constant `dt` a fixed number of frames — or add
|
||
a "single-step frame" debug key and advance exactly N steps before the grab. Now
|
||
"frame N" is reproducible → its screenshot is a golden image.
|
||
3. **Freeze and pose.** Add a pause/step debug key. Drive the game with a scripted input
|
||
sequence to a known state, pause, then grab — a paused frame is static and comparable.
|
||
4. **Pick naturally-static screens.** Menus, the level-1 spawn, pause overlays, and
|
||
game-over screens are often deterministic even when gameplay isn't. Script the input
|
||
to land on one of those and diff *that*.
|
||
5. **When you can't fully pin it, measure the noise floor.** Run the same binary twice,
|
||
`compare` the two shots — that AE is the inherent run-to-run variance. A refactor is
|
||
"preserving" if pre-vs-post AE stays within that floor. Use `compare -metric AE -fuzz
|
||
2%` to ignore sub-threshold per-pixel wobble, and/or crop to a stable region.
|
||
|
||
Even when pixel-diff isn't reliable, the screenshot is still worth grabbing: **eyeball
|
||
it** to confirm the refactor didn't break rendering wholesale.
|
||
|
||
---
|
||
|
||
## 6. Reaching states normal input can't
|
||
|
||
To test a code path you can't drive from the keyboard/mouse (an internal reset, a state
|
||
deep in a menu, a value only mouse-wheel-settable), **add a temporary keybinding** that
|
||
forces the state, verify, then remove it. Example pattern:
|
||
|
||
```c
|
||
// TEMP TEST KEYS (remove before commit)
|
||
if (IsKeyPressed(KEY_F2)) { /* force the app into the dirty/edge state */ }
|
||
if (IsKeyPressed(KEY_F3)) { TheFunctionUnderTest(); }
|
||
```
|
||
|
||
Drive F2 then F3 via xdotool, screenshot between, diff. This is how you exercise a
|
||
function end-to-end (e.g. "does load-new-file reset the view?") without the real trigger.
|
||
Build with the temp keys, verify, **delete them, rebuild, and re-confirm the baseline is
|
||
unchanged** before committing.
|
||
|
||
---
|
||
|
||
## 7. A reusable harness
|
||
|
||
Wrap §2–§4 in one script so a whole scenario is a single command. This repo's
|
||
[`shot_input.sh`](shot_input.sh) takes an output path plus a list of actions and runs
|
||
them in order, with `sleep N`, `key NAME`, `click X Y`, `drag X1 Y1 X2 Y2`, and
|
||
`rdrag` (modifier+drag) helpers that already bake in the frame-gap from §4:
|
||
|
||
```bash
|
||
./shot_input.sh /tmp/out.png "sleep 12" "drag 450 120 1000 400" "click 150 200"
|
||
```
|
||
|
||
The win for an agent: one Bash call launches, drives, screenshots, and tears down — no
|
||
multi-turn babysitting of background processes, and the action list reads like a test
|
||
case.
|
||
|
||
### Porting it to another raylib project
|
||
Change these and nothing else:
|
||
- `REPO` and the **run command** (binary path + how it takes/needs a data file).
|
||
- The **window title** in `xdotool search --name` (must match your `InitWindow` title).
|
||
- The **settle `sleep`** (§5) — or drop it to a small value if the app is static at once.
|
||
- The **coordinate cheat-sheet** in the header comment — recompute your own hit-rects
|
||
from the layout code at your window size, or just pass raw `mousemove`/`mousedown`
|
||
actions.
|
||
|
||
---
|
||
|
||
## 8. Refactor verification philosophy
|
||
|
||
- **Pure mechanical renames are compiler-proven.** When you rename a field/function and
|
||
the *old* name ceases to exist, a clean compile means every reference was updated — the
|
||
compiler found any you missed. For these, pixel-diff is a belt-and-suspenders check,
|
||
not the primary safety.
|
||
- **Behavior-changing edits need the visual gate.** Anything that alters control flow,
|
||
values, or layout: establish a baseline screenshot *before* the change, then prove the
|
||
intended pixels changed and nothing else did.
|
||
- **Keep a stable baseline** (`/tmp/base.png`) for the duration of a refactor arc so
|
||
every step diffs against the same known-good frame.
|
||
- **Commit at green points.** Build clean + diff clean = a safe checkpoint.
|
||
|
||
---
|
||
|
||
## 9. Quick-reference: traps that cost time
|
||
|
||
| Symptom | Cause | Fix |
|
||
|---|---|---|
|
||
| Black/empty frames | No GL under Xvfb | `LIBGL_ALWAYS_SOFTWARE=1`, install mesa-dri |
|
||
| `import` "not authorized" | ImageMagick policy | Use raylib `TakeScreenshot()` instead |
|
||
| Keys do nothing | Window not focused | `xdotool windowactivate $WID` first |
|
||
| Click does nothing | Press+release same frame | `mousedown` → `sleep` → `mouseup` |
|
||
| `Scroll_Lock`/`Caps` ignored | X swallows lock keys | Use letters / F-keys for temp hooks |
|
||
| Click "misses" | Wrong rectangle / dead margin / min-drag threshold | Aim at a hit-rect center from layout code |
|
||
| Diff nonzero with no code change | Async load or animation not settled | Settle longer; or determinize (§5) |
|
||
| Diff flaky run-to-run | RNG / wall-clock / variable timestep | Seed RNG, fixed `dt`, measure noise floor |
|