docs: AGENTS.md — headless run/see/drive playbook for raylib apps
Portable guide for an agent doing refactor/code-health work on a raylib (or any GLFW/OpenGL) app: Xvfb + screenshot + pixel-diff loop, xdotool key/mouse injection (incl. the press-release-must-span-frames gotcha), determinism strategies for static apps vs continuously-animating games, temp-keybinding state forcing, and a porting checklist for shot_input.sh. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,229 @@
|
||||
# Working on a raylib app as an agent: headless run, see, and drive it
|
||||
|
||||
This is a portable playbook for an AI agent doing code-health / refactor work on a
|
||||
**raylib** (or any GLFW/OpenGL) desktop app. The problem it solves: you can edit and
|
||||
compile the code, but you can't *see* the window or click it — so how do you prove a
|
||||
refactor didn't change behavior, and how do you exercise a UI path?
|
||||
|
||||
The answer is a tight loop: **run under a virtual display → capture a frame → diff it
|
||||
against a baseline → (optionally) inject keyboard/mouse → look at the PNG.** Everything
|
||||
below is the accumulated know-how for doing that *efficiently*, plus the traps.
|
||||
|
||||
> This repo's working reference implementation is [`shot_input.sh`](shot_input.sh).
|
||||
> Read it alongside this doc; the "Porting" checklist at the end says what to change.
|
||||
|
||||
---
|
||||
|
||||
## 0. The loop in one breath
|
||||
|
||||
```bash
|
||||
Xvfb :99 -screen 0 1280x800x24 & # 1. a fake screen
|
||||
DISPLAY=:99 ./bin/Debug/game & # 2. run the app on it
|
||||
sleep 2 # 3. let it reach a steady frame
|
||||
DISPLAY=:99 import -window root /tmp/a.png # 4. screenshot the root
|
||||
compare -metric AE /tmp/base.png /tmp/a.png /dev/null # 5. 0 == pixel-identical
|
||||
```
|
||||
|
||||
If step 5 prints `0`, the render is byte-for-byte what it was before your change. That
|
||||
single number is the workhorse of refactor verification.
|
||||
|
||||
---
|
||||
|
||||
## 1. Prerequisites (Debian/Ubuntu)
|
||||
|
||||
```bash
|
||||
sudo apt-get install xvfb imagemagick xdotool
|
||||
# If OpenGL fails under Xvfb (black frames / GLX errors), add software rendering:
|
||||
# sudo apt-get install libgl1-mesa-dri
|
||||
# and run the app with LIBGL_ALWAYS_SOFTWARE=1
|
||||
```
|
||||
|
||||
- **Xvfb** — an X server that renders to memory, no monitor. raylib's GL runs on it via
|
||||
Mesa's software rasterizer; you do not need a GPU.
|
||||
- **ImageMagick** — `import` (grab) + `compare` (diff). On ImageMagick v7 the commands
|
||||
are `magick import` / `magick compare`; v6 uses `import` / `compare` directly.
|
||||
- **xdotool** — synthesizes keyboard and mouse events into the X server.
|
||||
|
||||
---
|
||||
|
||||
## 2. Run headless
|
||||
|
||||
```bash
|
||||
Xvfb :99 -screen 0 1280x800x24 >/tmp/xvfb.log 2>&1 &
|
||||
DISPLAY=:99 ./bin/Debug/game [args] >/tmp/app.log 2>&1 &
|
||||
```
|
||||
|
||||
- Match the Xvfb screen size to the app's window so a root grab == the app's frame.
|
||||
- There is **no window manager**, so the window sits at `0,0` and fills the screen, and
|
||||
X input focus is unmanaged (see §4).
|
||||
- Capture stdout/stderr to a log — it's your only window into `TraceLog`/crashes.
|
||||
|
||||
---
|
||||
|
||||
## 3. Capture a frame — two ways
|
||||
|
||||
**A. Grab the X root (zero code change):**
|
||||
```bash
|
||||
DISPLAY=:99 import -window root /tmp/shot.png
|
||||
```
|
||||
If your ImageMagick policy blocks `import` ("not authorized"), use option B.
|
||||
|
||||
**B. raylib's own framebuffer dump (one line of code, more robust):**
|
||||
```c
|
||||
TakeScreenshot("/tmp/shot.png"); // writes exactly what GL rendered, no chrome/cursor
|
||||
```
|
||||
Trigger it from a temp keybinding or a `--shoot-after N frames` flag. This bypasses
|
||||
ImageMagick entirely and is the exact framebuffer — preferable when you can edit code
|
||||
(which, in a refactor task, you already are).
|
||||
|
||||
Then **look at it**: open the PNG with your image-reading tool. Vision catches "the HUD
|
||||
vanished" or "text is now black-on-black" that a pixel count won't explain.
|
||||
|
||||
---
|
||||
|
||||
## 4. Inject input (keyboard + mouse)
|
||||
|
||||
```bash
|
||||
WID=$(DISPLAY=:99 xdotool search --name "MyGameWindowTitle" | head -1)
|
||||
DISPLAY=:99 xdotool windowactivate "$WID" # give it focus first; keys need this
|
||||
DISPLAY=:99 xdotool key space # keys: by keysym (Return, Escape, F1, a, ...)
|
||||
DISPLAY=:99 xdotool mousemove 640 360
|
||||
DISPLAY=:99 xdotool mousedown 1 ; sleep 0.2 ; xdotool mouseup 1
|
||||
```
|
||||
|
||||
**Both keys and mouse buttons reach GLFW under Xvfb.** (If a guide tells you synthetic
|
||||
clicks are "dropped," it's wrong — see the next bullet for what actually bites you.)
|
||||
|
||||
### The one mouse gotcha that wastes an hour
|
||||
A button's **press and release must fall on different frames** of the app's loop.
|
||||
raylib detects a click as `current && !previous` (a rising edge). A bare
|
||||
`xdotool click 1` presses and releases inside ~1ms — if both land in the same
|
||||
`PollInputEvents()` tick, the app sees the button go down *and* up in one frame and the
|
||||
rising edge never registers. **Always** do `mousedown 1` → `sleep ≥0.15s` → `mouseup 1`.
|
||||
Same for drags: `mousemove A` → `mousedown` → `mousemove B` → `sleep` → `mouseup`.
|
||||
|
||||
### Lock/modifier keys
|
||||
`Scroll_Lock`, `Num_Lock`, `Caps_Lock` get eaten by X as LED/lock toggles and do **not**
|
||||
arrive as normal key presses. For temp test hooks use ordinary keys (letters, F-keys);
|
||||
`F1`–`F12` are reliable.
|
||||
|
||||
### Hit the right rectangle
|
||||
A click only does something if the pointer is over a live hit-target. Buttons and
|
||||
draggable regions are rectangles computed in code — read the layout code and aim at a
|
||||
rectangle's center, don't eyeball the screenshot. Many "the click didn't work" dead-ends
|
||||
are really "I clicked 10px into the sidebar / a dead margin / below a minimum-drag
|
||||
threshold." (This app, for instance, ignores selection drags under 5px.)
|
||||
|
||||
---
|
||||
|
||||
## 5. Determinism — the part that decides whether pixel-diff even works
|
||||
|
||||
`compare -metric AE` is only a *proof* if the same inputs produce the same pixels. Two
|
||||
cases:
|
||||
|
||||
### Mostly-static apps (viewers, editors, tools)
|
||||
Once loaded/settled the frame is constant, so two runs of the same binary give `AE 0`.
|
||||
Beware **async/background work**: if the app keeps computing after the window appears
|
||||
(streaming load, background workers, lazy caches), an early grab is nondeterministic.
|
||||
Capture *after it settles* — find the quiet point and add margin (this app needs ~12s
|
||||
for a background high-res pass; a 6s grab varied run-to-run and produced scary false
|
||||
diffs). Confirm your settle time by grabbing the **same binary twice** and checking the
|
||||
diff is 0.
|
||||
|
||||
### Games and anything continuously animating
|
||||
Frames differ every run even with zero code change, so `AE 0` is unattainable and the
|
||||
naive gate is useless. Make the test deterministic instead, roughly in this order:
|
||||
|
||||
1. **Pin the randomness and the clock.** `SetRandomSeed(N)` (raylib) / `srand(N)`; don't
|
||||
drive simulation off wall-clock `GetTime()` in test builds.
|
||||
2. **Fixed timestep.** Don't advance physics by `GetFrameTime()` (which jitters under
|
||||
software GL). In a test mode, step a constant `dt` a fixed number of frames — or add
|
||||
a "single-step frame" debug key and advance exactly N steps before the grab. Now
|
||||
"frame N" is reproducible → its screenshot is a golden image.
|
||||
3. **Freeze and pose.** Add a pause/step debug key. Drive the game with a scripted input
|
||||
sequence to a known state, pause, then grab — a paused frame is static and comparable.
|
||||
4. **Pick naturally-static screens.** Menus, the level-1 spawn, pause overlays, and
|
||||
game-over screens are often deterministic even when gameplay isn't. Script the input
|
||||
to land on one of those and diff *that*.
|
||||
5. **When you can't fully pin it, measure the noise floor.** Run the same binary twice,
|
||||
`compare` the two shots — that AE is the inherent run-to-run variance. A refactor is
|
||||
"preserving" if pre-vs-post AE stays within that floor. Use `compare -metric AE -fuzz
|
||||
2%` to ignore sub-threshold per-pixel wobble, and/or crop to a stable region.
|
||||
|
||||
Even when pixel-diff isn't reliable, the screenshot is still worth grabbing: **eyeball
|
||||
it** to confirm the refactor didn't break rendering wholesale.
|
||||
|
||||
---
|
||||
|
||||
## 6. Reaching states normal input can't
|
||||
|
||||
To test a code path you can't drive from the keyboard/mouse (an internal reset, a state
|
||||
deep in a menu, a value only mouse-wheel-settable), **add a temporary keybinding** that
|
||||
forces the state, verify, then remove it. Example pattern:
|
||||
|
||||
```c
|
||||
// TEMP TEST KEYS (remove before commit)
|
||||
if (IsKeyPressed(KEY_F2)) { /* force the app into the dirty/edge state */ }
|
||||
if (IsKeyPressed(KEY_F3)) { TheFunctionUnderTest(); }
|
||||
```
|
||||
|
||||
Drive F2 then F3 via xdotool, screenshot between, diff. This is how you exercise a
|
||||
function end-to-end (e.g. "does load-new-file reset the view?") without the real trigger.
|
||||
Build with the temp keys, verify, **delete them, rebuild, and re-confirm the baseline is
|
||||
unchanged** before committing.
|
||||
|
||||
---
|
||||
|
||||
## 7. A reusable harness
|
||||
|
||||
Wrap §2–§4 in one script so a whole scenario is a single command. This repo's
|
||||
[`shot_input.sh`](shot_input.sh) takes an output path plus a list of actions and runs
|
||||
them in order, with `sleep N`, `key NAME`, `click X Y`, `drag X1 Y1 X2 Y2`, and
|
||||
`rdrag` (modifier+drag) helpers that already bake in the frame-gap from §4:
|
||||
|
||||
```bash
|
||||
./shot_input.sh /tmp/out.png "sleep 12" "drag 450 120 1000 400" "click 150 200"
|
||||
```
|
||||
|
||||
The win for an agent: one Bash call launches, drives, screenshots, and tears down — no
|
||||
multi-turn babysitting of background processes, and the action list reads like a test
|
||||
case.
|
||||
|
||||
### Porting it to another raylib project
|
||||
Change these and nothing else:
|
||||
- `REPO` and the **run command** (binary path + how it takes/needs a data file).
|
||||
- The **window title** in `xdotool search --name` (must match your `InitWindow` title).
|
||||
- The **settle `sleep`** (§5) — or drop it to a small value if the app is static at once.
|
||||
- The **coordinate cheat-sheet** in the header comment — recompute your own hit-rects
|
||||
from the layout code at your window size, or just pass raw `mousemove`/`mousedown`
|
||||
actions.
|
||||
|
||||
---
|
||||
|
||||
## 8. Refactor verification philosophy
|
||||
|
||||
- **Pure mechanical renames are compiler-proven.** When you rename a field/function and
|
||||
the *old* name ceases to exist, a clean compile means every reference was updated — the
|
||||
compiler found any you missed. For these, pixel-diff is a belt-and-suspenders check,
|
||||
not the primary safety.
|
||||
- **Behavior-changing edits need the visual gate.** Anything that alters control flow,
|
||||
values, or layout: establish a baseline screenshot *before* the change, then prove the
|
||||
intended pixels changed and nothing else did.
|
||||
- **Keep a stable baseline** (`/tmp/base.png`) for the duration of a refactor arc so
|
||||
every step diffs against the same known-good frame.
|
||||
- **Commit at green points.** Build clean + diff clean = a safe checkpoint.
|
||||
|
||||
---
|
||||
|
||||
## 9. Quick-reference: traps that cost time
|
||||
|
||||
| Symptom | Cause | Fix |
|
||||
|---|---|---|
|
||||
| Black/empty frames | No GL under Xvfb | `LIBGL_ALWAYS_SOFTWARE=1`, install mesa-dri |
|
||||
| `import` "not authorized" | ImageMagick policy | Use raylib `TakeScreenshot()` instead |
|
||||
| Keys do nothing | Window not focused | `xdotool windowactivate $WID` first |
|
||||
| Click does nothing | Press+release same frame | `mousedown` → `sleep` → `mouseup` |
|
||||
| `Scroll_Lock`/`Caps` ignored | X swallows lock keys | Use letters / F-keys for temp hooks |
|
||||
| Click "misses" | Wrong rectangle / dead margin / min-drag threshold | Aim at a hit-rect center from layout code |
|
||||
| Diff nonzero with no code change | Async load or animation not settled | Settle longer; or determinize (§5) |
|
||||
| Diff flaky run-to-run | RNG / wall-clock / variable timestep | Seed RNG, fixed `dt`, measure noise floor |
|
||||
Reference in New Issue
Block a user