# Working on a raylib app as an agent: headless run, see, and drive it

This is a portable playbook for an AI agent doing code-health / refactor work on a
**raylib** (or any GLFW/OpenGL) desktop app. The problem it solves: you can edit and
compile the code, but you can't *see* the window or click it — so how do you prove a
refactor didn't change behavior, and how do you exercise a UI path?

The answer is a tight loop: **run under a virtual display → capture a frame → diff it
against a baseline → (optionally) inject keyboard/mouse → look at the PNG.** Everything
below is the accumulated know-how for doing that *efficiently*, plus the traps.

> This repo's working reference implementation is [`shot_input.sh`](shot_input.sh).
> Read it alongside this doc; the "Porting" checklist at the end says what to change.

---

## 0. The loop in one breath

```bash
Xvfb :99 -screen 0 1280x800x24 &              # 1. a fake screen
DISPLAY=:99 ./bin/Debug/game &                # 2. run the app on it
sleep 2                                        # 3. let it reach a steady frame
DISPLAY=:99 import -window root /tmp/a.png     # 4. screenshot the root
compare -metric AE /tmp/base.png /tmp/a.png /dev/null   # 5. 0 == pixel-identical
```

If step 5 prints `0`, the render is byte-for-byte what it was before your change. That
single number is the workhorse of refactor verification.

---

## 1. Prerequisites (Debian/Ubuntu)

```bash
sudo apt-get install xvfb imagemagick xdotool
# If OpenGL fails under Xvfb (black frames / GLX errors), add software rendering:
#   sudo apt-get install libgl1-mesa-dri
#   and run the app with  LIBGL_ALWAYS_SOFTWARE=1
```

- **Xvfb** — an X server that renders to memory, no monitor. raylib's GL runs on it via
  Mesa's software rasterizer; you do not need a GPU.
- **ImageMagick** — `import` (grab) + `compare` (diff). On ImageMagick v7 the commands
  are `magick import` / `magick compare`; v6 uses `import` / `compare` directly.
- **xdotool** — synthesizes keyboard and mouse events into the X server.

---

## 2. Run headless

```bash
Xvfb :99 -screen 0 1280x800x24 >/tmp/xvfb.log 2>&1 &
DISPLAY=:99 ./bin/Debug/game [args] >/tmp/app.log 2>&1 &
```

- Match the Xvfb screen size to the app's window so a root grab == the app's frame.
- There is **no window manager**, so the window sits at `0,0` and fills the screen, and
  X input focus is unmanaged (see §4).
- Capture stdout/stderr to a log — it's your only window into `TraceLog`/crashes.

---

## 3. Capture a frame — two ways

**A. Grab the X root (zero code change):**
```bash
DISPLAY=:99 import -window root /tmp/shot.png
```
If your ImageMagick policy blocks `import` ("not authorized"), use option B.

**B. raylib's own framebuffer dump (one line of code, more robust):**
```c
TakeScreenshot("/tmp/shot.png");   // writes exactly what GL rendered, no chrome/cursor
```
Trigger it from a temp keybinding or a `--shoot-after N frames` flag. This bypasses
ImageMagick entirely and is the exact framebuffer — preferable when you can edit code
(which, in a refactor task, you already are).

Then **look at it**: open the PNG with your image-reading tool. Vision catches "the HUD
vanished" or "text is now black-on-black" that a pixel count won't explain.

---

## 4. Inject input (keyboard + mouse)

```bash
WID=$(DISPLAY=:99 xdotool search --name "MyGameWindowTitle" | head -1)
DISPLAY=:99 xdotool windowactivate "$WID"   # give it focus first; keys need this
DISPLAY=:99 xdotool key space               # keys: by keysym (Return, Escape, F1, a, ...)
DISPLAY=:99 xdotool mousemove 640 360
DISPLAY=:99 xdotool mousedown 1 ; sleep 0.2 ; xdotool mouseup 1
```

**Both keys and mouse buttons reach GLFW under Xvfb.** (If a guide tells you synthetic
clicks are "dropped," it's wrong — see the next bullet for what actually bites you.)

### The one mouse gotcha that wastes an hour
A button's **press and release must fall on different frames** of the app's loop.
raylib detects a click as `current && !previous` (a rising edge). A bare
`xdotool click 1` presses and releases inside ~1ms — if both land in the same
`PollInputEvents()` tick, the app sees the button go down *and* up in one frame and the
rising edge never registers. **Always** do `mousedown 1` → `sleep ≥0.15s` → `mouseup 1`.
Same for drags: `mousemove A` → `mousedown` → `mousemove B` → `sleep` → `mouseup`.

### Lock/modifier keys
`Scroll_Lock`, `Num_Lock`, `Caps_Lock` get eaten by X as LED/lock toggles and do **not**
arrive as normal key presses. For temp test hooks use ordinary keys (letters, F-keys);
`F1`–`F12` are reliable.

### Hit the right rectangle
A click only does something if the pointer is over a live hit-target. Buttons and
draggable regions are rectangles computed in code — read the layout code and aim at a
rectangle's center, don't eyeball the screenshot. Many "the click didn't work" dead-ends
are really "I clicked 10px into the sidebar / a dead margin / below a minimum-drag
threshold." (This app, for instance, ignores selection drags under 5px.)

---

## 5. Determinism — the part that decides whether pixel-diff even works

`compare -metric AE` is only a *proof* if the same inputs produce the same pixels. Two
cases:

### Mostly-static apps (viewers, editors, tools)
Once loaded/settled the frame is constant, so two runs of the same binary give `AE 0`.
Beware **async/background work**: if the app keeps computing after the window appears
(streaming load, background workers, lazy caches), an early grab is nondeterministic.
Capture *after it settles* — find the quiet point and add margin (this app needs ~12s
for a background high-res pass; a 6s grab varied run-to-run and produced scary false
diffs). Confirm your settle time by grabbing the **same binary twice** and checking the
diff is 0.

### Games and anything continuously animating
Frames differ every run even with zero code change, so `AE 0` is unattainable and the
naive gate is useless. Make the test deterministic instead, roughly in this order:

1. **Pin the randomness and the clock.** `SetRandomSeed(N)` (raylib) / `srand(N)`; don't
   drive simulation off wall-clock `GetTime()` in test builds.
2. **Fixed timestep.** Don't advance physics by `GetFrameTime()` (which jitters under
   software GL). In a test mode, step a constant `dt` a fixed number of frames — or add
   a "single-step frame" debug key and advance exactly N steps before the grab. Now
   "frame N" is reproducible → its screenshot is a golden image.
3. **Freeze and pose.** Add a pause/step debug key. Drive the game with a scripted input
   sequence to a known state, pause, then grab — a paused frame is static and comparable.
4. **Pick naturally-static screens.** Menus, the level-1 spawn, pause overlays, and
   game-over screens are often deterministic even when gameplay isn't. Script the input
   to land on one of those and diff *that*.
5. **When you can't fully pin it, measure the noise floor.** Run the same binary twice,
   `compare` the two shots — that AE is the inherent run-to-run variance. A refactor is
   "preserving" if pre-vs-post AE stays within that floor. Use `compare -metric AE -fuzz
   2%` to ignore sub-threshold per-pixel wobble, and/or crop to a stable region.

Even when pixel-diff isn't reliable, the screenshot is still worth grabbing: **eyeball
it** to confirm the refactor didn't break rendering wholesale.

---

## 6. Reaching states normal input can't

To test a code path you can't drive from the keyboard/mouse (an internal reset, a state
deep in a menu, a value only mouse-wheel-settable), **add a temporary keybinding** that
forces the state, verify, then remove it. Example pattern:

```c
// TEMP TEST KEYS (remove before commit)
if (IsKeyPressed(KEY_F2)) { /* force the app into the dirty/edge state */ }
if (IsKeyPressed(KEY_F3)) { TheFunctionUnderTest(); }
```

Drive F2 then F3 via xdotool, screenshot between, diff. This is how you exercise a
function end-to-end (e.g. "does load-new-file reset the view?") without the real trigger.
Build with the temp keys, verify, **delete them, rebuild, and re-confirm the baseline is
unchanged** before committing.

---

## 7. A reusable harness

Wrap §2–§4 in one script so a whole scenario is a single command. This repo's
[`shot_input.sh`](shot_input.sh) takes an output path plus a list of actions and runs
them in order, with `sleep N`, `key NAME`, `click X Y`, `drag X1 Y1 X2 Y2`, and
`rdrag` (modifier+drag) helpers that already bake in the frame-gap from §4:

```bash
./shot_input.sh /tmp/out.png "sleep 12" "drag 450 120 1000 400" "click 150 200"
```

The win for an agent: one Bash call launches, drives, screenshots, and tears down — no
multi-turn babysitting of background processes, and the action list reads like a test
case.

### Porting it to another raylib project
Change these and nothing else:
- `REPO` and the **run command** (binary path + how it takes/needs a data file).
- The **window title** in `xdotool search --name` (must match your `InitWindow` title).
- The **settle `sleep`** (§5) — or drop it to a small value if the app is static at once.
- The **coordinate cheat-sheet** in the header comment — recompute your own hit-rects
  from the layout code at your window size, or just pass raw `mousemove`/`mousedown`
  actions.

---

## 8. Refactor verification philosophy

- **Pure mechanical renames are compiler-proven.** When you rename a field/function and
  the *old* name ceases to exist, a clean compile means every reference was updated — the
  compiler found any you missed. For these, pixel-diff is a belt-and-suspenders check,
  not the primary safety.
- **Behavior-changing edits need the visual gate.** Anything that alters control flow,
  values, or layout: establish a baseline screenshot *before* the change, then prove the
  intended pixels changed and nothing else did.
- **Keep a stable baseline** (`/tmp/base.png`) for the duration of a refactor arc so
  every step diffs against the same known-good frame.
- **Commit at green points.** Build clean + diff clean = a safe checkpoint.

---

## 9. Quick-reference: traps that cost time

| Symptom | Cause | Fix |
|---|---|---|
| Black/empty frames | No GL under Xvfb | `LIBGL_ALWAYS_SOFTWARE=1`, install mesa-dri |
| `import` "not authorized" | ImageMagick policy | Use raylib `TakeScreenshot()` instead |
| Keys do nothing | Window not focused | `xdotool windowactivate $WID` first |
| Click does nothing | Press+release same frame | `mousedown` → `sleep` → `mouseup` |
| `Scroll_Lock`/`Caps` ignored | X swallows lock keys | Use letters / F-keys for temp hooks |
| Click "misses" | Wrong rectangle / dead margin / min-drag threshold | Aim at a hit-rect center from layout code |
| Diff nonzero with no code change | Async load or animation not settled | Settle longer; or determinize (§5) |
| Diff flaky run-to-run | RNG / wall-clock / variable timestep | Seed RNG, fixed `dt`, measure noise floor |