01
Start state
Each task begins from a buildable Unreal project, no scaffolding to invent.
Benchmark 01
GameEngineBench
GameEngineBench evaluates whether frontier coding agents can make scoped C++ changes that compile and behave correctly inside real, running Unreal Engine 5 projects.
110UE5 C++ tasks
9Source projects
12Model configs evaluated
55.5%Best pass rate
Leaderboard
Measuring frontier coding agents on real UE5 tasks
Charts styled after public long-horizon agent leaderboards like DeepSWE, interactive pass@1 vs effort, then a ranked bar table.
01 · What it measures
Correctness in Unreal depends on the engine, not just the diff. A patch can compile, look plausible, and still fail because it misses replicated state, mishandles an actor lifecycle transition, or breaks a gameplay ability system it never touched directly.
Each task gives the agent a buildable start state, a behavior specification, and a scoped set of editable files, then programmatic tests and judge auditing check whether the resulting behavior is actually correct at runtime.
03 · Key takeaways
- The strongest configuration, Claude Fable 5 (max), now solves a majority of the task set at 55.5% calibrated pass@1, but every other evaluated configuration stays under 30%.
- The hard part is runtime integration, not syntax, strong models usually compile and reach execution; the remaining failures are authority mistakes, replication/state-sync errors, actor lifecycle bugs, and incomplete integration with surrounding gameplay systems.
- Failure is not evenly spread, save/persistence has the largest unresolved share, with weapons/combat, serialization, and AI/world orchestration also retaining unsolved tasks.
- 79 of 110 tasks are solved by at least one configuration; 31 remain unsolved by every configuration evaluated, model capabilities are complementary rather than strictly nested.
04 · Where agents struggle
05 · Dataset & task design
06 · Methodology
02
Scoped edit
The agent edits a constrained set of native C++ files against a behavior spec.
03
Programmatic tests
Tests run programmatically against the real Unreal build and runtime after the agent finishes.
04
Judge audit
LLM-as-judge checks requested behavior, not a match to a reference diff.
07 · Failure analysis
Task 01, Unsolved by all
Zombie System
Requires AI control, round-state updates, server-authoritative damage, and replicated feedback to all agree, models solve pieces but rarely the whole coordination.
Task 19, Unsolved by all
Map Orchestrator
Procedural generation, actor pooling, and readiness signaling, failures are almost always ordering and lifecycle bugs, not generation logic itself.
Tasks 76–80, SPUD persistence
Save / load lifecycle
Plausible serialization code that still loses actor identity or state across streaming levels and engine-managed teardown boundaries.