RESEARCH ARTICLEVersion 1.0

UnityBench-Mini3D:
Evaluating LLM Agents for 3D Game Development in Unity

AI Hub

UnityUnity China

About UnityBench-Mini3D

UnityBench-Mini3D evaluates AI agents on real Unity development workflows. It pairs 77 complete projects with natural-language requirements across seven development skills and three difficulty tiers. A total of 1,178 held-out, human-authored milestone tests check scene structure, component configuration, and runtime behavior directly in Unity.

Research overview
PROJECT-LEVEL UNITY DEVELOPMENTGolden reference projects
77Project-level tasks
1,178Held-out milestone tests
7Development skills

1 Beyond source-code correctness

In game development, correctness lives across code, scene hierarchies, component settings, and runtime behavior. A project can compile and still fail to work.

UnityBench-Mini3D puts agents inside complete Unity projects, then checks their final state with human-authored tests running in the engine.

01

Projects, with context.

Existing scenes, assets, scripts, and settings. Agents modify connected project state while preserving what already works.

COMPLETE UNITY PROJECTS
02

Skills, beyond genres.

Seven expert-defined categories measure the development capabilities behind a game, across three difficulty tiers.

EXPERT-CURATED TASKS
03

Evidence, in the engine.

Held-out EditMode and PlayMode tests verify structure and behavior, with partial credit for independently satisfied milestones.

DETERMINISTIC EVALUATION
Read the abstract

Game development stresses coding agents in ways general software engineering does not: correctness depends on how engine APIs, serialized scene state, and runtime behavior interact, not on source code alone. Existing benchmarks mainly target platforms such as Godot and heavily rely on LLM judges or coarse task-level outcomes. We present UnityBench-Mini3D, a benchmark of 77 project-level Unity tasks organized around seven expert-defined skill categories rather than tutorials or genres, with three difficulty tiers. Each task pairs a complete Unity project with natural-language requirements and is evaluated using 1,178 held-out, human-authored milestone tests covering scene structure, component configuration, and runtime behavior. This in-engine evaluation provides deterministic partial credit, localizes failures, and distinguishes partial implementations from complete solutions without relying on subjective judges. Across 21 models and 9 coding harnesses, the average pass rate reaches 80.9% on easy tasks but falls to 35.2% on hard tasks, with just 57.7% overall. UnityBench-Mini3D provides a reproducible, diagnostic testbed distinguishing systems across difficulty levels.

2 Task explorer

Explore the projects behind the scores. Open a task to compare its
initial scene and human-authored golden reference.

25 easy29 medium23 hard

Loading benchmark tasks…

Screenshots show initial and golden reference projects. They are not agent-generated outputs.

3 In-engine evaluation

No prescribed action sequence.
No LLM judge.
Just the final project state.

01 / INPUT

A project + a goal

An initial Unity project and a specification: goal, context, constraints, and resources.

02 / AGENT

Build, inspect, iterate

The agent uses its harness tools to edit code, assets, and engine state within a one-hour budget.

03 / HELD-OUT TESTS

Verify in Unity

Clean reimport and compilation, followed by injected EditMode and PlayMode milestone tests.

4 Benchmark results

Compare milestone pass rates across skills
and difficulty levels, alongside cost
and wall-clock time.

AVERAGE MILESTONE PASS RATE57.7%

Overall across coding configurations.
Performance drops as skills must work together.

Easy
80.9%
Medium
55.6%
Hard
35.2%
EVALUATION DIMENSIONS
01

Scene structure

EditMode tests inspect scene hierarchies, object transforms, and asset references to verify that required objects are present, correctly arranged, and connected.

02

Component configuration

Tests check attached components, serialized properties, and resource bindings against task requirements, including settings for physics, animation, rendering, and user interfaces.

03

Runtime behavior

PlayMode tests run the project in Unity to verify dynamic behavior, including physical interactions, animation, and interface responses.

Leaderboard

Paper v1.0 · Table 3

Model and harness performance on UnityBench-Mini3D. Select a column to sort.
#Model / Harness

Loading results…

Download data

Scores (%) are mean per-task milestone pass rates, not the percentage of fully solved projects. Cost is estimated API cost in USD; time is average wall-clock minutes. “—” indicates an unreported value. Computer use agents are evaluated separately from the 32 coding configurations.

5 Contributors

1 Unity China2 New York University3 University of Chinese Academy of Sciences4 Xiamen University5 Columbia University6 Peking University7 The Hong Kong University of Science and Technology

* Equal contribution.   † Corresponding authors: Minghao Shao, Heng Chang, Jie Feng

REFERENCE

Citation

If you use UnityBench-Mini3D in your research,
please cite the paper.

Read the full paper
BIBTEX
@misc{zhou2026unitybench,
  title = {UnityBench-Mini3D: Evaluating LLM Agents
    for 3D Game Development in Unity},
  author = {Zhou, Yijun and Shao, Minghao and
    Zhang, Ke and Yuan, Renkai and Yuan, Hongbo and
    Hu, Bin and Qian, Bozhao and Shen, Hongxia and
    Liu, Yanhong and Chen, Yuting and Shi, Yuntao and
    Xu, Zhongxuan and Mao, Shuo and Zhou, Haijia and
    Li, Zhendong and Zhang, Tao and Li, Chenggang and
    Wang, Yuxuan and Zheng, Zinan and Zhang, Calvin and
    Li, Jia and Chang, Heng and Feng, Jie},
  year = {2026},
  eprint = {2610.35980},
  archivePrefix = {arXiv},
  primaryClass = {cs.SE},
  url = {https://arxiv.org/abs/2610.35980}
}

Golden reference project
Initial challenge project
INITIAL PROJECTGOLDEN REFERENCE
← Reveal golden referenceDrag to compare →

A visual comparison of benchmark reference projects. Milestone tests evaluate scene structure, configuration, and runtime behavior.

Full specification in the paper appendix ↗