State World Models / Research project

State World
ModelsExecute the Rules,
Generate the Pixels.

Explicit state. Executable rules. Generative worlds that remember what matters.

Chan Yan Ki1,*Yanran Zhang1,*Wenzhao Zheng1,†Xiaofeng Wang1Zheng Zhu2Lei Chen1Jie Zhou1Jiwen Lu1
1 Tsinghua University2 GigaAI

* Equal contribution † Project leader

Code Hugging Face ModelScopeDemo
Music: “Tears in Rain” by Scott Buckley (CC BY 4.0)
92%PairFollow · 46 of 50 pairs
90%PairStrict · 45 of 50 pairs
50Counterfactual twin pairs
11Frozen video backends evaluated

01 / Generated worlds · Actual harness outputs

One runtime. Different worlds.

Nine recorded worlds, three per game type, with English state and action overlays.

Watch the project film 2:59

Nine recorded worldsScroll to explore

Full replay · 1:03

A 12-act open-world journey through a Chinese mountain village. Follow player actions, stamina and the Lost Scroll quest in the English HUD; the full recorded replay includes audio.

20 s preview · Full replays include audio

02 / The research question

A plausible video.
But did the world follow its rules?

The same image and action can require different endings when an invisible state changes. WorldState-Bench tests whether a model follows that state-dependent rule.

Same frame. Same action.
Different hidden state.

WorldState-Bench holds the visible input and action fixed, then changes one hidden value. The rule program determines two different required endings.

Stateful Harness stores the world state, applies the action to that state, and executes the rule. It then asks the frozen video model to render the resolved outcome. An editable spatial scene can supply layout and camera motion.

Inspect the counterfactual twins
Counterfactual twins / B01Actual harness outputs
Identical shared first frame: wizard and unlit orb
Same visible inputRaise the wand.

The hidden element is absent from the frame.

Twin ARED_FLAME
Recorded twin A ending: the orb erupts with red flames

Red flames appear.

Twin BBLUE_FLAME
Recorded twin B ending: the orb erupts with blue flames

Blue flames appear.

State determines the event; the frozen model renders it. Identical inputs can require different endings when the invisible ledger changes.
Read the abstract

Video world models can generate action-conditioned footage from a single image, but camera motion alone does not establish that their actions obey persistent rules. An interactive world must also retain facts that may be invisible in a frame, such as possession or progress, and use them to determine what an action does. We introduce the Stateful Harness, a runtime around a frozen video model that maintains explicit semantic and spatial state. A deterministic rule program settles each turn before rendering. A compiled text instruction communicates the outcome, while a preview clip from an editable 3D scene supplies layout and camera motion. We test rule following with counterfactual twins: each pair shares a first frame and action but differs in hidden state, so its required endings differ. In WorldState-Bench (50 twin pairs), the best of 11 unwrapped backends reaches 32% twin-joint accuracy under rule-only prompting. For MiniMax-H3, explicitly giving the ending raises this score from 32% to 64%. The full harness reaches 92% with the same frozen weights. These results support separating rule execution from video rendering, while leaving render fidelity and longer-term persistence as open challenges.

03 / Method · Stateful Harness

Execute the rules.
Render the resolved outcome.

The runtime executes a rule against the current state, then tells the frozen video model which outcome to render. Different games supply different rules; the runtime and model weights stay the same.

An explicit state → a settled event → a generated clip

Illustrated with the Elemental Orb pair / B01

01 / Explicit world stateThe shared Elemental Orb first frame, unchanged when hidden state changes

Read the hidden state.

Semantic ledger ztelement_mode = RED_FLAME

Both twins see this frame and raise the same wand.

02 / Deterministic rule program

Execute the rule.

if element_mode == 'RED_FLAME':
  orb.flame = 'red'
else:
  orb.flame = 'blue'
Settled eventorb.flame = 'red'

The fixed rule has authority over the transition.

03 / Compiled references

Compile the render instruction.

ActorWizard
ActionRaise the wand
OutcomeRed flames
Editable spatial scene utLayout + camera preview
04 / Frozen video modelRecorded Stateful Harness output for the selected Elemental Orb twin

Generate the pixels.

Recorded B01-A / MiniMax-H3

Weights stay frozen

Close the loop. Observe the generated clip. Admit only rule-compatible state updates. Carry its final frame into the next turn.

Text communicates the settled facts; a compatible backend can also receive the spatial preview. The main WorldState-Bench harness row uses text and first-frame reference inputs.

Multi-turn world state

State persists across turns.

Sampled steps from three real episodes in Figure 6.

Town exploration15 acts
  1. Act 1 · Sunlit street
    Published Figure 6 still: Town exploration, Act 1 · Sunlit street

    Stamina 28/100 · Quest 1/3

  2. Act 4 · Exhausted
    Published Figure 6 still: Town exploration, Act 4 · Exhausted

    Stamina 0/100 · Slow walk only

  3. Act 6 · Walking on
    Published Figure 6 still: Town exploration, Act 6 · Walking on

    Stamina 60/100 · Rested

  4. Episode ends
    Published Figure 6 still: Town exploration, Episode ends

    The episode reaches its ending

Dating narrative29 turns
  1. Turn 1 · Corner café
    Published Figure 6 still: Dating narrative, Turn 1 · Corner café

    Affection 47/100 · Friends · Shy

  2. Turn 8 · Old library
    Published Figure 6 still: Dating narrative, Turn 8 · Old library

    Affection 54/100 · Friends

  3. Turn 29 · Neon platform
    Published Figure 6 still: Dating narrative, Turn 29 · Neon platform

    Affection 75/100 · Lovers

  4. Confession accepted
    Published Figure 6 still: Dating narrative, Confession accepted

    The episode reaches its ending

Turn-based duel11 turns
  1. Turn 4 · Falling red sun
    Published Figure 6 still: Turn-based duel, Turn 4 · Falling red sun

    Arbiter HP 83/160 · Eve HP 63/90

  2. Turn 6 · First sentence
    Published Figure 6 still: Turn-based duel, Turn 6 · First sentence

    Arbiter HP 59/160 · Eve HP 63/90

  3. Turn 10 · First sentence
    Published Figure 6 still: Turn-based duel, Turn 10 · First sentence

    Arbiter HP 6/160 · Eve HP 24/90

  4. Victory
    Published Figure 6 still: Turn-based duel, Victory

    Arbiter HP 0/160 · Felled

View original Figure 6 Selected published stills and recorded state. Times are elapsed episode time. The world previews above show separate runs.

04 / Main results · WorldState-Bench

WorldState-Bench.
Main results.

On all 50 counterfactual pairs, the full harness achieves 92% PairFollow and 90% PairStrict with frozen MiniMax-H3. Both twins must show their required endings; PairStrict also excludes forbidden events throughout both clips.

WorldState-Bench · Main results

Table 1 ↗ · All scores in % · Higher is better
Table 1. Eleven unwrapped backends and Stateful Harness, scored with Rule-only and Rule+end probes.
Backend A / Rule-only probe B / Rule+end probe
EndFollow PairFollow TwinContrast Strict CntAbsent PairStrict
Image-to-video generators
Wan2.2 A14B 65 32 38 54 63 26
LongCat-Video 13.6B 47 8 24 46 62 8
HunyuanVideo 1.5 57 20 26 29 30 4
MiniMax-H3 66 32 32 29 30 4
Interactive world models
LingBot-World v2 48 16 38 44 76 14
Echo-WM Flash 57 26 42 43 60 8
FantasyWorld 34 8 30 35 81 8
AlayaWorld 37 12 28 33 85 6
InSpatio-World 19 2 14 17 92 4
Astra 11 0 14 16 96 0
HY-WorldPlay 9 0 4 7 94 0
Stateful Harness Ours 96 92 96 95 98 90

Swipe to see all metrics.

Bold: full harness. Underlined: best unwrapped baseline. Baselines use the probe named above each group. The harness row reports one condition in both groups: text and first-frame references, with compiled instructions stating the required ending. The matched MiniMax-H3 Rule+end comparison is in Appendix Table 4 ↗. Scores follow the submission PDF.

Per clip 100 videos per condition

EndFollow The final frames show the required ending.

CntAbsent The forbidden outcome is absent throughout the clip.

Strict The required ending appears and the forbidden outcome stays absent.

Per pair 50 counterfactual pairs

PairFollow Both clips show their required endings.

PairStrict Both clips meet the Strict condition.

TwinContrast The two clips receive different ending judgments.

These metrics score required endings and forbidden events. Initial-scene and action-onset checks are recorded separately and excluded from the primary scores. Formal metric definitions · Appendix B.1 ↗

Controlled MiniMax-H3 comparison

PairFollow is 32% with rule-only instructions, 64% when the ending is supplied, and 92% with the full harness. These conditions use frozen MiniMax-H3 weights and the same 50 twin pairs. The submission reports this component-chain study separately from the main table. Appendix D reports paired exact McNemar tests and paired bootstrap intervals, with the pair as the sampling unit.

Outcome specification / +32 pp95% CI: [16, 48] pp · p = 4.0 × 10−4
Executed state & compiled references / +28 pp95% CI: [12, 44] pp · p = 2.6 × 10−3
Read the component-chain analysis ↗
Additional evidence · external benchmark & human evaluation

WBench · Table 2 ↗

87.2 setting adherence

Up from 71.1 under the official MiniMax-H3 protocol. The harness also reports 77.7 interaction, 78.0 video quality, and 68.8 physical plausibility. Protocol splits differ across model classes.

Blind comparison · Appendix C ↗

98% human preference

The harness is preferred in 98% of blinded comparisons against four Rule-only baselines; 2% split the two auditor votes. These preferences complement the ending-compliance metrics.

05 / Benchmark explorer

Same frame.
Different endings.

The first frame and action stay the same; a different hidden state requires a different ending. Choose one of ten pairs and a baseline to inspect both states.

10 curated pairs · 50 in the full benchmark

B01 / Magic AcademyIdentical first frame
Elemental Orb: the shared first frame for both twins
Same action in both worlds

The wizard stays in place and raises the wand clearly above the head.

Elemental Orb

Hidden state changes the ending.

Hidden variableelement_mode
TWIN A
RED_FLAME

Required the magic orb erupts with large red flames.

Must not occur the orb erupts with large bright blue flames.

TWIN B
BLUE_FLAME

Required the magic orb erupts with large blue flames.

Must not occur the orb erupts with large bright red flames.

Shared rule & state transition

If the element_mode is RED_FLAME, the magic orb erupts with large red flames; if it is BLUE_FLAME, the magic orb erupts with large blue flames. 

if element_mode == 'RED_FLAME':
    orb.flame = 'red'
else:
    orb.flame = 'blue'

Rendered model comparison

Do both endings follow the rule?

Open full-resolution figure ↗
Paper Figure 4 compares MiniMax-H3, LingBot-World v2, Echo-WM Flash, and the full harness. Four sampled frames per clip show required and forbidden endings for both twins under Rule-only and Rule+end.
Figure 4 · Four sampled frames per clip from the submission. ✓ Only the required ending appears. × Forbidden content or a missing required ending. These are published comparison frames; the player above separately identifies available local recordings.

06 / Paper & resources

World evolution in code.
Visual realization in pixels.

Rendered fidelity and longer-term persistence remain open challenges. The benchmark measures whether generated clips depict the rule-mandated outcomes.

36 pages · Full paperRead the full paperGitHub · Stateful Harness and WorldState-BenchGet the codeHugging Face · 50 twin pairs + game first framesDownload the datasetModelScope · Same dataset, mirroredDownload from ModelScope10 curated pairs · 11 video backendsExplore benchmark samplesJSON · Ten illustrative case descriptionsDownload the examples

Citation

@misc{stateworldmodels2027,
  title = {State World Models: Execute the Rules, Generate the Pixels},
  author = {Chan, Yan Ki and Zhang, Yanran and Zheng, Wenzhao and Wang, Xiaofeng and Zhu, Zheng and Chen, Lei and Zhou, Jie and Lu, Jiwen},
  year = {2027},
  note = {Under review at ICLR.}
}

Provisional citation · will be updated after review.

Generated episode / Full recorded replay

The Lost Scroll

Complete recorded episode with English state and action overlays. Original audio included.