OrigamiTrace

Origami as a Real-Image Benchmark for Procedural State-Transition Reasoning

NeurIPS 2026 Evaluations and Datasets Track

Ikumi Numaya1 Reina Akama1,2,3 Kise Yoshida1 Yuka Saito1 Sumika Tsuda1 Takahiro Shimizu1 Miori Sagara4 Jun Suzuki1,3,5

1Tohoku University 2NINJAL 3RIKEN 4Baobab, Inc. 5NII LLMC

is-failab-research@grp.tohoku.ac.jp

Overview of the OrigamiTrace benchmark: a reference image sequence with textual instructions from a start state to a target state, Task 1 where the agent directly receives the procedure, Task 2 where a human instructor guides the agent, and a comparison with prior work.
Overview of the proposed benchmark tasks and how they relate to typical manipulation and origami benchmarks.

Abstract

General-purpose embodied AI systems are increasingly expected to interpret multimodal instructions and perform fine-grained physical tasks. However, many manipulation benchmarks provide limited support for jointly evaluating procedural correctness, local geometric sensitivity, and human-guided correction. We propose origami as a benchmark based on real images for procedural state-transition reasoning. Origami requires precise operations such as tucking, inserting, reversing, and folding overlapping layers, making it a focused domain for studying physical state transitions. We introduce OrigamiTrace, a dataset of step-by-step folding images, textual entries, and difficulty annotations, and define two tasks, i.e., autonomous state transitions from multimodal instructions and human-guided collaborative transitions. Unlike crease-pattern- or diagram-based origami benchmarks, the proposed benchmark is grounded in photographs of actual intermediate folding states. In an initial simulator-based evaluation, we use three multimodal LLM controllers to evaluate procedural understanding, action-interface grounding, and retry-based correction.

Dataset

348origami designs
7,221real state images
2state-transition tasks
5difficulty levels

Each origami design is an ordered sequence of photographs of intermediate paper states, from paper preparation to the completed origami. For each non-final step, a human-written textual entry describes the operation required to reach the next state; the final step marks completion. Images were captured in a controlled, fiducial-assisted setup, and every transition was checked for foldability, specification conformance, and text quality.

Example from the dataset: the Long-Horned Samurai Helmet, shown as a sequence of real images with textual entries.
Example from the dataset: “Tsunonaga-Kabuto” (long-horned samurai helmet).

Difficulty levels

Example of a Level 1 designLevel 1 – Beginner58 designs (16.7%)
Example of a Level 2 designLevel 2 – Elementary131 designs (37.6%)
Example of a Level 3 designLevel 3 – Intermediate144 designs (41.4%)
Example of a Level 4 designLevel 4 – Advanced15 designs (4.3%)

Dataset Examples

Browse one origami design from each difficulty level. Each step pairs a captured state image with the textual entry describing the operation that leads to the next state. Dot stickers on the paper serve as fiducial cues: red and blue stickers indicate source and target positions for folds when needed.

Textual entry

Images are downscaled for the web. Use the arrow keys to step through a design. The full dataset is available on Harvard Dataverse.

Tasks

A benchmark episode is formed by selecting a start state and a later target state from the same procedure, together with the corresponding state images and textual entries.

Task 1: Autonomous Origami State Transition

Task 1: the agent directly receives the procedure from the dataset, with no human instructor involved.

The agent directly receives the reference image sequence, textual instructions, and target state, and must execute each transition. After a failed attempt it receives fixed feedback and may retry.

Task 2: Human-Guided Collaborative Origami State Transition

Task 2: a human instructor who sees the procedure guides the agent in natural language; the agent has no direct access to the dataset.

The agent has no direct access to the reference procedure. A human instructor who sees the procedure guides it step by step in natural language, answers clarification questions, and gives corrective feedback.

Example: one Task 1 transition

Current state + entry

Current state: step 5 of the Long-Horned Samurai Helmet, with red and blue dot stickers.

“Make valley folds diagonally from the red dot stickers to the blue dot sticker.”

Agent output

{"command_name": "fold",
 "fold_infos": [{
  "fold_type": "valley",
  "source_point": [-0.77,0.24],
  "target_point": [0,-0.53],
  ...}, {...right side...}]}

A JSON simulator command with two simultaneous valley folds.

Visualized command

The command drawn on the current image: crease lines marked with crosses, source points with circles, and the shared target point with a diamond.

The command drawn on the current image: crease ×, source ● → target ◆.

Human judge

Next reference state: step 6 of the Long-Horned Samurai Helmet.

Attempt 1 ✗ “The fold is incorrect. Please regenerate it.”
Attempt 2 ✓ → next state

Results

We evaluate three multimodal LLM controllers that operate an origami simulator through structured commands, on all 348 procedures with the same start–target pairs in both tasks.

ModelEpisode-SRTrans-SRFirst-Att-SRAvg Att/Trans ↓Retry-SR
Task 1
GPT 5.6 Sol22.7%34.1%61.3%1.6838.2%
Gemma 4 31B3.7%6.4%26.3%2.448.2%
Gemma 3 12B IT1.7%0.8%2.8%2.932.6%
Task 2
GPT 5.6 Sol25.6%40.3%58.5%1.6950.9%
Gemma 4 31B3.4%8.5%28.6%2.3614.1%
Gemma 3 12B IT2.9%3.0%14.3%2.694.8%

SR: success rate. Episode-SR counts procedure segments completed from the start state to the target state; transition-level metrics diagnose partial progress and retry-based correction. See the paper for definitions and significance tests.

BibTeX

@inproceedings{numaya2026origamitrace,
  title     = {Origami as a Real-Image Benchmark for Procedural State-Transition Reasoning},
  author    = {Ikumi Numaya and Reina Akama and Kise Yoshida and Yuka Saito and
               Sumika Tsuda and Takahiro Shimizu and Miori Sagara and Jun Suzuki},
  booktitle = {The Fortieth Annual Conference on Neural Information Processing Systems
               Evaluations and Datasets Track},
  year      = {2026}
}