SpatialBench

Spatial reasoning is the unsaturated axis.

SpatialBench is a family of spatial-reasoning benchmarks for vision-language models. Every item is generated from a renderer with exact ground truth, scored by string equality (no LLM judges), and the test set is a seed, not a file.

Explore CubeBench Run it on your model

Rendered, not scraped

Scenes come from a physically-based renderer, so the label is the scene graph itself. No annotation noise, no web contamination.

Verified, not judged

Every answer is checked by string equality against renderer state. Accuracy and format-failure rate are published side by side.

A seed, not a file

Packs are regenerable from a seed at any size or difficulty. Contaminated? Bump the seed and re-render.

Benchmarks

One family, one contract

Frontier models sit within ~2 points of each other on general multimodal benchmarks. On CubeBench the same field spreads 14–85%, and the ordering upends the price list. Each benchmark below probes a different spatial skill through the same pack format.

Live · 1,040 items · 19 models

CubeBench

Can your model read a Rubik's cube? Face reading, view anchoring, hidden-face inference, and move simulation on rendered cubes with exact state ground truth.

Full leaderboard →
rendered Rubik's cube
Planned

Dice

Opposite-face constraints and hidden-value inference across multiple dice: the minimal probe of "what can't I see, and what must it be?"

Planned

Multi-view

Two cameras, one scene. Fuse views to recover state neither camera sees alone, and flag when the views are inconsistent.

Planned

Video

Track object state through a sequence of manipulations with occlusion: spatial memory, not just spatial perception.

Planned

Rotation

Mental rotation and re-identification: is this the same object seen from a different pose, or a mirrored impostor?

Shared harness

Add an object, get a benchmark

Every SpatialBench suite emits the same pack format (items, images, conventions, ground truth) and runs on the same open harness, benchkit, in Bun/npm or Python. The harness is part of the measurement:

  • Raw and tool-assisted tracks separate "can't see" from "can't reason".
  • Named prompt profiles are stamped into every results row.
  • Full response traces ship with every result, downloadable per model.
bunx benchkit --pack packs/cube-pilot \
  --models your-org/your-model \
  --leaderboard

# or python
python run_eval_or.py --pack packs/cube-pilot