MMDM Benchmark: 28 Decision Models Cap at 76% Accuracy

A new benchmark for System-1 decision models reveals a stark performance ceiling across the current generation of AI systems. MMDM, officially released on November 11, 2026, evaluates 28 models through 5,448 questions spanning 16 task categories, with option spaces ranging from 2 to 1,024 choices and input contexts stretching to 64,817 tokens. WeMM-JEV 27B claims the top position on hard single-select tasks at 76.00% accuracy, but the entire field collapses on multi-select challenges, where no system exceeds 36.81%.

Benchmark Design and Scope

MMDM targets a specific category of models the authors define as System-1 decision makers: systems that directly produce structured probabilistic decisions over query-specified options in a single forward pass, without generating intermediate reasoning chains or textual responses. The benchmark's architecture addresses gaps in existing evaluation suites by combining three dimensions that prior work rarely tests together: long contexts, large option spaces, and complex logical dependencies.

The 5,448-question suite spans 16 task categories, with option counts scaling from binary choices to sets of 1,024 alternatives. Input contexts reach 64,817 tokens at the upper end, forcing models to maintain coherence across document-length inputs while mapping decisions to hundreds of possible outputs. The authors published the benchmark in a Hugging Face paper listing, establishing a standard for evaluating decision-making architectures distinct from generative language models.

Release Date and Availability

The MMDM benchmark was officially released on November 11, 2026, with full specifications and leaderboard data made public through the Hugging Face repository. The timing places it in a dense evaluation season: our tracker logs 569 agent stories in the past 90 days, including yesterday's WikiTrace paper on memory-induced regression in agent systems.

Single-Select Performance and Leaders

WeMM-JEV 27B establishes the performance ceiling for hard single-select tasks at 76.00%, but the margin separating top systems narrows quickly. The leaderboard shows competitive clustering among several architectures, with the gap between first and fifth place spanning fewer than 10 percentage points on the aggregate single-select metric. The relative simplicity of choosing one correct answer from a predefined set allows most modern decision models to extract signal reliably when option counts stay low and context remains manageable.

Multi-Select Collapse

Performance disintegrates when tasks require selecting multiple correct answers from large option sets. No model in the 28-system field breaks 36.81% on hard multi-select questions, and every tested architecture scores below 0.20 on the Separation-Balance F metric, which measures how well a model recovers the full correct set without false positives. WeMM-JEV 27B posts the highest SB-F at 0.1905, still landing below the 0.20 threshold the authors identify as minimal competence.

The multi-select failure exposes a fundamental limitation in current decision architectures. Unlike single-select tasks where a model can rely on comparative ranking to identify the strongest option, multi-select requires absolute threshold judgments: determining not just which options are better than others, but which cross the bar for inclusion in the correct set. The benchmark reveals that today's models lack robust calibration for this kind of binary membership decision at scale.

Context and Option Scaling Effects

Aggregate rankings collapse when sliced by context length and option count. OpenJev outperforms Intern-Decision across every option-count bin until input contexts exceed 8,000 tokens, where the order reverses to 15.5% versus 34.5%. On option sets of 129 to 1,024 choices, OpenJev flips the result again, scoring 26.0% against Intern-Decision's 8.3%. Jev and Microsoft Decision-1 sit within 7 percentage points of each other on 2-to-4-option questions but diverge by more than 37 points on the largest option bin.

These inversions suggest that no single architecture dominates across the full range of decision scenarios. Models optimized for long-context reasoning sacrifice performance on large option spaces, while systems tuned for high-cardinality choices struggle when input length grows. The benchmark exposes the absence of a unified decision-making architecture that scales gracefully along both dimensions simultaneously.

What This Means

MMDM establishes a new performance baseline for System-1 decision models and highlights a critical gap in multi-select reasoning. The sub-37% ceiling on hard multi-select tasks and the universal failure to reach 0.20 SB-F indicate that current architectures lack the calibration mechanisms required for reliable set-membership decisions at scale. The dramatic ranking shifts across context and option dimensions suggest that researchers face a fundamental trade-off: optimizing for one scaling axis degrades performance on the other. Organizations deploying decision models in production should audit task profiles carefully, matching architecture strengths to the specific combination of context length and option cardinality their applications demand.