A model can solve one step correctly and still lose the thread across fifty steps. That distinction matters when we want AI to handle work that takes sustained planning, state tracking, and error correction. The LongCoT benchmark is an attempt to measure that specific ability.
The paper's question is straightforward: how reliably can a model reason through a long chain of dependent steps? A problem may fit in a short prompt, yet solving it can require tens of thousands of output tokens. The challenge is keeping earlier results consistent, choosing what to do next, and noticing when a wrong turn requires backtracking. The paper calls this long-horizon reasoning.
What the benchmark tests
LongCoT contains 2,500 problems across chemistry, mathematics, computer science, chess, and logic. Each has a verifiable final answer. Some expose their dependencies directly as a graph of subproblems; others make the solver discover the structure through search or simulation.
Imagine a task where the answer to step A determines which version of step B you must solve, and that result changes a constraint in step C. Getting A, B, and C right in isolation is not enough. The solver has to preserve the right values, follow the dependencies in order, and recover if an intermediate result is wrong. That is the kind of failure LongCoT is designed to make visible.
The authors also test isolated subproblems. In their analysis, GPT 5.2 solved 95.7% of those isolated steps, while performance on the composed benchmark was far lower. This supports the view that many failures come from coordinating the steps over a long horizon, rather than from every individual step being impossibly hard. It does not mean all errors have a single cause; the paper examines several failure modes.
How the tasks were made
The human design work here is important. Groups of domain experts designed parameterized task templates with meaningful dependencies and verifiable answers. The released set uses 10 templates in each of five domains, with 50 parameter settings per template: 500 problems per domain, or 2,500 in total. The mathematics templates incorporate existing Olympiad-level questions before composing them into larger tasks, and the paper says the problems are verified programmatically.
The questions are human-designed at the template and composition level, then instantiated and checked programmatically. That is different from saying each of the 2,500 final questions was independently written by hand. The design gives the benchmark meaningful problems and enough instances to test performance across different horizons.
How it compares with GPQA
GPQA consists of individually expert-written, graduate-level multiple-choice science questions. LongCoT is also expert-designed and difficult, but its main test is different: the individual steps are meant to be tractable, while getting through a long chain of dependent steps is hard. So I think of it as another demanding reasoning benchmark, rather than simply GPQA with harder standalone questions.
What the first results show
In the original paper, the best reported model, GPT 5.2, reached 9.83% accuracy, using about 62,000 output tokens per problem on average. Gemini 3 Pro reached 6.1%. These are results from the paper's release, not a live ranking of current models. The benchmark site and repository provide the current benchmark materials.
The authors describe errors such as drifting from a plan, losing intermediate results, failing to backtrack, and letting mistakes compound. Their primary evaluation gives models no tools or scaffolding so that the score reflects reasoning within the model's own output. A separate experiment with code execution improves some procedural tasks, but leaves the compositional mathematics, chemistry, and computer science tasks close to zero in that setup.
Why it is a useful indicator for longer tasks
LongCoT isolates a capability that many longer tasks need: carrying a coherent plan through interdependent work. A short-input problem with a long, checkable solution makes it easier to distinguish this from simply retrieving information from a long document. If a model cannot keep track of a dependency graph in a controlled setting, that is a useful warning about its reliability on longer work.
There is a limit to the inference. A LongCoT score is not a complete measure of an AI agent's real-world ability. Agents can use tools, external memory, collaborators, and feedback from an environment. Conversely, real work can introduce ambiguity and changing goals that these problems do not capture. The paper itself notes that not every LongCoT question directly resembles a realistic workflow. I see the benchmark as one strong signal about a necessary underlying skill.
References
- S. R. Motwani et al. (2026). LongCoT: Benchmarking Long-Horizon Chain-of-Thought Reasoning. arXiv:2604.14140.