About Testing the Limits of Chain-of-thought with Multistep Soft Reasoning
Definition and scoring
- Organisation
- Zayne Sprague, Xi Ye, Kaj Bostrom, Swarat Chaudhuri, Greg Durrett
- Category
- Reasoning
- Version
- 2023
- Direction
- higher is better
- Ranking use
- Reference
- Contamination risk
- Unknown
A dataset for evaluating language models on multistep soft reasoning tasks specified in natural language narratives. Tests the ability to perform complex, structured reasoning. Results stay tied to the exact model variant and evaluation system. Multiple systems for the same model use the best published score on this page; overall Lumina scoring uses the median of ranking-eligible rows.
Open benchmark source ↗