SIGTBD 2026

I submitted two papers to SIGTBD 2026, MIT CSAIL’s annual conference for work of debatable research taste. One got a talk, and one got a poster.

BENCH-PRESS: A Benchmark for Measuring Strength in Large Language Models

As researchers, we noticed that the term “strong” appears frequently in model evaluation literature. Models are described as strong at reasoning, strong at code, or strong on general benchmarks — but nobody had stopped to ask the obvious question: how much can they bench press? This seemed like a significant gap. If the field is going to use strength as a primary descriptor of model capability, it should at minimum be measuring strength in pounds.

The results confirmed that the gap was worth closing. We found substantial and structured variation in barbell performance across the model landscape, including clear family-level training effects, uneven lift allocation, and a surprising non-monotonic relationship between reasoning effort and strength. We also identified a state-of-the-art bench press result from Mistral Medium 3.1 at 472,153 pounds, which establishes a considerable margin over competing systems and suggests that current training pipelines may be substantially under-optimized for barbell performance. Taken together, these results suggest that barbell performance remains a productive and underexplored axis of model evaluation, and we hope they encourage other researchers to take model strength more literally.

The Model Is Getting Better At Its Job

This work began as a routine observation. During a standard evaluation cycle, we noticed that the model had improved. We then checked previous evaluations and found that the model had also improved during those. A pattern emerged. We felt it was important to document this pattern, not because improvement is unusual in machine learning — it is, in fact, the point — but because at some stage we realized we had stopped asking whether the model would get better and started asking whether it would stop. We could not find a satisfying answer. The model, for its part, did not seem interested in the question.

We believe this result is significant in the way that all obvious things become significant once you write them down and add a graph. The field has spent considerable effort making models better, and comparatively little effort deciding what to do when that effort succeeds. We do not claim this is a problem. We only note that the model continues to improve, that it does so without apparent hesitation, and that our paper about this phenomenon was, at one point during the drafting process, partially written by a model that was better at writing it than we were. We present these findings without recommendation.