Stanford study finds LLM findings expire within months

Stanford researchers Owen Queen and James Zou examined 50 widely cited claims about large language models against 20 current systems and reported that many of the findings no longer held. They said 44 percent had expired and that the average lifespan among the findings with dates attached was 15.7 months.

The paper said claims about what models could not do faded fastest. It listed examples such as letter counting, unfamiliar arithmetic, information loss in long contexts and resistance to user pushback, and said these results were often tied to tasks with a clear target.

Findings about behaviour and preferences lasted longer. The study said models kept showing patterns such as favouring their own answers, rating accepted and rejected papers similarly, and being influenced by wording in medical abstracts, while other results involved sycophancy, political lean and random-number generation.

As a case study, Queen and Zou revisited citation fabrication from a 2023 paper by Walters and Wilder. They said GPT-6.1 Sol and Claude Sonnet 5.5 produced almost no false references in their test, while Qwen3.7-Flash still fabricated many.

The researchers also said performance still varied widely between models released in the same period, and that release date, provider and size did not reliably predict the pattern. They argued that such findings should be treated as model-specific observations and should be published with prompts and analysis code so they can be retested later.