Compress the filing to 20 bullet points and the AI flips its callFlippin’ heck

Researchers had an AI read a full earnings filing, then had it judge a 20-point summary of the same text: the buy or sell call flipped on a quarter to a third of the stocks.

Analysts are increasingly putting AI pipelines to work: one model reads a long filing and boils it down to bullet points, a second model takes those bullets and makes the call. It saves time, and nobody checks every step in between.

Same filing, different wording, different answer

Researchers from the US, the UK and China, with people from JP Morgan, BlackRock and State Street, ran an experiment: ChatGPT, Gemini, Qwen and DeepSeek were asked to summarise the management discussion section of the quarterly reports for the 100 largest US stocks into 20 key points.

That summary was then handed to Gemini to make a buy, hold or sell call. The full filing was also handed to Gemini for the same call. The answers should have matched.

They did not: on a quarter to a third of the stocks, the call flipped from buy to sell or from sell to buy. All that changed was the length of the input.

One wrong answer in ten is just normal

You could argue models get things wrong sometimes. The New York Times tested Google's AI Overviews and found Gemini 2 was accurate 85% of the time and Gemini 3, 91% — so even the best model still invents roughly one answer in ten.

And even when it is right, it either cites nothing or cites a source unrelated to the answer 56% of the time.

Who signs off on the pipeline

Each step looks roughly good enough on its own. Chained together, nobody can say where the final recommendation actually came from.

The tool is fine for going faster. Just do not let it be the one signing your name.