Brave New Backtest: Why LLMs Have Three Structural Limitations in Trading Research

Drawing on two academic papers, the author identifies three structural limitations of using LLMs in trading research: First, training data is saturated with trading common sense and lacks mechanism-driven edge thinking. Second, across 35+ models, accuracy in time-series value tracking declines logarithmically as the number of historical updates increases, and neither prompt engineering nor expanding the context window can fix this. Third, outputs from 25+ models on open-ended questions show 71%–82% similarity, and models with different architectures converge to the same common answers.