Audit Study Finds That Scores in the FLARE Financial Large Model Benchmark Are Partly Determined by Answer Extraction Rules Rather Than Model Capability

This 12-page arXiv paper audits the scoring pipeline of the FLARE financial task suite, finding that a substantial portion of scoring cell results are determined by answer extraction rules rather than the actual capability of the model under test. The paper notes that FLARE and the lm-evaluation-harness it forked from use different first-match rules on the same generated outputs, with accuracy differences reaching up to 0.4216, and no leaderboard states which rule set it uses.