arXiv Releases Research Paper on Risk Calculation for Candidate Scoring Decision Models
Get up to speed
Agus Sudjianto and Denis Burakov have released a research paper on arXiv on exact risk calculation for candidate scoring decision models, using a Gaussian surrogate model to precisely compute the finite-sample risk of such models and analyzing the boundary of effectiveness of three training elements: teacher soft targets, frozen pretrained backbones, and temperature calibration. The paper notes that after a student model inherits teacher bias, additional student data cannot eliminate that bias; fixed representations only reduce risk below an explicit sample size threshold; and under a probit model, exact recalibration maps to temperature scaling, which can eliminate all excess risk only when the teacher's coefficient direction is correct. The study is validated using 231,521 credit card complaint records from the U.S. Consumer Financial Protection Bureau, with whether a complaint ultimately received monetary compensation as the decision target. Results show that a student model trained with logistic regression teacher probabilities as supervision achieves decision costs on par with the teacher; a student model trained with true outcomes as supervision has decision costs 8% lower than the teacher; and a 30B-parameter large model, after zero-shot reading and even recalibration, still underperforms the teacher model, deviating from consistency conditions by 17.5 decians.
AI-generated from reports · 3 hr agoUpdated
Coverage timeline
Follow the coverage to understand different sides of the event.
- arXiv q-finExact Risk Calculation for Candidate Scoring Decision Models: Analysis of the Impact of Teacher Supervision, Fixed Representations, and Calibration
A new arXiv paper uses Gaussian surrogate models to precisely calculate the finite-sample risk of candidate scoring decision models, analyzing the boundaries of the role of three training elements: teacher soft targets, frozen pre-trained backbones, and temperature calibration. The paper points out that after a student model inherits teacher bias, more student data cannot eliminate that bias; fixed representations only reduce risk below an explicit sample size threshold; under the probit model, exact recalibration maps to temperature scaling, and can only eliminate all excess risk when the direction of the teacher's coefficients is correct.
Trend of this event
Not enough consecutive observations yet; trend is not drawn.