If predictions assigned a given probability are correct at roughly that rate, the system is calibrated for that group and setting. Reliability diagrams and proper scoring rules can reveal overconfidence or underconfidence.
Calibration describes whether predicted confidence levels correspond to observed outcome frequencies.
If predictions assigned a given probability are correct at roughly that rate, the system is calibrated for that group and setting. Reliability diagrams and proper scoring rules can reveal overconfidence or underconfidence.