Most prediction systems ship two numbers: a call, and a confidence. The confidence is usually an integer — 1 to 10, 60 to 95, whatever reads well in a UI. Everyone downstream then treats it as a probability, because it is a number between 0 and 100 and probabilities are numbers between 0 and 100.
That inference is where audits die. In our production table, confidence is an integer from 1 to 10 assigned at call time. It is not a probability, it never was, and the table below is the proof.
The dataset
The numbers come from one PostgreSQL table, ai_predictions, joined to matches for settlement fields. Nothing is simulated; every figure is recomputable below.
- 46,196 prediction rows across 16,732 distinct matches, created 2026-06-21 → 2026-11-09
- 45,722 rows are settled and carry a binary outcome (
hit_1x2) plus a confidence - 3,715 distinct
model_idvalues produced them - Predictions per match: mean 2.77, median 1, p90 7, max 358
That last line matters. The unit of analysis is the call, not the row, and one match with 358 rows will silently dominate any bucket it lands in.
What the score actually predicts
Here is the honest bucket table. n is row count, hit% is empirical accuracy on the binary outcome, and the interval is a 95% Wilson score interval — not a normal approximation, because at 57 rows the normal approximation is a work of fiction.
SELECT confidence,
count(*) AS n,
sum(CASE WHEN hit_1x2 THEN 1 ELSE 0 END) AS hits,
round(100.0 * avg(CASE WHEN hit_1x2 THEN 1 ELSE 0 END)::numeric, 2) AS hit_pct
FROM ai_predictions
WHERE settled = true
AND hit_1x2 IS NOT NULL
AND confidence IS NOT NULL
GROUP BY confidence
ORDER BY confidence;conf n hits hit% 95% Wilson CI
1 251 118 47.01 40.93 – 53.19
2 1108 604 54.51 51.57 – 57.42
3 8043 4327 53.80 52.71 – 54.89
4 8750 5255 60.06 59.03 – 61.08
5 11375 6305 55.43 54.51 – 56.34
6 9260 5063 54.68 53.66 – 55.69
7 4292 2575 60.00 58.52 – 61.45
8 2037 1335 65.54 63.45 – 67.57
9 548 404 73.72 69.88 – 77.23
10 57 33 57.89 44.98 – 69.81There are three separate failures in it.
One: the middle of the scale is not monotonic. A call labelled 5 hits 55.43%; a call labelled 4 hits 60.06%, and the intervals do not overlap (59.03–61.08 vs 54.51–56.34). Claiming more confidence produced a worse result by 4.6 points across 20,125 rows, and the inversion recurs at 6 vs 5 (54.68% vs 55.43%). Whatever the model emits when it says "5", it is not a probability — it is a label that merely looks ordinal.
Two: the top of the scale collapses. Confidence 9 gives 73.72% on 548 rows; confidence 10 gives 57.89% on 57 rows. The loudest tier is both the worst and the thinnest — 0.12% of the data, with an interval of 44.98–69.81. The claim you can least verify is the claim published loudest.
Three: the aggregate hides all of it. Mean confidence is 4.94 (49.4%), overall accuracy 56.91%. So the system is under-confident by 7.5 points at the mean while its top tier is over-confident by 42. A single averaged "calibration" number reports a timid herd and hides a reckless tail.
Expected Calibration Error, and why it is not enough
The standard summary is ECE: bucket the predictions by stated probability, average the absolute gap between stated and observed, weight by bucket size.
WITH s AS (
SELECT confidence::numeric / 10 AS p,
CASE WHEN hit_1x2 THEN 1 ELSE 0 END::numeric AS y
FROM ai_predictions
WHERE settled = true AND hit_1x2 IS NOT NULL AND confidence IS NOT NULL
), b AS (
SELECT width_bucket(p, 0, 1.0001, 10) AS k,
count(*) AS n, avg(p) AS mp, avg(y) AS my
FROM s GROUP BY 1
)
SELECT round(sum(n * abs(mp - my)) / sum(n), 4) AS ece,
round(sum(n * my) / sum(n), 4) AS overall_hit,
sum(n) AS n
FROM b;Result: ece = 0.1333, overall_hit = 0.5691, n = 45722.
13.3 points of average miscalibration, a Brier score of 0.2689, and a mean signed bias of −0.0751. Now the baseline nobody computes: a constant forecast at the base rate, 0.5691, scores 0.5691 × 0.4309 = 0.2453. The raw score's 0.2689 is worse than predicting the same number every time. That is the whole argument in one line — a column that cannot beat a constant is not carrying probability information, it is carrying formatting.
Note also that ECE is weighted by bucket size, so 45% of it comes from confidence 5 and 6. The two tiers anyone would actually audit — 9 and 10 — contribute 1.3% of the weight. A metric weighted by frequency will always under-report the failures that matter most.
The fix we shipped, and the trap inside it
We added a fitted calibration layer. Each settled row now carries confidence_calibrated, a continuous probability. Re-running the same two queries against it:
- ECE: 0.0017 (down from 0.1333)
- Brier: 0.2439 (down from 0.2689)
- Mean bias: +0.0011 (down from −0.0751)
- Range across all 45,722 settled rows: 0.4741 to 0.7286, mean 0.5701
That last bullet is the one to stare at. A near-zero ECE looks like a triumph. What it actually means is that the calibrated map refuses to output anything outside a 25-point band, and then the data obliges — because the map was fitted on the data it is being scored on. An ECE of 0.0017 in-sample is a statement about the fit, not the skill. The honest reading is the range: this system's real probability band is 47% to 73%, and nothing it says should be read as an 85% claim. Even the improved Brier, 0.2439, is only 0.6% better than the constant-forecast baseline of 0.2453 — all the gain is in the level, none of it in the ordering.
If you take one engineering lesson from this post: fitting a calibration layer always makes your calibration metric look good. What survives is the width of the output band plus an out-of-sample check. We report the band, not the ECE.
Two contamination sources that quietly inflate everything
Two-way calls. 32,147 settled rows are single-direction calls and hit 58.35%; 13,546 are "either of two outcomes" calls and hit 53.50%. Counter-intuitively the easier category scores worse — conditioning on two of three outcomes still loses ground. Any hit-rate table mixing the two arities measures the mix, not the skill.
Per-match multiplicity. With a mean of 2.77 predictions per match and a maximum of 358, one fixture can contribute dozens of correlated rows. A confidence-9 bucket can read "548 rows" while holding far fewer independent calls. Deduplicate to one row per match before computing any interval, or your Wilson bounds are narrower than reality — the one direction of error an audit must never make.
Per-model, the audit is brutal: of 3,715 model ids, only 7 have 200+ settled rows, and the worst of those claims 63.6% average confidence over 306 rows while delivering 55.23% — an 8.4-point overstatement no leaderboard sorted by raw hit rate would surface.
What an automated audit actually looks like
Calibration decays quietly. A single check is worthless; the useful artifact is a function that runs daily and fails loudly on inversion. This is the shape we use — the interesting part is not the ECE calculation, it is the INVERSION flag and the minimum-sample guard:
import math
from collections import defaultdict
MIN_N, INVERSION_GAP = 30, 0.04 # no claim below 30 rows; 4 pts back = failure
def wilson(k, n, z=1.96):
if n == 0:
return (0.0, 1.0)
p, d = k / n, 1 + z * z / n
c = (p + z * z / (2 * n)) / d
h = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / d
return (c - h, c + h)
def audit(rows):
"""rows: (match_id, model_id, stated_p, outcome_0_1).
Buckets hold [n, hits, sum_p] so ECE uses the real mean, not the midpoint --
midpoints hide exactly the skew we are hunting."""
seen, buckets = set(), defaultdict(lambda: [0, 0, 0.0])
for mid, _model, p, y in rows:
if mid in seen: # one independent observation per match
continue
seen.add(mid)
b = buckets[round(p * 10)]
b[0] += 1
b[1] += int(y)
b[2] += p
total = sum(b[0] for b in buckets.values())
ece = sum(b[0] * abs(b[2] / b[0] - b[1] / b[0]) for b in buckets.values())
report = {"n_independent": len(seen), "buckets": {}, "inversions": [],
"ece": round(ece / total, 4) if total else None}
prev = None
for decile, (n, k, _sp) in sorted(buckets.items()):
if n < MIN_N:
report["buckets"][decile] = {"n": n, "skipped": "n < MIN_N"}
continue # no claim, not a 50% guess
hit = k / n
lo, hi = wilson(k, n)
report["buckets"][decile] = {"n": n, "hit": round(hit, 4),
"ci95": [round(lo, 4), round(hi, 4)]}
if prev and prev[1] - hit > INVERSION_GAP:
report["inversions"].append((prev[0], decile))
prev = (decile, hit)
if report["inversions"]:
report["status"] = "FAIL: non-monotonic confidence"
return reportTwo design choices there matter. MIN_N = 30 with an explicit skipped marker means a bucket that cannot support a claim is reported as having no claim, not as a 50% guess. And inversion detection is a hard failure, not a warning: monotonicity is the minimum contract between a model and everyone reading its output.
The checklist
- Never publish an integer confidence as a probability. Store it as a label; derive a probability in a separate, versioned column with a fit date in its name.
- Bucket-then-diff, never average-then-report. Under-confident at the mean and 42 points over-confident at the top is not "slightly miscalibrated".
- Weight the audit by cost, not frequency. ECE's weighting makes the tiers that matter invisible — report the tail separately, always.
- Deduplicate to one observation per match, and state the deduplication in the report.
- Guard thin buckets explicitly. 57 rows is not evidence; n=57 with a 45–70% interval is a press release, not a measurement.
None of this requires new modelling — just admitting that the number the model hands you is a label needing a schema around it, the same conclusion we reached about immutability and leaderboards, from a third direction. Every number that leaves a prediction system needs a contract, and confidence is the one everyone forgets.
The table above is reproducible from the queries in this post against the public dataset snapshot; where they disagree, the dataset is the authority.
Analytics and research content. Not betting advice. 18+. 本内容仅供数据分析与研究参考，不构成任何决策建议。