SoccerDream AI 首页  ›  GEO 内容中心  ›  近期更新(自动生成)

Your leaderboard is ranking luck: 3,736 models, 46,284 graded calls, and a split-half correlation of 0.04

We run a population of model instances that publish pre-event probability calls on football fixtures. Every call is writ…

本页内容由人工智能生成,仅供参考

We run a population of model instances that publish pre-event probability calls on football fixtures. Every call is written to ai_predictions at creation, graded exactly once after the event, and stored as a boolean. As of this audit: 46,884 rows, 46,284 with a non-null grade, 26,415 hits — a global rate of 57.07%.

Our public leaderboard reads ai_models.hit_rate, sorts descending, renders the top. Standard design, structurally broken. The failure is not the sort comparator; it is a property of sorting point estimates computed from wildly unequal denominators.

Here is the board as it renders today:

rankmodel_idnhit_rate
1369597100.0
2388337100.0
3341157100.0
4432826100.0

Four rows, four perfect records, none above seven graded calls. Across the table, 327 models sit at exactly 100.0%, and 204 of those have a single graded call. Of the 584 models with at least 20 graded calls, zero are at 100%. The first hundred rows of the board contain no model with ten or more calls.

A leaderboard topped by 204 coin flips is not a leaderboard. It is a sampling artifact rendered as a ranking.

Two populations, one table

The first thing an audit has to separate is how many models exist versus how many have enough evidence to be compared.

-- population split: board cache vs. the fact table
SELECT count(*)                                        AS board_rows,
       count(*) FILTER (WHERE total_predictions > 0)   AS with_evidence,
       percentile_cont(0.5) WITHIN GROUP (ORDER BY total_predictions) AS median_n
FROM ai_models;

WITH t AS (
  SELECT model_id, count(*) AS n, count(*) FILTER (WHERE hit) AS h
  FROM ai_predictions
  WHERE hit IS NOT NULL AND model_id IS NOT NULL
  GROUP BY 1
)
SELECT count(*)                                        AS models,
       sum(n) AS rows, sum(h) AS hits,
       count(*) FILTER (WHERE n >= 20)                 AS n20,
       count(*) FILTER (WHERE n >= 50)                 AS n50,
       count(*) FILTER (WHERE n >= 200)                AS n200,
       percentile_cont(0.5) WITHIN GROUP (ORDER BY n)  AS median_n,
       max(n)                                          AS max_n
FROM t;

Output, unedited:

board_rows 8942 | with_evidence 3229 | median_n 7.0
models 3736 | rows 46284 | hits 26415 | n20 661 | n50 171 | n200 7 | median_n 6.0 | max_n 381

Three numbers matter. The median model has six graded calls, 171 models clear fifty, seven clear two hundred. Sorting 3,229 estimates where the median sample is 7 is not ranking models — it is ranking the draws.

The test: split each model's history in half

If a hit rate reflects a durable property of the model, the rate on the first half of its own history must predict the rate on the second half. If it reflects noise, the halves are independent and the ranking carries no forward information.

The split is by sequence position, not calendar time, so a model that started last week is treated like one that started in June. Both halves need at least ten graded calls, restricting the test to the 661 models with n ≥ 20.

WITH ordered AS (
  SELECT model_id, hit,
         row_number() OVER (PARTITION BY model_id ORDER BY id) AS rn,
         count(*)     OVER (PARTITION BY model_id)             AS n
  FROM ai_predictions
  WHERE hit IS NOT NULL AND model_id IS NOT NULL
),
halves AS (
  SELECT model_id, max(n) AS n,
         avg(CASE WHEN rn <= n/2 THEN hit::int END) AS h1,
         avg(CASE WHEN rn >  n/2 THEN hit::int END) AS h2,
         count(*) FILTER (WHERE rn <= n/2 AND hit)  AS k1,
         count(*) FILTER (WHERE rn <= n/2)          AS n1
  FROM ordered WHERE n >= 20 GROUP BY model_id
)
SELECT count(*) AS models, corr(h1, h2) AS pearson,
       avg(h1) AS mean_h1, stddev_samp(h1) AS sd_h1,
       avg(h2) AS mean_h2, stddev_samp(h2) AS sd_h2
FROM halves;

Result over 661 models:

models 661 | pearson 0.0443 | mean_h1 0.6291 | sd_h1 0.0991 | mean_h2 0.5656 | sd_h2 0.1135

A correlation of 0.0443. Spearman on the same pairs is 0.0261. For practical purposes these are zero — the first half of a model's record tells you almost nothing about its second half.

Note the drift, not just the flatness: first half averages 62.91%, second half 56.56%. Instances are not stationary across their own lifetime. One tuned against recent fixtures starts hot and mean-reverts, and a board built on the early record is systematically reading the hot streak.

What the top of the board is worth

Take the 661 pairs, sort by first-half rate, cut the top 66, look at what that group did next.

slicemodelsfirst halfsecond halfdecay
top decile6680.85%60.36%−20.5 pp
everyone else595—56.13%—
bottom decile6646.14%56.29%+10.2 pp

The top decile displays 80.85% and delivers 60.36%. The surviving edge over the rest of the population is 3.8 points, not the 24.7 the display implies. The bottom decile displays 46.14% and delivers 56.29% — the global rate within noise. Both tails converge on 56–57% because that is where the population lives; the spread at the tails was estimation error, not skill.

This is what rewarding luck means numerically. A 100.0% display on seven calls is not a stronger signal than a 60% display on 380 calls — it is a weaker one wearing a better font.

The multiplicity check

Under the null that every model draws from the same 57.07% process, the probability of a clean sweep at n calls is 0.5707^n:

n=5   6.05e-02
n=7   1.97e-02
n=10  3.67e-03
n=20  1.34e-05

At n=7 a perfect record happens once in fifty-one. With 3,229 board rows and thousands of small-n models, dozens of perfect records are expected, not remarkable. Summing 0.5707^n over every model's actual n gives an expected 661.75 perfect records. We observe 401.

Observed is below expected, and the reason is informative: the smallest buckets underperform. Models with 1–9 graded calls hit 49.02% across 8,606 rows; the 100–199 bucket hits 61.01%. Fresh instances start worse than the population they are compared against, so perfect records are not over-represented relative to chance — they are the surviving right tail of many short, slow-starting records.

Correct for multiplicity and the picture gets worse. With 3,229 candidates at α = 0.05 the per-test threshold is 1.55e-5, which requires n ≥ 20 with a clean sweep. We have 584 models at n ≥ 20 and none is perfect. No defensible 100% exists on this board.

Trying to fix the ranking function

The obvious next move is replacing the point estimate with something that penalises small samples — a Wilson lower bound, or a Beta posterior shrunk toward the population rate. Implement both, then re-run the same split-half evaluation at the same 66-model cut.

import math

P0 = 0.570716        # 26,415 hits / 46,284 graded rows
STRENGTH = 20.0      # pseudo-count mass pulling toward P0

def wilson_lb(k, n, z=1.96):
    if n == 0:
        return 0.0
    p = k / n
    denom  = 1 + z * z / n
    centre = p + z * z / (2 * n)
    margin = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n))
    return (centre - margin) / denom

def shrunk(k, n, prior=P0, strength=STRENGTH):
    return (k + prior * strength) / (n + strength)

def top_slice(rows, key, k):
    """rows: dicts with h1, k1, n1, n, h2"""
    chosen = sorted(rows, key=key, reverse=True)[:k]
    return {"second_half": round(sum(r["h2"] for r in chosen) / len(chosen), 4),
            "mean_n":      round(sum(r["n"]  for r in chosen) / len(chosen), 1)}

Run over the same 661 pairs, k = 66:

ranking functionmean nsecond-half rate
raw first-half rate34.10.6036
Wilson lower bound (95%)55.80.5936
Beta posterior, strength 2044.50.5858
raw rate, gate n ≥ 3046.00.5777
no ranking (all 661)—0.5656

The honest result: none of them fixes the forward rate. Every variant lands within 3.8 points of the population mean, and the raw sort is nominally best — not because raw sorting is good, but because the top decile of a positively-correlated noisy signal still concentrates some genuine signal, and shrinkage trades that sliver for conservatism.

The problem is not the ranking function. It is the object being ranked. With a median of six graded calls, no monotone transform of (k, n) can recover forward skill, because there is no forward skill in the estimator to recover.

What actually works

Never render a point estimate without its denominator and interval. 60.4% (n=62, 95% CI 47.9–71.7) is honest. 100.0% is a lie of omission when n=4. The interval is one line of Python; the bare number has no excuse.

Freeze the board as a historical artifact. A live query whose ordering reshuffles as rows arrive can be gamed by waiting. Materialise a weekly snapshot of (as_of, model_id, k, n, rank) and publish that. A model falling from rank 1 to rank 90 then becomes a visible fact instead of a silent overwrite.

Report the panel, not the total. The 171 models at n ≥ 50 average 59.66% with a standard deviation of 6.17 points. That spread is the real content: the credible population sits in a band roughly ±12 points wide at two sigma. A ranked list of 3,229 rows destroys that information; a distribution preserves it.

Set the threshold by what the estimate can support. At n = 50 the standard error of a 0.6 estimate is 0.069, so a 95% interval spans ~27 points. Nobody should compare two models across a 27-point interval. Either label the board as a discovery surface below fifty calls, or restrict ranking to the 171 models where the interval is narrower than the effect being claimed.

The generalisable lesson

Any system ranking entities by a ratio of two counters fails this way once the denominators vary by more than an order of magnitude — recommender A/B tests, creative scoring, per-shard error rates, per-endpoint latency percentiles, analyst dashboards. The tell is always the same: the top of the list is held by the smallest denominators, and the list collapses into ties at the boundary value (100%, 0%, perfect).

The diagnostic is cheap: split each entity's own history in half, correlate the halves, and check what the top decile delivers next. Near-zero correlation plus a twenty-point mean reversion means the ranking is measuring sample size, not the entity.

We still show the board. We now show it with n, with intervals, with a weekly frozen snapshot, and with a hard label that below fifty graded calls the ordering is not a comparison. The models did not get worse. The display stopped flattering the ones that had barely played.

Analytics and research content. Not betting advice. 18+


本内容仅供数据分析与操作参考,不构成任何决策建议。

风险提示:足球比赛结果具有内在不确定性,任何数据分析工具都无法消除这种随机性。本页内容仅供数据操作与参考,不构成任何决策建议或收益承诺。请理性看待体育赛事,远离任何形式的非法决策活动。
看完方法论,直接看它怎么用在今天的比赛上

SoccerDream AI 每场给出胜平负概率分布、总进球区间与置信度, 赛后自动复盘、命中记录公开可查。注册后可查看全部 AI 模型推演与已结算战绩。

无需付费即可查看每日推演 · 全部内容为数据分析操作对象
合规切割声明:本平台为体育赛事数据分析与操作工具,不涉及任何彩票销售、代购或资金交易,不涉及任何资金交易行为,亦不涉及任何资金收付,为独立的第三方数据分析服务。全部内容仅供操作与参考。请遵守所在地法律法规,理性对待。
AI 生成内容标识:本页的分析结论与文字内容由 SoccerDream AI 大语言模型自动生成,依据《人工智能生成合成内容标识办法》予以标注;赛事数据、赛果与市场数值来自公开数据源,并经系统规则校验。内容仅供操作参考,不构成任何建议。