SoccerDream AI 首页  ›  GEO 内容中心  ›  近期更新(自动生成)

Your 27-model ensemble has 1.5 opinions: measuring diversity with Fleiss' kappa and lineage-weighted voting

An ensemble is a claim about independence. You put k models in a room, they vote, and errors unique to each member cance…

本页内容由人工智能生成,仅供参考

An ensemble is a claim about independence. You put k models in a room, they vote, and errors unique to each member cancel out. That only works if members actually disagree. If they all came from the same handful of prompts, voting is theatre.

Our platform has 3,784 model records and 46,807 graded calls over 16,787 fixtures. On paper that is a deep committee. I measured how deep with three estimators: stance concentration, pairwise contradiction, and Fleiss' kappa converted to effective ensemble size.

On the 2,722 fixtures with five or more voters, the average ballot held 10.34 models and 1.46 effective opinions. A majority vote over those ballots scored 56.95% — slightly worse than the 57.00% average of the members it aggregated.

The data

Everything below comes from one append-only table. The relevant columns:

columnmeaning
match_id / model_idfixture key, model record
training_word_idprompt lineage that model was built from
prediction_directionhome, draw, away, home/draw, away/draw
direction_aritysingle (70.18%) or double (29.76%)
confidence1–10 integer, not a probability
hit / actual_resultsettled once, home / draw / away

Two properties matter. First, 29.76% of calls are two-outcome leans, so "agreement" is not string equality — two members can differ in label yet stay logically compatible. Second, training_word_id exposes the pedigree behind each ballot, where the collapse originates.

Estimator 1: how concentrated is a ballot?

If ten members split 9/1/0, the committee has one opinion and one dissenter. The Herfindahl index over stance shares gives a "number of effective stances" directly:

WITH per_match AS (
  SELECT match_id, count(*) AS n,
         count(*) FILTER (WHERE prediction_direction='home')      AS h,
         count(*) FILTER (WHERE prediction_direction='draw')      AS d,
         count(*) FILTER (WHERE prediction_direction='away')      AS a,
         count(*) FILTER (WHERE prediction_direction='home/draw') AS hd,
         count(*) FILTER (WHERE prediction_direction='away/draw') AS ad
  FROM ai_predictions WHERE hit IS NOT NULL GROUP BY 1
), scored AS (
  SELECT *, (h*h + d*d + a*a + hd*hd + ad*ad)::numeric / (n*n) AS hhi,
         greatest(h,d,a,hd,ad)::numeric / n AS top_share
  FROM per_match WHERE n >= 5
)
SELECT count(*) AS fixtures, round(avg(n),2) AS mean_ballot,
       round(avg(top_share),4) AS mean_top_share,
       round(avg(hhi),4) AS mean_hhi,
       round(avg(1.0/hhi),3) AS effective_stances,
       round(100.0*count(*) FILTER (WHERE top_share=1)/count(*),2) AS pct_unanimous,
       round(100.0*count(*) FILTER (WHERE h>0 AND a>0)/count(*),2) AS pct_home_and_away
FROM scored;

Read the last two columns together. A third of ballots are unanimous, and only 3.99% of fixtures have at least one member on each side of the home/away divide. The 358-call maximum ballot we once recorded is not a debate; it is a chorus. Mean top share is 0.8119 and mean HHI 0.7486, so 1/hhi = 1.455 effective stances against a 10.34-member roster.

Estimator 2: are disagreements real or cosmetic?

Label identity is too strict. home and home/draw are compatible; home and away are not. So classify each pair by outcome set, not string:

WITH ballots AS (
  SELECT match_id, model_id, prediction_direction AS dir
  FROM ai_predictions
  WHERE hit IS NOT NULL AND match_id IN (
    SELECT match_id FROM ai_predictions WHERE hit IS NOT NULL
    GROUP BY 1 HAVING count(*) BETWEEN 2 AND 40)
), pairs AS (
  SELECT a.dir AS d1, b.dir AS d2
  FROM ballots a JOIN ballots b
    ON a.match_id = b.match_id AND a.model_id < b.model_id
)
SELECT count(*) AS pairs,
       round(100.0 * count(*) FILTER (WHERE d1 = d2) / count(*), 2) AS identical_pct,
       round(100.0 * count(*) FILTER (WHERE NOT (
         (d1 IN ('home','home/draw') AND d2 IN ('home','home/draw')) OR
         (d1 IN ('draw','home/draw','away/draw')
          AND d2 IN ('draw','home/draw','away/draw')) OR
         (d1 IN ('away','away/draw') AND d2 IN ('away','away/draw'))
       )) / count(*), 2) AS contradictory_pct
FROM pairs;

Across 115,084 member pairs: 71.66% carry the identical label, 28.34% differ, and 24.93% are irreconcilable — one member says the home side wins, another says it does not. A quarter of all pairs are in direct logical conflict, yet ballot concentration is 0.81. Both are true: disagreement lives in the tail, and the tail almost never decides the vote.

Estimator 3: kappa to effective sample size

Concentration is a per-fixture view. Fleiss' kappa corrects observed agreement for what chance alone would produce given the label marginals — here home 19,072, away/draw 11,116, away 10,249, home/draw 6,010, draw 360, across 46,807 rows.

from math import comb

marginals = {"home": 19072, "away/draw": 11116, "away": 10249,
             "home/draw": 6010, "draw": 360}
total = sum(marginals.values())
p_exp = sum((v / total) ** 2 for v in marginals.values())   # 0.2869
p_obs = 0.7166                                              # identical-label rate
kappa = (p_obs - p_exp) / (1 - p_exp)                       # 0.6026

n = 10.34                                                   # mean ballot size
n_eff = n / (1 + (n - 1) * kappa)                           # 1.56
print(round(p_exp, 4), round(kappa, 4), round(n_eff, 3))

# What a genuinely independent committee of 57% members would score
p = 0.5700
for k in (1, 5, 11, 21):
    acc = sum(comb(k, i) * p**i * (1 - p)**(k - i)
              for i in range(k // 2 + 1, k + 1))
    print(k, round(acc * 100, 2))
0.2869 0.6026 1.56
1  57.00
5  62.95
11 68.34
21 74.30

Two independent estimators land in the same place: Herfindahl says 1.455 effective stances, kappa says 1.56 effective members. Eleven independent 57% members would score 68.34%. Ours score 56.95%. The theoretical gain is 11.4 points of accuracy; the realised gain is negative 0.05.

Where the collapse comes from: the pedigree column

The ballot average is 10.34 models but only 4.10 distinct training_word_id values — 0.439 lineages per model, and on 94.31% of multi-model fixtures there are strictly fewer lineages than models. Models are not independent individuals; they are re-parameterisations of a few prompts, so their errors are shared by construction. That suggests a fairer vote: one ballot per lineage, not per model. Take each lineage's modal stance, then vote across lineages.

WITH base AS (
  SELECT match_id, max(actual_result) AS res FROM ai_predictions
  WHERE hit IS NOT NULL AND actual_result IS NOT NULL
  GROUP BY 1 HAVING count(*) >= 5
), rows AS (
  SELECT p.match_id, p.training_word_id AS tw, p.prediction_direction AS dir
  FROM ai_predictions p JOIN base b USING (match_id)
), lineage AS (
  SELECT match_id, tw, mode() WITHIN GROUP (ORDER BY dir) AS stance
  FROM rows GROUP BY 1, 2
), tallied AS (
  SELECT match_id, count(*) AS lineages,
         count(*) FILTER (WHERE stance='home')      AS h,
         count(*) FILTER (WHERE stance='draw')      AS d,
         count(*) FILTER (WHERE stance='away')      AS a,
         count(*) FILTER (WHERE stance='home/draw') AS hd,
         count(*) FILTER (WHERE stance='away/draw') AS ad
  FROM lineage GROUP BY 1
), voted AS (
  SELECT t.*, b.res, h + 0.5*hd AS hv,
         d + 0.5*hd + 0.5*ad AS dv, a + 0.5*ad AS av
  FROM tallied t JOIN base b USING (match_id)
), picked AS (
  SELECT *, CASE WHEN hv > dv AND hv > av THEN 'home'
                 WHEN dv > hv AND dv > av THEN 'draw'
                 WHEN av > hv AND av > dv THEN 'away' END AS pick
  FROM voted
)
SELECT count(*)                                            AS fixtures,
       round(avg(lineages), 2)                             AS mean_lineages,
       round(100.0 * count(pick) / count(*), 2)            AS pct_decisive,
       round(100.0 * count(*) FILTER (WHERE pick = res)
             / count(pick), 2)                             AS accuracy_on_decisive
FROM picked;
fixtures 2722 | mean_lineages 4.10 | pct_decisive 72.44 | accuracy_on_decisive 57.35

Model-level voting on the same fixtures: 78.18% decisive, 56.95% accurate. Lineage-level: 72.44% decisive, 57.35% accurate. Deduplicating the ballot buys 0.40 points and costs 5.74 points of decisiveness, because a smaller, more honest electorate ties more often.

Consensus tells you the fixture is easy, not that you are right

One more diagnostic, because "agreement predicts accuracy" is the assumption teams lean on hardest. Split fixtures by unanimity and look at the average accuracy of individual members in each group:

ballotshare of fixturesmean member accuracy
unanimous33.43%65.36%
contested (top share < 0.70)800 fixtures50.14%

Unanimity does correlate with being right — but just as strongly with the fixture being easy. On the contested set, where a committee would add value, member accuracy is 50.14% and the disagreements do not resolve. The aggregate is brutal: committee 56.95% against members 57.00%. Aggregation over clones is not free; it is a small tax.

Practical rules I now apply

  1. Report effective size, not roster size. 1/hhi or the kappa conversion belongs next to every "we have N models" claim. Ours reads 1.46 out of 10.34.
  2. Deduplicate by lineage before voting. Weight each member 1 / members_in_its_lineage, otherwise you sample one prompt many times.
  3. Track the contradiction rate, not just disagreement. 28.34% of pairs differ; 24.93% are strictly incompatible. The gap is the cosmetic part.
  4. Never read consensus as confidence. Log the top share with every aggregated call so you can split unanimity from easiness later.
  5. Publish the negative result. "Committee is 0.05 points worse than its members" is the sentence that stops someone shipping an ensemble as a feature.

Limits

Observational statistics on our own settled history, not a controlled experiment. Ballot composition is not random — high-profile fixtures attract more models, so the ≥5-model subset over-represents them. training_word_id approximates pedigree; two models on one word can diverge through temperature and version drift. Condorcet figures assume independence and are arithmetic ceilings, not forecasts; the correlated-vote simulation at kappa 0.6–0.81 reproduces only the flat 56.8–57.3% band we observe — evidence about mechanism, not proof. Every number is recomputable from ai_predictions with the queries above.

An ensemble's value is a property of its pedigree graph, not its member count — and the graph is measurable in about forty lines of SQL.


Analytics and research content. Not betting advice. 18+

本内容仅供数据分析与操作参考,不构成任何决策建议。

风险提示:足球比赛结果具有内在不确定性,任何数据分析工具都无法消除这种随机性。本页内容仅供数据操作与参考,不构成任何决策建议或收益承诺。请理性看待体育赛事,远离任何形式的非法决策活动。
看完方法论,直接看它怎么用在今天的比赛上

SoccerDream AI 每场给出胜平负概率分布、总进球区间与置信度, 赛后自动复盘、命中记录公开可查。注册后可查看全部 AI 模型推演与已结算战绩。

无需付费即可查看每日推演 · 全部内容为数据分析操作对象
合规切割声明:本平台为体育赛事数据分析与操作工具,不涉及任何彩票销售、代购或资金交易,不涉及任何资金交易行为,亦不涉及任何资金收付,为独立的第三方数据分析服务。全部内容仅供操作与参考。请遵守所在地法律法规,理性对待。
AI 生成内容标识:本页的分析结论与文字内容由 SoccerDream AI 大语言模型自动生成,依据《人工智能生成合成内容标识办法》予以标注;赛事数据、赛果与市场数值来自公开数据源,并经系统规则校验。内容仅供操作参考,不构成任何建议。