SoccerDream AI 首页  ›  GEO 内容中心  ›  近期更新(自动生成)

Deleting your losers buys 4.1 points of hit rate: an audit protocol for model retirement

Every prediction system eventually grows a lifecycle manager: models get born, promoted, demoted and killed. Ours has 8,…

本页内容由人工智能生成,仅供参考

Every prediction system eventually grows a lifecycle manager: models get born, promoted, demoted and killed. Ours has 8,789 model rows and a retirement pipeline that runs every two hours. For a long time the number we published was computed over whatever models happened to still exist at query time — a mistake whose size is measurable.

46,160 graded predictions carry a boolean hit. The all-cohort rate is 57.05% (26,334 correct). The rate over models still in a live lifecycle state is 61.17% (19,281 of 31,520). Nothing about the models changed between those queries — only the WHERE clause did. A 4.12-point swing produced entirely by bookkeeping. In a system that retires models automatically, survivorship bias is not an edge case; it is the default output. What follows is the audit protocol we now run before any retirement decision.

Where a model actually goes when it dies

There is no single "deleted" state. A model leaves the visible cohort through six different doors, and the aggregate rate treats each differently.

lifecycle_statusmodelsgraded predshitshit rate
evolving2,16531,10519,00161.09%
eliminated2,7087,9503,58645.11%
demoted2,9371,01434534.02%
retiring58933019659.39%
pending_retrain563189028.30%
archived170———
hibernating1001487550.68%
active2441528067.47%
retraining6251248.00%
(no ai_models row)525 ids4,8542,74856.61%

Three things stand out. First, the worst cohorts are also the smallest. demoted holds 2,937 model rows but only 1,014 graded predictions — 0.35 per model. pending_retrain holds 56 models and 318 predictions at 28.30%. These are not measured failures; they are barely-measured ones. A rule that fires on 0.35 observations per model is not selecting on quality, it is selecting on noise.

Second, eliminated is a real signal. 801 models, 7,950 graded predictions, 45.11% — 11.98 points below the evolving cohort on a sample worth reading.

Third: 4,854 graded predictions reference 525 model_id values that no longer exist in ai_models at all. Their score, 56.61%, is close to the pool average, and any metric written as JOIN ai_models silently drops every one of them. We also keep an ai_models_archive table with 4,054 rows, while archived_at is NULL on all 8,789 live rows — the primary table cannot tell you when anything was retired.

The audit query: publish two numbers, never one

The protocol: publish two numbers from one query so they cannot drift apart — the headline, and its complement over the full historical cohort including rows whose model row is missing.

-- Cohort audit: the same graded rows, seen through two different filters.
WITH graded AS (
    SELECT p.id,
           p.model_id,
           p.hit,
           p.created_at,
           COALESCE(m.lifecycle_status, 'hard_deleted') AS lc
    FROM ai_predictions p
    LEFT JOIN ai_models m ON m.id = p.model_id
    WHERE p.settled = TRUE
      AND p.hit IS NOT NULL
)
SELECT 'all_cohort'      AS view,
       COUNT(*)          AS n,
       SUM(hit::int)     AS hits,
       ROUND(100.0 * SUM(hit::int) / COUNT(*), 2) AS hit_rate
FROM graded
UNION ALL
SELECT 'live_only', COUNT(*), SUM(hit::int),
       ROUND(100.0 * SUM(hit::int) / COUNT(*), 2)
FROM graded
WHERE lc IN ('evolving', 'active', 'incubating', 'validating')
UNION ALL
SELECT 'retired_cohort', COUNT(*), SUM(hit::int),
       ROUND(100.0 * SUM(hit::int) / COUNT(*), 2)
FROM graded
WHERE lc IN ('eliminated', 'demoted', 'archived');
     view      |   n   | hits  | hit_rate
---------------+-------+-------+----------
 all_cohort    | 46160 | 26334 |    57.05
 live_only     | 31520 | 19281 |    61.17
 retired_cohort|  8964 |  3931 |    43.85

The rule we now enforce: the delta between all_cohort and live_only is itself a monitored metric. Call it the retirement gap. When it grows, the pipeline is deleting measured failures faster than unmeasured models — flattering the published number without improving the system. Month by month:

monthgradedall-cohortlive-onlygapretired share of rows
2026-062,73562.30%62.94%+0.641.50%
2026-072,63750.89%58.94%+8.051.82%
2026-0816462.80%70.37%+7.5732.32%
2026-099,73958.51%62.13%+3.6226.43%
2026-1021,82456.28%60.85%+4.5722.99%
2026-119,06057.43%60.95%+3.5213.58%

The gap is not constant. It is smallest in the month when almost nothing had been retired and largest when a quarter of the pool was already gone. The full series runs between 50% and 62%, and part of the recent stability is the deletion of 23% of the evidence.

Why the survivors look better than they are

Hard deletion destroys the denominator. 525 model ids vanished from ai_models while their predictions stayed, so every JOIN-based metric silently excludes 4,854 rows — worse than a wrong number, because nothing in the output declares it.

Status filters are applied at read time, not write time. The same physical rows produce 57.05% or 61.17% depending on a WHERE clause chosen by a dashboard author, and no schema-level signal marks that the choice exists.

The leaderboard adds a minimum-sample filter on top. Of 4,681 models with any graded prediction, only 660 reach 20 and 170 reach 50; the median is 6. A "top models by hit rate" view is therefore almost entirely models whose whole record is under ten games, and sorting it descending returns 100% records: 388 with fewer than five graded predictions, 18 more between five and nine.

Here is the part that should end the argument. If every model were exchangeable at the pool rate of 0.5705, the expected number of perfect records is Σ p^n over the actual sample sizes:

models with n < 5   : expected 634.5 perfect records, observed 388
models with n >= 5  : expected  25.3 perfect records, observed  18

Observed is below the coin-flip expectation in both bands. A 4-for-4 record is not evidence of skill here — it is less impressive than chance would produce. Any rule that ranks on raw rate ranks a mixture of noise and sample size, and the models it rewards are the ones that guessed few times.

Make retirement append-only and auditable

The fix is not "stop filtering" — some views legitimately want the live cohort. The fix is to make the filter explicit, permanent and reversible, so excluded rows stay addressable. Retirement becomes an event, never a delete.

"""Retirement ledger: retirement is an event, never a delete."""
import hashlib, json, datetime as dt

def _h(prev: str, payload: dict) -> str:
    body = json.dumps(payload, sort_keys=True, separators=(",", ":"))
    return hashlib.sha256((prev + body).encode()).hexdigest()

def retire(cur, model_id: int, new_state: str, reason: str, evidence: dict):
    """Append one row. Never UPDATE ai_models.status, never DELETE."""
    cur.execute(
        "SELECT state, entry_hash FROM model_lifecycle_ledger "
        "WHERE model_id = %s ORDER BY id DESC LIMIT 1", (model_id,))
    row = cur.fetchone()
    prev_state, prev_hash = (row if row else ("born", "0" * 64))

    payload = {
        "model_id": model_id, "from_state": prev_state,
        "to_state": new_state, "reason": reason,     # 'eliminated_low_n'
        "evidence": evidence,                        # n, hits, rate, window
        "at": dt.datetime.utcnow().isoformat(timespec="seconds"),
    }
    cur.execute(
        "INSERT INTO model_lifecycle_ledger "
        "(model_id, from_state, to_state, reason, evidence, entry_hash, prev_hash) "
        "VALUES (%s,%s,%s,%s,%s,%s,%s)",
        (model_id, prev_state, new_state, reason,
         json.dumps(evidence), _h(prev_hash, payload), prev_hash))
    # projections may change; history may not
    cur.execute("UPDATE ai_models SET lifecycle_status = %s WHERE id = %s",
                (new_state, model_id))

With that ledger, the cohort view stops being a guess:

CREATE OR REPLACE VIEW v_model_cohort AS
SELECT m.id AS model_id,
       m.lifecycle_status AS current_state,
       COALESCE(f.first_retired_at, TIMESTAMPTZ 'infinity') AS retired_at,
       f.retirement_reason
FROM ai_models m
LEFT JOIN LATERAL (
    SELECT MIN(at) AS first_retired_at,
           (ARRAY_AGG(reason ORDER BY at))[1] AS retirement_reason
    FROM model_lifecycle_ledger l
    WHERE l.model_id = m.id
      AND l.to_state IN ('eliminated', 'demoted', 'archived')
) f ON TRUE;

Now "as of date D" is a temporal join instead of a status filter: a model retired in September still contributes its September predictions to every point-in-time metric, and the historical series can no longer be rewritten by a later decision. Three invariants belong in CI, each of which our bad state violated:

  1. COUNT(*) FROM ai_predictions p WHERE NOT EXISTS (SELECT 1 FROM ai_models m WHERE m.id = p.model_id) must be zero. Ours was 4,854.
  2. archived_at must be non-NULL for every row whose state is archived. Ours was 0 of 170.
  3. Every state change needs a ledger row, and the chain must verify from genesis. Ours had 1,185 rows with reactivated_at set and no recorded reason.

What a retirement decision actually needs

Judge a retirement rule on the predictions it would have removed, not on the leaderboard it produces afterwards. Report four numbers together for every candidate cohort: n, hits, rate, and the rate of the cohort that survives. A rule removing 8,964 rows at 43.85% from a pool running at 57.05% raises the published number by 3.18 points while deleting 19.4% of the evidence. Whether that trade is defensible depends on whether the rule fired on measured failure — and with demoted averaging 0.35 graded predictions per model, ours mostly did not.

The honest summary of our pool: 63.6% of model rows (5,587 of 8,789) have never produced a single graded prediction, the median model has six, only 170 clear fifty, and 253 models were fathered by parents that were later eliminated. A system that kills a parent but keeps its descendants runs a selection loop it cannot audit, because the descendants inherit a training word whose measured performance was negative and nothing records the inheritance.

Retirement is a data operation, not a UI operation. If deleting a row changes the headline metric, that metric is a function of your bookkeeping rather than your model quality — and the cheapest way to improve it is always to delete more.

Analytics and research content. Not betting advice. 18+

Disclaimer: This content is provided for data analysis and research purposes only and does not constitute any decision-making advice.

风险提示:足球比赛结果具有内在不确定性,任何数据分析工具都无法消除这种随机性。本页内容仅供数据操作与参考,不构成任何决策建议或收益承诺。请理性看待体育赛事,远离任何形式的非法决策活动。
看完方法论,直接看它怎么用在今天的比赛上

SoccerDream AI 每场给出胜平负概率分布、总进球区间与置信度, 赛后自动复盘、命中记录公开可查。注册后可查看全部 AI 模型推演与已结算战绩。

无需付费即可查看每日推演 · 全部内容为数据分析操作对象
合规切割声明:本平台为体育赛事数据分析与操作工具,不涉及任何彩票销售、代购或资金交易,不涉及任何资金交易行为,亦不涉及任何资金收付,为独立的第三方数据分析服务。全部内容仅供操作与参考。请遵守所在地法律法规,理性对待。
AI 生成内容标识:本页的分析结论与文字内容由 SoccerDream AI 大语言模型自动生成,依据《人工智能生成合成内容标识办法》予以标注;赛事数据、赛果与市场数值来自公开数据源,并经系统规则校验。内容仅供操作参考,不构成任何建议。