Every prediction platform grows a gate. Not the model — the filter after the model, the code deciding which predictions the public may see. Ours is one boolean, is_visible, on a table with 47,150 rows. 8.98% are visible; the other 91% are computed, stored, graded, and hidden.
A gate is a claim: the rows I publish are better than the rows I suppress. That claim is testable. Almost nobody tests it. This is the audit, run against production data — including the part where the gate fails.
The setup: six gate-related columns, three of them dead
ai_predictions holds one row per model per match. Columns that look like they participate in the publish decision:
is_visible boolean 47,150 populated
rec_level varchar 46,721 populated (A/B/C/D)
bet_worthiness smallint 21,002 populated (1..8)
veto_reason text 10,223 populated
gate_reason text 0 populated
gate_want_visible boolean 0 populated
direction_gate varchar 0 populated
Three of six are NULL on every row. That is the first finding — gate_want_visible at 0% occupancy says the gate was refactored and the audit trail did not survive. veto_reason survives on 10,223 rows, which is a reason code we can work with.
Baseline: 47,150 rows, 46,996 settled, 46,877 graded, 26,773 hits (57.11%), 3,796 models, 16,898 matches.
Step 1 — the pooled comparison, and why it is nearly useless
Start with the obvious query — the one that will mislead you.
SELECT is_visible,
count(*) AS n,
count(*) FILTER (WHERE hit) AS hits,
round(100.0 * count(*) FILTER (WHERE hit)
/ nullif(count(*) FILTER (WHERE hit IS NOT NULL), 0), 2) AS hit_pct
FROM ai_predictions
WHERE hit IS NOT NULL
GROUP BY 1;
is_visible | n | hits | hit_pct
false | 42684 | 24360 | 57.07
true | 4231 | 2413 | 57.53
A lift of +0.48pp. With 4,231 published rows against 42,684 suppressed, the standard error of that difference is 0.80pp, so z = 0.60. Wilson intervals: visible 56.05–59.04, hidden 56.60–57.54 — overlapping across almost their entire width.
The gate's output is not measurably different from its own reject pile. Publish a random 8.9% sample instead and you would land within noise of what the gate actually publishes.
Two caveats. The gate need not be useless to produce this number — only diffuse: 4,231 rows from 543 models, selected by a mechanism that varies per model, pool flat even if each local gate works. And the audit was possible only because 42,684 of 42,918 suppressed rows (99.45%) were graded. Stop grading what you hide and the gate becomes unfalsifiable.
Step 2 — stratify by model, and watch the sign flip
The pooled query collapsed 3,796 models into one number. Redo it per model, keeping only models that publish some and suppress some — else it degrades to "always shows" vs "never shows".
WITH per_model AS (
SELECT model_id,
count(*) FILTER (WHERE hit IS NOT NULL AND is_visible) AS vis_graded,
count(*) FILTER (WHERE hit AND is_visible) AS vis_hits,
count(*) FILTER (WHERE hit IS NOT NULL AND NOT is_visible) AS hid_graded,
count(*) FILTER (WHERE hit AND NOT is_visible) AS hid_hits
FROM ai_predictions
GROUP BY 1
)
SELECT count(*) AS models_compared,
count(*) FILTER (WHERE hid_hits * 1.0 / hid_graded
> vis_hits * 1.0 / vis_graded) AS gate_is_worse,
round(avg((vis_hits * 1.0 / vis_graded
- hid_hits * 1.0 / hid_graded) * 100), 2) AS mean_lift_pp
FROM per_model
WHERE vis_graded >= 5 AND hid_graded >= 5;
models_compared | gate_is_worse | mean_lift_pp
218 | 146 | -7.92
On 146 of 218 models, the published rows are worse than the hidden ones. Mean per-model lift is −7.92pp. Across the 194 models publishing at least 5 graded rows, visible rows average 56.05% and their suppressed siblings average 63.15%.
This is Simpson's paradox, and it is why a pooled gate audit is worthless. The pooled number looks fine only because visible rows are unevenly distributed: 387 of the 675 models with ≥20 graded rows (57.3%) never publish a single row, contributing zero to the "visible" bucket. The pooled average is a weighted sum whose weights have nothing to do with the gate's competence.
Step 3 — the tier system does not rank what it claims to rank
rec_level is the human-facing grade: A strongest, D weakest. It is populated on 46,721 rows.
| rec_level | graded rows | hit % | 95% Wilson | published share |
|---|---|---|---|---|
| A | 1,052 | 52.95 | 49.93–55.95 | 15.97% |
| B | 2,027 | 63.15 | 61.06–65.19 | 14.41% |
| C | 15,859 | 57.66 | 56.91–58.41 | 9.46% |
| D | 27,514 | 56.56 | 56.00–57.12 | 7.78% |
A is the worst grade in the table — 3.61pp below D, z = −2.31, p ≈ 0.021. B is the best, 6.6pp over A. The ordering is B > C > D > A: rec_level does not encode "how likely this row is to be right", and 1,052 rows carry the strongest label while being the weakest cohort.
Now the gate's behaviour inside each tier:
| tier | published hit % | suppressed hit % | gate lift |
|---|---|---|---|
| A | 58.33 (n=168) | 51.92 (n=884) | +6.41pp |
| B | 62.33 (n=292) | 63.29 (n=1,735) | −0.96pp |
| C | 57.50 (n=1,501) | 57.68 (n=14,358) | −0.18pp |
| D | 56.14 (n=2,141) | 56.60 (n=25,373) | −0.46pp |
Inside tier A the gate works. Inside B, C and D it does not. And the published mix is 168 A / 292 B / 1,501 C / 2,141 D — 86.86% of published rows come from the two lowest grades. The gate operates where it has no edge and stays quiet where it has one: an A row is barely more likely to surface than a D row (15.97% vs 7.78%), and that 2× ratio cannot rescue an inverted tier.
Step 4 — a confidence inversion visible in one table
If the gate were a confidence threshold, publication would rise with confidence. That column is an integer 1–10, not a percentage.
| confidence | graded | published % | hit % |
|---|---|---|---|
| 3 | 8,212 | 4.46 | 53.64 |
| 4 | 8,978 | 7.03 | 60.16 |
| 5 | 11,780 | 12.57 | 56.04 |
| 6 | 9,534 | 12.48 | 55.07 |
| 7 | 4,329 | 8.45 | 60.04 |
| 8 | 2,041 | 4.26 | 65.51 |
| 9 | 549 | 1.09 | 73.77 |
| 10 | 58 | 0.00 | 56.90 |
Publication peaks at 5–6, where accuracy sits below average, and collapses to 1.09% at 9 — the most accurate bucket in the table at 73.77%. The 58 rows at 10 are never published.
So it is not a confidence filter. rec_reason says what it actually is:
"结构未知(样本不足或无此组合)" 12,932 rows
"结构B但非赛事分析/赛事盘面(观察)" 7,490 rows
"低质结构 ROI-8.3% (n=222)" 3,043 rows
"低质结构 ROI-4.8% (n=31)" 2,399 rows
"低质结构 ROI-0.2% (n=493)" 2,075 rows
The gate keys off a structure bucket and labels it with a bucket-level return figure computed over n=31, n=222, n=493 prior rows. That is the failure mode in one line: a decision about a single match made by a statistic computed over a different set of matches. It also explains the negative per-model lift — bucket membership is a property of market shape, not of the evidence for that particular match.
Step 5 — two integrity failures that block any future audit
Published rows carrying a veto. veto_reason is populated on 10,223 rows; 803 of them sit where is_visible = true. 19.0% of the published set has a recorded reason for being rejected. Either the flag is written after publication, or the veto is advisory and not wired into the publish path. Both are unverifiable from outside, and both mean the gate has no single source of truth.
Partial slates. 4,232 published rows span 4,212 of 16,898 matches — 24.93% carry at least one published row, and exactly 4 matches are fully published. A reader is not seeing "this match has a published view" but "this match had one model whose row survived", with the model varying per match. Any comparison built on that surface compares different things under one label.
Step 6 — what an auditable gate looks like
None of this is fixable with a threshold. The gate has to become a decision with a recorded counterfactual and a published scorecard.
import math
def wilson(k, n, z=1.96):
p = k / n
d = 1 + z * z / n
c = (p + z * z / (2 * n)) / d
h = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / d
return c - h, c + h
def audit_gate(vis_hits, vis_n, hid_hits, hid_n, min_lift_pp=3.0, z_crit=1.96):
"""PASS only if the gate beats its own reject pile by a margin
that this sample can actually resolve."""
pv, ph = vis_hits / vis_n, hid_hits / hid_n
pooled = (vis_hits + hid_hits) / (vis_n + hid_n)
se = math.sqrt(pooled * (1 - pooled) * (1 / vis_n + 1 / hid_n))
lift_pp = (pv - ph) * 100
lo, hi = wilson(vis_hits, vis_n)
need_n = 2 * pooled * (1 - pooled) / (min_lift_pp / 100) ** 2
return {
"lift_pp": round(lift_pp, 2),
"z": round((pv - ph) / se, 3),
"vis_ci": (round(lo * 100, 2), round(hi * 100, 2)),
"n_needed_per_arm": round(need_n),
"verdict": "PASS" if (lift_pp >= min_lift_pp
and abs((pv - ph) / se) >= z_crit
and min(vis_n, hid_n) >= need_n) else "FAIL",
}
print(audit_gate(2413, 4231, 24360, 42684))
# {'lift_pp': 0.46, 'z': 0.578, 'vis_ci': (56.05, 59.01),
# 'n_needed_per_arm': 544, 'verdict': 'FAIL'}
The n_needed_per_arm term matters: at a 57% base rate, resolving a 3pp lift takes ~544 graded rows per arm. Both arms clear that bar, so the effect is genuinely absent — which blocks the usual answer ("we need more data").
Four changes follow: publish the gate's own scorecard, not only its output; keep grading suppressed rows, since 99.45% coverage is what made this audit possible; wire up the dead columns or delete them; and make the decision and the label one event, since 803 published rows with a veto reason mean the publish and skip paths can disagree.
The uncomfortable summary
A gate publishing 8.9% of 47,150 graded rows, with a pooled lift of +0.48pp (z = 0.60), a per-model lift of −7.92pp across 218 models where it fails on 146, an A-tier 3.6pp below its own D-tier, a confidence curve that publishes most where accuracy is lowest, and 19% of its output carrying a veto reason.
That is not a tuning failure. It is what happens when a publication decision is made by a bucket statistic and never scored against its own counterfactual. The fix is not a better threshold — it is making the gate answerable to the rows it threw away, which requires having graded them.
Analytics and research content. Not betting advice. 18+
本内容仅供数据分析与操作参考,不构成任何决策建议。