The draw is 24% of the outcomes and 0.8% of the labels: auditing class collapse in a three-way model

A three-class classifier that reports one label per row is not a three-class classifier. It is a ranking function collapsed to a decision, and the collapse is where the information dies. On our platform, 11,177 of 46,189 settled predictions ended in a draw; the label draw was emitted 360 times.

Every number below comes from the live production table, queried today, not estimated.

Two distributions that are required to match, and don't

The base table stores one row per model per match with the realised outcome attached. Two distributions matter: what happened, and what the model said would happen. If the second is a faithful compression of the first, they should be close.

-- D0: outcome distribution vs emitted-label distribution
WITH actual AS (
  SELECT actual_result AS cls, count(*) AS n
  FROM ai_predictions
  WHERE settled AND actual_result IS NOT NULL
  GROUP BY 1
),
emitted AS (
  SELECT prediction_direction AS cls, count(*) AS n
  FROM ai_predictions
  WHERE settled
  GROUP BY 1
)
SELECT 'actual' AS side, cls, n,
       round(100.0 * n / sum(n) OVER (), 2) AS pct
FROM actual
UNION ALL
SELECT 'emitted', cls, n,
       round(100.0 * n / sum(n) OVER (), 2)
FROM emitted
ORDER BY side, n DESC;

The draw happens 24.20% of the time and is named as the sole answer 0.78% of the time — a 31× under-representation. 16,925 rows name it as one of two outcomes; 27,647 rows chose strictly between home and away and could not be correct if the match finished level.

That last set is the interesting one. Of those 27,647 binary picks, 6,269 (22.68%) finished level. The ceiling was set by the label vocabulary, not the analysis.

Why the headline accuracy hides it

Reported accuracy by output arity on rows with a settled outcome:

58.52% on single-label rows looks like a margin over chance. But chance for a 44/31/24 class split under a majority-class strategy is 44.26% — and the "single" bucket is not homogeneous: 4,490 of those rows had a compound single label (home/draw, away/draw) and 27,647 were strictly two-class. Averaging them reports neither.

This is a binary classifier wearing a three-class schema: any binary classifier over football outcomes pays a fixed 22.68% tax on the subset it is applied to. A genuine three-class system moves that tax into calibrated probability on the draw.

The audit query that should be a unit test

Aggregate accuracy is what lets this bug live for 46,000 rows. Per-class precision and recall make it visible in one screen:

-- D1: per-class precision / recall on the collapsed label set
WITH base AS (
  SELECT actual_result, string_to_array(prediction_direction, '/') AS picked
  FROM ai_predictions
  WHERE settled AND actual_result IS NOT NULL
),
exploded AS (
  -- a compound label is a set; a single label is a set of one
  SELECT actual_result, unnest(picked) AS cls FROM base
),
per_class AS (
  SELECT cls,
         count(*)                                    AS emitted_n,
         count(*) FILTER (WHERE actual_result = cls) AS correct
  FROM exploded
  GROUP BY cls
),
totals AS (
  SELECT actual_result AS cls, count(*) AS actual_n FROM base GROUP BY 1
)
SELECT c.cls, c.emitted_n, c.correct,
       round(100.0 * c.correct / c.emitted_n, 2) AS precision_pct,
       round(100.0 * c.correct / t.actual_n,  2) AS recall_pct
FROM per_class c
JOIN totals t USING (cls)
ORDER BY c.emitted_n DESC;

Result on the 45,860 rows with a non-null outcome:

Draw precision of 26.81% hides the shape of the failure: draw alone hit 78 of 360 (21.67%), while away/draw hit 5,920 of 10,997 (53.83%) and home/draw hit 3,272 of 5,928 (55.20%). The compound labels are not predicting a draw — they express uncertainty between two outcomes, and the draw is an escape hatch. Recall on the draw is 41.16%: 6,576 draws were named by no label at all.

The most damning row: 283 distinct models emitted draw at least once across 46,189 predictions, and the busiest did it 4 times. No model here has learned to identify draws. Some occasionally guess.

Scoring both systems on one yardstick

Accuracy cannot compare a one-hot label with a distribution. Log loss can. The emitted label is scored as a probability vector (1.0 on the named class, split evenly for compounds) against the market-implied probability from the quoted three-way price, normalised to sum to one. 45,110 of 46,189 rows survive the price sanity filter.

import math

def implied(prices):
    inv = [1.0 / p for p in prices]
    s = sum(inv)
    return [x / s for x in inv]

LL_MARKET = LL_MODEL = BRIER_DRAW = 0.0
n = 0
for actual, ph, pd, pa, direction, arity in rows:
    mh, md, ma = implied((ph, pd, pa))
    LL_MARKET += -math.log(max({"home": mh, "draw": md, "away": ma}[actual], 1e-12))
    BRIER_DRAW += (md - (1.0 if actual == "draw" else 0.0)) ** 2

    if arity == "single":
        q_true = 1.0 if direction == actual else 0.0
    else:
        parts = direction.split("/")
        q_true = 1.0 / len(parts) if actual in parts else 0.0
    LL_MODEL += -math.log(max(q_true, 1e-12))
    n += 1

print(n, round(LL_MARKET / n, 4), round(LL_MODEL / n, 4), round(BRIER_DRAW / n, 4))
# 45110  0.9683  13.4625  0.1812

Log loss 0.9683 for the market-implied probabilities, 13.4625 for the emitted label — a factor of 13.9. That is what a zero looks like inside a logarithm: every row where the model picked a compound and the outcome was the excluded class contributes -log(0), clamped at 1e-12, or 27.6 nats.

This is not a claim that market probabilities are a good model. It is a claim about representation: a probability vector can be scored and improved; a one-hot label can only be counted, and counting rewards the majority class. The draw Brier of 0.1812 is what a real three-class output would have to beat.

What actually carries draw information

If confidence carried draw-risk signal, high-confidence binary picks would convert to draws less often. They don't. Draw rate among single picks restricted to home or away (27,647 rows, 6,269 draws):

Flat, and mildly inverted at the top: a model that says 9 is no less likely to be level than one that says 4. That is the expected failure when confidence is attached to a binary comparison and the third class was pruned before scoring — the pruning discards the evidence.

The market-implied draw probability, on the same 45,110 rows, is calibrated in the ordinary sense:

Mean implied 0.2423 against an observed 24.20%. The information is already in a column we store, and it is thrown away at the label step.

The imbalance is structural, not random

Draw rates are not uniform across competitions. On 78,553 finished matches the overall draw rate is 24.69% (19,392 draws), and competition buckets with at least 500 settled rows span 15.49% to 30.42%:

A single global threshold for "is this a draw?" is therefore wrong by about 15 percentage points. A per-class correction must be conditioned on the competition, and that requires a calibrated probability rather than a relabelled decision — you cannot attach a competition prior to a one-hot output.

Six rules that would have prevented this

1. Store the distribution, not the decision. A row should carry p_home, p_draw, p_away; the label is a rendering derived at read time. Ours stores prediction_direction and a 1–10 integer, so every metric above had to reconstruct intent from a string.
2. Never score a three-class problem by accuracy. Log loss and per-class recall, in the same query as the headline number.
3. Assert a per-class floor in CI. If the draw is emitted in under 10% of rows while the observed base rate is 24%, fail the build.
4. Measure compound labels separately. home/draw at 55.20% precision is not three-class capability; it inflates the "single" hit rate whenever the draw lands.
5. Condition on competition before claiming a threshold. A 15.49%–30.42% spread makes one global cut miscalibrated for every league it touches.
6. Score against market-implied probability. It is the only genuine distribution in the schema and it is already calibrated on our own rows — without it, a 0.78% draw rate looks like a modelling choice rather than a defect.

Two integrity facts

The 46,189 settled predictions cover 16,724 distinct matches, and only 3,736 (8.09%) join to matches on id; the rest carry a match_id from an older identifier space. Every per-competition number above was computed from matches directly — joining predictions to competitions would have dropped 92% of rows and produced a confident, wrong table.

Second: 329 settled rows have actual_result IS NULL and 121 have hit IS NULL while settled is true. They are excluded explicitly with FILTER.

The point

A 24% versus 0.8% gap is not a dataset problem. Twenty-four percent is an ordinary share for the third class of a football outcome. The imbalance was manufactured by the output format: the model emitted a label, the vocabulary had three slots, and the scoring loop only rewarded the slot that appeared most. Fixing it needs no better model — three numbers instead of one string, and a refusal to ship a three-class system whose per-class recall is never measured for one of its classes.

The draw was never hard to predict. It was hard to record.

Analytics and research content. Not betting advice. 18+

本内容仅供数据分析与操作参考，不构成任何决策建议。