I run a platform where a couple of hundred independent models publish probability estimates on football fixtures. They aren't variants of one model with different seeds — each one has its own training corpus, its own feature set, its own retraining schedule, and its own life cycle. At the time of writing: 209 have existed, 27 are active, 102 are retired, and 25,963 predictions have been graded.
The first version of the admin panel had a leaderboard. Sort by accuracy, descending, take the top one. It took about three weeks to learn that a leaderboard over a population that large is not a ranking — it's a multiple-comparisons machine with a UI.
Here's what replaced it, and the four bugs that forced each piece.
The problem with sorting by accuracy
Two hundred models, dozens of fixtures a day. Somebody will hit a hot streak. It is not a hypothesis that one of them will look excellent over a 40-fixture window; it is arithmetic. A leaderboard has no memory of how many candidates were tried, so a 40-fixture window of luck is indistinguishable from a 40-fixture window of skill — and the leaderboard will promote it to the top of the page, where it gets attention, which is the only reward the system offers.
The failure isn't subtle. In the early build the "top model" changed almost weekly, always upward, never stable. When I pulled the histories, the models that topped the board were systematically the ones with the fewest graded predictions. Small sample, large variance, and a ranking that treats variance as merit.
So the ranking had to stop being a ranking and start being a life cycle.
Four states, one direction
probation ──▶ active ──▶ degraded ──▶ retired
│ │ │
└────────────┴───────────┘
(demotion allowed, promotion re-earned)- probation — new model, publishing publicly but with a visible "unproven" badge. It exists to accumulate a sample.
- active — passed the promotion gate. Eligible for prominence.
- degraded — was active, drifted. Still publishing, no longer promoted. This state exists because the alternative is a binary that oscillates, which I'll get to.
- retired — stopped. Terminal in practice, but see the section on why the rows stay.
One rule I'd defend hard: state changes are events, not columns.
The obvious schema is models.status plus a cron job that reassigns it. That's what I wrote first, and it cost me every bit of history. Once the cron overwrites the field, you cannot answer "when did this model die, and on what evidence" — which is the only question worth asking when you're deciding whether a retirement was a mistake. A status column is a snapshot pretending to be a record.
Append-only transitions
CREATE TABLE model_transitions (
id bigserial PRIMARY KEY,
model_id bigint NOT NULL REFERENCES models(id),
from_state text NOT NULL CHECK (from_state IN ('probation','active','degraded','retired')),
to_state text NOT NULL CHECK (to_state IN ('probation','active','degraded','retired')),
reason text NOT NULL,
evidence jsonb NOT NULL, -- the actual numbers that justified it
decided_at timestamptz NOT NULL DEFAULT now()
);
CREATE INDEX ON model_transitions (model_id, decided_at DESC);Current state is a view, never stored:
CREATE VIEW model_state AS
SELECT DISTINCT ON (model_id)
model_id, to_state AS state, decided_at AS state_since
FROM model_transitions
ORDER BY model_id, decided_at DESC;And the promotion guard lives in the writer, not the reader:
def transition(conn, model_id, to_state, reason, evidence, min_dwell_days=7):
cur = conn.execute(
"SELECT state, state_since FROM model_state WHERE model_id = %s", (model_id,)
).fetchone()
if cur and cur.state == to_state:
return None # no-op, not an error
if cur and (now() - cur.state_since) < timedelta(days=min_dwell_days):
return None # too soon, by design
conn.execute(
"INSERT INTO model_transitions (model_id, from_state, to_state, reason, evidence) "
"VALUES (%s, %s, %s, %s, %s)",
(model_id, cur.state if cur else "probation", to_state, reason, json.dumps(evidence)),
)evidence is the part people skip and then regret. It stores the window size, the metric value, and the comparison baseline at the moment of the decision. Six months later, "why was this model retired" has an answer that isn't my memory.
Flapping, and why degraded exists
The first demotion rule was a single threshold. Below it, demote. Above it, promote. A model sitting near the line generates a state change every single evaluation tick, and each change fires notifications, rewrites the public badge, and generally makes the system look broken.
Three fixes, all necessary:
- Two thresholds, not one. Promote at the upper band, demote only below the lower band. The gap is dead space where nothing happens. This is just hysteresis and I should have started there.
- Minimum dwell time. Even inside the dead space, refuse any transition within 7 days of the last one. Short-lived states are almost never information.
degradedas a buffer. A model coming offactivegoes todegradedfirst, not straight toretired. Retirement requires it to fail again fromdegraded. One bad window shouldn't be terminal; two consecutive bad windows in different states is a pattern.
The counter I keep is the number of transitions per model per month. Anything above ~2 is either a threshold sitting exactly at the population mean or a genuinely unstable model — both worth knowing.
Retiring means stopping, not deleting
This is the one that actually corrupted a number I had published.
When a model failed, the cleanup job deleted its predictions. Reasonable-sounding: dead models don't need rows. The effect was that the platform-level accuracy figure — computed over whatever predictions still existed — silently excluded every model that performed badly enough to be retired. The number went from 54% to 61% and I briefly believed it, because the computation was correct and the denominator was a lie.
The platform-level metric is only meaningful if the population includes the failures:
-- every model that ever published, including the dead
SELECT count(*) FILTER (WHERE g.outcome = argmax_of_prediction)::float / count(*) AS accuracy
FROM predictions p
JOIN models m ON m.id = p.model_id
LEFT JOIN grades g ON g.prediction_id = p.id
WHERE p.created_at BETWEEN %s AND %s;No WHERE m.state = 'active'. If you filter to survivors and call the result platform performance, you're not measuring the system, you're measuring the system after it selected for the outcome you wanted.
Retired models keep their predictions, their grades, and their transition history. 102 of them at the moment. They are the control group.
Promotion gates that survive contact with reality
Absorbing everything above, the gate for probation → active is:
- at least 50 graded predictions in the current window (below that, the interval on any rate is wider than the differences we're trying to detect)
- performance measured within the same competition and compared against that competition's contemporaneous median, never against a global figure
- no promotion within 7 days of the last transition
The competition scoping is not pedantry. A model covering one evenly-matched league and a model covering forty leagues of wildly different predictability are answering different questions. Blending them produces a number that describes neither, and the model that happened to pick the easy competition wins the leaderboard for reasons that have nothing to do with modelling.
I also stopped showing a single "platform accuracy" number on the site. The honest answer to "how accurate is the platform" is a distribution across models and competitions, and the moment you collapse it to one figure you've started marketing instead of measuring.
Failure modes, honestly
- The cron that ate the history. Described above. Append-only transitions, view for current state. Cost: one extra join on read, worth it every time.
- Thresholds set at the population mean. Guarantees flapping, because half the population is always on the wrong side of the mean. Set bands from the distribution's interquartile range, not its centre.
- Comparing a model to itself across regimes. A model's own history includes retraining. Comparing this month to six months ago compares two different objects wearing the same
model_id. Any retraining that changes weights is a new model row. retiredused as a soft delete. If retirement hides a model from every query, you've rebuilt survivorship bias with a nicer name.- No cap on promotions. Without a dwell time, a noisy window promotes, the next window demotes, and the public page flickers. Users read flicker as unreliability, correctly.
What this buys
The leaderboard answered "which model looks best right now," which is a question about noise. The life cycle answers "which models have earned attention, which are being watched, and which are gone" — and every answer has a timestamp, a reason, and the numbers that produced it.
The schema is thirty lines. The hard part is accepting that a model's status is a claim you have to defend later, with evidence you kept at the time.
The platform this comes from is SoccerDream AI — independent models publishing timestamped probability estimates on football fixtures, graded in the open, retired models included in the record. Per-model pages show the full life cycle rather than a current score.
Analytics and research content. Not betting advice. 18+