tailspec

The problem

The public AI-weather benchmarks measure mean skill and event contingency extremely well, yet none of them characterise the distributional tail or ask whether it is calibrated. WeatherBench scores mean RMSE and CRPS; ExtremeWeatherBench scores impact contingency over curated cases. Both are quiet exactly where the decisions that pay for a forecast are made: reinsurance pricing, energy hedging, civil protection all live in the far tail, where a thin bias is not a rounding error but a mispriced risk.

tailspec asks a single sharp question. Can AI weather emulators be trusted for high-stakes tail-risk estimation, marginally, spatially, and decisionally? It answers it across 6 emulators spanning 3 DL stacks (JAX · PyTorch · ONNX), turning a known qualitative weakness into per-cell, per-model, per-lead-time numbers a risk owner can act on.

Approach

The engine evaluates forecast against reference in the tail rather than in the mean. For every grid cell it fits the extreme-value distribution itself (GEV block maxima and GPD peaks-over-threshold), reads off the shape parameter that controls how heavy the tail is, and compares it against date-matched ERA5. It then layers tail calibration on the ensemble emulators, conservation-budget drift, spectral effective resolution, and decision value (Brier skill, CRPS, and relative economic value). Where a claim touches money, the design is pre-registered and frozen before any evaluation data lands.

Every emulator’s fitted tail sits thinner than ERA5’s. This is not a discovery: MSE-trained models regress towards the conditional mean, an effective soft cap on intensity, and the mechanism is established in the literature (Zhang et al. 2026; Brenowitz et al. 2024). What tailspec contributes is the systematic, integrated diagnosis across architectures, which is what lets it separate a genuine, defensible discriminator from an artefact of one training objective.

Results

Record extremes carry a real decision-value edge. On the forecast-conditioned Brier skill of record-breaking events, GenCast (a diffusion model) beats AIFS (trained on CRPS) by a margin whose permutation test does not close over zero (ΔBSS +0.085, p<10⁻⁴). It is a sharpness-versus-recall trade-off, so which model to trust depends on the loss function: the relative-economic-value crossover sits at a cost-loss ratio of α* = 0.025 [0.018, 0.078], and AIFS wins only for the most miss-averse users while GenCast wins across the rest of the decision spectrum.

The ordering flips with the book. Two operational-grade AI ensembles that already agree on the marginal extreme tail part ways the moment the forecast is turned into money. On a pre-registered day-ahead wind-power backtest (66 paired delivery days), the sign of the imbalance-cost difference reverses between the German and the British book (+0.0070 DE / −0.0122 GB (CF)), under the same frozen decision rule. The mechanism is not mean accuracy. It is where each ensemble’s lower tail sits relative to the newsvendor commitment quantile: neither is calibrated there, and they miss in opposite directions, GenCast over-committing and AIFS under-committing. In the low-wind German regime AIFS’s lower tail is nearly right and GenCast’s over-commitment dominates; in the windy British regime AIFS’s lower tail collapses and GenCast wins. Crucially, a single “which model is better” ranking does not survive a change of portfolio geography.

Beside the flip sits the study’s other first-of-kind result: the first tail-calibration assessment of AI weather ensembles benchmarked against the operational IFS ENS physics ensemble. On the occurrence-ratio diagnostic the AI ensembles stand beside the physics baseline rather than behind it (|ratio − 1| ≈ 0.16 vs 0.217 IFS ENS). And where the tail shape has converged, the discriminator that survives is per-member physical consistency: diffusion conserves on average but not per member, carrying markedly more budget scatter than AIFS across mass, moisture, and energy (2.9× [2.69, 3.13] per-member scatter).

The nulls are reported as features. On the four-city temperature book the two models are statistically indistinguishable, and the spatial tail-dependence ordering that looks clean in point estimates does not survive the date bootstrap. An honest evaluation states where the error bars close over the effect, and says so before it is asked.

> ENGINE_ROOM — Extreme-value fitting, pre-registration, and decision value

The marginal tail is characterised per grid cell by fitting the generalised extreme value distribution to block maxima,

$$G(z) = \exp\!\left\{-\left[1 + \xi\left(\frac{z-\mu}{\sigma}\right)\right]^{-1/\xi}\right\},$$

whose shape parameter $\xi$ sets the tail weight: $\xi < 0$ is a bounded, thin tail, and every emulator sits more negative than ERA5’s ξ ≈ −0.11. A peaks-over-threshold GPD fit gives the same reading from the exceedances above a high quantile, and a single composite fit is reused across the derived tail metrics rather than refitted.

Pre-registration is what makes the decision-value claims load-bearing. The hypotheses, the decision rules, and every amendment were dated and frozen before any season data was read, so the wind-power flip cannot be a garden-of-forking-paths artefact. Decision value is then measured with relative economic value $V(\alpha)$ over the cost-loss ratio $\alpha$, which converts a probabilistic forecast into the fraction of the perfect-forecast value a user with that loss ratio would capture. Reading the model comparison as a function of $\alpha$, rather than at a single operating point, is what exposes the crossover that a scalar skill score hides.

The engine itself is GPU-free and xarray/zarr-native, built on a four-part spine (sources, diagnostics, orchestration, results) and running on arbitrary forecast-against-reference stores. The emulator-inference backends are isolated behind optional extras, so import tailspec stays light and the diagnostics need no ML stack to run.

What this demonstrates