In the past decades, ensemble forecasting (EF) became a ubiquitous part of Numerical Weather Prediction worldwide. Yet some basic questions have evaded the scrutiny of investigators: What value EF can deliver, given all perturbed forecasts perform significantly below the level of a single “control” forecast started from the best (unperturbed) analysis? Will probabilistic information derived from such degraded forecasts suffer in quality? Why the mean of such degraded forecasts exhibits an error smaller than the control forecast? A careful look at questions like these reveal a perplexing behavior, played out in multidimensional space with characteristics distinctly different from the low dimension of the physical world we inhabit or our simple models exemplify.
When assessed variable by variable in 1 dimension, ensembles behave as intended – truth appears as a random draw from the ensemble members, roughly half of which are superior to the control. The atmosphere, however, is governed by multidimensional dynamics. Through theoretical considerations and the evaluation of both perfect and operational ensemble forecasts, we find that in such systems the addition of random initial perturbations always degrades the control forecast. As the entire ensemble is further displaced from truth, the mean, as well as any other ensemble-derived probabilistic products suffer. The results show that our failure with ensemble forecasting is not due to a lack of preparedness or poor techniques but rather to the inherent geometry of high dimensional spaces. That leaves the observer with a question – if our dream is unattainable – why keep trying?