Quantifying the skill of convection-permitting ensemble forecasts for the sea-breeze occurrence

Email: carlo.cafaro@pgr.reading.ac.uk

On the afternoon of 16th August 2004, the village of Boscastle on the north coast of Cornwall was severely damaged by flooding (Golding et al., 2005). This is one example of high impact hazardous weather associated with small meso- and convective-scale weather phenomena, the prediction of which can be uncertain even a few hours ahead (Lorenz, 1969; Hohenegger and Schar, 2007). Taking advantage of the increased computer power (e.g. https://www.metoffice.gov.uk/research/technology/supercomputer) this has motivated many operational and research forecasting centres to introduce convection-permitting ensemble prediction systems (CP-EPSs), in order to give timely weather warnings of severe weather.

However, despite being an exciting new forecasting technology, CP-EPSs place a heavy burden on the computational resources of forecasting centres. They are usually run on limited areas with initial and boundary conditions provided by global lower resolution ensembles (LR-EPS). They also produce large amounts of data which needs to be rapidly digested and utilized by operational forecasters. Assessing whether the convective-scale ensemble is likely to provide useful additional information is key to successful real-time utilisation of this data. Similarly, knowing where equivalent information can be gained (even if partially) from LR-EPS using statistical/dynamical post-processing both extends lead time (due to faster production time) and also potentially provides information in regions where no convective-scale ensemble is available.

There have been many studies on the verification of CP-EPSs (Klasa et al., 2018, Hagelin et al., 2017, Barret et al., 2016, Beck et al., 2016 amongst the others), but none of them has dealt with the quantification of the skill gained by CP-EPSs in comparison with LR-EPSs, when fully exploited, for specific weather phenomena and for a long enough evaluation period.

In my PhD, I have focused on the sea-breeze phenomenon for different reasons:

  1. Sea breezes have an impact on air quality by advecting pollutants, on heat stress by providing a relief on hot days and also on convection by providing a trigger, especially when interacting with other mesoscale flows (see for examples figure 1 or figures 6, 7 in Golding et al., 2005).
  2. Sea breezes occur on small spatio-temporal scales which are properly resolved at convection-permitting resolutions, but their occurrence is still influenced by synoptic-scale conditions, which are resolved by the global LR-EPS.
Figure 1: MODIS visible of the southeast of Italy on 6th June 2018, 1020 UTC. This shows thunderstorms occurring in the middle of the peninsula, probably triggered by sea-breezes.
Source: worldview.earthdata.nasa.gov

Therefore this study aims to investigate whether the sea breeze is predictable by only knowing a few predictors or whether the better representation of fine-scale structures (e.g. orography, topography) by the CP-EPS implies a better sea-breeze prediction.

In order to estimate probabilistic forecasts from both the models, two different methods have been applied. A novel tracking algorithm for the identification of sea-breeze front, in the domain represented in figure 2, was applied to CP-EPSs data. A Bayesian model was used instead to estimate the probability of sea-breeze conditioned on two LR-EPSs predictors and trained on CP-EPSs data. More details can be found in Cafaro et al. (2018).

Figure 2: A map showing the orography over the south UK domain. Orography data are from MOGREPS-UK. The solid box encloses the sub-domain used in this study with red dots indicating the location of synoptic weather stations. Source: Cafaro et al. (2018)

The results of the probabilistic verification are shown in figure 3. Reliability (REL) and resolution (RES) terms have been computed decomposing the Brier score (BS) and Information gain (IGN) score. Finally, scores differences (BSD and IG) have been computed to quantify any gain in the skill by the CP-EPS. Figure 3 shows that CP-EPS forecast is significantly more skilful than the Bayesian forecast. Nevertheless, the Bayesian forecast has more resolution than a climatological forecast (figure 3e,f), which has no resolution by construction.

Figure 3: (a)-(d) Reliability and resolution terms calculated for both the forecasts (green for the CP-EPS forecast and blue for LR-EPSs). (e) and (f) represent the Brier score difference (BSD) and Information gain (IG) respectively. Error bars represent the 95th confidence interval. Positive values of BSD and IG indicate that CP-EPS forecast is more skilful. Source: Cafaro et al. (2018)

This study shows the additional skill provided by the Met Office convection-permitting ensemble forecast for the sea-breeze prediction. The ability of CP-EPSs to resolve meso-scale dynamical features is thus proven to be important and only two large-scale predictors, relevant for the sea-breeze, are not sufficient for skilful prediction.

It is believed that both the methodologies can, in principle, be applied to other locations of the world and it is thus hoped they could be used operationally.


Barrett, A. I., Gray, S. L., Kirshbaum, D. J., Roberts, N. M., Schultz, D. M., and Fairman J. G. (2016). The utility of convection-permitting ensembles for the prediction of stationary convective bands. Monthly Weather Review, 144(3):1093–1114, doi: 10.1175/MWR-D-15-0148.1

Beck,  J., Bouttier, F., Wiegand, L., Gebhardt, C., Eagle, C., and Roberts, N. (2016). Development and verification of two convection-allowing multi-model ensembles over Western europe. Quarterly Journal of the Royal Meteorological Society, 142(700):2808–2826, doi: 10.1002/qj.2870

Cafaro C., Frame T. H. A., Methven J., Roberts N. and Broecker J. (2018), The added value of convection-permitting ensemble forecasts of sea breeze compared to a Bayesian forecast driven by the global ensemble, Quarterly Journal of the Royal Meteorological Society., under review.

Golding, B. , Clark, P. and May, B. (2005), The Boscastle flood: Meteorological analysis of the conditions leading to flooding on 16 August 2004. Weather, 60: 230-235, doi: 10.1256/wea.71.05

Hagelin, S., Son, J., Swinbank, R., McCabe, A., Roberts, N., and Tennant, W. (2017). The Met Office convective-scale ensemble, MOGREPS-UK. Quarterly Journal of the Royal Meteorological Society, 143(708):2846–2861, doi: 10.1002/qj.3135

Hohenegger, C. and Schar, C. (2007). Atmospheric predictability at synoptic versus cloud-resolving scales. Bulletin of the American Meteorological Society, 88(11):1783–1794, doi: 10.1175/BAMS-88-11-1783

Klasa, C., Arpagaus, M., Walser, A., and Wernli, H. (2018). An evaluation of the convection-permitting ensemble cosmo-e for three contrasting precipitation events in Switzerland. Quarterly Journal of the Royal Meteorological Society, 144(712):744–764, doi: 10.1002/qj.3245

Lorenz, E. N. (1969). Predictability of a flow which possesses many scales of motion.Tellus, 21:289 – 307, doi: 10.1111/j.2153-3490.1969.tb00444.x

A New Aviation Turbulence Forecasting Technique

Anyone that has ever been on a plane will probably have experienced turbulence at some point. Most of the time it is not likely to cause injury, but during severe turbulence unsecured objects (including people) can be thrown around the cabin, costing the airline industry millions of dollars every year in compensation (Sharman and Lane, 2016). Recent research has also indicated that in the future the frequency of clear-air turbulence will increase with climate change. Forecasting turbulence is one of the best ways to reduce the number of injuries by giving pilots and flight planners ample warning, so they can put on the seat-belt sign or avoid the turbulent region altogether. The current method used in creating a turbulence forecast is a single ‘deterministic’ forecast – one forecast model, with one forecast output. This shows the region where they suspect turbulence to be, but because the forecast is not perfect, it would be more ideal to show how certain we are that there is turbulence in that region.

To do this, a probabilistic forecast can be created using an ensemble (a collection of forecast model outputs with slightly different model physics or initial conditions). A probabilistic forecast essentially shows model confidence in the forecast, and therefore how likely it is that there will be turbulence in a given region. For example, if all 10 out of 10 forecast outputs predict turbulence in the same location, the pilots would be confident in taking action (such as avoiding the region altogether). However, if only 1 out of 10 models predict turbulence, then the pilot may choose to turn on the seat-belt sign because there is still a chance of turbulence, but not enough to warrant spending time and fuel to fly around the region. A probabilistic forecast not only provides more information in the certainty of the forecast, but it also increases the chances of forecasting turbulence that a single model might miss.

Gill and Buchanan (2014) showed this ensemble forecast method does improve the forecast skill. In my project we have taken this one step further and created a multi-model ensemble, which is combining two different ensembles, each with their own strengths and weaknesses (Storer et al., 2018). We combine the Met Office Global and Regional Ensemble Prediction System (MOGREPS-G), with the European Centre for Medium Range Weather Forecasting (ECMWF) Ensemble Prediction System (EPS).

Figure 1: Plot of a moderate-or-greater turbulence event over the possible sources of turbulence: top left: orography, shear turbulence (bottom left: MOGREPS-G and bottom right: ECMWF EPS probability forecast), and top right: convection from satellite data (colour shading indicates deep convection). Both the MOGREPS-G and ECMWF-EPS ensembles forecast the shear turbulence event. The circles indicate turbulence observations with grey indicating no turbulence, orange indicating light turbulence and red indicating moderate or greater turbulence. The convective classification can be found in Francis and Batstone (2013).

There are three main sources of turbulence. The first is mountain wave turbulence, where gravity waves are produced from mountains that ultimately lead to turbulence. The second is convectively-induced turbulence, which includes in-cloud turbulence and also gravity waves produced as a result of deep convection that also lead to turbulence. The third is shear-induced turbulence, which is the one we are trying to forecast in this example. Figure 1 is an example plot showing orography and thus mountain wave turbulence (top left), convection and thus convectively induced turbulence (top right), the MOGREPS-G ensemble forecast of shear turbulence (bottom left) and the ECMWF ensemble forecast of shear turbulence (bottom right). The red circle indicates a ‘moderate or greater’ turbulence event, and we can see that because it is over the North Atlantic it is not a mountain wave turbulence event, and there is no convection nearby, but both the ensemble forecasts correctly predict the location of the shear-induced turbulence. This shows that there is high confidence in the forecast, and action (such as putting the seat-belt sign on) can be taken.

Figure 2: Value plot with a log scale x-axis of the global turbulence with the 98 convective turbulence cases removed showing the forecast skill of the MOGREPS-G (dot-dash), ECMWF (dot), combined multi-model ensemble (dash) and the maximum value using every threshold of the combined multi-model ensemble (solid). The data used has a forecast lead time between +24 hours and +33 hours between May 2016 and April 2017.

To understand the usefulness of the forecast, Figure 2 is a relative economic value plot. It shows the value of the forecast for a given cost/loss ratio (which will vary depending on the end user). The multi-model ensemble is more valuable than both of the single model ensembles for all cost/loss ratios, showing that every end user will benefit from this forecast. Although our results do show an improvement in forecast skill, it is not statistically significant. However, by combining ensemble forecasts we gain consistency and more operational resilience (i.e., we are still able to produce a forecast if one ensemble is not available), and is therefore still worth implementing in the future.

Email: luke.storer@pgr.reading.ac.uk


Gill PG, Buchanan P. 2014. An ensemble based turbulence forecasting system. Meteorol. Appl. 21(1): 12–19.

Sharman R, Lane T. 2016. Aviation Turbulence: Processes, Detection, Prediction. Springer.

Storer, L.N., Gill, P.G. and Williams, P.D., 2018. Multi-Model Ensemble Predictions of Aviation Turbulence. Meteorol. Appl., (Accepted for publication).

Atmospheric blocking: why is it so hard to predict?

Atmospheric blocks are nearly stationary large-scale flow features that effectively block the prevailing westerly winds and redirect mobile cyclones. They are typically characterised by a synoptic-scale, quasi-stationary high pressure system in the midlatitudes that can remain over a region for several weeks. Blocking events can cause extreme weather: heat waves in summer and cold spells in winter, and the impacts associated with these events can escalate due to a block’s persistence. Because of this, it is important that we can forecast blocking accurately. However, atmospheric blocking has been shown to be the cause of some of the poorest forecasts in recent years. Looking at all occasions when the ECMWF model experienced a period of very low forecast skill, Rodwell et al. (2013) found that the average flow pattern for which these forecasts verified was an easily-distinguishable atmospheric blocking pattern (Figure 1). But why are blocks so hard to forecast?

Figure 1:  Average verifying 500 hPa geopotential height (Z500) field for occasions when the ECMWF model experienced very low skill. From Rodwell et al. (2013).

There are several reasons why forecasting blocking is a challenge. Firstly, there is no universally accepted definition of what constitutes a block. Several different flow configurations that could be referred to as blocks are shown in Figure 2. The variety in flow patterns used to define blocking brings with it a variety of mechanisms that are dynamically important for blocks developing in a forecast (Woollings et al. 2018). Firstly, many phenomena must be well represented in a model for it to forecast all blocking events accurately. Secondly, there is no complete dynamical theory for block onset and maintenance- we do not know if a process key for blocking dynamics is missing from the equation set solved by numerical weather prediction models and is contributing to the forecast error. Finally, many of the known mechanisms associated with block onset and maintenance are also know sources of model uncertainty. For example, diabatic processes within extratropical cyclones have been shown to contribute substantially to blocking events (Pfahl et al. 2015), the parameterisation of which has been shown to affect medium-range forecasts of ridge building events (Martínez-Alvarado et al. 2015).

Figure 2: Different flow patterns, shown using Z500 (contours), that have been defined as blocks. From Woollings et al. (2018).

We do, however, know some ways to improve the representation of blocking: increase the horizontal resolution of the model (Schiemann et al. 2017); improve the parameterisation of subgrid physical processes (Jung et al. 2010); remove underlying model biases (Scaife et al. 2010); and in my PhD we found that improvements to a model’s dynamical core (the part of the model used to solved the governing equations) can also improve the medium-range forecast of blocking. In Figure 3, the frequency of blocking that occurred during two northern hemisphere winters is shown for the ERA-Interim reanalysis and three operational weather forecast centres (the ECMWF, Met Office (UKMO) and the Korean Meteorological Administration (KMA)). Both KMA and UKMO use the Met Office Unified Model – however, before the winter of 2014/15 the UKMO updated the model to use a new dynamical core whilst KMA continued to use the original. This means that for the 2013/14 the UKMO and KMA forecasts are from the same model with the same dynamical core whilst for the 2014/15 winter the UKMO and KMA forecasts are from the same model but with different dynamical cores. The clear improvement in forecast from the UKMO in 2014/15 can hence be attributed to the new dynamical core. For a full analysis of this improvement see Martínez-Alvarado et al. (2018).

Figure 3: The frequency of blocking during winter in the northern hemisphere in ERA-Interim (grey shading) and in seven-day forecasts from the European Centre for Medium-Range Weather Forecasts (ECMWF), the Met Office (UKMO) and the Korean Meteorological Administration (KMA). Box plots show the spread in the ensemble forecast from each centre.

In the remainder of my PhD I aim to investigate the link between errors in forecasts of blocking with the representation of upstream cyclones. I am particularly interested to see if the parameterisation of diabatic processes (a known source of model uncertainty) could be causing the downstream error in Rossby wave amplification and blocking.

Email: j.maddison@pgr.reading.ac.uk.


Rodwell, M. J., and Coauthors, 2013: Characteristics of occasional poor medium-range weather  forecasts for Europe. Bulletin of the American Meteorological Society, 94 (9), 1393–1405.

Woollings, T., and Coauthors, 2018: Blocking and its response to climate change. Current Climate Change Reports, 4 (3), 287–300.

Pfahl, S., C. Schwierz, M. Croci-Maspoli, C. Grams, and H. Wernli, 2015: Importance of latent  heat release in ascending air streams for atmospheric blocking. Nature Geoscience, 8 (8), 610– 614.

Mart´ınez-Alvarado, O., E. Madonna, S. Gray, and H. Joos, 2015: A route to systematic error in forecasts of Rossby waves. Quart. J. Roy. Meteor. Soc., 142, 196–210.

Mart´ınez-Alvarado, O., and R. Plant, 2014: Parametrized diabatic processes in numerical simulations of an extratropical cyclone. Quart. J. Roy. Meteor. Soc., 140 (682), 1742–1755.

Scaife, A. A., T. Woollings, J. Knight, G. Martin, and T. Hinton, 2010: Atmospheric blocking and mean biases in climate models. Journal of Climate, 23 (23), 6143–6152.

Schiemann, R., and Coauthors, 2017: The resolution sensitivity of northern hemisphere blocking in four 25-km atmospheric global circulation models. Journal of Climate, 30 (1), 337–358.

Jung, T., and Coauthors, 2010: The ECMWF model climate: Recent progress through improved physical parametrizations. Quart. J. Roy. Meteor. Soc., 136 (650), 1145–1160.