Quantifying the skill of convection-permitting ensemble forecasts for the sea-breeze occurrence

Email: carlo.cafaro@pgr.reading.ac.uk

On the afternoon of 16th August 2004, the village of Boscastle on the north coast of Cornwall was severely damaged by flooding (Golding et al., 2005). This is one example of high impact hazardous weather associated with small meso- and convective-scale weather phenomena, the prediction of which can be uncertain even a few hours ahead (Lorenz, 1969; Hohenegger and Schar, 2007). Taking advantage of the increased computer power (e.g. https://www.metoffice.gov.uk/research/technology/supercomputer) this has motivated many operational and research forecasting centres to introduce convection-permitting ensemble prediction systems (CP-EPSs), in order to give timely weather warnings of severe weather.

However, despite being an exciting new forecasting technology, CP-EPSs place a heavy burden on the computational resources of forecasting centres. They are usually run on limited areas with initial and boundary conditions provided by global lower resolution ensembles (LR-EPS). They also produce large amounts of data which needs to be rapidly digested and utilized by operational forecasters. Assessing whether the convective-scale ensemble is likely to provide useful additional information is key to successful real-time utilisation of this data. Similarly, knowing where equivalent information can be gained (even if partially) from LR-EPS using statistical/dynamical post-processing both extends lead time (due to faster production time) and also potentially provides information in regions where no convective-scale ensemble is available.

There have been many studies on the verification of CP-EPSs (Klasa et al., 2018, Hagelin et al., 2017, Barret et al., 2016, Beck et al., 2016 amongst the others), but none of them has dealt with the quantification of the skill gained by CP-EPSs in comparison with LR-EPSs, when fully exploited, for specific weather phenomena and for a long enough evaluation period.

In my PhD, I have focused on the sea-breeze phenomenon for different reasons:

  1. Sea breezes have an impact on air quality by advecting pollutants, on heat stress by providing a relief on hot days and also on convection by providing a trigger, especially when interacting with other mesoscale flows (see for examples figure 1 or figures 6, 7 in Golding et al., 2005).
  2. Sea breezes occur on small spatio-temporal scales which are properly resolved at convection-permitting resolutions, but their occurrence is still influenced by synoptic-scale conditions, which are resolved by the global LR-EPS.
Figure 1: MODIS visible of the southeast of Italy on 6th June 2018, 1020 UTC. This shows thunderstorms occurring in the middle of the peninsula, probably triggered by sea-breezes.
Source: worldview.earthdata.nasa.gov

Therefore this study aims to investigate whether the sea breeze is predictable by only knowing a few predictors or whether the better representation of fine-scale structures (e.g. orography, topography) by the CP-EPS implies a better sea-breeze prediction.

In order to estimate probabilistic forecasts from both the models, two different methods have been applied. A novel tracking algorithm for the identification of sea-breeze front, in the domain represented in figure 2, was applied to CP-EPSs data. A Bayesian model was used instead to estimate the probability of sea-breeze conditioned on two LR-EPSs predictors and trained on CP-EPSs data. More details can be found in Cafaro et al. (2018).

Figure 2: A map showing the orography over the south UK domain. Orography data are from MOGREPS-UK. The solid box encloses the sub-domain used in this study with red dots indicating the location of synoptic weather stations. Source: Cafaro et al. (2018)

The results of the probabilistic verification are shown in figure 3. Reliability (REL) and resolution (RES) terms have been computed decomposing the Brier score (BS) and Information gain (IGN) score. Finally, scores differences (BSD and IG) have been computed to quantify any gain in the skill by the CP-EPS. Figure 3 shows that CP-EPS forecast is significantly more skilful than the Bayesian forecast. Nevertheless, the Bayesian forecast has more resolution than a climatological forecast (figure 3e,f), which has no resolution by construction.

Figure 3: (a)-(d) Reliability and resolution terms calculated for both the forecasts (green for the CP-EPS forecast and blue for LR-EPSs). (e) and (f) represent the Brier score difference (BSD) and Information gain (IG) respectively. Error bars represent the 95th confidence interval. Positive values of BSD and IG indicate that CP-EPS forecast is more skilful. Source: Cafaro et al. (2018)

This study shows the additional skill provided by the Met Office convection-permitting ensemble forecast for the sea-breeze prediction. The ability of CP-EPSs to resolve meso-scale dynamical features is thus proven to be important and only two large-scale predictors, relevant for the sea-breeze, are not sufficient for skilful prediction.

It is believed that both the methodologies can, in principle, be applied to other locations of the world and it is thus hoped they could be used operationally.


Barrett, A. I., Gray, S. L., Kirshbaum, D. J., Roberts, N. M., Schultz, D. M., and Fairman J. G. (2016). The utility of convection-permitting ensembles for the prediction of stationary convective bands. Monthly Weather Review, 144(3):1093–1114, doi: 10.1175/MWR-D-15-0148.1

Beck,  J., Bouttier, F., Wiegand, L., Gebhardt, C., Eagle, C., and Roberts, N. (2016). Development and verification of two convection-allowing multi-model ensembles over Western europe. Quarterly Journal of the Royal Meteorological Society, 142(700):2808–2826, doi: 10.1002/qj.2870

Cafaro C., Frame T. H. A., Methven J., Roberts N. and Broecker J. (2018), The added value of convection-permitting ensemble forecasts of sea breeze compared to a Bayesian forecast driven by the global ensemble, Quarterly Journal of the Royal Meteorological Society., under review.

Golding, B. , Clark, P. and May, B. (2005), The Boscastle flood: Meteorological analysis of the conditions leading to flooding on 16 August 2004. Weather, 60: 230-235, doi: 10.1256/wea.71.05

Hagelin, S., Son, J., Swinbank, R., McCabe, A., Roberts, N., and Tennant, W. (2017). The Met Office convective-scale ensemble, MOGREPS-UK. Quarterly Journal of the Royal Meteorological Society, 143(708):2846–2861, doi: 10.1002/qj.3135

Hohenegger, C. and Schar, C. (2007). Atmospheric predictability at synoptic versus cloud-resolving scales. Bulletin of the American Meteorological Society, 88(11):1783–1794, doi: 10.1175/BAMS-88-11-1783

Klasa, C., Arpagaus, M., Walser, A., and Wernli, H. (2018). An evaluation of the convection-permitting ensemble cosmo-e for three contrasting precipitation events in Switzerland. Quarterly Journal of the Royal Meteorological Society, 144(712):744–764, doi: 10.1002/qj.3245

Lorenz, E. N. (1969). Predictability of a flow which possesses many scales of motion.Tellus, 21:289 – 307, doi: 10.1111/j.2153-3490.1969.tb00444.x

1st Met Office Training and Research Summer School

Email: carlo.cafaro@pgr.reading.ac.uk

From the last week of June until the 1st September I took part in the Met Office Training and Research (MOTR), as part of the Mathematics of Planet Earth CDT.

Inspired by the highly popular and successful Geophysical Fluid Dynamics Summer School at the Woods Hole Oceanographic Insitution in the USA, it is a 10 week-programme, hosted by Met Office in Exeter. The PhD students have the opportunity to handle an applied research topic outside the area of their PhD, diversify their portfolio and experience the working and social life at the Met Office.

In the first two weeks we participated in a summer school. In particular in the first week there was a lecture course on ”Regional Climate Variability and Change”. In the morning the lectures were given by David Karoly from University of Melbourne on “patterns of climate change”, starting from the basic concepts of the climate systems and then expanding to the climate change attribution. In the afternoon we had specialist lectures by Met Office and University of Exeter scientists about El Nino, modelling paleoclimates and attribution of extreme weather events. 

David Karoly giving one of his lecture on attribution of regional climate change patterns.

In addition, in the afternoon we had to do lab work working in pairs, using Climate Explorer, choosing a specific continent of the world and investigating past climate and future climate projections for that area. My colleague and I selected South America and we gave a presentation about that.

During the second week we participated in the workshop ”Future opportunities to inform UK regional projections”, with a lecture given by Ed Hawkins, amongst  others, talking about sources of uncertainty.

From the third week onward each student started a research project in different Met Office research groups. A different colleague and I worked within the Atmospheric Processes and Parametrization group (APP), supervised by Gabriel Rooney. My project was on numerical simulations and theoretical aspects of colliding density currents. Other colleagues were placed within the Climate Science, Dynamics groups and Informatics Lab, a partner of Met Office.

A typical day for us at Met Office started at 9am, meeting almost every day with Gabriel at 9.45 am, coffee break at 10.30 for half an hour or so (where I also met Annelize, previously in Reading), 1 hour lunch break and then “working” again until 5.30 pm or so.

Not quite our typical day at Met Office (found on a desk at Met Office)

Also, once a week we had the meeting with the smaller convection group, where everyone was asked to give an update of their own work. We also attended journal club sessions on Fridays and a brainstorming meeting on 21st July. It was nice to take part in these events, even being only summer interns. During the project phase we had also weekly advanced seminars by Glenn Shutts and Mike Cullen, mainly about large-scale dynamics and hierarchies of operational models used in NWP.

Glenn Shutts giving a lecture on Rossby waves breaking.

Personally, it was a wonderful experience for several reasons. The Met Office is a very pleasant place to work, with very friendly and flexible people. Since I think my project was quite academic I did not find many differences with working at university itself. Nevertheless, interacting with new people in a new environment has provided me with new inspirations and insights. I had the chance to talk with several scientists and also a chief Meteorologist, since I was curious about the activities carried out in the Operational room and how much communication there is with the research side. There were some social and sports events organised by MOSSA (Met Office Social and Sports Association) which I really enjoyed (picnic and sports day), getting the chance to meet and to talk with people of other research divisions.

Finally, to top it off, I visited Exeter a lot and the area around, mainly during the weekends and the 5 days of holidays agreed at the beginning, even going to Cornwall for 2 days. 

In the end I would like to thank all of the organisers, my supervisor and all the people I talked with for giving me and my colleagues this very valuable opportunity which I will keep always in mind for my future career.