Forecast Verification Summer School

Lily Greig – l.greig@pgr.reading.ac.uk

A week-long summer school on forecast verification was held jointly at the end of June by the MPECDT (Mathematics of Planet Earth Centre for Doctoral Training) and JWGFVR (Joint Working Group on Forecast Verification Research). The school featured lectures from scientists and academics from many different countries around the world including Brazil, USA and Canada. They each specialised in different topics within forecast verification. Participants gained a large overview of the field and how the fields within it interact.

Structure of school

The virtual school consisted of lectures from individual members of the JWGFVR on their own subjects, along with drop-in sessions for asking questions and dedicated time to work on group projects. Four groups of 4-5 students were given an individual forecast verification challenge. The themes of the projects were precipitation forecasts, comparing high resolution global model and local area model wind speed forecasts, and ensemble seasonal forecasts. The latter was the topic of our project.

Content

The first lecture was given by Barbara Brown, who provided a broad summary of verification and gave examples of questions that verifiers may ask themselves as they attempt to assess the “goodness” of a forecast. The next day, a lecture by Barbara Casati covered continuous scores (verification of continuous variables e.g., temperature), such as linear bias, mean-squared error (MSE) and Pearson coefficient. She also outlined the deficits of different scores and how it is best to use a variety of them when assessing the quality of a forecast. Marion Mittermaier then spoke about categorical scores (yes/no events or multi category events such as precipitation type). She gave examples such as contingency tables which portray how well a model is able to predict a given event, based on hit rates (how often the model predicted an event when the event happened), and false alarm rates (how often the model predicted the event when it didn’t happen). Further lectures were given by Ian Joliffe on methods of determining the significance of your forecast scores, Nachiketa Acharya on probabilistic scores and ensembles, Caio Coelho on sub-seasonal to seasonal timescales, and then Raghavendra Ashrit, Eric Gilleland and Caren Marzban on severe weather, spatial verification and experimental design. The lectures have been made available online and you can find them here.

Forecast Verification

So, forecast verification is as it sounds: a part of assessing the ‘goodness’ of a forecast as opposed to its value. Verification is helpful for economic purposes (e.g. decision making), as well as administrative and scientific ones (e.g. identifying model flaws). The other aspect of measuring how well a forecast is performing is knowing the user’s needs, and therefore how to apply the forecast. It is important to consider the goal of your verification process beforehand, as it will outline your choice of metrics and your assessment of them. An example of how forecast goodness hinges on the user was given by Barbara in her talk: a precipitation forecast may have a spatial offset of where a rain patch falls, but if both observation and forecast fall along the flight path, this may be all the aviation traffic strategic planner needs to know. For a watershed manager on the ground, however, this would not be a helpful forecast. The lecturers also emphasised the importance of performing many different measures on a forecast and then understanding the significance of your measures in order to help you understand its overall goodness. Identifying standards of comparison for your forecast is also important, such as persistence or climatology. Then there are further challenges such as spatial verification, which requires methods of ‘matching’ the location of your observations with the model predictions on the model grid.

Figure 1: Problem statement for group presentation on 2m temperature ensemble seasonal forecasts, presented by Ryo Kurashina

Group Project

Our project was on verification of 2 metre temperature ensemble seasonal forecasts (see Figure 1). We were looking at seasonal forecast data with a 1-month lead time for the summer months for three different models and investigating ways of validating the forecasts, finally deciding which one was the better. We decided to focus on the models’ ability to predict hot and cold events as a simple metric for El Nino. We looked at scatter plots and rank histograms to investigate the biases in our data, Brier scores for assessing model accuracy (level of agreement between forecast and truth) and Receiver Operating Characteristic curves to look model skill (the relative accuracy of the forecast over some reference forecast). The ROC curve (see Fig. 2) refers to the curve formed by plotting hit rates against false alarm rates based on probability thresholds. The further above the diagonal line your curve lies, the better your forecast is at discriminating events compared to a random coin toss. The combination of these verification methods were used to assess which model we thought was best.

Of course, virtual summer schools are less than ideal compared to the real (in person) deal, but with Teams meetings, shared code and chat channel we made the most of it. It was fun to work with everyone, even (or especially?) if the topic was new for all of us.

Figure 2: Presenting our project during group project presentations on Friday

Conclusions

The summer school was incredibly smoothly run, very engaging to people both new and experienced in the topic and provided plenty of opportunity to ask questions to the enthusiastic lecturers. Would recommend to PhD students working with forecasts and wanting to assess them!

SWIFT and YESS International Summer School, Kumasi, Ghana

Email: a.j.doyle@pgr.reading.ac.uk

Last month, from the 21st July until the 3rd August 2019, I was in Ghana attending the African SWIFT and YESS International Summer School. What a catchy name you are probably thinking. SWIFT, or Science for Weather Information and Forecasting Techniques, is a programme of research and capability building, led by the National Centre for Atmospheric Science (NCAS), and funded by the UK Research and Innovation Global Challenges Research Fund. The project aims to improve African weather forecasting, especially on seasonal timescales, as well as build capability in related research. It’s worth a quick Google search at some point, and there are several people involved in the project at the University of Reading. YESS, the Young Earth System Scientists community, is an international, multidisciplinary network of early career researchers. Catchy!

Anyway, all this contributed toward a really remarkable summer school in tropical West Africa, with people from many different institutions and nations across Europe and Africa attending the summer school and science meeting alongside. It was also another chance to get consistently barraged with Brexit questions by a baffled international audience.

The days were long but engaging, with lectures, practical sessions and workshops on a huge variety of topics in tropical meteorology – from Rossby waves, to the monsoon, to remote sensing applications. It was quickly evident how tricky tropical African weather is to forecast. It is largely driven by convection, which is very difficult to forecast accurately on a small spatio-temporal scale, unlike nice, large mid-latitude weather systems. Furthermore, several different atmospheric features are at play. This is where we were introduced to the wonderful West African synoptic analysis/forecast charts (see below for an example of mine). We also had a chance to present our posters, with many of those from the science meeting – experts in their fields – coming round to look, and this was a fantastic networking opportunity. It was really beneficial being around other early career scientists in the same specific field as me, from different places around the world. It cannot be said enough how important this is for PhD students, who for the most part live quite an isolated existence where when you switch from your native English language to your ‘PhD language’, only your supervisors and a few select others can understand you!

For me, it is in the people attending where the strength of the summer school really lies. The people in Kumasi, Ghana were amazing people. They not only keep you going through 2 weeks of long days, 3 dozen lectures, and 400 meals of rice, but they reminded me what it meant to be a scientist. I found, to my discredit, that most of the students there were far more studious than I was, not because they were any less clever or anything like that, but because they simply loved knowledge, and loved applying it (meteorology is great for quickly being able to see how what we know manifests itself in the real world). On reflection, I think they are more aware of the fact millions of people (moreso in Africa than any other continent) simply do not have access to such knowledge, but in Kumasi we were learning about African meteorology from world experts. They did not take it for granted, in fact, it was clearly what drove them. Science wasn’t just an occupation for them, it had tangible importance, which came across in the way they spoke about their science, but also their future ambitions, hopes and plans.

Further to this, meeting people across universities, countries, and continents also brings a different perspective on your work and where it fits into the wider collection of research in the area. One sad point was learning how hard it was for African students to get PhDs. Not only do they typically have to travel much further (i.e. typically to Europe or the US) in order to get one, but they also rely on getting funding, which is often the final obstacle even after they have found the right PhD project. It’s a real shame.

So after 2 long weeks (and a very hot football game on a gravelly pitch with no shoes) I came back physically exhausted, but academically I was refreshed with lots of new ideas floating round, but even more importantly newfound inspiration. In the now famous words of the provost of the college during the closing ceremony, “let your research be SWIFT and YESS.”

Fluid Dynamics of Sustainability and the Environment Summer School

Email: m.prosser@pgr.reading.ac.uk

From the 1st – 12th of July 2019, I was fortunate enough to be able to attend the Fluid Dynamics of Sustainability and the Environment (FDSE) summer school held at Ecole Polytechnique on the southern outskirts of Paris. Although it was held at Ecole Polytechnique this year, it alternates with the University of Cambridge, where it will be held in 2020.

As hinted at in the title, the summer school explores the fluid dynamical aspects of planet Earth, including, but not limited to: the atmosphere, the ocean, the cryosphere and the solid Earth, and was of particular relevance to me because I study clear-air turbulence (a fluid dynamical phenomenon) and its impact on aviation. To get a better sense of the summer school, have a watch of this 3-minute promotion video: https://www.youtube.com/watch?v=TGoF0L8gqXw

It was a busy, action-packed 2 weeks. The days consisted of: 4 hours of lectures held each morning (coffee was provided), followed by either lab or numerical practical sessions in the afternoons and something social (wine was provided) such as a poster session, barbecue, and an environmentally-themed film night followed by a discussion of the film’s (The Day After Tomorrow) fluid dynamical accuracy (or not, as the case may be!). During the mid-programme weekend, we were put up in a hostel in central Paris, treated to an evening on a moored boat on the Seine (champagne was provided) and then left to our own devices to explore Paris.

The boat on the Seine even had its own dance floor.

The other students were great, with all sorts of backgrounds/PhD projects that linked in one way or another to the FDSE theme. Many interesting and diverse conversations were had, as well as a great deal of fun and laughter! No doubt many of the people who met here both this year and others will collaborate scientifically in the future.

Not having come from a maths/physics background, I found a lot of the mathematical content quite challenging, but I made copious notes and my interest in and appreciation for the subject greatly increased. As I progress throughout my PhD (I am currently still in my first year), I feel many of the concepts that I encountered here are likely to resurface in a slow-burn fashion and I can see myself returning to the lecture material as and when I meet related concepts.

In particular, gaining an understanding of what an instability is and studying the different types was eye-opening, and seeing Kelvin-Helmholtz instabilities — which cause the shear that generates the clear-air turbulence I study in my PhD — form in a tube of dyed fluid was a particularly memorable moment for me.

Kelvin-Helmholtz billows forming in a tube.

Apart from being very interesting theoretically, fluid dynamics also has many practical applications. For example, insufficient understanding and modelling of the behaviour of plumes at the Fukushima nuclear reactor led to hydrogen gas concentrations exceeding 8%, resulting in dangerous explosions. Many other such examples could be given.

The summer school was well-organised and many of the lecturers and guest speakers were both highly entertaining and informative, and really bought the subject to life with their enthusiasm for it. I highly recommend it to anyone with a related PhD!

The 2019 cohort in front of Ecole Polytechnique.

The 2nd ICTP Summer School in Hierarchical Modelling of Climate Dynamics

Between the 1st and 12th July 2019, I attended the 2nd International Centre for Theoretical Physics (ICTP) Summer School in Hierarchical Modelling of Climate Dynamics at the ICTP guesthouse in Trieste, Italy. The focus of this summer school was on convective organisation and climate sensitivity, which is incredibly relevant to my PhD topic: Interactions between Radiation and Convective Organisation. So, I felt I had to attend this summer school (and not just because my lead supervisor, Chris Holloway, was one of the lead directors).

This was an international conference with staff and students coming together from all corners of the globe. In total there were 111 people attending the school, made up of 84 participants, 20 speakers and 7 directors. Without knowing anyone else going to this school (except my supervisor), I was initially a little apprehensive as I didn’t know what to expect but as soon as I met some of the other students I was put at ease. It was amazing to meet other people working on very similar projects to me, especially since my supervisor was the only other person I previously knew working on this convective organisation topic. So, it was great to not only make new friends but also meet potential future colleagues.

Group photo of all those involved in the summer school.

As expected, the schedule was pretty intense, with most days working from 9am until 6pm except for lunch and a couple of coffee breaks. The mornings consisted of a couple of lectures given by some of the leading experts in the field including Kerry Emanuel, Bjorn Stevens and Sandrine Bony, then in the afternoons we would do some group project work. In our groups of 4 or 5, we analysed some numerical model data, to study how convection organises within our model. I was surprised to find that our group tasks were very similar to what I’ve been doing for my first year, so I was a bit worried that we’d manage to do what I’ve been working on this past year within a couple of weeks! But actually, it ended up giving me almost too many new ideas for my own research! In the second week, each group then had to give a quick presentation on their work.

Talk by Kerry Emanuel about the consequences of climate change on our weather.

Each day, after the lectures and the group work, we were free to do what we wanted for the rest of the evening. With the venue being right on the coast, and with temperatures consistently between 26 – 32C in the day, it was perfect to relax by the sea or go for a swim. Or, if we were bored with the relentless supply of pasta in the canteen then we’d often go into town in search of pizza and of course gelato!

At the start of the second week, there was a poster session in which a lot of the participants brought posters to showcase their projects. This was the first time I’d presented my research at an event like this, so it was great to show what I’ve been working on in front of so many people. It was exciting to see so many people genuinely interested in my work and I got lots of useful feedback and ideas.

Presenting my work at the poster session.

So overall, this summer school far surpassed my expectations and I would strongly recommend attending a summer school if you get the chance. I learned so much through the lectures, the group work, through chatting to the professors and students and through presenting my work. I now have far too many ideas to explore with my research, probably more than I can realistically achieve! Perhaps the most valuable aspect of the school was being able to meet so many people working in this field. Since this topic is very niche, I have been very lucky to meet a very large proportion of the people working in the topic so I’m sure some of our paths will cross in the future and we will be able to collaborate on future projects.