What is validation?
Validation is assessing data based on real-world expectations. Data or products that calculate the risk of events happening in the present-day can use historic or observational data to compare predictions against expectations. However, data or products that incorporate the potential impact of climate change provide an estimate of risk in the future. If the risk of an event is influenced by something that has not happened yet, how can we vouch for the quality of a product?
Validation is an important part of the product development process that ensures quality. Validation should be used to assess whether input data and model results are realistic and conform to expectations through scientific analysis and comparison with high-quality benchmarking data. JBA approaches data validation in two ways:
- Process validation, which ensures that the process is appropriate and that the component datasets meet expectations
- Product validation, which ensures that the overall product produces results that meet expectations
For climate change products, validation is more challenging. It is impossible to validate something that hasn’t yet happened, such as the climate trends by 2050! However, it is really important to critically evaluate our data. There are some things we can do. This blog will use an example from JBA’s global climate change work to demonstrate valuable ways to evaluate and build confidence in the data that represent future climates.
Validation example: JBA’s Global Change Factors
Climate projections offer insights into the climatic outcomes of different greenhouse gas emissions pathways. JBA develops climate change data that follows multiple trajectories of global warming.
Global Change Factors
At JBA, we estimate the predicted change to flood risk as a result of changes in the climate for any scenario and time horizon. These estimates of change – change factors – are available globally for river and surface water by statistically comparing a baseline view of precipitation and temperature with the future projections. The global change factors are generated using multidimensional data derived from a Global Climate Model (GCM) to interpret future extremes and calculate the proportional change at a location for a given climate scenario and time period. For river change factors, precipitation and temperature data are used as input to a rainfall-runoff model. Surface water change factors use precipitation data only. As these change factors underpin JBA’s global climate change products, it is really important to evaluate them! So how do we do that?
Firstly, we approach scientific literature and see if our projections for future extremes match other studies. One source of information is peer-reviewed journal articles, though you do have to consider the challenge of data scarcity for certain regions of the globe that are not studied as thoroughly as others due to socio-cultural biases. The IPCC AR6 regional fact sheets are a great source of expectations as:
- They have global coverage
- They do not use the exact same set of models that we have used in our products
The second point is critical to our evaluation: it is very easy to “mark your own homework” when relying on peer-reviewed literature because the majority of climate change research uses the same set of GCMs that we have used to develop the JBA view of risk. So we need to be mindful not to fall into that trap.
With that in mind we take another approach to evaluating the change factors. This involves deriving an alternative variable to use as a proxy for extreme precipitation, which is the climate variable that is of most interest for flooding. We did this by first calculating the total rainfall for the wettest five-day period (pentad) in a calendar year, RX5day. The wettest pentad is used by the IPCC as a proxy for extreme rainfall. When we calculate the difference between the RX5day in the present-day (typically 1980-2010) and a time period in the future, that gives us the anomaly.
The anomaly tells us how a variable will change between two time periods. The benefits of using the RX5day anomaly are:
- It is quick and relatively simple to calculate
- It is a proxy for how extreme events will change in the future
- It is a variable that is independent of the workflow and process used to estimate change factors
Example of Change Factor Evaluation
Below is an example of how we might evaluate change factors. In this example, we evaluate change factors calculated for Europe under an intermediate climate scenario (RCP4.5) with a medium-term time frame.
Figure 1 shows the river and surface water change factors for Europe under RCP4.5 by 2050. Any change factors below 1 indicate a decrease in extreme river flow or rainfall from the present day, whilst change factors above 1 suggest an increase.
Figure 1: Change factors for Europe under RCP4.5 by 2050. (Left:) river change factors. (Right:) surface water change factors.
Figure 2: The RX5day anomaly for 1980-2005 to 2036-2065 for Europe under RCP4.5.
Using the available literature and data to evaluate the RCP4.5 2050 change factors for Europe, we can start assessing our predictions for the future against benchmark expectations. Figure 2 is the RX5day anomaly for Europe under RCP4.5. The figure shows the average change in total precipitation for the wettest five-day period from the historic period (1980-2005) to the future time period (2036-2065). Green colours signify an increase in rainfall totals over the RX5day, whilst brown suggests a decrease in rainfall totals.
Spatially, the surface water change factors match the changes to the RX5day anomaly, with increases in precipitation observed across the northern European states whilst Mediterranean regions observe less precipitation in its annual extreme events. This is corroborated in literature, with extreme precipitation and pluvial flooding projected to increase at global warming levels exceeding 1.5°C in all regions except the Mediterranean (IPCC, 2021).
Comparison of the river change factors with the IPCC Fact Sheet for Europe (IPCC, 2021) show that changes in river discharge corresponding to the return period of 1-in-100 for the mid-21st century present very similar patterns to the river change factors.
However, we observed differences between the river change factors and the RX5day anomaly across northern Scandinavia. The Arctic and sub-Arctic river change factors show a pattern that suggests river flows will be less extreme in the future (change factors < 1). The drying pattern is not observed in either the RX5day anomaly or the surface water change factors. The IPCC Fact Sheet advises with high confidence that glaciers, permafrost, and snow-covered systems in high latitudes and altitudes are already in decline, and that this trend will continue in a warming world. This observation, coupled with the occurrence of “drying” patterns in river change factors in other regions that experience seasonal snow accumulation and subsequent melting leads us to confirm a new hypothesis: the temperature data that is used as part of the rainfall-runoff model impacts the predicted changes to river flows in the future. This demonstrates how validation enables us to question and understand the data in much greater detail.
(We will explore how climate will drive flooding from snowmelt in a separate blog to be published shortly).
Summary
You cannot validate something that has not yet happened, but you can evaluate your predictions based on scientific literature and a solid understanding of the climate model data.
This blog illustrates how we have evaluated one important aspect in the development of our global climate change products – we have applied similar approaches to evaluating our climate change event sets, loss modelling, and climate change datasets (UK and global). We are always happy to discuss these results with clients if you are interested in learning more.
References
IPCC, 2021. Sixth Assessment Report WG1 – The Physical Science Basis, Regional fact sheet – Europe.
https://www.ipcc.ch/report/ar6/wg1/downloads/factsheets/IPCC_AR6_WGI_Regional_Fact_Sheet_Europe.pdf. Accessed 11/10/22.