“All models are wrong, but some are useful” was a term first used by George Box, and whilst there is some truth in this statement, at JBA we need to rely on models to produce world-leading flood maps that are as useful as possible. So how do we ensure that our maps are not “wrong”?
To begin with, we need to acknowledge that the data that we’re putting into our hydraulic model (JFlow) is as accurate as possible. As touched on in one of our previous blogs, the different components used to create a flood map play a major part in the quality and accuracy of the finished product. So it follows that it is not enough just to take data that we believe to be accurate. Throughout this blog I will highlight some of the key stages of validation that we undertake throughout our mapping process, which give us the highest confidence in the flood maps that we release.
What is validation? Validation should be used to assess whether input data and model results are realistic and conform to expectations. This can be achieved through scientific analysis and comparison with high-quality benchmarking data. At JBA Risk Management we validate the components and product throughout the development cycle so that we are confident that we are creating the highest quality data.
Component Validation
The greater confidence we have in the data we put into the model, the greater the confidence we can have in the output data that we’re producing. Therefore, it’s important for us to undertake validation on each of the key components of flood maps.
Digital Elevation Model:
The Digital Elevation Model (DEM) is one element that has a significant influence on the accuracy and detail of a hydraulic flood map. With this being said, we need to ensure that the information being shown in the DEM is in line with the actual elevation readings of the ground. When embarking upon an update for a new country, the first step is to acquire new elevation data, ideally the best available at the time. After acquiring a new elevation model, we assess surface elevation values using Ice, Cloud, and land Elevation Satellite – 2 (ICESAT-2) Ground Control Points, a global dataset provided by NASA. These ground control points provide an accurate measurement for what the elevation is in the real world. This dataset has been validated independently in literature to demonstrate its accuracy over land, whilst the pre-filtering technique that we use has been adapted from academic papers. (Science Direct, 2022; Springer Link, 2022; Optica, 2019). We run a statistical analysis to compare the DEM to the ICESAT points to determine how realistic the DEM really is.
Hydrology:
The hydrological data that we use as an input to our model is another significant factor in the quality and accuracy of our flood maps. Realistic estimates of river flow and rainfall represents what could happen in real life events, so it's important to validate this data with historic peak flows to ensure our inputs are accurate. Once we have sourced validation data, we compare it to our own hydrological data and determine the coefficient of determination, R2. The R2 value can range from 0-1. A high value means we can be confident in the accuracy of our data. A low value would prompt us to investigate the cause.
2022 Canada Flood Map Update Example:
Validation of our river flow values was carried out against the HYDAT database’s peak flow values provided and maintained by the Government of Canada. Below is a table demonstrating the R2 values of the comparison. The R2 values are all greater than 0.8, so we can be confident that our hydrological data is positively comparable to the Hydat data. (See Fig.1)
Fig. 1. R2 values of comparison.
Land Use:
Land use data is used to determine run off rates and roughness coefficients in our hydraulic models. Run off rates determine how much water will run over the surface, as some will infiltrate into the ground or evaporate into the air. Roughness coefficients determine how the roughness of the land varies across the surface, and so affects the flow of the water over land. This data is relatively simple to validate as we generally overlap the raster with aerial imagery and visually confirm that the land use depicted in the dataset is true to the satellite imagery. Although this is a relatively fast and simple check, it once again gives us confidence in the data that we’re inputting into our model.
Fig.2. The above example shows the ESRI land cover dataset compared to ESRI Aerial Imagery (This dataset was produced by Impact Observatory for Esri. © 2023 Esri. This dataset is based on the dataset produced for the Dynamic World Project by National Geographic Society in partnership with Google and the World Resources Institute.)
Product Validation
As validation is completed on our inputs throughout the modelling process, the next logical step is to validate the product that we release to clients. This includes river, surface water and coastal flood maps where applicable. There are two main ways that we can accomplish this: by using other models as a source where available or by using observed events.
There’s a lot of uncertainty in models, and so when it comes to comparing a model to a model, a cautious approach must be taken. Some countries have well respected flood maps, an example of being the FEMA national flood maps of the United States. Though not without their flaws as outlined in an earlier blog, they can provide a good baseline for validation at a basic level.
A more desirable source of validation would be a direct comparison to real life events. Ideal sources for this can include high water marks, earth observation data containing the extents of flood events, and studies carried out by local authorities following such events. Each of these can provide a great insight into how indicative a flood map can be.
Japan satellite imagery example:
Fig.3. (Above left) JBA 100-year fluvial flooding, 2022. (Above right) GSI flooding near Mogami River Basin (Source: Geospatial Information Authority of Japan website (https://maps.gsi.go.jp/).
What happens if the maps don’t perform well following validation
Validation is not only helpful as a mechanism to reinforce our flood maps, it’s also a great guide as to when something hasn’t quite gone as expected. But what do we do in those situations? In the case of elevation data, we can make minor adjustments to the data we have, for example in Galway, Republic of Ireland, during the update process we identified that our flooding in the coastal maps was significantly less extensive than our validation sources. After an investigation into the data, we discovered that the channel in our DEM was deeper than it was in our previous DEM, resulting in less flooding. We were able to raise the bed of the channel in the DEM and rerun our model, which resulted in a similar likeness to the validation source.
The process of validation within our modelling cycle is ever evolving. Over the past 12 months we’ve introduced validation leads within each team to help identify improvements to what can be validated throughout the flood mapping process, and what else is available to use as validation. It’s a welcome addition to the methodology and builds an additional level of confidence in the maps we produce.
References
Science Direct, 2022. Effects of environmental conditions on ICESat-2 terrain and canopy heights retrievals in Central European mountains
https://www.sciencedirect.com/science/article/abs/pii/S0034425722002267
(Accessed 24 July 2023)
Springer Link, 2022. Accuracy assessment of ICESat-2 ATL08 terrain estimates: A case study in Spain
https://link.springer.com/article/10.1007/s11771-022-4896-x
(Accessed 24 July 2023)
Optica, Vol 27, 2019. Ground elevation accuracy verification of ICESat-2 data: a case study in Alaska, USA
https://opg.optica.org/oe/fulltext.cfm?uri=oe-27-26-38168&id=424608
(Accessed 24 July 2023)