Using Local Storm Reports and National Weather Service Warnings as Verification Proxies for Warn-On Forecast Ensemble Guidance

In 2019, I accepted an internship at NOAA's National Weather Center in Norman, Oklahoma, which houses many well-known entities of meteorological studies including the National Severe Storms Laboratory (NSSL), the Storm Prediction Center (SPC), and the Cooperative Institute for Mesoscale Meteorological Studies (CIMMS). For the beginning weeks of my internship, I got to interact with employees from all these areas as forecasters, researchers, and members of academia came together to create real-time forecasts during the Spring Forecasting Experiment. I got to take part in this as an intern (which is awesome because it's pretty exclusive!), and draw weather maps, fill out surveys of probabilistic model performance, and be apart of the first SPC-issued "high risk" day in over 2 years! (May 20th, 2019...the first day of my internship. You could say I was excited.)

Check out this video for a taste of what it was the experience was like. I'm on the right in the beginning in a polka-dot shirt ^____^

After participation in the Spring Forecasting Experiment, I worked on research with the NSSL related to data that derived directly from the experiment. This project is still in the works, and will become my master's thesis. This page will be updated as more results are gathered, but, for now, here are preliminary results!

What is the Warn-On Forecast System?

Images from: nssl.noaa.gov and link.springer.com/chapter/10.1007/978-3-030-21565-1_39

The Warn-On Forecast system (WoFS) is an ensemble model designed to provide probabilistic, short-term guidance for thunderstorm hazards (Skinner et al. 2016). The goal of this forecast system is to support watch-to-warning hazardous weather such as severe convective wind gusts, hail, flash flooding, and tornadoes. Evaluation of model guidance can be assessed against event observations, so I sought out conducting research where I assist in correcting them. For this study, datasets of storm observations are put in place as verification metrics to assess the reliability of WoFS.

Specifically, these datasets are:

- Local Storm Reports (LSRs)
- National Weather Service (NWS) warnings
- Multi-Radar/Multi-Sensor (MRMS) System

Traditionally, convection-allowing models (CAMs) have been verified using LSRs, which are useful in terms of supplying human confirmation about extreme weather in a specific location. However, LSRs have limitations: they require a human to be in an area to report said storm making them highly influenced by population density. Is it possible to leverage other sources of datasets as verification proxies? For now, NWS Warnings and the MRMS system are used to determine this, because they can help rectify issues of obtaining LSRs in sparsely populated areas (they are broader in scale). MRMS Composite Reflectivity and MRMS Azimuthal Shear in particular are the research focus.

Below are specifics on the data and methods used.

=============================================================================================================

1) Forecast Data

WoFS Forecasts: May 2017, 2018, and 2019 (19-02Z cycles, incremented every hour)

i. 18 member forecasts
ii. 19 cases for 2017, 22 cases in 2018, 24 cases in 2019 (no data available on certain days)

2) Observational/Verification Data

Local Storm Reports (LSRs): May 2017, 2018, and 2019 (19-02Z cycles, incremented every hour)

i. Raw LSR data obtained from SPC
ii. 19 cases for 2017, 22 cases in 2018, 24 cases in 2019 (no data available on certain days)

NWS issued severe thunderstorm and tornado warning polygons:

i. Shapefiles obtained from Iowa State University IEM
ii. 19 cases for 2017, 22 cases in 2018, 24 cases in 2019 (no data available on certain days)

Variable of interest: Vertical Velocity at 0-2 km above ground level (threshold > 0.003 s^-1)

=============================================================================================================

Methods

1) Verify data

- LSRs and NWS severe thunderstorm and tornado warning polygons are gridded to daily WoFS domain
- “Practically perfect forecast” generated (9km and 27km neighborhood sizes used to match forecast probabilities)

2) Assess skill and reliability of 4-hour, 3-hour, 2-hour, and 1-hour forecasts

- Reliability diagrams (Perfect forecast: straight diagnol line from bottom left to top right. Below line = overforecasting. Above line= underforecasting).
- Fractions skill score (FSS) where 0: no skill and 1: perfect skill
with different sigma smoothing weights applied - Variable explored: Vertical Velocity with the Neighborhood maximum ensemble probability (NMEP) method. This represents the probability a selected threshold will be exceeded anywhere within the chosen neighborhoods (Roberts et al. 2019) which is 9km and 27km in this case.

*Since these are premiliminary results, only vertical velocity is explored. But other variables that will be applied are hail, 0-2 km and 2-5 km updraft helicity, and vertical updraft velocity. Applying differeing neighborhood sizes and times will help determine the best set metrics for determining reliability, and various thresholds will be applied in the near future. For now, the fractions skill score and reliability diagrams are shown below for vertical velocity.

=============================================================================================================

Results

The best FSS scores generally occur with NWS warnings for a 9 km NMEP (all sigmas) and MRMS Azimuthal Shear fir a 27 km NMEP at sigma 39, both at 4-hour forecast periods. The worst FSS generally occurs with LSRs at 9 km and 27 km NMEPs. They are also notably low at sigma values of 9 and 1-hour forecast periods.

The best performing reliability diagram is NWS warnings with sigmas 9 and 15 for earlier forecast hours and sigmas 22 and 27 for later forecast hours. MRMS Azimuthal Shear also performs well, especially at a sigma of 15 for a 4-hour period. Both yield ideal results when the NMEP is 27 km. The worst performing reliability diagrams were LSRs at 9 km and 27 km for 1-hour forecasts and sigma values of 9. However, MRMS Composite Reflectivity was also prone to under-forecasting throughout the runs.

=============================================================================================================

Conclusions

1) FSS for NWS Warnings and MRMS Azimuthal Shear proved to be more relevant information for the spatial scales at which WoFS provides probabilistic guidance. On the other hand, LSRs are not shown to be spatially compatible with WoFS, as expected. These outputs are representative in both the 9 km and 27 km NMEP. It is probable that increased sigma values (hence, increased smoothing) also increased the effective neighborhood length scale (Schwartz and Sobash, 2017). Additionally, longer forecasting time periods are favorable as this allows more time for storm evolution and can give the user more information about the storm.

2) In regards to the reliability diagrams, the results depict 27 km for both NWS warnings and MRMS Azimuthal Shear to be the best perfoming NMEP. This makes sense as a higher neighborhood radius increases the chance of capturing the points where hazards like hail and tornadoes have formed. Though high sigma values were favored for FSS measurements, they may cause removal of important details near storm initiation time, so sigmas 9 and 15 early in the period can best mitigate this. LSRs again are not ideal verification metrics without the combination of observations with better reliability, like NWS Warnings or MRMS Azimuthal Shear. It is important to note that the MRMS Composite Reflectivity was not always identifiable, which should be kept in mind when choosing different MRMS products to use as verification.



Future Work

1) Use NWS severe thundertorm warnings for hail and wind verification, not just tornadoes

2) Test additional neighborhood sizes and thresholds (what are optimal NMEP and thresholds to maximize skill and reliability?)

3) More years for May will be added to compare to May 2019 (specifically 2017 and 2018).

Link to Poster

=============================================================================================================

References

Roberts, B., I. L.Jirak, A. J.Clark, S. J.Weiss, and J. S.Kain, 2019: Postprocessing and visualization techniques for convection-allowing ensembles. Bull. Amer. Meteor. Soc., 100, 1245–1258, https://doi.org/10.1175/BAMS-D-18-0041.1

Schwartz, C. S., and R. A. Sobash, 2017: Generating probabilistic forecasts from convection-allowing ensembles using neighborhood approaches: A review and recommendations. Mon. Wea. Rev., doi:https://doi.org/10.1175/MWR-D-16-0400.1

Skinner P, Wicker L, Wheatley D, Knopfmeier K. Application of Two Spatial Verification Methods to Ensemble Forecasts of Low-Level Rotation. Weather and Forecasting. 2016;31(3):713-735.

Frequently Asked Questions At The Poster Session

1) When combining LSRs and NWS Warnings, doesn't it double-count values where storms occur?

This isn't the case - in the generals sense of what is being coded in Python, all values that fall within either LSRs or NWS warnings are assigned values of 1. If the datasets happen to fall within the same place, that value will still remain 1. If this were not the case, the combined LSRs/warning diagrams and FSS scores would most liekly not look similar to singular NWS warning results.