cloud image for UX Researcher Behavioral Scientist Portfolio case research study

SIZING THE IMPACT OF HIGH-STAKES WEATHER FORECAST VOLATILITY VS. ACCURACY ON USER RETENTION

The Impact of Forecast Consistency and Accuracy on Trust: Projects

ROLE

Lead Researcher

METHODS

Quantitative/Experimental

DEVELOPMENT TIME

4 months

The Impact of Forecast Consistency and Accuracy on Trust: List

Problem

Forecasts for major weather events often begin days in advance. Although weather models constantly change, growing more accurate on average as lead times decrease (Lazo, Morss, & Demuth, 2009; Wilson & Giles, 2013), forecasters are often hesitant to update the forecasts they provide the public out of fear that inconsistency (mismatching forecast values) in subsequent forecasts will result in user skepticism and churn. Because accuracy generally increases as lead times decrease, the choice to maintain consistency can be at tradeoff with accuracy. The purpose of this study was to directly compare the effects of forecast inaccuracy and inconsistency on trust to inform whether retention is best preserved by updating forecasts (to be more accurate) or artificially maintaining consistency.

The Impact of Forecast Consistency and Accuracy on Trust: Text

Method

Study Design

Forecast accuracy and consistency were manipulated within-groups using a computer-based task in which participants monitored sequences of snow forecasts in order to make school closure decisions. There were 24 trials (including 8 filler trials) of 24-hour snowfall accumulation forecasts at 48 and 24 hours in advance of the target date. This allowed me to assess the impact of consistency and accuracy on trust ratings reported after learning the outcome on each trial, as well as on snow accumulation estimates, and closure decisions.

Participants were told to advise closing if they expected six or more inches of snow accumulation.

The Impact of Forecast Consistency and Accuracy on Trust: Text

Stimuli

To ensure the effects observed were due to accuracy and consistency alone, I controlled for a number of forecast characteristics. In particular, I attempted to maintain:

The Impact of Forecast Consistency and Accuracy on Trust: Testimonials

Forecast Data: Experiments 1-3

Because of the tight control of extraneous variables required to examine the effects of forecast accuracy and consistency, no individual experiment was capable of fully addressing them. Therefore, these effects were addressed by the combined results of three experiments in which uncontrolled characteristics were traded across.

The Impact of Forecast Consistency and Accuracy on Trust: Text

Insights

Inaccuracy and inconsistency both reduced trust.
However, the magnitude of the reduction due to inaccuracy was considerably larger than that due to inconsistency.
Therefore, where trust is a concern, forecasters should update forecasts to be more accurate regardless of consistency.
In addition, inconsistency may have other benefits as well. In particular, it can:
- communicate uncertainty
- encourage more cautious decisions

Limitations

The tight control of extraneous variables was done at a loss to ecological validity.

The Impact of Forecast Consistency and Accuracy on Trust: Text

Outcome

Increased user trust, understanding, and appropriate protective action.