snow image for UX Researcher Behavioral Scientist Portfolio case study

MITIGATING THE IMPACT OF HIGH-STAKES WEATHER FORECAST VOLATILITY ON USER RETENTION: EXPLICIT UNCERTAINTY ESTIMATES

The Effect of Uncertainty Information on Trust and Decision Making: Projects

The Effect of Uncertainty Information on Trust and Decision Making: List

ROLE

Lead Researcher

METHODS

Quantitative/Experimental

DEVELOPMENT TIME

3 months

Problem

Although accurate numeric uncertainty estimates are often available, they're infrequently relayed to the general public. Like many other domains, the majority of public weather forecasts provide only single-value forecast (e.g. "4 inches of snow"). Uncertainty estimates (e.g., a 30% chance of observing 6 or more inches of snow) have demonstrated advantages over single-value forecasts alone in a number of situations, but it’s unclear whether these benefits extend equally to naturalistic forecasts with different magnitudes of inconsistency. The purpose of this study was to understand whether inconsistency is in fact a lesser threat to trust than inaccuracy in the context of naturalistic forecasts, and whether uncertainty estimates are effective strategies for preserving trust when forecasts are inconsistent.

The Effect of Uncertainty Information on Trust and Decision Making: Text

Method

Participants

419 University of Washington psychology students participated for course credit and the opportunity to earn a cash bonus.

The Effect of Uncertainty Information on Trust and Decision Making: Text

Stimuli

Instead of utilizing highly controlled forecast and accumulation stimuli (as I had in previous experiments), the present experiment used forecast stimuli based on real forecasted and observed snow accumulation values provided by the National Weather Service (NWS). This allowed me to retest and extend prior research questions with a more realistic set of forecast data.

Anchor 1

The Effect of Uncertainty Information on Trust and Decision Making: Text

RESULTS

Contrary to what we expected, inconsistency actually increased trust.
However, somewhat in line with what I expected, The negative effect of forecast inconsistency on user trust was greater for deterministic forecasts than for probabilistic forecasts.
The effect of forecast inconsistency on accumulation estimates did not vary across the levels of forecast format.
The negative effect of forecast inconsistency on expected value of decisions (decision quality) was greater for deterministic forecasts than for probabilistic forecasts.

Study Design

Forecast format was manipulated between-groups using a computer-based task in which participants monitored sequences of snow forecasts in order to make school closure decisions. There were 130 trials of 24-hour snowfall accumulation forecasts at 48 and 24 hours in advance of the target date. This allowed us to assess the impact of uncertainty estimates and consistency on trust ratings, snow accumulation estimates, and closure decisions.

Participants were told to advise closing if they expected six or more inches of snow accumulation.

The Effect of Uncertainty Information on Trust and Decision Making: Text

Insights

Forecasts should be updated to be more accurate, even if that means being inconsistent.
Access to explicit uncertainty estimates can preserve trust under both inaccuracy and inconsistency, by dampening reductions in trust due to inaccuracy and amplifying slight increases in trust due to inconsistency.

Limitations

It's possible some of our results may be due to the characteristics of the specific set of naturalistic forecasts utilized. Future research should retest these questions with a unique set of naturalistic forecasts.

Outcome

Increased user trust, understanding, and appropriate protective action.

Anchor 2