JESSICA BURGENO, PhD
Design Researcher + Behavioral Scientist
THE EFFECT OF EXPLICIT UNCERTAINTY ON USER TRUST
ROLE
Lead Researcher
METHODS
Quantitative/Experimental
DEVELOPMENT TIME
3 months
INTRODUCTION
Problem Statment
Although accurate numeric uncertainty estimates are often available, they're infrequently relayed to the general public. Like many other domains, the majority of public weather forecasts provide only single-value forecast (e.g. "4 inches of snow"). Uncertainty estimates (e.g., a 30% chance of observing 6 or more inches of snow) have demonstrated advantages over single-value forecasts alone in a number of situations, but it’s unclear whether these benefits extend equally to naturalistic forecasts with different magnitudes of inconsistency.
​
Research Questions
-
Do the results of prior studies (with highly controlled forecast data) extend to real historical, naturalistic forecast data? That is, does inconsistency reduce trust but to a lesser extent than inaccuracy does?
-
Does the inclusion of a threshold probability attenuate the negative effect of forecast inconsistency (mismatch between forecasts) on trust?
​
Research Goal
Provide insight as to whether inconsistency is in fact a lesser threat to trust than inaccuracy in the context of naturalistic forecasts, and whether there are effective strategies for preserving trust when forecasts are inconsistent.
​
METHODOLOGY
Participants
419 University of Washington psychology students participated for course credit and the opportunity to earn a cash bonus.
Stimuli
Stimuli
Instead of utilizing highly controlled forecast and accumulation stimuli (as I had in previous experiments), the present experiment used forecast stimuli based on real forecasted and observed snow accumulation values provided by the National Weather Service (NWS). This allowed me to retest and extend prior research questions with a more realistic set of forecast data.
​
RESULTS
-
Contrary to what we expected, inconsistency actually increased trust.
-
However, somewhat in line with what I expected, The negative effect of forecast inconsistency on user trust was greater for deterministic forecasts than for probabilistic forecasts.
-
The effect of forecast inconsistency on accumulation estimates did not vary across the levels of forecast format.
-
The negative effect of forecast inconsistency on expected value of decisions (decision quality) was greater for deterministic forecasts than for probabilistic forecasts.
​
The School Closure Paradigm
Forecast format was manipulated between-groups using a computer-based task in which participants monitored sequences of snow forecasts in order to make school closure decisions. There were 130 trials of 24-hour snowfall accumulation forecasts at 48 and 24 hours in advance of the target date. This allowed us to assess the impact of uncertainty estimates and consistency on trust ratings, snow accumulation estimates, and closure decisions.
​
Participants were told to advise closing if they expected six or more inches of snow accumulation.
​
Results
-
In line with prior research, inaccuracy reduced trust.
-
However, counter to prior research, inconsistency actually increased trust slightly.
-
Probabilistic forecasts:
-
increased trust
-
dampened reductions in trust due to inaccuracy
-
amplified increases in trust due to inconsistency
-
​
Limitations​
It's possible some of our results may be due to the characteristics of the specific set of naturalistic forecasts utilized. Future research should retest these questions with a unique set of naturalistic forecasts.
CONCLUSIONS
-
Trust is a concern in high-stakes contexts like extreme weather, as people may not respond to available information if they don't trust it. This research suggests that forecasts should be updated to be more accurate, even if that means being inconsistent.
-
Access to explicit uncertainty estimates can preserve trust under both inaccuracy and inconsistency, by dampening reductions in trust due to inaccuracy and amplifying slight increases in trust due to inconsistency.