cloud image for UX Researcher Behavioral Scientist Portfolio case research study

THE IMPACT OF INCONSISTENT FORECASTS ON USER TRUST AND DECISION MAKING

The Impact of Forecast Consistency and Accuracy on Trust: Projects

ROLE

Lead Researcher

METHODS

Quantitative/Experimental

DEVELOPMENT TIME

4 months

The Impact of Forecast Consistency and Accuracy on Trust: List

INTRODUCTION

Problem Statment

Forecasts for major weather events often begin days in advance. Although weather models constantly change, growing more accurate on average as lead times decrease (Lazo, Morss, & Demuth, 2009; Wilson & Giles, 2013), forecasters are often hesitant to update the forecasts they provide the public out of fear that inconsistency in subsequent forecasts will negatively affect user trust. Because accuracy generally increases as lead times decrease, the choice to maintain consistency can be at tradeoff with accuracy.

Research Questions

Does forecast inconsistency (mismatching forecast values) actually reduce user trust in forecasts?
- If so, how does it compare to reductions in trust due to inaccuracy (mismatching values between most recent forecast and observed accumulation)?

Primary Research Goal

Directly compare the effects of forecast inaccuracy and inconsistency on trust in order to inform whether trust is best preserved by updating forecasts (to be more accurate) or maintaining consistency. To this end, it was necessary to tightly control a number of forecast characteristics in order to ensure effects observed were due to inconsistency and inaccuracy. It wasn't possible to control for all forecast characteristics in a single experiment; therefore, the answers to our research questions are contained in the combined results of the three experiments described below.

The Impact of Forecast Consistency and Accuracy on Trust: Text

METHODOLOGY

The Impact of Forecast Consistency and Accuracy on Trust: Text

Participants

ALL EXPERIMENTS

- University of Washington psychology students
- Participated for course credit and the opportunity to earn a cash bonus.

EXPERIMENT 1

N=368
67% Female
Mean Age = 19.1 years

EXPERIMENT 2

N=164
49.1% Female
Mean Age = 19.7 years

EXPERIMENT 3

N= 160
50.6% Female
Mean Age = 19.9 years

The Impact of Forecast Consistency and Accuracy on Trust: List

The School Closure Paradigm

Forecast accuracy and consistency were manipulated within-groups using a computer-based task in which participants monitored sequences of snow forecasts in order to make school closure decisions. There were 24 trials (including 8 filler trials) of 24-hour snowfall accumulation forecasts at 48 and 24 hours in advance of the target date. This allowed me to assess the impact of consistency and accuracy on trust ratings reported after learning the outcome on each trial, as well as on snow accumulation estimates, and closure decisions.

Participants were told to advise closing if they expected six or more inches of snow accumulation.

The Impact of Forecast Consistency and Accuracy on Trust: Text

Stimuli

To ensure the effects observed were due to accuracy and consistency alone, I controlled for a number of forecast characteristics. In particular, I attempted to maintain:

The Impact of Forecast Consistency and Accuracy on Trust: Testimonials

Forecast Data: Experiments 1-3

Because of the tight control of extraneous variables required to examine the effects of forecast accuracy and consistency, no individual experiment was capable of fully addressing them. Therefore, these effects were addressed by the combined results of three experiments in which uncontrolled characteristics were traded across.

The Impact of Forecast Consistency and Accuracy on Trust: Text

RESULTS

Inaccuracy reduced trust.
Inconsistency also reduced trust.
However, the magnitude of the reduction due to inaccuracy was considerably larger than that due to inconsistency.
In addition, inconsistency may have other benefits as well. In particular, it can:
- communicate uncertainty, and
- encourage more cautious decisions

The Impact of Forecast Consistency and Accuracy on Trust: Text

CONCLUSIONS

Forecast inconsistency reduces trust in forecasts, but not to the extent that inaccuracy does. If trust is a concern, it is inadvisable to sacrifice accuracy in favor of consistency. Forecasters should update forecasts to be more accurate regardless of consistency.

Inconsistent forecasts may provide users with important information about forecast uncertainty that can be applied to decision-making. Participants regard inconsistent forecasts as indicating greater uncertainty and are more likely to protect themselves.

Participants may have an intuitive understanding that the most recent forecast is likely to be more accurate.

Limitations

The tight control of extraneous variables was done at a loss to ecological validity.