Misuse of automated decision aids: Complacency, automation bias and the impact of training experience

an article by J Elin Bahner, Anke-Dorothea Hüper and Dietrich Manzey (Berlin Institute of Technology, Department of Psychology and Ergonomics) in International Journal of Human-Computer Studies Volume 66 Issue 9 (September 2008)

Abstract
The present study investigates automation misuse based on complacency and automation bias in interacting with a decision aid in a process control system. The effect of a preventive training intervention which includes exposing participants to rare automation failures is examined. Complacency is reflected in an inappropriate checking and monitoring of automated functions. In interaction with automated decision aids complacency might result in commission errors, i.e., following automatically generated recommendations even though they are false. Yet, empirical evidence proving this kind of relationship is still lacking. A laboratory experiment (N=24) was conducted using a process control simulation. An automated decision aid provided advice for fault diagnosis and management. Complacency was directly measured by the participants’ information sampling behaviour, i.e., the amount of information sampled in order to verify the automated recommendations. Possible commission errors were assessed when the aid provided false recommendations. The results provide clear evidence for complacency, reflected in an insufficient verification of the automation, while commission errors were associated with high levels of complacency. Hence, commission errors seem to be a possible, albeit not an inevitable consequence of complacency. Furthermore, exposing operators to automation failures during training significantly decreased complacency and thus represents a suitable means to reduce this risk, even though it might not avoid it completely. Potential applications of this research include the design of training protocols in order to prevent automation misuse in interaction with automated decision aids.

References and further reading may be available for this article. To view references and further reading you must purchase [this was a link but it didn't] this article. Or ask Hazel to go and read it in the British Library!

Hazel's comment:
Actually you don't need to ask – I'll be doing it anyway because I want to see whether any of the references look at decision-making aids in a careers context rather than a manufacturing one. It would also be interesting, to me anyway, to see how this complacency affects automated classification systems where you have to accept or reject the programs "suggestion".
Given that most of the mistakes I both made and found when doing quality assurance on learning opportunity data were connected with complacently copying down through the data rows I suspect that we'll all go for the easy option!