The public understanding of error in educational assessment

an article by John Gardner (University of Stirling, UK) published in Oxford Review of Education Volume 39 Issue 1 (2013)

Abstract

Evidence from recent research suggests that in the UK the public perception of errors in national examinations is that they are simply mistakes; events that are preventable.

This perception predominates over the more sophisticated technical view that errors arise from many sources and create an inevitable variability in assessment outcomes. The public perception also seems to invest assessment grades and marks with the precision and accuracy of scientific measurements; a perception that does not sit easily with the academic and professional understanding that grades and marks are assigned rather than measured.

However, growing numbers of successful challenges by students to their examination results present an interesting challenge for examination bodies. Such evidence could point to a number of possible causes.

For example, there might be an increasing awareness among students of the uncertainty surrounding the grades assigned to their work; or their confidence in the capacity of such assessments to reflect the true quality of their performance may be decreasing. As the numbers of challenges increase year on year, there is growing consensus among assessment experts that public confidence needs to be strengthened.

Clearly this may be achieved in a number of ways, e.g. by reducing the incidence of human errors and system breakdowns or by improving the public understanding of the assessment process. The latter tactic prompts calls for greater openness and transparency in all aspects of assessment design and process.

However, such calls rarely touch on one of the most enduring dimensions of the problem: the public’s perception of precision and accuracy in educational assessments.

This paper argues that this problem partly arises from a misuse of the term ‘measurement’ in educational assessment and that in addition to openness and transparency, any public understanding strategy should seek to reduce the misconceptions it causes.

---

Informing in the Information Age: How to Communicate Measurement Concepts to Education Policy Makers

an article by Stephen G. Sireci (University of Massachusetts Amherst) and Ellen Forte (edCount, LLC) published in Educational Measurement: Issues and Practice Volume 31 Issue 2 (Summer 2012)

Abstract

Current educational policies rely on educational assessments. However, the technical aspects of assessments are often unknown to policy makers, which is dangerous because sound assessment policy requires knowledge of the strengths and limitations of educational tests.

In this article, we discuss the importance of informing policy makers of important psychometric issues that should be considered whenever tests are proposed for specific purposes. We discuss the types of information that are important to communicate to policy makers, how to best convey this information in a manner in which it can be understood, and how to be seen as a valuable source of information to education policy makers.

We end with some specific steps organizations such as NCME can take to inform policy makers and advocate for valid educational assessment policies.

Hazel’s comment:
This is definitely one occasion that I wish, I wish very hard, that an American concept would come across the pond. However, it appears from this article that even the USA’s policy-makers are not taking much notice.
Reminds me of the not entirely untruthful comment: “the consultants are there to provide evidence for the determined policy”.

---