A Study on the Influence of Rater Effect on the Fairness of Oral Examination Scoring
Min-Ning Yu
Abstract
Oral examinations have been used in selecting personnel for many years, and relevant research reports over the years have also supported the feasibility and effectiveness of adopting “structured oral examinations.” When implementing “structured oral examination” measures, most of the many factors that may interfere with the fairness of oral examination scoring can be eliminated or improved through standard operating procedures. Only one of them -- the “rater effect” -- is difficult to exclude or control from standard operating procedures. This article puts forward a suggestion for the application of “many-facet model” (MFM) scoring to objectively analyze the estimated values of various factors in the oral examination committee scoring data, including the examinee abilities, the difficulties of the oral exam questions, and the severity/lenient parameters of the raters’ grading. According to relevant literature, applying the MFM model to such data analysis involving rater effects can not only improve the accurate estimation of examinee abilities, but also improve the reliabilities of the overall data analysis, and promote more accurate scoring of oral exams to achieve impartiality, fairness, and correctness. In order to ensure that the MFM model can be applied to the analysis of subsequent oral examination score data, this article also suggests that the scoring method of the ready-made oral examination score sheet be changed from an interval scale score (continuous data attribute) (e.g., assign a score within the score range of 50-59 points) into an ordinal scale score of discrete data attributes (e.g., only evaluate the score into excellent, good, average, and poor grades), in order to reduce the interference of the “rater effect” on the fairness of oral examination scoring.
Keywords: rater effect, structured oral examinations, many-facet model, grading fairness, grading severity/lenient parameter