Assistant Professor Western University London, Ontario, Canada
Abstract Body : Introduction & Objective: Does proctoring matter? With the mass movement of courses online to support emergent remote pandemic learning, so too did assessment, often with very little consideration for the changes in assessment administration necessitated by being online. Especially for large classes, multiple choice question exams are an assessment staple, and are one example of direct transfer that occurred from classroom to digital spaces in light of the pandemic. These question types are familiar, easy to administer and automatically graded by most learning management systems, making them a natural preference for instructors and students alike. But does student performance change alongside the environment in which we ask them to write, and are some question levels better suited for different environments? This study set out to determine just that, using real course assessment data collected across three terms in which students completed nearly identical assessments on the same foundational content. We hypothesized that with increased proctoring, performance scores would decrease, especially for questions of lower Bloom’s Taxonomy levels.
Methods: In this QI study, midterm examination data from 3 successive cohorts in a large undergraduate anatomy course were compared via a two-way repeated-measures ANOVA to analyze differences between test item performance (question difficulty and discrimination index) on varying question levels (higher and lower levels of bloom’s taxonomy) as the assessment mode changed (2020: online unproctored, 2021: online remote proctored, and 2022: in-person). Critically, due to the nature of the course, lecture content was held constant throughout the duration of the study period.
Results: 28 questions (out of a possible 70) were repeated on the midterm examination across the 2020, 2021 and 2022 academic years and were answered by 865, 437 and 391 students respectively. 21 questions were identified as being lower level and 7 as higher level by two independent researchers (IRR = 0.89). For both question difficulty and discrimination index, there were simple main effects of cohort year (difficulty: F(2,25) = 10.984, p < 0.001, discrimination: F(2,25) = 4.277, p = 0.025) with no effect of question level (p>0.05) or interaction (p>0.05) such that question difficulty increased with increased proctoring across all three cohorts (2022>2021>2020) and discrimination index increased between 2020 and 2022.
Conclusion: Assessment modality is an important consideration for faculty and students, especially as courses, and their assessments, look to stay online. Based on the current findings, virtual proctoring seems to increase test item difficulty while only in-person proctoring influences discrimination indices. This may indicate the need for students to write exams in person, or to create explicitly open book-style assessments to preserve the academic integrity of multiple-choice style assessments.
Significance/Implications: As online learning surfaces as the new model of contemporary education, ensuring exam integrity while meeting the needs and challenges faced by the student is paramount. Instructors should approach online testing thoughtfully if the intention is to maintain grade equivalency with traditional in-person examinations. These results have implications for online course design, online pedagogy, grade inflation, and online grade equivalency.
Funding Sources: There is no funding for this project.
