137 - Evaluating Student Perceptions of the AI-OSPE Tool Using Q-Methodology

Poster Board Number: 137
There are separate poster presentation times for odd and even posters.
Odd poster #s – first hour
Even poster #s – second hour

Abstract Body : Introduction:
The use of artificial intelligence (AI) to provide rapid, personalized feedback to students has the potential to promote individualized education and improve student learning. To further this improvement and alleviate the time-consuming and resource-intensive demands of marking Objective Structured Practical Examinations (OSPE), McMaster University’s Education Program in Anatomy was prompted to develop the AI-OSPE tool. The tool aims to provide anatomy students with an accessible online study tool powered and graded by AI, to help in their preparation for OSPEs.

Objective:
The current study seeks to utilize Q-methodology to assess student perceptions of the AI-OSPE tool on learning, and to inform the improvement of a publicly available, open source, free, web-based, AI-graded OSPE tool in fostering equity for access to virtual education resources.

Method:
A comprehensive list of statements was developed based on feedback obtained from a focus group with year 2 McMaster MSc Occupational Therapy students who previously used the AI-OSPE tool. The final statements are based on student opinions ranging from the practical utility of questions, utility of the feedback/AI-grading, accessibility of the tool, limitations of technology, study preferences, user preferences, and trust in AI.

Subsequently, a class of 68 year 1 McMaster MSc Occupational Therapy students will be introduced to the AI-OSPE tool and given the opportunity to use it as an additional study resource in their own time. Access to the tool will be provided to the students during an 8-week period in which the students are enrolled in a musculoskeletal anatomy course. At the end of the course, students will be asked to complete a Q-sort survey in which they will sort the Q-statements in a quasi-normal grid based on their degree of agreement with the statements. Participants will also be asked to provide qualitative feedback justifying their most extreme (strongly agree and strongly disagree) rankings. The rankings will be analyzed using a by-person factor analysis which categorizes students with shared perceptions into groups. This will allow our team to better understand and respond to the needs of students, and further improve the AI-OSPE tool.

Results:
Data collection and analysis are anticipated to be completed by February of 2023.

Results from our study will establish how Q-methodology factors, obtained from a common Q-sample, reflect student perceptions of the AI-OSPE tool as a study tool. This is critical in providing an evidence-based approach to directing AI-OSPE improvements and informing practical use of the tool.

Conclusion:
We hypothesize that Q-methodology will enable us to assess the efficacy of the current AI-OSPE tool and provide us with a rich and informative dataset that will allow for honing the study tool prior to its launch. These results stand to improve our understanding of students’ preferences related to the use of a virtual tool for study of anatomy and receiving AI-based feedback.