161 - Transforming Anatomy OSPE Grading with Decision Tree Learning AI

Poster Board Number: 161
There are separate poster presentation times for odd and even posters.
Odd poster #s – first hour
Even poster #s – second hour

Abstract Body : Objective structured practical examinations (OSPEs) are a standard method for testing anatomical knowledge. Grading OSPEs poses a variety of challenges, as it requires a significant amount of time, skilled personnel, and monetary investment to ensure evaluators are consistent and accurate in their feedback. These challenges inevitably increase the lag between evaluation and feedback. Our laboratory has developed a system for grading OSPEs using artificial intelligence (AI) that overcomes these challenges, allowing for increased accuracy, consistency, and speed in evaluations while reducing required resource investment.

Our AI tool is built using decision tree (DT) learning , a classical form of supervised machine learning. In order for the AI to grade, it must first build a model of the students’ answers using a training dataset provided to the AI algorithm consisting of predetermined correct and incorrect OSPE responses. Using the DT learning algorithm, the AI identifies patterns in the answers and splits the dataset iteratively based on distinctive features of the input data. For example, all answers that include the word “muscle” versus those that do not. Each split creates a node, starting from the root node which represents the entire dataset, branching into increasingly specific subsets until the final subsets, called leaf nodes, are created. The data is divided so that each split maximizes information gain and minimizes the entropy of the data. Splitting continues until all leaf nodes’ subset data is pure (all correct or all incorrect) or no further splits provide any information gain. The AI’s confidence in grading an answer is directly based on the nodal purity with pure nodes having 100% confidence, and impure nodes having confidence in proportion to ratio of correct to incorrect answers. The AI must be trained to mark each unique question and is retrained to incorporate more answers from students over time. The DT works particularly well for datasets with specific language like anatomical structures. Future steps are to include an unsupervised model into the classification and confidence model based on student feedback to the AI tool. When a student agrees with the grade provided by the AI, the tool will gain confidence.

Due to the stepwise nature of DT-based grading process, this model is suitable for grading OSPEs as it provides instructors with the ability to trace the results in an interpretable manner. Additionally, the DT is able to make predictions to handle missing/incomplete data, so it has some capacity to judge answers it has not previously been trained on. Provided a rigorous and extensive training data set, with correct and incorrect answers reinforced by human revision and reinforcement, the AI is capable of refining the accuracy of its output with high consistency.

Ultimately, our AI tool demonstrates the viability of decision tree learning-based AI in grading evaluations with an expected range of answers, that is not limited to anatomy but applicable in any educational discipline. Its integration with online study resources enables the output of adaptive and immediate feedback, empowering students and educators alike.