2023-07-06 ジョージア大学 (UGA)
◆研究では、評価結果に影響を与える要素や審査員間の意見の相違点を分析し、AIシステムの改善に役立てることが目指されています。この研究は、創造性評価の精度向上と信頼性向上に向けた一歩となるものです。将来的には、学校教育における創造性の評価がより重要な要素となることが期待されています。
<関連情報>
- https://news.uga.edu/ai-creativity-assessments/
- https://onlinelibrary.wiley.com/doi/10.1002/jocb.588
創造性評価に対する子どもの反応を信頼性高く判断することが難しいのはなぜか? What Makes Children’s Responses to Creativity Assessments Difficult to Judge Reliably?
Denis Dumas, Selcuk Acar, Kelly Berthiaume, Peter Organisciak, David Eby, Katalin Grajzel, Theadora Vlaamster, Michele Newman, Melanie Carrera
The Journal of Creative Behavior Published: 26 May 2023
DOI:https://doi.org/10.1002/jocb.588
ABSTRACT
Open-ended verbal creativity assessments are commonly administered in psychological research and in educational practice to elementary-aged children. Children’s responses are then typically rated by teams of judges who are trained to identify original ideas, hopefully with a degree of inter-rater agreement. Even in cases where the judges are reliable, some residual disagreement on the originality of the responses is inevitable. Here, we modeled the predictors of inter-rater disagreement in a large (i.e., 387 elementary school students and 10,449 individual item responses) dataset of children’s creativity assessment responses. Our five trained judges rated the responses with a high degree of consistency reliability (α = 0.844), but we undertook this study to predict the residual disagreement. We used an adaptive LASSO model to predict 72% of the variance in our judges’ residual disagreement and found that there were certain types of responses on which our judges tended to disagree more. The main effects in our model showed that responses that were less original, more elaborate, prompted by a Uses task, from younger children, or from male students, were all more difficult for the judges to rate reliably. Among the interaction effects, we found that our judges were also more likely to disagree on highly original responses from Gifted/Talented students, responses from Latinx students who were identified as English Language Learners, or responses from Asian students who took a lot of time on the task. Given that human judgments such as these are currently being used to train artificial intelligence systems to rate responses to creativity assessments, we believe understanding their nuances is important.
