To determine the impacts of using ChatGPT to assist English as a foreign language (EFL) English college majors in revising essays and the possibility of leading to higher scores and potentially causing unfairness. A prospective, double-blinded, paired-comparison study was conducted in Feb. 2023. A total of 44 students provided 44 original essays and 44 ChatGPT-assisted revised essays, which were rated by two independent graders in a randomized and crossover fashion to minimize grading bias. The original and revision scores were paired for before-after comparison. Eight control essays were also rated by both graders to ensure inter-rater reliability. This study used a rigorous experimental design to confirm that ChatGPT-assisted revised essays led to significantly higher scores for EFL college English majors. Significant improvements were observed in all four dimensions of writing quality assessment, with the largest effects observed in vocabulary, followed by grammar, organization, and content. ChatGPT-assisted revised essays shifted the score curve from a normal distribution to a skewed distribution towards higher grades, with the greatest increase in revision scores seen among students who had lower original scores. This disproportionate improvement raises concerns about fairness in evaluation. The findings suggest that ChatGPT is effective in providing timely feedback to EFL English majors in an affordable manner, but it also highlights the potential for unfairness in writing evaluation. We should note that ChatGPT-assisted revisions do not reveal learners’ writing competence. Therefore, new forms of writing performance assessment should be implemented in EFL composition classes in this AI era.
Relation:
Education and Information Technologies 29, p.22427-22445
