Human-mediated AI writing assistance: A case study with Japanese English Language Learners. John Maurice Gayed *, Jeffrey S. Cross ** and Angelu Mari Oriola *** * Tokyo Institute of Technology gayed.j.aa@m.titech.ac.jp ** Tokyo Institute of Technology cross.j.aa@m.titech.ac.jp *** L.A. English Academy angelu.laenglish@gmail.com Abstract Increasing use of English as a Lingua Franca worldwide has brought much attention to tools that can assist English language learners (ELLs) in their journey to fluency. Much research has shown that ELLs who struggle with English sentence production rarely have sufficient cognitive resources available to work on higher level writing tasks such as organization and revision. The researchers developed the AI- based web application “AI-KAKU” to assist ELLs in reducing the cognitive barriers they face when producing written text in English. While there has been much research and discussion on Automated Writing Evaluation (AWE) technologies or older technologies such as spell check and grammar check, few studies have attempted to use AI-based tools as learning instruments outside assessments. In order to evaluate the potential impact of AI-KAKU on student writing, this study recruited adult ELLs in a counter- balanced experiment. Preliminary results indicate that this is a potentially useful tool for English language learners who need more structured assistance than what traditional word processors can provide. Keywords: CALL, AI in education, L2 writing, cognitive load, artificial intelligence 1. Introduction According to the British Council, it is estimated there are some 1.5 billion ELLs in the world. A common struggle for second language (L2) learners is the tip-of-the-tongue (TOT) state, a temporary mental state in language production where there is difficulty in retrieving an intended word. ELLs who are tasked with producing text in English often compose their ideas in their first language (L1) and then struggle mentally to translate those ideas into English while attempting to complete the writing task (Wolfersberger, 2003). Regardless of the approach, writing in a second language involves considerable low level cognitive stress such as dealing with TOT. This cognitive stress might hinder learners from focusing on higher level writing tasks such as organization and revision (Kellogg, 2008). When faced with a seemingly insurmountable task, ELLs might turn to less scrupulous techniques to complete their work, such as wholesale machine translation of their writing in L1 into the target language. To assist ELLs in the writing process, the researchers have developed an online writing application called AI-KAKU. The application has two distinct features. First, an AI-based word suggestion engine gives users word recommendations based on the user’s input. This is commonly seen in text predication applications on smart devices. The second feature is a reverse translation function. As users write in English, a simultaneous translation in the user’s L1 is displayed under their writing. This encourages the user to think in the L2 first while still giving validation while they are writing. AI advancements have led to more sophisticated intelligent writing assistants that offer synchronous feedback to the writer (such as Grammarly and Microsoft Editor) than traditional word processors. However, there has been little development into word processors that are aimed for ELLs’ usage and little research is done into their potential impact on the student’s writing proficiency. A directly observable measure of student writing ability is fluency, or the total words written. Another commonly used measure is Lexical Diversity (LD) or the range of different words used in a text. Texts with a lower range tend to use the same words repeatedly. LD is commonly used in second language research and LD indices have been found to be suggestive of writing quality, vocabulary knowledge and speaker competence (McCarthy & Jarvis, 2010). This research aims to address the following research questions: 1. How has AI-KAKU impacted the LD and fluency of student writing? 2. What was the users’ impression of the utility of AI- KAKU? 2. Methodology This case study enrolled ten Japanese adult students that are at the EIKEN 2, pre-2 level, equivalent to CEFR B1/A2, TOEFL iBT score of 45/32. Students were divided into two groups and a counter-balanced research design was used to conduct the experiment. The control condition using standard word processing software (Google Docs) while timed (thirty minute) and goal limited (three-hundred word) writing assignment is compared to a second writing condition of using AI-KAKU with the same parameters. The writing prompts were chosen from sample independent writing tasks from the TOEFL iBT exam of English, a commonly used test of English for students wishing to enter tertiary education in the United States. A mixed method approach where the researchers used automated text analysis to detect differences between the two writing conditions followed by quantitative and qualitative analysis of post-activity survey data was conducted. The researchers conducted text analysis of the participants’ writing with the Text Inspector, a web-based text analysis tool (Text Inspector, 2020) to analyze the