A Preliminary Analysis of Students’ Help Requests with an LLM-powered Chatbot when Completing CS1 Assignments

Multiple recent studies have integrated large language models (LLMs) into diverse educational contexts, including CS1 classrooms. One common application is integrating a chatbot to serve as a teaching assistant. In this preliminary analysis, we explored four methods (correlation analysis, Latent Dirichlet Allocation, expert evaluation, LLM labeling, and evaluation) with multiple levels of data to analyze students’ help requests with a basic chat-based LLM tutor when completing CS1 assignments. This dataset contains 73 initial help-seeking conversation sessions with corresponding student self-reported survey answers. It also included 18 hallucinating responses from all the conversation sessions. Our results indicate that students with lower self-efficacy tended to create longer help requests, while students with higher self-efficacy tended to conduct more concise ones. Other than this, we found that learners shared more commonalities than differences when conducting help requests, including the length of turn-taking and the struggle to locate LLM hallucinations. As AI-based chatbots become prevalent in education settings, this preliminary analysis sheds light on what types of learner data can be collected, and what analytic approaches can be leveraged to unpack students’ help-seeking with these LLM-based learning systems.