RESEARCH ARTICLE
Eye-tracking Study on Students’ Reading Process of Multimodal Texts: Focusing on Graphic Novels and Picturebooks
Seok-ju Yoon1, Kyeong-hwa Lee2*
1Hagnam Elementary School
2Korea National University of Education
Correspondence to leadread@hanmail.net
Brain, Digital, & Learning. Volume 15, Number 3, 309–330, September 2025. https://doi.org/10.31216/BDL.2025.15.3.1
Received on July 25, 2025, Revised on August 11, 2025, Accepted on August 17, 2025, Published on September 30, 2025.
Copyright © 2025 Institute of Brain based Education, Korea National University of Education This is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (https://creativecommons.org/licenses/by-nc/4.0/) which permits unrestricted non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited.
Abstract
The purpose of this study was to compare and analyze the gaze movement patterns of sixthgrade elementary school students while reading multimodal texts—graphic novels and picturebooks—in order to investigate their reading strategies and meaning-making processes. A crossover experimental design was employed in which the same participants read both types of texts. Quantitative and qualitative data were collected using an eye-tracking device, focusing on gaze path, heatmap, and areas of interest (AOI), and were supplemented by retrospective think-aloud interviews. The analysis of gaze paths revealed that during graphic novel reading, students followed a structured pattern in line with the sequential flow between speech balloons and images, indicating a text-centered information processing tendency. In contrast, during picturebook reading, students exhibited more fluid gaze patterns led by image exploration, highlighting the construction of meaning through the interaction between visual and verbal information. Heatmap analysis showed strong fixation on speech balloon texts and characters’ faces in graphic novels, whereas in picturebooks, fixations were concentrated on text and images marking narrative transitions. In AOI analysis, the text area in graphic novels showed the highest dwell time and number of fixations, while in picturebooks, significant gaze distribution was also observed in image areas related to the text. Interpreting these results through Serafini’s (2012) reader resource theory for successful multimodal text reading, readers of graphic novels primarily activated roles as “navigators” and “designers,” whereas readers of picturebooks more prominently exhibited the roles of “interpreters” and “navigators.” This study provides an empirical, process-oriented approach to multimodal text reading and offers meaningful implications for reading education and research in multimodal literacy by identifying genre-specific differences in information integration strategies.
Keywords
Multimodal texts, eye-tracking, graphic novels, picturebooks, gaze path, heatmap, AOI