Publications
SVG Decomposition for Enhancing Large Multimodal Models Visualization Comprehension
Under Review
Abstract: Large multimodal models (LMMs) are increasingly capable of interpreting visualizations, yet they continue to struggle with spatial reasoning. One proposed strategy is decomposition, which breaks down complex visualizations into structured components. In this work, we examine the efficacy of scalable vector graphics (SVGs) as a decomposition strategy for improving LMMs' performance on floor plans comprehension.
Floor plans serve as a valuable testbed because they combine geometry, topology, and semantics, and their reliable comprehension has real-world applications, such as accessibility for blind and low-vision individuals. We conducted an exploratory study with three LMMs (GPT-4o, Claude 3.7 Sonnet, and Llama 3.2 11B Vision Instruct) across 75 floor plans. Results show that combining SVG with raster input (SVG+PNG) improves performance on spatial understanding tasks but often hinders spatial reasoning, particularly in pathfinding. These findings highlight both the promise and limitations of decomposition as a strategy for advancing spatial visualization comprehension.
Read more
Agentic Accessibility: A New Paradigm for Graphics Accessibility for Blind and Low-Vision
Under Review
Abstract: We introduce agentic accessibility, a conceptual paradigm for graphics accessibility for blind and low-vision individuals (BLV). In this paradigm, an AI-enabled conversational agent analyzes a graphic, such as a chart, map, or floor plan, and engages in open-ended dialogue with BLV to support progressive exploration and information-seeking. We situate agentic accessibility within the broader landscape of graphics accessibility and highlight how it complements and extends existing approaches.
Building on this foundation, we propose an initial research agenda spanning the technological, human, and regulatory fronts of agentic accessibility. We then empirically ground the concept through a user study comparing how BLV and sighted participants engaged with a prototype conversational accessibility agent to seek descriptive and navigational information about a residential floor plan. Our findings reveal differences in information-seeking strategies and in participants’ ability to conversationally construct accurate mental representations of a test floor plan.
Read more
[C.1] From Text to Visuals: Using LLMs to Generate Math Diagrams with Vector Graphics
26th International Conference on Artificial Intelligence in Education, 2025
Jaewook Lee, Jeongah Lee, Wanyong Feng, Andrew Lan
Abstract: Advances in large language models (LLMs) offer new possibilities for enhancing math education by automating support for both teachers and students. While prior work has focused on generating math problems and high-quality distractors, the role of visualization in math learning remains under-explored.
Diagrams are essential for mathematical thinking and problem-solving, yet manually creating them is time-consuming and requires domain-specific expertise, limiting scalability. Recent research on using LLMs to generate Scalable Vector Graphics (SVG) presents a promising approach to automating diagram creation. Unlike pixel-based images, SVGs represent geometric figures using XML, allowing seamless scaling and adaptability. Educational platforms such as Khan Academy and IXL already use SVGs to display math problems and hints. In this paper, we explore the use of LLMs to generate math-related diagrams that accompany textual hints via intermediate SVG representations. We address three research questions: (1) how to automatically generate math diagrams in problem-solving hints and evaluate their quality, (2) whether SVG is an effective intermediate representation for math diagrams, and (3) what prompting strategies and formats are required for LLMs to generate accurate SVG-based diagrams. Our contributions include defining the task of automatically generating SVG-based diagrams for math hints, developing an LLM prompting-based pipeline, and identifying key strategies for improving diagram generation. Additionally, we introduce a Visual Question Answering-based evaluation setup and conduct ablation studies to assess different pipeline variations. By automating the math diagram creation, we aim to provide students and teachers with accurate, conceptually relevant visual aids that enhance problem-solving and learning experiences.
Read more
PDF
[J.1] IoT Idge-Cloud: An Internet-of-Things Edge-Empowered Cloud System for Device management in Smart Spaces
IEEE Network Magazine, 2023
Yoseop Joseph Ahn, Minje Kim, Jeongah Lee, Yiwen Shen, Jaehoon Paul Jeong
Abstract: This paper proposes an Internet-of-Things (IoT) Edge-Empowered Cloud System (called IoT Edge-Cloud) for the visual control of IoT devices in a user’s smartphone. This system uses the combination of existing technologies (e.g., DNSNA, SALA, SmartPDR, and PF-IPS), for DNS naming and indoor localization to support the visual control of IoT devices.
For the visual control of IoT devices, the IoT devices register their auto-generated DNS names and the corresponding IPv6 addresses with the IoT Edge-Cloud. Each DNS name embeds an IoT device’s type (e.g., fire sensor, television, refrigerator, or air conditioner) and its location information, which is obtained through an Indoor Positioning System (IPS). With the DNS name, a user’s smartphone can display each IoT device and its location in an indoor place (e.g., home, office, and classroom), so that the IoT device can be located in the smartphone’s screen. Through performance evaluation, this paper proposes a localization scheme for a smartphone with average localization error of 1.08 meters. Also, it proposes a localization scheme for IoT devices (especially, at the center area in a testbed) with average localization error of 1.11 meters.
Read more
PDF
