LLM4Vis: Large Language Models for Information Visualization
Half-day tutorial at IEEE Vis Conference 2024.
Overview
This tutorial will provide an introduction to Large Language Models (LLMs) for interested researchers in the visualization (Vis) community. It will first motivate why LLM4Vis is an important area of research and how Large Language Models (LLMs) can be lever- aged to solve various NLP tasks for visualizations. We will delve into the basics of language models, covering model architectures, including the Transformer architecture, and discuss various train- ing methodologies, from pre-training to fine-tuning. We will then dive deeper into Large Language Models, elucidating their emergent abilities and practical applications in visualization tasks, including prompt engineering, instruction tuning, and model variations for processing text, tables, and images. In the final part, we will focus on applying LLMs for information visualization, covering an array of applications such as visual text analytics, natural language inter- faces, chart question answering, text generation, visual analytics, automatic visual story generation, and addressing issues of acces- sibility and inclusivity in visualization. We will conclude with an interactive discussion of future challenges for NLP+Vis applications. The audience will include researchers interested in applying NLP for visualizations as well as others who focus more generally on the intersection of AI and visualization.
Materials
Tutorial Overview
Part 1: Introduction [15 mins]
- Why LLM + Vis?
- An overview of LLM + Vis Research
- An overview of the tutorial
Part 2: LLMs for Visualizations [60 mins]
- Basics of language models
- Language modeling
- Model architectures
- Transformer architecture
- Encoder, decoder, encoder-decoder
- Model training
- Pre-training
- Fine-tuning
- Large language models (LLMs)
- Scaling LMs to LLMs
- Prompt engineering
- In-context learning
- Chain-of-thought prompts
- Program-aided language models
- ReAct: Reasoning + Action
- Multi-agent systems
- Multimodal LLMs
Coffee Break
Part 3: LLM4Vis applications [50 mins]
- LLM4Vis Design space
- Benchmark development
- Model development
- Evaluation
Part 4: Challenges and research opportunities [15 mins]
- Open research questions
- Research opportunities