LLM4Vis: Large Language Models for Information Visualization

Half-day tutorial at IEEE Vis Conference 2024.

Overview

This tutorial will provide an introduction to Large Language Models (LLMs) for interested researchers in the visualization (Vis) community. It will first motivate why LLM4Vis is an important area of research and how Large Language Models (LLMs) can be lever- aged to solve various NLP tasks for visualizations. We will delve into the basics of language models, covering model architectures, including the Transformer architecture, and discuss various train- ing methodologies, from pre-training to fine-tuning. We will then dive deeper into Large Language Models, elucidating their emergent abilities and practical applications in visualization tasks, including prompt engineering, instruction tuning, and model variations for processing text, tables, and images. In the final part, we will focus on applying LLMs for information visualization, covering an array of applications such as visual text analytics, natural language inter- faces, chart question answering, text generation, visual analytics, automatic visual story generation, and addressing issues of acces- sibility and inclusivity in visualization. We will conclude with an interactive discussion of future challenges for NLP+Vis applications. The audience will include researchers interested in applying NLP for visualizations as well as others who focus more generally on the intersection of AI and visualization.

Materials

Tutorial Overview

Part 1: Introduction [15 mins]

  • Why LLM + Vis?
  • An overview of LLM + Vis Research
  • An overview of the tutorial

Part 2: LLMs for Visualizations [60 mins]

  • Basics of language models
    • Language modeling
    • Model architectures
      • Transformer architecture
      • Encoder, decoder, encoder-decoder
    • Model training
      • Pre-training
      • Fine-tuning
  • Large language models (LLMs)
    • Scaling LMs to LLMs
    • Prompt engineering
      • In-context learning
      • Chain-of-thought prompts
      • Program-aided language models
      • ReAct: Reasoning + Action
    • Multi-agent systems
    • Multimodal LLMs

Coffee Break  

Part 3: LLM4Vis applications [50 mins]

  • LLM4Vis Design space
  • Benchmark development
  • Model development
  • Evaluation

Part 4: Challenges and research opportunities [15 mins]

  • Open research questions
  • Research opportunities

Slides

Part 1: Introduction

Part 2: LLMs for Visualizations

LLM4Vis applications, Future Challenges

Organizer

Enamul Hoque
Enamul Hoque

Associate Professor, York University

Previous Versions of the Tutorial

Related tutorials were presented at the IEEE Vis 2023 and EMNLP 2023