TeleopWM-Dataset

TeleopWM-Dataset is a large-scale collection of CARLA driving rollouts used for training and evaluating TeleopWM, a predictive latent world model for latency-resilient vision-based teleoperation.

The dataset contains synchronized RGB observations, vehicle controls, speed measurements, and metadata used to construct short-horizon future prediction tasks for both visual rollout prediction and future-action forecasting.

Related Resources

• Project page: https://bimilab.github.io/paper-TeleopWM/
• Model checkpoint: https://huggingface.co/bimilab/TeleopWM
• GitHub repository: https://github.com/bimilab/paper-TeleopWM
• YouTube demo: https://youtu.be/WeKqqZuwBl0

Overview

TeleopWM-Dataset was designed for research on:

• latency-resilient teleoperation
• predictive display
• future observation prediction
• future action prediction
• world models for driving
• autonomous and teleoperated vehicle systems

The dataset follows a CARLA/MILE-style rollout format and contains driving data collected across multiple CARLA towns and driving scenarios.

Dataset Structure

mile_action_diverse/
├── train/
│   ├── Town01/
│   ├── Town03/
│   └── Town04/
│
├── val/
│   └── Town02/
│
└── test/
    └── Town05/

The official TeleopWM experiments use:

This split was selected to evaluate generalization to previously unseen environments.

Data Contents

Each rollout contains:

• RGB camera images

• vehicle controls:

  • throttle
  • steering
  • brake

• vehicle speed

• route metadata

• rollout metadata stored in:

pd_dataframe.pkl

The TeleopWM pipeline constructs:

• 9 pas