ThreeDWorld (TDW): A High-Fidelity, Multi-Modal Platform for Interactive Physical Simulation

TitleThreeDWorld (TDW): A High-Fidelity, Multi-Modal Platform for Interactive Physical Simulation
Publication TypeCode
Year of Publication2020
AuthorsSchwartz, J, Alter, S, DiCarlo, JJ, McDermott, JH, Tenenbaum, JB, Yamins, DLK, Gutfreund, D, Gan, C, Traer, J, Kubilius, J, Schrimpf, M, Bhandwaldar, A, De Freitas, J, Mrowca, D, Lingelbach, M, Sano, M, Bear, D, Kim, K, Haber, N, Fan, C
Date Published07/2020

TDW is a 3D virtual world simulation platform, utilizing state-of-the-art video game engine technology

A TDW simulation consists of two components: a) the Build, a compiled executable running on the Unity3D Engine, which is responsible for image rendering, audio synthesis and physics simulations; and b) the Controller, an external Python interface to communicate with the build.

Researchers write Controllers that send commands to the Build, which executes those commands and returns a broad range of data types representing the state of the virtual world.

TDW provides researchers with:

  • A general, flexible design that does not impose constraints on the types of use-cases it can support, nor force any particular metaphor on the user.
  • Support for multiple modalities -- visual rendering with near-photoreal image quality, coupled with superior audio rendering fidelity.
  • A comprehensive, highly extensible and thoroughly documented command and control Python API.
  • Multiple paradigms for object interaction, capable of generating physically-realistic behavior.

TDW is being used on a daily basis in multiple labs, supporting research that sits at the nexus of neuroscience, cognitive science and artificial intelligence.

Find out more about ThreeDWorld on the project weobsite using the link below.

Custom 1

ThreeDWorld on Github -

Citation Key4633

Associated Module: