Abstract Perception

Understanding and tokenizing abstract concepts in videos

Long-range Compositional Reasoning

Planning and integrating intermediate reasoning steps for managing long-range videos with numerous input

Overview

Trope in Movies (TiM) comprises (1) 684 movies, each annotated with per-shot keyframes, subtitles, and trope labels, and (2) 95 trope identification queries accompanied by their definitions.

TiM is designed as a testbed for exploring two critical yet previously overlooked video reasoning skills: (1) Abstract Perception: understanding and tokenizing abstract concepts in videos, and (2) Long-range Compositional Reasoning: planning and integrating intermediate reasoning steps for managing long-range videos with numerous input.

Comparison of challenges between TiM and existing datasets

Statistics

Tropes

We selected 95 tropes from TiMoS datasets and there're four category of them: Character-Trait(CT), Role-Interaction(RI), Situation(ST) and Storyline(SL)

Movies

We selected 684 movies from MovieNet, each movie is annotated with per-shot keyframes, subtitles (246 of them have), and trope labels.

Download

  1. Annotation of QA
    • It contains three JSON files, corresponding to three experiment setups
    • Each JSON file can be loaded as a JSON object, the format is shown as follows:
    • [
          {
          "video": "tt1637725",          # IMDB ID
          "num_option": 2,
          "a0": "yes",
          "a1": "no",
          "trope": "Big Bad",            # Trope name
          "category": "Character Trait", # Trope category
          "qid": "tt1637725_0",          # {IMDB ID}_{Trope ID}
          "answer": 1,
          "question": "Is the trope '{Trope name}', which means '{Trope definition}', relevant to the movie?"
          }
      ]
  2. MovieNet per-shot Keyframes
  3. Preprocessed Subtitles
    • subtitles.zip contains 246 JSON files, each corresponding to subtitles in a movie in TiM
    • Each JSON file can be loaded as a JSON object, the format is shown as follows.
    • Note that the key is corresponding to the image filename under MovieNet per-shot Keyframes
    • {
          "shot_{shot_idx}_img_{img_idx}": {
              "bboxes": [], # deprecated
              "subtitles": [
                  subtitle1,
                  subtitle2,
                  ...
              ] # subtitles within the shot
          }   
      }

Github

We provide some useful script to pre-process, access and manage the TiM dataset in our repository

Contact Us

Hung-Ting Su: htsu@cmlab.csie.ntu.edu.tw

Chun-Tong Chao: adnchao@cmlab.csie.ntu.edu.tw