Understanding and tokenizing abstract concepts in videos
Planning and integrating intermediate reasoning steps for managing long-range videos with numerous input
Trope in Movies (TiM) comprises (1) 684 movies, each annotated with per-shot keyframes, subtitles, and trope labels, and (2) 95 trope identification queries accompanied by their definitions.
TiM is designed as a testbed for exploring two critical yet previously overlooked video reasoning skills: (1) Abstract Perception: understanding and tokenizing abstract concepts in videos, and (2) Long-range Compositional Reasoning: planning and integrating intermediate reasoning steps for managing long-range videos with numerous input.
We selected 95 tropes from TiMoS datasets and there're four category of them: Character-Trait(CT), Role-Interaction(RI), Situation(ST) and Storyline(SL)
Trope | Category | Definition |
---|---|---|
Big Bad | Character Trait | The character who is the direct cause of all of the bad happenings in a story. |
Jerkass | Character Trait | A character who is so completely obnoxious that it is unbelievable anyone would willingly interact with them. |
Faux Affably Evil | Character Trait | A villain whose politeness is an act that only serves to enhance their evil. |
Smug Snake | Character Trait | A type of character (usually a villain) who tends to treat friends and enemies alike with equal disdain. |
Abusive Parents | Character Trait | This includes parents who are emotionally, verbally, physically, or mentally abusive, or who neglectfully allow their children to be abused \nby others if they don't abuse the child themselves |
Would Hurt a Child | Character Trait | People that do hurt children, sometimes even kill them. |
Action Girl | Character Trait | An Action Girl is a female badass who is tough and kicks butt. |
Reasonable Authority Figure | Character Trait | Authirity that is open to heros' inasane ideas. Often, they'll ask for proof and facts rather than follow baseless accusations, but even then, \nthey'll usually humor the heroes and go check out their theories. |
Papa Wolf | Character Trait | Paternal instinct can transform a Bumbling Dad into an Action Dad. If someone threatens his kids they will soon wish \nthey'd never come within a mile of them. |
Deadpan Snarker | Character Trait | A character prone to gnomic, sarcastic, sometimes bitter, occasionally whimsical asides. |
Determinator | Character Trait | A character \ good or evil, male or female, young or old \ who never gives up. Ever. No matter what. |
Only Sane Man | Character Trait | When there is a group of characters who are all just totally weird, either in general or in a particular scenario, the Only Sane Man is the only \none who, well, isn't. |
Anti-Hero | Character Trait | An antihero is a protagonist who has the opposite of most of the traditional attributes of a hero. |
Asshole Victim | Character Trait | When the victim is a bad guy. |
Jerk with a Heart of Gold | Character Trait | A person you would expect to be a big Jerkass has some redeeming qualities behind their tough demeanor. |
Even Evil Has Standards | Character Trait | Keep a villain safely on the "still sympathetic" side of the Moral Event Horizon; give him something that he simply will not do. |
Affably Evil | Character Trait | Villains that are not too bad. There is absolutely nothing separating them from being normal, polite people except for the fact that they want \nto take over the world. |
Too Dumb to Live | Character Trait | People who are so stupid that they simply do not deserve to survive. |
Butt-Monkey | Character Trait | The character who is always the butt of the demeaning joke or the "put them through hell" plotline. |
Ax-Crazy | Character Trait | An "ax-crazy" character is someone who is psychologically unstable and presents a clear and immediate danger to others. |
Adorkable | Character Trait | An adorkable character might be socially inept, shy or clumsy. However, rather than making them an outcast, these quirks give the character \nan endearing vulnerability. |
Berserk Button | Character Trait | Some characters always get enraged when given a certain, minor trigger. |
Ms. Fanservice | Character Trait | A female character who provides a significant amount of Fanservice (generally for males), ranging from simple revealing of cleavage, midriff, \nshoulders or legs, to shots of her curves, hips or derriere (even when covered). |
The Alcoholic | Character Trait | This person likes their liquor, and in large amounts. |
Disappeared Dad | Character Trait | The father of a character or characters is missing or absent. |
Would Hit a Girl | Character Trait | A guy (good or evil) has no problems with violence against women. |
Oh, Crap! | Role Interaction | The moment at which a character realizes that something really, REALLY bad is about to happen. |
Driven to Suicide | Role Interaction | Whatever the reason, a character may be driven to commit suicide. |
Adult Fear | Role Interaction | Things that mature, well-adjusted adults generally are concerned about, as opposed to supernatural fears. |
Not So Different | Role Interaction | When one character in a pair (who'd otherwise be different and/or unrelated) discovers that they actually have some things in common, more \nthan the other character would want to believe. |
Heroic BSoD | Role Interaction | A stunning revelation or horrible event affects a character or someone they care deeply about, leaving them shocked to the point of mentally\n shutting down for a while. |
Big \"NO!\" | Role Interaction | The moment when a character realizes that something awful has happened, or notices that something awful is about to happen, and screams \n"NOOOOOOOOOOOOOOO!" |
Eye Scream | Role Interaction | When the creator of a work takes advantage of our Primal Fear of having something utterly horrible happen to our eyes, usually involving the \ninvasion of the eye sockets by a foreign body. |
Gory Discretion Shot | Role Interaction | Blood or brains are seen splattering against a wall and the rest is left to the imagination. |
Impaled with Extreme Prejudice | Role Interaction | When someone gets stabbed in a really extreme way.. |
Off with His Head! | Role Interaction | When someone is going to get his head chopped off. |
Disney Villain Death | Role Interaction | Villains exit the film by falling off or out of things. |
Your Cheating Heart | Role Interaction | Cheating on your partner. |
\"The Reason You Suck\" Speech | Role Interaction | One character delivers a speech to another character on all the reasons why that other character sucks or is a horrible person. |
Tempting Fate | Role Interaction | Tempting Fate is when a character says something that dares the universe into making things miserable for them, and the universe takes them up \non the challenge. |
Disproportionate Retribution | Role Interaction | the situation where a person let the opponent know that if they so much as sneeze on someone you protect, it will cost them a limb. |
Badass Boast | Role Interaction | The character has got to be pretty impressive to be able to make this kind of boast convincingly, though sometimes a weaker character will \nbluff like this. |
Groin Attack | Role Interaction | Groin attack |
Roaring Rampage of Revenge | Role Interaction | The Roaring Rampage of Revenge differs from the standard Revenge plot in that our hero dispenses with the Machiavellian plotting that would \ndefine a classic revenge tale and goes straight for the bloodshed. |
Big Damn Heroes | Role Interaction | Any time the heroes/anti heroes get to save the day in a big, awesome manner. |
Heroic Sacrifice | Role Interaction | A character saves another/others from harm and is killed, crippled, or maimed as a result. |
Screw This, I'm Outta Here! | Role Interaction | Screw you guys, I'm going home. |
Kick the Dog | Role Interaction | When a character does something evil for no apparent gain, because the author wants to demonstrate that he's not a nice guy and shift \naudience sympathy away from him. |
Pet the Dog | Role Interaction | Show the nasty old crank petting a dog, and you show the audience, aw shucks, they're not that bad of a person after all. |
Villainous Breakdown | Role Interaction | The tendency of relatively calm and composed villains to lose their cool over the course of the story. |
Precision F-Strike | Role Interaction | When a character suddenly swears in an uncharacteristically strong manner, or when a swear is unexpectedly used in a work with mild language. |
Cluster F-Bomb | Role Interaction | Lots of swearing occur. |
Jerkass Has a Point | Role Interaction | When a jerkass is right. |
Idiot Ball | Role Interaction | A moment where a character's stupidity fuels an episode, or a small plot line. |
Batman Gambit | Role Interaction | A plan that revolves entirely around people doing exactly what you'd expect them to do. |
Police are Useless | Situation | Whenever someone in film or on TV reports a murder, or a monster, or a stalker or whatever, the police come as close to ignoring them \nas procedure will allow. |
The Dragon | Situation | There is almost always someone, person or monster, The Hero has to get past to get at the Big Bad. |
Cool Car | Situation | The hero's ride has to be just as unique and awesome as the hero is. Enter a Cool Car! |
Body Horror | Situation | This is any form of horror or squickiness involving body parts, parasitism, disfigurement, mutation, or unsettling bodily configuration. |
The Reveal | Situation | A character is revealed as another character's father, a god, or secret suitor or arch nemesis in disguise. |
Curb-Stomp Battle | Situation | A fight that's extremely one-sided, where one side just absolutely trashes the other with little-to-no effort |
Cassandra Truth | Situation | Sometimes people just won't believe you even though you are telling the truth, because the truth sounds insane. |
Blatant Lies | Situation | A glaringly obvious lie. |
Crapsack World | Situation | A Crapsack World is a horrible setting of the world where it's dark and hopeless |
Comically Missing the Point | Situation | A character completely misses a really obvious point for comic effect. |
Fanservice | Situation | The use of sex or sexualized situations to reward or entice viewers. |
Fan Disservice | Situation | Fan Disservice looks like regular garden variety Fanservice, except that the creators intentionally use the circumstances around it to make the \nscene outright creepy or disgusting. |
Brick Joke | Situation | When two seemingly unrelated jokes work together to create a comedic effect. |
Hypocritical Humor | Situation | A type of gag in which a character's actions or words contradict each other and this is Played for Laughs. |
Does This Remind You of Anything? | Situation | Any situation which is made, sometimes for comic effect, sometimes for dramatic effect, to look like another situation. |
Black Comedy | Situation | A sub-genre of comedy and satire where topics and events that are usually treated seriously are treated in a satirical manner, while still being \nportrayed as the negative events that they are. |
Irony | Situation | The intended meaning is an inversion of the plain meaning. |
Exact Words | Situation | When the exact wording of a rule, promise, prophecy, etc. matters more than the spirit of the wording, it's an Exact Words situation. |
Stealth Pun | Situation | The writers put in a joke (almost always a pun), but never make or put in a Punch Line or explicit statement, hiding it in the set up of the joke. |
Bittersweet Ending | Story Line | When victory came at a harsh price, when, for whatever reason, the heroes cannot fully enjoy the reward of their actions, when some \nirrevocable loss has happened during the course of the events, and nothing will ever be the same again. |
Karma Houdini | Story Line | The character has done a number of things that deserve a karmic comeuppance, most importantly things that caused harm to the innocent. But \nwhen the time comes for the hammer to fall, that's not what happens. At least, not on them. They don\t get what they deserve. |
Downer Ending | Story Line | A finale to a movie that ends things in a way that varies from sad to tragic. |
Laser-Guided Karma | Story Line | If you mistreat the people and things around you, karma will strike you down. |
Earn Your Happy Ending | Story Line | A cast of characters in a story go through a lot more hardship, anguish and grief than is really necessary. In the end, however, you see them get \ntheir happily-ever-after. |
Karmic Death | Story Line | When the villain dies in a manner that is completely their own fault. Or, at least, obviously not the hero's. |
Nice Job Breaking It, Hero! | Story Line | The hero has accomplished their goal, but it turns out that by the very act of success, they've unwittingly made things worse. |
My God, What Have I Done? | Story Line | Well-intentioned but misguided character (very likely a Tragic Hero, a Tragic Villain or Anti-Villain) comes to realize that his actions have\n caused incredible damage, physical pain, emotional pain, or even loss of life. |
What the Hell, Hero? | Story Line | When characters In-Universe call out one of the heroes for doing something clearly unheroic, if not outright heinous. |
Hope Spot | Story Line | The Protagonist is about to face utter defeat. Suddenly, he finds a glimmer of hope what have you only to be crushed cruelly by the bad guys \nreturning in force. |
Heel Face Turn | Story Line | When a bad guy turns good. |
Took a Level in Badass | Story Line | A charcter who is transformed into someone stronger and tougher that the viewer almost can't recognize him. |
Chekhov's Gun | Story Line | An insignificant object that later turns out to be important. |
Foreshadowing | Story Line | A clue or allusion embedded in the narrative that predicts some later event or revelation. |
Chekhov's Skill | Story Line | Training in any kind of skill, ability, or knowledge that will likely later come in handy. |
Chekhov's Gunman | Story Line | Any character who is innocuously and unimportantly introduced to the viewer, but who later proves to be important by the end of the episode. |
Red Herring | Story Line | A clue that leads in the wrong direction. |
Ironic Echo | Story Line | When a character repeats a comment they received in a different situation, giving it a flipped or surprising new meaning. |
Hoist by His Own Petard | Story Line | The villain's own weapon or malicious plan is the cause of their downfall and/or death. |
Meaningful Echo | Story Line | It occurs when characters, in their usual way, say things to one another, and repeat it later on to give extra meaning. |
Freudian Excuse | Story Line | The villain had an abusive and particularly violent childhood (such as Abusive Parents, being bullied by peers, etc.), making them insane and warping their perception on the universe |
We selected 684 movies from MovieNet, each movie is annotated with per-shot keyframes, subtitles (246 of them have), and trope labels.
[ { "video": "tt1637725", # IMDB ID "num_option": 2, "a0": "yes", "a1": "no", "trope": "Big Bad", # Trope name "category": "Character Trait", # Trope category "qid": "tt1637725_0", # {IMDB ID}_{Trope ID} "answer": 1, "question": "Is the trope '{Trope name}', which means '{Trope definition}', relevant to the movie?" } ]
subtitles.zip
contains 246 JSON files, each corresponding to subtitles in a movie in TiM{ "shot_{shot_idx}_img_{img_idx}": { "bboxes": [], # deprecated "subtitles": [ subtitle1, subtitle2, ... ] # subtitles within the shot } }
We provide some useful script to pre-process, access and manage the TiM dataset in our repository
Hung-Ting Su: htsu@cmlab.csie.ntu.edu.tw
Chun-Tong Chao: adnchao@cmlab.csie.ntu.edu.tw