Movie affective characterization using physiological signals and content analysis

Table of Contents

Movie affective characterization

We propose an approach for affective representation of movie scenes based on the emotions that are actually felt by spectators. Such a representation can be used for characterizing the emotional content of video clips for e.g. affective video indexing and retrieval, neuromarketing studies, etc. A dataset of 64 different scenes from eight movies was shown to eight participants. While watching these clips, their physiological responses were recorded. The participants were also asked to self-assess their felt emotional arousal and valence for each scene. In addition, content-based audio- and video-based features were extracted from the movie scenes in order to characterize each one. Degrees of arousal and valence were estimated by a linear combination of features from physiological signals, as well as by a linear combination of content-based features. We showed that a significant correlation exists between arousal/valence provided by the spectator's self-assessments, and affective grades obtained automatically from either physiological responses or from audio-video features. This demonstrates the ability of using multimedia features and physiological responses to predict the expected affect of the user in response to the emotional video content.

The video clips of the dataset

To create the video dataset, we extracted video scenes from eight movies selected either according to similar studies, or from recent famous movies. The movies included four major genres: drama, horror, action, and comedy. Video clips used for this study are from the following: Saving Private Ryan (action), Kill Bill, Vol. 1 (action), Hotel Rwanda (drama), The Pianist (drama), Mr. Bean’s Holiday (comedy), Love Actually (comedy), The Ring, Japanese version (horror) and 28 Days Later (horror). The extracted scenes, eight for each movie, had durations of approximately one to two minutes each and contained an emotional event. The following dataset has been used in the research published in:

[2008, inproceedings] (BIB)
Soleymani, M., Chanel, G., Kierkels, J., & Pun, T. (2008). Affective Ranking of Movie Scenes Using Physiological Signals and Content Analysis. Paper presented at the 2nd ACM Workshop on the Many Faces of Multimedia Semantics, ACM MM08, Vacnouver, Canada.
[2008, inproceedings] (BIB)
Soleymani, M., Chanel, G., Kierkels, J., & Pun, T. (2008). Affective Characterization of Movie Scenes Based on Multimedia Content Analysis and User's Physiological Emotional Responses. Paper presented at the IEEE International Symposium on Multimedia, Berkeley, US.
The detailed timings of the used scenes are given in the following tables:

28 days later

The version of the movie we used starts by the “Fox searchlights pictures” Intro

Scene number Start time Stop time Description of the start of the scene
1 00:09:05 00:10:09 Jim (Cillian Murphy) walks in the deserted streets of London with plastic bag in his hand
2 00:13:39 00:15:15 Starts with a close shot on Jim's face in the church
3 00:42:33 00:44:55 The taxi starts ascending the trash dump in the tunnel
4 00:49:04 00:50:02 Inside the grocery store in the deserted gas station
5 01:08:29 01:09:24 Major Henry West (Christopher Eccleston) and Jim are walking in a corridor
6 01:34:58 01:35:46 Hannah (Megan Burns) is sitting in a red dress frightened in a room
7 01:36:09 01:37:58 A fight between the black soldier and a zombie
8 01:39:14 01:39:48 Jim opens the taxi's door facing Major Henry West

Hotel Rwanda

The version of the movie we used starts by the “METROPOLITAN FILMS” Intro

Scene number Start time Stop time Description of the start of the scene
1 00:07:09 00:07:50 General Bizimungu (Fana Mokoena) and Colonel Oliver(Nick Nolte) are talking in the hotel's garden (while tourists are in the pool in the background)
2 00:09:37 00:10:58 In the Paul's (Don Cheadle) house, their son run into the living room frightened
3 00:23:04 00:24:21 Discussion between Paul and the Rwandan officer and he askes for Paul's ID
4 00:51:04 00:54:33 French soldiers are checking tourists' passports
5 01:07:45 01:09:16 The hotel's van is passing by is passing by burning houses at night
6 01:11:41 01:12:25 The hotel's van is on the road in a foggy dawn
7 01:27:03 01:28:40 Rebells are dancing around the road waiting for the UN trucks
8 01:28:15 01:29:39 Rebells are hitting the refugees in the truck

Kill Bill VOL 1

The version of the movie we used starts by the “MIRAMAX” Intro

Scene number Start time Stop time Description of the start of the scene
1 00:05:46 00:06:46 Uma Thrman is fighting with Vernita Green (Vivica A. Fox)
2 00:58:50 01:01:00 The Japanese gangs are sitting around a black table
3 01:10:13 01:11:40 Japanese gangs are drinking in a room
4 01:15:02 01:17:07 Gogo Yubari (Chiaki Kuriyama) starts fighting with Uma Thurman
5 01:18:24 01:22:50 The fight scene of Uma Thurman and Japanese fighters in black suits
6 01:24:43 01:25:40 The fight scene of Uma Thurman and the Japanese bald fighter (Kenji Ohba)
7 01:28:48 01:30:53 The final fight scene between Uma Thurman and O-Ren Ishii(Lucy Lin)
8 01:02:10 01:03:35 Motorbikes are escorting a Mercedes in Tokyo streets

Love Actually

The version of the movie we used starts by the “UNIVERSAL” Intro

Scene number Start time Stop time Description of the start of the scene
1 00:12:19 00:13:57 Colin Frissell (Kriss Marchall) serves at the party
2 00:47:37 00:49:09 Aurelia (Lúcia Moniz) is bringing a cup of coffee for Jamie Bennett (Colin Firth) in the garden
3 01:18:28 01:20:33 The jewellery salesman (Rowan Atkinson) is rapping a necklace
4 01:23:45 01:27:04 Colin Frissell arrives at Milwaukee
5 01:43:19 01:44:57 The old lady opens the door and surprises by seeing the prime minister (Hugh Grant)
6 01:51:38 01:54:14 School's Christmas concert starts
7 01:58:15 02:05:00 Jamie Bennett arrived at Portugal
8 00:28:26 00:29:52 Daniel (Liam Neeson) and Sam (Thomas Sangster) are sitting on a bench

Mr. Bean's Holiday

The version of the movie we used starts by the “UNIVERSAL” Intro

Scene number Start time Stop time Description of the start of the scene
1 00:05:39 00:07:22 Mr. Bean (Rowan Atkinson) takes a taxi at the train station
2 00:10:41 00:12:59 Mr. Bean is being served in a French restaurant
3 00:31:52 00:33:50 Mr. Bean is trying to raise money by dancing and imitating singer's acts
4 00:37:33 00:39:19 Mr. Bean rides a bike on a road
5 00:40:37 00:42:35 Mr. Bean tries to hitchhike
6 00:45:45 00:47:15 Mr. Bean wakes up in a middle of the shooting of a commercial
7 01:08:04 01:09:02 Dressed as a woman tries to get into the theater with a fake ID
8 01:11:35 01:13:22 Mr. Bean is changing the projecting movie to his webcam videos

Ringu

The version of the movie we used starts by the “STUDIO CANAL” Intro

Scene number Start time Stop time Description of the start of the scene
1 00:05:32 00:08:15 School girls are frightened by hearing the ring tone
2 00:27:39 00:29:38 Reiko (Nanako Matsushima) watches the video alone in an empty room
3 00:59:39 01:01:52 Reiko sees the past in black and white
4 01:19:10 01:21:48 Reiko is descending into the well
5 01:25:24 01:27:54 Ryuji (Hiroyuki Sanada) is writing at home and he notices that the TV is on and showing the terrifying video
6 01:29:56 01:31:55 Reiko seats on the sofa in her house
7 01:12:05 01:14:48 Reiko and Ryuji are pushing the well's lead
8 00:48:20 01:49:42 Reiko is sleeping in her father's house

Saving private Ryan

The version of the movie we used starts by the “Paramount pictures” Intro

Scene number Start time Stop time Description of the start of the scene
1 00:04:29 00:06:08 Start scene of the approaching of boats with these words apear “June 6, 1944”
2 00:06:09 00:08:15 Landing on the Omaha beach
3 00:09:33 00:12:56 Combat scene on the beach
4 02:13:38 02:14:23 The sniper is praying when he is on a tower
5 00:18:51 00:21:38 The commander is looking into a mirror to see the source of the gunfire in a combat scene
6 02:26:05 02:27:21 The combat scene where Capt. John H. Miller (Tom Hanks) was shot
7 00:56:04 00:57:25 While they are looking for private ryan, by accident a wall collapses and they face a group of German soldiers on the other side of the destroyed wall
8 01:20:00 01:20:45 Group of soldiers are walking on a green field

The pianist

The version of the movie we used starts by the “BAC films” Intro

Scene number Start time Stop time Description of the start of the scene
1 00:00:24 00:02:09 Warsaw in 1939 (black and white shots)
2 00:21:10 00:22:34 Szpilman (Adrien Brody) is playing in a restaurant
3 00:24:28 00:25:54 Szpilman walks in the streets of Warsaw
4 00:32:57 00:34:11 A crazy man and children on the street
5 01:56:12 01:58:13 Szpilman (with long hair and beards) tries to open a can
6 00:44:34 00:47:07 Jewish families are waiting to be sent to concentration camps
7 00:58:01 01:59:27 Szpilman in a construction site
8 01:50:38 01:51:52 German soldiers are burning everything with flamethrower