Neuroscientific techniques applied to audiovisual stimuli: cognitive-emotional analysis of Heineken advertisements during COVID-19

1. Introduction

The consumer's brain understands, feels or remembers an advertisement due to several cognitive-emotional factors (Spence, 2016). It is essential to investigate how advertising messages that influence the subject's behaviour are produced and how they are oriented towards decision-making regarding a brand, product or service (Cherubino et al., 2019).

Consumer neuroscience poses the challenge of understanding consumers through their unconscious processes. In other words, to explain their preferences, motivations and expectations in order to predict their behaviour and assess the success of advertising messages (Bercea, 2012). Neuroscientific techniques, combined with traditional social science research methods, can provide more satisfactory results than those obtained by traditional marketing procedures (Singh and Jain, 2018).

The study of perception, experience and recall are key to this neuromarketing research (Halls, 2002), bringing subjects' unconscious responses to the fore (Poels and Dewitte, 2006), and, when contrasted with conscious responses, completing a comprehensive analysis of advertising effects (Varan et al., 2015).

Neuroscience applied to the field of audiovisual advertising uses biometric techniques to study in real-time what processes are triggered in the consumer's brain and thus obtain objective, instantaneous and continuous data that allow, from such procedures, indicators or metrics, to draw relevant conclusions that measure, interpret and predict behaviours based on direct emotional responses (Winer, 2011).

Following this approach, advertising messages aim to achieve the highest possible levels of attention and emotion because these cognitive responses are strongly correlated with recall and decision-making (Le Blanc et al., 2014). That is, advertising stimuli with emotional content are easier to remember than purely informative ones (Page et al.,1990). Thus, advertising productions are always closely linked to current affairs and to their informative and emotional relationship with the subjects. Advertising with themes related to the SARS-CoV-2 coronavirus pandemic is a prime example of this.

The changes that COVID-19 has brought about in our everyday life have led to uncertainty and stress, thereby affecting mental health (Fiorillo and Gorwood, 2020; Holmes et al., 2020; Pfefferbaum and North, 2020). It has also led to emotional distress, social disorder, defensive responses and fear (Cullen et al., 2020), among other psychological disorders.

This research analyses the emotion and recall of a series of Heineken brand advertisements that have introduced COVID-19 as a recurrent theme during the different phases of the pandemic until today, when it remains a global health, economic and social catastrophe. Several neuro-marketing techniques and a recall test were used to analyse these audiovisual stimuli.

Neuroscience tools measure the effectiveness of advertising. In particular, eye tracking (ET) (Wedel and Pieters, 2008, 2014; Ramsøy et al., 2012; De Oliviera et al., 2015) and facial coding (FC) (Teixeira et al., 2012; Taggart et al., 2016) are used. Other techniques include functional magnetic resonance imaging (fMRI) (Bakalash and Riemer, 2013; Couwenber et al., 2017), experiments in virtual reality environments (Bigné et al., 2016) or galvanic skin response (GSR) (Tapia-Frade et al., 2017; Cuesta et al., 2018). Due to the nature of this research, both eye tracking and facial coding techniques have been used.

Eye tracking (ET) is used for the analysis of visual attention and, from a consumer behaviour perspective, seeks to associate visual attention with consumers' cognitive and emotional responses.

Vision, and in particular visual attention, plays a very important role in this type of experiment. Visual attention processing begins when the eyes receive light signals. This information received by the eyes is directed to the brain via specialised neurons called photo-receptors that convert the light signals into coded electrochemical signals (Zurawicki, 2010). According to Russo (1978), eye movements can be considered valid behavioural candidates for measuring visual attention and information acquisition because they are closely related to higher-order cognitive processes. Therefore, understanding and monitoring pupil dilation and other eye movement patterns are very prominent cues (De Oliveira et al., 2015).

According to researchers Hoffman and Subramaniam (1995), eye movements can be seen as an objective indicator of where a person's overt attention is focused and help filter visual information. Different ET systems are able to estimate the fixation point of an eye on a screen, eye movement in relation to the head, pupil size and dilation (attention and emotion identifier) or eyelid closure (drowsiness control), and can accurately determine where the user's attention is directed (Duchowski, 2003). Eye fixation typically ranges from approximately 200 milliseconds when reading text to 350 milliseconds when viewing a scene.

In consumer neuroscience, ET has been used to study brand equity (Brasel and Gips, 2008; Kotler and Keller, 2009; Martinez, 2011; Hurley et al., 2013; Chae and Lee, 2013); segmentation (Hawkins et al., 2009; Zurawicki, 2010; Venkatraman et al., 2012); product (Fenko et al., 2010; Fiszman et al., 2013; Ares et al., 2013; Khushaba et al., 2013); price (Bizer and Schindler, 2005; Grewal et al., 2011); promotion (Chandon et al., 2009; Cyr et al., 2010; Rowe and Burridge, 2012; Lee and Ahn, 2012; Bix et al., 2013); and social marketing (Grier and Bryant, 2005; Thomsen and Fulton, 2007; Baschnagel, 2013; Maynard et al., 2013).

Regarding the analysis of facial micro-expressions (FC), different computer modules can be used to assess advertising effectiveness using the Facial Action Coding System (FACS) or FaceReader-FEBE system (Lewinski et al., 2014), the GfK-EMO Scan software (Hamelin et al., 2017) and FACET and AFFDEX (Stöckli et al., 2018; Magdin et al., 2019; Kulke et al., 2020). The fundamentals, characteristics and key techniques of micro-expression analysis with AFFDEX are explained in Baños-González et al. (2021).

2. Materials and methods

This research's main objective is to detect what cognitive-emotional responses are elicited by Heineken beer advertisements related to COVID-19 and to assess whether the narrative construction of the productions achieves their goals in conveying their advertising messages in an audiovisual form.

In this sense, the selection of the advertisements that are going to serve as stimuli for the experiment is fundamental. Eight Heineken brand commercials produced in the period 2020-2021 were chosen, lasting between 0:40 and 1:10 minutes and representing the different phases of the pandemic (confinement, remote working, return to public spaces, etc.).

Based on this selection, a group of 15 experts (professors, researchers and advertising professionals) were asked to evaluate the 8 advertisements according to three criteria: narrative value (qualities of the story and audiovisual expression), advertising value (qualities as a persuasive message) and relationship with COVID-19 (connection with different themes of the pandemic).

For the neuroscience experiment with facial coding and eye tracking, the 4 highest-rated ads were chosen in the aggregate of the 3 categories. Those selected were: Connections, Back to the Bars, Home Gatherings and Smart Working.

This was followed by the experiment in the Brain Research Lab at the URJC. The study involved 30 subjects, 25 women and 5 men aged 20-25, all of them students of the Degree in Advertising and Public Relations at the Universidad Rey Juan Carlos. The intention to restrict it to this specific area was determined both by the format of the stimuli (adverts broadcast on YouTube) and by the search for participants who, although they are not yet professionals in the advertising medium, nevertheless have a solid knowledge of the audiovisual languages, narrative techniques and marketing communication strategies used in the adverts-stimuli, so that viewing the different pieces would be meaningful for them. In order to voluntarily participate in the study, all subjects signed a consent form covering all aspects of the experimental components, phases and protocols, including COVID-19 special measures, research objectives and data protection. This consent ensured the non-identification of participants and the non-dissemination of individual personal data.

In this study, the Tobii Pro X3-120 ET model was used to measure participants' visual attention. The analysis of facial micro-expressions was carried out using the AFFDEX programme, a computer module that records people's faces in front of stimuli and provides 7 indicators of the probability of the emotional response occurring in relation to Ekman's (1978) basic emotions, plus 3 additional indicators of the individual's emotional involvement: attention (based on the position of the head), engagement (the emotional response of commitment or involvement that triggers the stimulus) and valence (the positive or negative nature of their experience). The neuroscience analysis platform used was iMotions version 9 (https://imotions.com/platform/) and, in addition to the two ET and FC tools, a recall test was incorporated to contrast emotional and cognitive responses.

The recall test model used has been put into practice in different experiments (Hamelin et al., 2017; Baraybar et al., 2017), in which it has been shown to be a relevant technique for the cognitive analysis of advertising messages. Thus, declarative response techniques (questionnaire) complement uncontrolled response techniques (FC and ET).

3. Results

3.1. Analysis of facial micro-expressions

The analysis with the AFFDEX module yields the following significant results.

Firstly, the level of attention for all four advertisements is very high, exceeding a value of 97 out of 100 points for all four stimuli. In terms of the 7 emotions that iMotions differentiates, Joy stands out as a significantly predominant emotion in all four examples: Joy's scores for Connections (7.62), Smart Working (7.23), Back to the Bars (3.98) and Home Gathering (3.95) are far higher than any of the other emotions. In second place, surprise, with Connections (1.31) and Smart Working (1.21), standing out as the next predominant emotion.

Contempt, disgust, fear, sadness and anger, in decreasing order, offer much lower levels than the above. Back to the Bars is the ad with the highest level of Contempt (0.76) and Sadness (0.36, the same value as Home Gathering); Smart Working, in Disgust (0.62) and Anger (0.15) and Connections in Fear (0.52).

In that sense, Connections offers the highest values in three emotions (joy, surprise and fear) and the lowest in two others (contempt and disgust); Back to the Bars has the highest in two (contempt and sadness) and the lowest in two others (surprise and anger); Home Gathering the lowest in three (joy, surprise and anger) and Smart Working the highest in two (disgust and anger) and the lowest in two others (fear and sadness).

In the aggregate of emotions analysed, therefore, the four advertisements show similar results: joy and surprise (the two most positive emotions) are predominant, while the remaining five emotions are recorded to a much lesser extent.

In terms of the particular analysis of each advertisement, the following results are presented¹.

The Back to the Bars advert has a series of quite pronounced peaks of joyful emotion. The longest, at 0:14, reaches 14.69. It corresponds to the moment when a character observes something out of field that has not yet been seen (shot-counter-shot editing); two other strong moments of joy are at 0:10.5, when two characters "bump elbows" (11.10 Joy) and the second 0:34 (10.62), when another character gestures to another character to give way. It is noteworthy that this image and the one at 0:14 are quite similar, both in the narrative motif (character in the middle shot) and visual composition (centred, looking to the left, out of frame), as well as in their function in the montage (they are shots that precede counter-shots).

The rest of the emotions remain more stable throughout the 62 seconds of the commercial. The first 4.5 seconds of the advert, which correspond to contextualisation shots (exteriors, posters and the arrival of a character at the bar), present comparatively high levels of contempt (average = 3.46) which drop considerably thereafter (practically to 0) and which are not reached again except at specific moments such as 0:33.5 with a value of 3.43 (when two characters meet in a corridor); from the second 0:42 to 0:44 with values ranging between 3.42 and 3.57 (when the plots of blowing out the candles and that of the characters giving way to each other are resolved); or at 0:53.5/0:54, with values of 3.28 and 3.17 respectively (when the first sentence of the claim appears: “There is only one thing better than going out at night").

This ad had the highest average sadness (0.36) along with Home Gathering. Although this is a very low value, there is a moment at 0:30.5 when it peaks at 2.81. The image in question is when the character cannot touch the door to open it because he is afraid of being infected (the same character who will then pull up his mask).

Connections is, as noted above, the ad with the highest average level of joy (7.62) and surprise (1.31). In this sense, there are moments in the story (0:42) when the joy register reaches a value of 19.46. It corresponds to one of the most humorous and paradoxical scenes in the advert: a character walking the dog inside the house while making a video call. Two other moments that increase in intensity can be seen at 0:33.5 (15.52) and 0:52 (15.33): the images in which a character is lying on a bed making a video call with her mobile phone raised, and the moment in which another character is hiding behind a table in order to be able to make another call.

As for surprise, there are two highlights that exceed four points: 0:04.5-0:05 (from 3.88 to 4.12) and 0:65.5 (4.77). In the first case, this corresponds to the image in which the telephone suddenly falls, and the on-screen character can no longer be seen, only the ceiling, and instantly a character is holding a mobile phone upright; the second in which the Heineken logo is shown. This last surprise may be caused by the fact that it what is being advertised is not explicit until that very moment.

Finally, this commercial also registered the highest values, although much lower on average than the previous ones, concerning the emotion of fear. At 0:38.5 it peaks at 4.33. It corresponds to the image (seen from the point of view of a badly framed webcam) of an Asian man staring in surprise at an Asian woman with a facial beauty mask.

Home Gatherings shows a considerable number of peaks of joy which, without reaching the levels of the other ads (it is the commercial with the lowest average in this section), does stand out for its number and for its abrupt ups and downs, all of which are due to its narrative structure of parallel stories. It is also remarkable that these peaks of joy are higher at the beginning than at the end of the advert. Thus, the two highest peaks are at 0:01 (9.61) and 0:06 (12.49). The first corresponds to the moment when a character is about to press the button on an intercom with his elbow; the second corresponds to the moment when, for the first time, two bottles of Heineken are explicitly seen on the screen, and the spectator discovers that it is an advertisement for that brand. It is also significant that the other peaks in the succession of scenes, 0:13.5 (character hesitates which bottle of beer to take); 0:16.5 (a detail shot of two identical, indistinguishable bottles); 0:26 (a character puts four bottles together); 0:47.5 (a lone character holds a bottle); 0:50.5 (the same character places a headband on his beer) or 0: 53 (a character places one bottle adjacent to another), have to do with moments that refer to the main conflict of the advert: when putting bottles together it is not possible to know which one corresponds to each one and, from there, solutions must be posed to either distinguish them or keep them apart.

It is also remarkable that, despite maintaining low levels of disgust (0.56 on average), this advert has a comparatively sharp peak near the end (0:61) of 3.38. Once it seemed that the ad was over, showing the Heineken logo on a green background, a final story is added as a coda or epilogue: two characters are once again faced with the dilemma of determining which beer is theirs.

The Smart Working advertisement first of all presents very distinct moments of peaks of joy: 0:01 (17.24), a shot in which a split-screen video conference begins; 0:25 (16.85), a character is caught drinking; from 0:28 to 0: 30 (18.48 to 19.11), another is caught red-handed taking a drink and in the next shot one uses a bottle opener to open a Heineken in front of a crowded video conference; and 0:32.5 (16.34), in which a character looks at the camera and the first part of the claim statement is superimposed: “If it´s smart working”. Also significant are the abrupt drops in joy: for example, very sharp, down to a value of only 0.20 at 0:36.5, when a character starts to remove the bottle from his mouth while the second part of the claim is shown: “Make it smart drinking”.

At the same time, there is a comparatively high register of surprise at 0:0 (4.19), the video conference image at the beginning of the advert, and a sudden register of disgust at 0:33.5 (6.10): the character looking at the camera pouts as he continues the first sentence of the claim.

3.2. Eye tracking analysis

The eye-tracking analysis of the four advertisements provides significant results in terms of the exact point and direction of the subjects' gaze.

Firstly, the heat maps indicate that when human figures are present, the eye's attention is focused on them. Specifically, on the eyes (somewhat less so on the mouth, and only when the character speaks, laughs or drinks). If, in any shot, the human figure shares the frame with any of the visual manifestations of Heineken beer (bottle, beer tap, packaging, etc.), it can be seen that this remains subordinate to the presence of the characters, even if it is not perceived by the subjects.

The Heineken-object, on the other hand, captures the gaze mostly when one of these three situations occurs: it moves in the image, forms part of the action (Images 16-17); the characters look at it, direct their attention towards it (Images 18-19); or, when a new shot is introduced, the object appears in the same place where the most prominent point of the gaze of the previous shot was located (Images 20-25).

However, the time that the viewers' eyes remain on the Heineken beer (which is an inert object, even if it is intended to provide dynamism with the movement of the liquid, the foam, or some other technical-expressive feature of recording the product) is substantially less than the time devoted to the faces or expressions of the characters.

Similarly, when there is written text on the screen (e.g. the slogan), the eye focuses on the letters and not on the product, indicating a subordination of the visual elements to the written content on the screen.

Regarding the analysis of AFFDEX facial micro-expressions with respect to the results obtained with eye tracking, it can be seen that the most emotionally powerful moments detected through face coding indicated in the previous section correspond to images with very well-defined heat maps, that is, shots in which the gaze of the subjects is limited to very specific points in the frame. Thus, the highest peak of joy detected in Connections (0:43 - 19.46) shows a focus of the gaze on the two essential points of the brief visual gag: as seen, a character walks the dog inside the house while videocalling (Image 5).

Similarly, in the second 0:61 of Home Gatherings, with a peak disgust of 3.38 (in an ad that showed practically no increase in value in this section), this same focus of the gaze on the three fundamental elements can also be seen (Image 10).

Many similar moments could be pointed out, but in short, it can be said that there is a correspondence between the emotion perceived in the facial micro-expressions and the register of the gaze, in such a way that the shots that are more confusing in terms of visual composition, or that have more centres of attention on which to divert the gaze or very fast cuts in the shot, coincide with images that present less high emotional values than those where the aggregate gaze is located at very marked points.

In turn, the visual elements most followed by the eye (faces, direction of the characters' gaze, spoken or written word, product), are concentrated in the moments with the most outstanding narrative techniques (story progression, twists, construction of subplots, etc.) of the stories told.

However, there are narrative elements which depend on visual elements shown in the shot and which, because they are not perceived by the eye, are not appreciated on first viewing. The viewer would need to watch the ad a second time in order to decode the information offered. As an example, in Connections (0:23) no one looks at the crossed-out microphone icon that indicates that the character who is speaking cannot be heard. The viewer is provided with the audio information (either because the character is seen articulating words but not making sound, or because the other character indicates that they cannot hear them), but the direct visual information provided by the red button with the microphone off has been lost.

Finally, eye tracking shows that strong actions (eye-catching, humorous or exciting moments; characters' points of view and reactions; overprinting of text or iconic messages) will attract the viewers' gaze to them. In that sense, change and movement are two explicit vectors for capturing the viewer's attention.

Following this approach, it is noteworthy that the shots that include elements related to the COVID-19 pandemic (masks, gels, social distance greetings, strange behaviour) quickly attract the gaze of the subjects in the shots in which they are shown.

3.3. Recall test

After viewing all 4 Heineken ads, the 30 subjects took a suggested recall test regarding the emotions they had experienced with each ad. The questionnaires, using a Likert scale, determined which emotions and to what degree of intensity they were felt.

Of all the participants in the study, 7 out of 10 had had some contact with coronavirus. In this study, the participant was considered to have had some contact with the illness if the subject or a family member had suffered from it.

The emotion most frequently reported by participants with some form of contact with COVID-19 was joy, followed by surprise. In contrast, the emotions of anger, sadness and disgust were the least remembered.

As can be seen, Home Gatherings and Back to Bars are remembered as more cheerful and Smart Working (the only one with a surprise twist at the end) as the most surprising, as well as conveying, along with Back to Bars, a certain sense of sadness.

Among the COVID-19 non-contact subjects, the results are quite similar. The most prominent emotion was again joy, followed by surprise. In addition, the emotion of fear was less reported among subjects with no previous contact with coronavirus.

Therefore, it is observed in both cases that the emotional level data obtained with AFFDEX coincides with the emotional recall of the advertisements.

4. Discussion

The analysis of the four Heineken commercials using neuroscientific tools provides consistent and significant results in terms of the study of facial micro-expressions and eye-tracking. In addition to achieving a high level of attention, the predominant emotions are mostly joy and surprise, the two basic emotions that the brand wanted to highlight in the four stories. In other words, the creators manage to convey, informatively and emotionally, the desired message in order to influence consumer behaviour. According to the analysis of the AFFDEX module and eye tracking, the content has been captured by the subjects without any disconnections, distortions or misreadings.

Likewise, the analysis of the images presents interesting insights from the point of view of audiovisual creation, both in its narrative and aesthetic aspects, and opens up exciting lines of research to explore which audiovisual construction techniques (scenography, camera, sound, editing, etc.) provoke certain reactions and how to reproduce them. Three aspects stand out in this respect: in a shot/counter-shot structure, the shot with the subject looking at the object achieves a higher level of emotion than the shot with the object being looked at by the subject; the product-beer, in its various visual forms, captures the attention through various procedures, either by being looked at by a character, by being integrated into the action or by being located in the strong point of the previous shot's frame. If there is a correspondence between what is told and how it is told (e.g. by directing, sequencing the viewer's gaze within the margins of the frame, whether in simple or complex compositions with several centres of attention), a certain sensation is induced, a direct emotional reaction which, as can be seen in the post-viewing test, also affects the recall of the advertisements. The subject's response can translate into behaviour or decisions in favour of the brand, product or service.

Similarly, in relation to the way in which the stories were connected to COVID-19, it was found that the narrative and visual elements that refer to the pandemic were recognised and evaluated by the subjects. Introducing this type of socio-health component associated with a product or brand, as Heineken has done in this series of advertisements, allows us to establish links of attention and emotion with target audiences. By depicting various stages of the coronavirus (confinement, remote working, the return to some sort of normality, etc.), these advertisements connect with the viewer by appealing to their emotions and their own cognitive perception of the pandemic and the product being advertised.

Therefore, as explained in the methodology, the selection of students from the Degree in Advertising and Public Relations was motivated by the didactic, formative nature that guides this research: future creators of advertising stories will learn to analyse, interpret and apply narrative techniques supported by neuroscientific experiments to create more effective, more enjoyable and more satisfactory stories from the point of view of marketing communication. In this sense, the subjects of the experiment were the first to see that there are a series of tools that are being perfected day by day and that allow us to unravel which stories move us, which messages fail, which specific techniques work best and, in short, which ways exist to enhance the effects and expand the creative limits in the production of advertising content.

Authors' contribution

Mario Rajas Fernández: Conceptualisation, Formal analysis, Research, Methodology, Resources, Validation, Visualisation, Software, Original writing-drafting and Writing-revising and editing. Lucía Sutil Martín: Conceptualisation, Formal Analysis, Research, Methodology, Resources, Monitoring, Software and Drafting-revision and editing. Héctor Canorea Tiralaso: Research, Methodology, Resources, Software, Visualisation, Original writing-drafting, and Writing-revising and editing.

All authors have read and accepted the published version of the manuscript. Conflicts of interest: The authors declare that they have no conflict of interest.

Referencias

Notes

¹ As each ad was introduced with one second of blank image at the beginning, the ads are one second longer. Therefore, in the YouTube versions, for example, second 0:09 is second 0:10 in the following graph. For ease of reading and to be able to compare the analyses with the Heineken advertisements broadcast on YouTube, we subtract that extra second from the explanatory texts, but not from the graphs based on the AFFDEX data.

Author notes

* Professor of Audiovisual Script and Audiovisual Language and Technologies (Department of Audiovisual Communication and Advertising, Faculty of Communication Sciences, Universidad Rey Juan Carlos, Madrid), Spain.

** Associate Professor PhD (Department of Business Economics. Market Commercialisation Area. Universidad Rey Juan Carlos, Madrid), Spain.

*** Pre-doctoral scholarship holder at the Universidad Rey Juan Carlos (Department of Audiovisual Communication and Advertising, Faculty of Communication Sciences, Universidad Rey Juan Carlos, Madrid), Spain.

Additional information

To cite this article : Rajas Fernández, Mario; Sutil Martín, Lucia and Canorea Tiralaso, Hector (2022). Neuroscientific techniques applied to audiovisual stimuli: cognitive-emotional analysis of Heineken advertisements during COVID-19. ICONO 14 journal Scientific Journal of Communication and Emerging Technologies, 20(2). https://doi.org/10.7195/ri14.v20i2.1836

ICONO 14, Revista de comunicación y tecnologías emergentes

ISSN: 1697-8293

Vol. 20

Num. 2

Año. 2022

Neuroscientific techniques applied to audiovisual stimuli: cognitive-emotional analysis of Heineken advertisements during COVID-19

Mario Rajas Fernández ¹, Lucía Sutil Martín ², Héctor Canorea Tiralaso ¹