Lateralization of facial emotion processing and facial mimicry

ABSTRACT The two halves of the brain are believed to play different roles in emotional processing. In studies involving chimeric faces, emotional expressions in the left visual field are more strongly perceived as emotional than those in the right visual field. Notably, the role of facial mimicry has not been studied in relation to hemispheric lateralization. In the current study, which used a novel stimulus set of chimeric faces, we proposed and found that emotional intensity judgments replicate the left visual field bias for facial expressions of emotions. While a general facial mimicry effect to the chimeric faces occurred for the corrugator muscle, these mimicry effects were not related to the visual field bias. The results suggest that encoding the emotionality of another person’s facial expression might occur independent from the mere mimicry of the facial expression itself.


Introduction
Social interaction and our connections with other people are a vital part of human life. In order for these social interactions to work smoothly, it is crucial that we understand what our communication partner feels. One of the most important manners in which people express their feelings in social interactions is by facial expressions. Facial expressions thus play a crucial role in the understanding of the other. There is a fair amount of evidence showing that the facial expression of an emotional state can lead observers' corresponding facial muscles to be directly activated, often referred to as "facial mimicry" (e.g., Dimberg, Thunberg, & Grunedal, 2002). The automatic nature of facial mimicry suggests the existence of a process that supports the interpretation and encoding of emotions during social interactions with other people (e.g., Niedenthal, 2007).
An interesting finding in research on the encoding of emotional expressions concerns the observation that when people attend more to the left visual field, rather than the right visual field, emotional facial expressions are perceived as more intense. This bias of perceiving an expression seen in the left (vs. right) visual field as more emotional is interpreted as supporting evidence for the idea that the right-brain hemisphere is involved more strongly in emotion processing than the left-brain hemisphere (e.g., Bourne, 2010). Hemispheric processing is often studied by using the chimeric faces test, a behavioural test of lateralization in which an emotional facial expression is presented in either the left or right visual field.
The relation between right hemispheric processing of emotional expressions and perceptions of emotional intensity raises the intriguing question of whether this relation might be modulated by facial mimicking processes. The role of the facial muscles of the observer in relation to hemispheric lateralization has been partly in studies with patients with unilateral facial paralysis (Korb et al., 2016). While some tasks showed no difference between patients with a left vs. right facial paralysis, one difference that was found in these studies was that left-sided paralysis patients showed more errors for onset of happiness on left vs. the right visual field. Though the current study focuses on perceived emotionality of facial emotional expressions on the left vs. right visual field and not on accuracy in judging the onset of facial emotional expressions on the left vs. right visual field, the finding of this study does suggest that there might be an association between facial mimicry and lateralization of emotion processing. Surprisingly, the role of facial mimicry has not been studied in relation to hemispheric lateralization directly.
Accordingly, the present study serves two main goals. First of all, we aim at replicating the results of the chimeric faces test. Secondly, we aim to examine whether facial mimicry occurs for chimeric faces-e.g., faces showing an emotional expression only in one half of the face-something which has not been tested before.

Hemispheric lateralization of emotion processing
Theories concerning the hemispheric lateralization of emotion processing often consider the viewpoint that (1) all emotions are, by and large, processed in the right hemisphere-the right hemisphere hypothesis, or that (2) positive emotions are processed in the left hemisphere, and negative emotions in the right hemisphere-the valence hypothesis 1 (e.g., Bourne, 2010). Each of these viewpoints is partially supported empirically, suggesting that neither hypothesis has unambiguous evidence (cf. Fridlund, 1988). However, in general, the predominant evidence suggests that the right hemisphere plays a more important role in emotion processing than the left hemisphere (Murray et al., 2015). For example, right hemispheric processing of emotional facial expressions often leads to better recognition, discrimination, as well as a stronger perceived emotionality of these expressions compared to left hemispheric processing and has been reported for facial expressions of various emotions (e.g., Bourne, 2010). Research has also shown that left hemiface composites of faces tend to be rated as more similar to the full face as well as being perceived as more expressive than right hemiface composites (e.g., Sackeim, Gur, & Saucy, 1978). This relates to the expressiveness of the poser, and is in line with the right hemisphere hypothesis as it suggests that our left hemiface (which is primarily controlled by the right hemisphere) is more expressive than our right hemiface (e.g., Bourne, 2010). Moreover, deficiencies in right hemispheric areas have been associated with difficulties in recognition of emotional facial expressions, and with social and emotional functioning in general (e.g., Meletti et al., 2003;Murray et al., 2015). In line with this, the left visual field bias tends to be reduced or reversed for people with right brain damage (e.g., Kucharska-Pietura & David, 2003), further supporting the role of the right hemisphere in processing emotional facial expressions. The current's study's focus is not on lateralization in expressiveness of a poser but on lateralization of perceived emotionality and emotion processing in the observer.
A great deal of evidence for the lateralization of emotion processing is derived from the chimeric faces test. This test measures the possible bias in the observer relating to the perception of emotional expressions presented in the left vs. the right visual field (e.g., Bourne, 2010;Bourne & Gray, 2011;Levy, Heller, Banich, & Burton, 1983). In studies using the chimeric faces test, the classic and repeatedly reported finding is that expressions shown in the left visual field are perceived and judged as more emotive than expressions shown in the right visual field, in line with the right-hemisphere hypothesis. 2 Different versions of this test exist. For example, participants can be presented with two faces at a time, one above the other, with one face depicting an emotional expression in the left visual field while the other depicts it in the right visual field (e.g., Bourne & Vladeanu, 2011), after which participants are asked to choose which of the two faces they find more emotive. Another option is to present one chimeric face at a time, at fixation, with an emotional expression depicted in the part of the face that falls in either the left or right visual field. The participant is asked how emotive they find each face, and comparisons can be made across faces depicting the expression in the left vs. right visual field (e.g., Bourne & Gray, 2011).

Facial mimicry and emotion processing
People's facial muscles often automatically respond when processing emotional facial expressions. Seeing a negative emotional expression, for example, tends to increase activity of the frowning muscle-corrugator supercilii-and seeing a positive emotional expression tends to increase activity of the smiling muscle-zygomaticus major. This process of facial mimicry (e.g., Dimberg et al., 2002) is suggested to relate to the valence of the facial expression one sees (Hess & Fischer, 2013). Facial mimicry is one way in which bodily processes that are active when people experience a state can also become activated to a certain extent when people perceive and process a similar state in the environment (e.g., Winkielman, Niedenthal, Wielgosz, Eelen, & Kavanagh, 2015). Facial mimicry is thought to play a functional role, supporting our recognition, processing, understanding and encoding of emotional information (e.g., Niedenthal, 2007).

Lateralization and facial mimicry
While voluntarily induced movements of the face stem from the cortical motor strip-a pathway called the pyramidal tract-and are likely to be lateralized, emotionally induced facial movements are thought to arise from an older motor system-the extrapyramidal tract-(for a review, see: Rinn, 1984) and are thus unlikely to be lateralized. Evidence for this double dissociation shows that neurological damage can disrupt either one of these systems (Blair, 2003;Rinn, 1984). Furthermore, little asymmetry is reported in studies using spontaneous emotional expressions, with any present asymmetry divided equally over the two sides of the face (cf. Ekman, Hager, & Friesen, 1981;Gazzaniga & Smylie, 1990). Only one study, to our knowledge, reported facial mimicry to be somewhat stronger in the left than the right side of the perceiver's face (Dimberg & Petterson, 2000). Another study did not find differences based on the side of the muscles but did show a relationship between stronger facial mimicry and right-brain activity (Achaibou, Pourtois, Schwartz, & Vuilleumier, 2008), suggesting that the strength of facial mimicry and hemispheric emotion processing could be related. Taken together, it can be argued that right hemispheric processing of emotional expressions can be expected to be related to stronger facial mimicry, but facial mimicry itself will likely not differ between the left or right side of the observer's face.

Current study
While many effects reported in psychology have been difficult to replicate, the chimeric faces test seems to show a reliable and medium to large effect. Most of the previous tests of hemispheric processing of emotional expressions used chimeric faces that are based on the database developed by Ekman and Friesen (1976) or the Karolinska Directed Emotional Faces database (Lundqvist, Flykt, & Öhman, 1998). In the current study, we firstly aim to further generalize these effects by replicating the previous findings of hemispheric processing with a more recently developed stimulus set based on the Dutch Radboud Faces Database (Langner et al., 2010). Validation of this database revealed that the overall agreement rate between intended and chosen expression was around 11% higher than reported in a recent validation study of the Karolinska database (Langner et al., 2010).
Secondly, we aim to examine whether facial mimicry occurs for chimeric faces. Lastly, if facial mimicry occurs to chimeric faces, we will further examine whether an individual's facial muscles react differently to faces showing the emotional expression in the left vs. the right visual field. If such lateralization of emotion processing shows differential effects in facial mimicry, then this would allow for a test of the mediating role of facial muscle activity in the effects of hemispheric processing and perceptions of emotionality.
The relationship between lateralization of processing emotional facial expressions and emotional facial mimicry has not been studied directly. In the study reported here, this relationship is examined with the aid of the chimeric faces test of angry and happy facial expressions. The one-face chimeric faces test (see Bourne & Gray, 2011 for a similar procedure) was chosen in order to allow for informative facial EMG measurements per type of stimulus. Faces were presented at fixation. Participants were shown one chimeric face at a time-with the emotional expression depicted either in the left or the right visual field-and had to indicate the emotionality of the face 3 while both left and right facial muscle activity of the corrugator and zygomaticus were measured. Attempting to replicate previous findings with a new stimulus set, we first tested whether there would be stronger emotionality judgments when emotional expressions are shown in the left vs. the right visual field. Furthermore, we examined whether a general emotional facial mimicry effect would show despite the fact that the stimuli consist of chimeric faces. If facial mimicry occurred, we would examine if emotional facial mimicry would be stronger for left vs. right visual field facial emotional expressions.

Study overview
The study had a within participants' design with emotional expression (angry vs. happy) and emotional half (left vs. right visual field) of the stimulus as repeated measures.

Method
Participants A sample of 23 right-handed University students (12 female, M age = 24.74, SD age = 3.59) participated in this experiment. 4

Stimuli
Chimeric faces were created using images of four male and four female faces from the Dutch Radboud Faces Database (Langner et al., 2010). Chimeric faces were created by slightly blending the faces at the midline. This is in line with the approach of a recent study (Innes, Burt, Birch, & Hausmann, 2016) which showed that chimeric faces blended at the midline are equally effective as previous versions while providing the advantage of avoiding possibly inducing atypical emotion processing because of a visible midline. Each chimeric face was composed of an emotional (angry or happy) half face and a neutral half face (see Figure 1 for an example). We made use of both the original pictures and the mirrored picture. 5 The final stimulus set existed of 64 unique chimeric faces, differing in gender (4 male, 4 female), emotion (happy vs. angry), emotional half (left vs. right visual field), and version (original vs. mirrored). The images of faces had a resolution of 462 × 562 pixels and an absolute size of 11.3 × 15.0 cm, and were presented in grayscale on a grey background. 6

Procedure
Participants were told that the study involved measurement of facial EMG to assess muscle activity during exposure to visual stimuli on the computer screen. The EMG procedure followed the typical protocol of facial muscle activity assessment that had received approval from the ethics commission at Utrecht University. The study was conducted and written informed consent of each participant was obtained in compliance with the principles contained in the Declaration of Helsinki.
Electrodes to measure facial muscle activity were placed on the participant's face, after which they were seated in an individual cubicle in which the experiment was completed. Participants were told that they were to rate pictures of faces presented on the screen on a 9-point scale regarding how emotional they found each face. They were asked to trust their first impression and to not think too long about their rating. Mean ratings of emotionality were calculated per stimulus type.
The experimental task was presented in two blocks, each consisting of the same 64 trials, randomly presented. A trial started with a blank screen (2s), after which a fixation point appeared (1s), followed by the chimeric face (3s). After 3 s the rating scale appeared below the chimeric face and the scale remained on screen until participants rated the emotionality of the face.

Facial EMG
Facial muscle activity at the corrugator supercilii and zygomaticus major sites was measured using bipolar placements of Ag/AgCl miniature surface electrodes filled with electrode gel attached on the left and right side of the face. The skin was cleansed and prepared with alcohol prep pads and semi abrasive lotion. The electrodes were placed following the methods described by Fridlund and Cacioppo (1986), and all pairs were referenced to a forehead electrode placed near the midline. The raw EMG signal was measured with a BioNex Bio-Potential amplifier and stored with a sampling frequency of 1,000 Hz. Raw data were filtered with a 30-300 Hz band pass filter and a 50 Hz notch filter and then rectified. Facial muscle activity recorded during the last 500 milliseconds of each blank screen at the start of a trial was used as baseline. Difference scores were calculated using this baseline. Prior to statistical analysis, data were collapsed over trials with the same emotional expression shown in the same half face and averaged over steps of 200 ms for a total of 1,000 ms. 7 Equivalence testing in analyses Reported non-significant effects for this study were further explored by equivalence testing based on the confidence interval approach (see Lakens, 2016 8 ). This meant that if the range of the confidence interval of the effect size fell completely within the range of indifference-the equivalence range-the reported effect was considered not meaningful. If the range of the confidence interval of the effect size fell partly, but not completely, within the range of indifference, it was concluded that it was undetermined if there was or was not a meaningful effect. The range of indifference for this study was defined by a medium effect d z of .5 (range of -.5 until .5), and a medium effect h 2 p of .06 (range of .00 until .06). 9 Bayesian tests were also performed for non-significant effects in order to assess the weight of evidence in the data set for the null hypothesis. If the visual field bias would be different based on the emotional expression (happy vs. angry chimeric faces), this should show in an interaction between emotional expression and emotional half. This interaction was not close to 7 The first second of seeing the stimulus is chosen as it can be assumed that facial muscle activity during this time window is considered to be spontaneous, while after the first second more deliberate processes can occur (Häfner & Ijzerman, 2011). 8 In line with recommendations by Lakens (2014), 90% CI is reported for eta-squared. 9 As reported earlier, the data for this study was collected in 2012. At that time equivalence testing for effect sizes were not common practice yet in psychological research, therefore the range of indifference for the effect sizes was set after obtaining the data. The effect sizes for the equivalence bounds are based on Cohen (1988 [.00-.22]. Equivalence testing revealed that the range of this confidence interval lies partly, but not completely, within the zone of indifference defined by a medium (h 2 p of .06) effect size. This indicates that it is undetermined if there was or was not a difference in visual field bias based on the type of emotional expression being a happy or an angry one. Therefore, Bayesian paired samples t-tests were performed in order to assess the weight of evidence of the non-significant interaction effect (emotional expression x emotional half) provided by the set of data. Comparing emotionality ratings for chimeric faces showing anger in the left versus in the right visual field, showed that the data were 6.24 times more likely to reflect a null effect than to reflect a difference (BF 01 = 6.24). Comparing emotionality ratings for chimeric faces showing happiness in the left versus in the right visual field, showed that the data were 5.71 times more likely to reflect a null effect than to reflect a difference (BF 01 = 5.71).

Physiological data: facial muscle activity
Two separate repeated measures analyses were executed to examine the facial muscle activity data, one for the corrugator muscles (left and right), and one for the zygomaticus muscles (left and right) with the following repeated measures: emotional expression (happy vs. angry), emotional half (left vs. right visual field), block of presentation (first vs. second block), and time (5 steps of 200 ms).

Emotional facial mimicry: corrugator muscles
We first examined whether participants would show emotional facial mimicry to the chimeric faces, expecting stronger activation of the corrugator to angry compared to happy chimeric faces. The interaction between emotional expression and time was indeed significant, F(1.97, 41.96) = 14.97, p < .001, h 2 p = .40, 90% CI h 2 p [.20-.54]. 10 This interaction revealed increasing reactivity of the corrugator to the differential emotional expressions at later time points. Stronger corrugator activation to angry than to happy chimeric faces showed (see Figure 2), revealing that an emotional facial mimicry effect indeed occurred even for chimeric faces showing an emotional expression in only one half of the face. In line with expectations, no differences in emotional mimicry revealed based on the side of the corrugator muscle.
Because emotional facial mimicry occurred to the chimeric faces for the corrugator muscle, we continued analyses checking if the process of facial mimicry were related to hemispheric processing of emotional expressions. If so, then emotional facial mimicry should be stronger to emotional expressions shown in the left vs. right half face. This was not confirmed; the interaction between emotional half and emotional expression was not significant for the corrugator muscle, F(1,22) = 3.97, p = .06, h 2 p = .15, 90% CI h 2 p [.00-.36]. Equivalence testing by use of the confidence intervals revealed that the range of this confidence interval lies partly, but not completely, within the zone of indifference defined by a medium (h 2 p of .06) effect size. The fact that no interaction showed for the corrugator based on the emotional facial expression being shown in the left vs. right visual field could either reflect that (1) no such interaction existed and that lateralization of emotion processing and facial mimicry are in fact not related, or (2) that said interaction would be relatively small meaning we did not have enough power to detect its occurrence.
Bayesian paired samples t-tests were performed in order to assess the weight of evidence of the interaction effect (visual field of emotion x emotional expression) provided by the set of data. Comparing corrugator activity for chimeric faces showing happiness in the left versus in the right visual field, it showed that the data were 6.22 times more likely to reflect a null effect than to reflect a difference (BF 01 = 6.22). Comparing corrugator activity for chimeric faces showing anger in the left versus in the right visual field, revealed that the data were 1.87 times more likely to reflect a null effect than to reflect a difference (BF 01 = 1.87). Emotional facial mimicry: zygomaticus muscles We again first examined whether participants would show emotional facial mimicry to the chimeric faces, with stronger zygomaticus activity to happy compared to angry chimeric faces. This was not confirmed. No interaction between emotional expression and time emerged for the zygomaticus, F (4,88) = .18, p = .95, h 2 p = .01, 90% CI h 2 p [.00-.01]. Equivalence testing by use of the confidence intervals revealed that the entire range of this confidence interval is completely contained within the zone of indifference that we defined by a medium (h 2 p of .06) effect size, indicating that the zygomaticus muscle indeed did not show a meaningful facial mimicry effect of increasing reactivity to the differential emotional expressions at later time points. A Bayesian paired samples t-test comparing zygomaticus activity to happy and angry chimeric faces revealed that the data were indeed 3.46 times more likely to be observed under the null hypothesis (BF 01 = 3.46). In line with expectations, no differences in emotional mimicry revealed based on the side of the zygomaticus muscle.
While no emotional facial mimicry occurred to the chimeric faces for the zygomaticus muscle, we did check if there might be a differentiation in facial mimicry related to hemispheric processing of emotional expressions (see Figure 3). If so, then emotional facial mimicry should be stronger to emotional expressions shown in the left vs. right visual field. This was not confirmed; the interaction between emotional half and emotional expression was not significant for the zygomaticus muscle, F(1,22) = 1.02, p = .323, h 2 p = .04, 90% CI h 2 p [.00-.23]. Equivalence testing by use of the confidence intervals revealed that the range of this confidence interval lies partly, but not completely, within the zone of indifference defined by a medium (h 2 p of .06) effect size.
Bayesian paired samples t-tests were performed in order to assess the weight of evidence of the non-significant interaction effect (emotional expression x emotional half) provided by the set of data for the zygomaticus. Comparing zygomaticus activity for chimeric faces showing happiness in the left versus in the right visual field, showed that the data were 5.32 times more likely to reflect a null effect than to reflect a difference (BF 01 = 5.32). Comparing zygomaticus activity for chimeric faces showing anger in the left versus in the right visual field, revealed that the data were 4.63 times more likely to reflect a null effect than to reflect a difference (BF 01 = 4.63).

General discussion
The present study had two main goals: First, replicating the findings of the chimeric faces test by use of images created with a more recent facial stimulus set (Langner et al., 2010); and second, examining the role of facial muscle activity (i.e., facial mimicry) in these effects. In line with the idea that the right hemisphere is more involved in processing emotional information, we replicated previous findings of the chimeric faces test and found that participants indeed perceived emotional expressions presented in the left visual field-processed first in the right hemisphere-to be more emotional than those presented in the right visual field. This further generalizes the effects the chimeric faces test, showing that the newly developed stimulus set can be used to measure hemispheric processing of emotional facial expressions.
Moreover, a general emotional facial mimicry effect to the chimeric faces was found for the corrugator muscles, which were activated more in response to angry than to happy chimeric faces. This suggests that even for faces that only show an emotional expression in one half of the face, participants' frowning muscle activity responded in line with the emotional expression shown, showing stronger activation to angry than to happy chimeric faces. Unfortunately, with the current data, we were unable to determine if facial mimicry for the corrugator was due to the visual field in which the emotional expression was shown. The zygomaticus muscle did not show any emotional facial mimicry effect. Lastly, as expected, we observed no differences in emotional facial mimicry between the left and right side of the facial muscles.
While the corrugator revealed the expected emotional facial mimicry effect, the zygomaticus did not show any meaningful facial mimicry effect.
One possible reason for the absence of the zygomaticus activity effect concerns the notion that the faces in this study were not full emotional faces, only showing an emotional expression in one half of the face. It could thus be the case that the general emotional facial mimicry effect for the zygomaticus could be weaker than when people view full emotional facial expressions. Indeed, emotional stimuli that are less straightforward or intense have been found to induce weaker emotional facial mimicry effects, in particular for the zygomaticus (e.g., Larsen, Norris, & Cacioppo, 2003). Furthermore, it has been found that the zygomaticus shows increased activity to positively valenced stimuli only when these stimuli are strongly positive, while the corrugator shows differences in activity to emotional stimuli at various levels of intensity (Larsen et al., 2003). The absence of facial mimicry effects for the zygomaticus in the current study is thus in line with previously observed differences between the zygomaticus and the corrugator in sensitivity to emotional stimuli.
While the facial mimicry effect observed in the present study was found for the corrugator, the current data did not reveal if this mimicry effect was different for faces with the emotional expression shown in the right vs. left visual field. While this study is the first to examine the relationship between hemispheric processing of chimeric faces and facial mimicry, we do realize that leaving this undetermined leaves an open question. However, we hope that by providing information on this effect, future studies can provide further insight into the possible connection between facial mimicry and lateralization of emotion processing.
A limitation of the current study is that the facial expression stimuli were of static nature. Use of static stimuli keeps in line with the common chimeric faces test as used by Bourne (2010), hence providing the opportunity to first replicate the basic finding of visual field bias while simultaneously adding the measure of facial muscle activity. However, dynamic facial stimuli (as for example addressed in Carr, Korb, Niedenthal, & Winkielman, 2014) have the obvious advantage of being more representative of real-life situations. Future research could-in line with the study by Korb et al. (2016)-further extend the current study by employing dynamic facial stimuli and measures of facial muscle activity in order to provide more insight into the working of lateralization and facial mimicry.
The general gist of our findings suggests that the chimeric faces test is a reliable measure of hemispheric processing and our replication with a newly developed stimulus set further generalizes its findings. Moreover, our study shows that, for the corrugator, a reliable emotional facial mimicry effect was found, revealing that the corrugator responds sensitively to perceived facial hemi-expressions. This opens up possibilities for further investigation into the relationship between facial mimicry and hemispheric processing. The zygomaticus muscle, however, did not show a facial mimicry effect in the current study, further supporting the differences between the corrugator and zygomaticus muscle in sensitivity to intensity of emotional information.
In closing, we would like to stress that the present study provides a first test for the relationship between the perception of chimeric faces and facial muscle activity measures. In our view, the combined methods of hemispheric encoding of facial emotional expressions and the ability of displaying muscle activity in response to these facial expressions presents an interesting avenue for future scientific exploration.