Color Perception in Virtual Characters and Communicating Emotion Using Color, Pose, and Face of Static Virtual Character
It is widely accepted that people are willing to apply social characteristics to virtual characters. Having environments that are able to adjust in harmony with the user has been sought for many years. Human beings have been shown to apply social characteristics to virtual characters. The study of environments that have the ability to recognize the emotions of the user and react in consequence has drawn a lot of interest from researchers in the field of affective computing. Recent development support the increase use of naturalistic virtual characters to allow for a better understanding of how emotions can be communicated while maintaining experimental control. While virtual characters are suitable for the study of human cognition, they cannot be equated to human emotions.
The recognition of human emotions is one of the interesting domains in academic literature and a number of top-ranking corporations have announced their plans to integrate these technologies in their products. This development is inseparably linked to the development of new input devices that would facilitate the process of emotion recognition. Some solutions focus on audio-visual channels with an aim of recognizing emotions expressed by the user through their facial expressions, gestures, and body posture whereas other focus on processing of physiological signals including temperature, rate of heartbeat and neurological signs (Szwoch, 2015).
It has been identified as one of the most active research fields in human behavior studies and studies in this area have profound importance to many fields, such as education, entertainment and healthcare. It focuses on how to automatically detect, interpret, and process human emotions by assessing the sensory data that is available (Szwoch, 2015). In addition, affect-aware applications have a functionality to react to user emotions. The six basic emotions of human beings are: happiness, sadness, anger, disgust, fear, and surprise (McDonnell, Jorg, Hodgins, Newell, & O’Sullivan, 2017).
The design of affect-aware games is not an easy task and it comes with the responsibility of making several important decisions. Even though people are generally able to express a broad range of emotions, not all of these emotions can be observed when playing games. The expressed emotions are affected by various factors including player’s gender, experience, type of game, among others. Besides, emotional reaction in a specific situation is highly dependent on individual personality, current mood as well as other unpredictable external factors.
The role of affective computing in a game varies depending on many factors including game genre, target audience, type of gameplay, or the game objectives. Nonetheless, we can define a number of roles of affective computing in game design. One of the development propensities of modern games is to avoid frustrating the players. This tendency is manifested in various aspects of the game design such as careful design of interface, avoiding the design of impossible levels, and introducing gameplay facilities (Szwoch, 2015). Another significant role of affective computing is keeping the game interesting and averting any form of boredom.
Affective gaming has embraced emotional evaluation to personalize feedback during interactive gameplay but as a result, extracts complex emotional states into simple inputs. In their research on “Playing with emotions: sentiment design for public space” Sun, Matthews, Gay, Morris, & Carmean (2014) focused on games in public spaces to investigate how people’s emotions might affect each other. They test four games namely Escalating Emotions, Mood Match, Make a Face, and Clairsentience and the study’s result show that image-based are a rich aspect for these kinds of games. Further, the study outlines the significance of ambiguity and conflicting ideas amongst players to facilitate sharing of personal stories (Sun, Matthews, Gay, Morris, & Carmean, 2014)
Affective learning is an important element of education that can be supported through games (Dormann & Biddle, 2008). The process of using games to support affective learning is complex since player emotions need to be identified and interpreted, and an emotional experience must be created to motivate players and deepen learning. Furthermore, there is a need to evaluate affective representations and mechanisms that games support. Dormann & Biddle (2008) present the different perceptions on affective learning, and then focuses on the socio-emotional aspect of the affective response. They identify the key principles and create a repertory of affective learning game patterns, as well as utilizing techniques to enhance learning and contextualize gameplay.
McDonnell, Jorg, McHugh, Newell & O’Sullivan (2009) state that “Virtual characters in animated movies and games can be very expressive and have the ability to convey complex emotions” (p.1)Virtual characters play a significant role in filling virtual worlds, whether they are automatically controlled by machines or act as channels for human expressions in the form of avatars (Palmber, Peters, & Qureshi, 2017). Developments in modern game design have made it more economical to assemble virtual characters from various sources of motions and appearances. However, the creation of virtual characters from varying sources of motions and appearances may have unintended effects in relation to how individuals perceive their expressions.
Color selection in game design
Researchers in the field of color believe that color influences behavior and cognition through learned connections. When individuals repeatedly come across situations where different colors are accompanied by specific concepts or experiences, they create specific connections to colors. Blue and red colors have been shown to possess different connections within the cognitive domain. For example, in their study Mehta and Zhu (2009) propose that a different connection related to red versus blue color can induce alternative motivations.
A number of studies in this area of study have asked the question of how colors influence the emotions of people who are exposed to them and how, for example, a picture can reveal the emotions expressed by the person in it (Lambrant, Luro & Sundstedt, 2015; Kim and Suk, 2016; Mauderer, Flatla, & Nacenta, 2016). In their study of key color generation for affective multimedia production, Kim and Suk (2016) report that key colors can be used to construct an affective and aesthetic harmony with visual content. The study reveals that given an image and an affective term, a key color could be identified by combining a dominant hue of the image in a distinctive tone associated with the affective word (Kim & Suk, 2016).
The vast majority of empirical research in the recent years has supported the notion that communication is an attention-directing mechanism in the cognitive processing of color in both offline similarity judgments and online perceptual discrimination. Bartram, Patra & Stone (2017) suggest that communicating the right affect or emotion is essential in development of engaging visual communications. The results of their study on affective color in visualization reveals how color and palette properties can be manipulated to accomplish affective expressiveness even in the small series of colors utilized for the encoding of data in information visualization (Bartram, Patra & Stone, 2017).
In gameplay, colors can be used to control the concentration of the player, for instance with the use of contrasting colors. The Portal game developed by Valve presents a good example whereby the game uses blue and bright orange portals in their low saturated environments showing the user that the portals are objects to interact with (Lambrant, Luro & Sundstedt, 2015). Colors can be remembered and associated with certain events. Therefore, game designers can use colors to elicit a certain feeling in the player. One of the important tasks in game design is the selection of color schemes. Principles of color theory are essential to game designers because they help them to chooses and know the appropriate and preferred colors. In their article on avatar preference selection in game design based on color theory, Lambrant, Luro & Sundstedt (2015) carry out perceptual experiments that assess some basic principles of color theory applied to game objects to investigate if a specific combination is preferred. The result of the study showed that particular color harmonies were usually preferred over others illustrating the importance of considering these aspects in game design. Lambrant, Luro & Sundstedt, (2015) found out that that color harmonies with a base in green were less popular when compared to blue and red harmonies.
The influence of color on cognitive performances has been an area of interest for many stakeholders in the field of game design. Kao and Harell (2016) posit that the color red hinders motivation, performance, and affect in a variety of contexts that involve cognitively demanding tasks. They explain that teams wearing read have a tendency of impairing the performance of their opponents and this is also present in online gaming (Kao & Harrell, 2016). This effect is often attributed to the sub conscious section of the brain and this impact is powerfully manifested in other primates such as the Rhesus Macaques. The study showed a correlation between color red and a decrease in competence, flow and immersion. In another study, Mehta and Zhu (2009) conducted a study to explore influence of color on cognitive task performances and the results showed that color red primarily induces an avoidance motivation while blue enhanced performance on creative task.
According to Birk, Lacovide, Johnson, & Mandryk (2015) overlaid reference components need to be adequately visible to efficiently related to the underlying information, but not too prominent that they muddle the presentation. Their study focused in the works of designers who utilize transparency to incorporate overlaid grids with their underlying imagery by comparing black grids to blue and red ones on diverse types of images. The study results show minimal differences between boundaries set for black and red grids. However, the boundaries for blue grids were set consistently more opaque (Birk, Lacovide, Johnson, & Mandryk, 2015).
The use of real-time eye tracking, gaze-contingent displays can alter their content to enable the representation of depth or enhance the rendering performance. In their article, Mauderer, Flatla, & Nacenta, (2016) present two experiments in color matching and sorting. The two experiments manipulated object colors and peripheral background to influence the color perception of the user. The findings of their study showed that gaze-contingent can be used in conjunction with contrast to affect the appearance of color. They suggest that gaze-contingent color has the potential to expand the perceived color scope of current display technology thereby making it possible for individuals to discriminate color with better accuracy.
A large section in the male population is affected by color vision impairment and they experience a host of difficulties when distinguishing colors (Jenny & Kelso, 2007). In their article “Color Design for the Color Vision Impaired” they discuss some of the prevalent forms of color vision impairment. Further, they introduce a new software tool referred to as Color Oracle which helps designers in the verification of color schemes. Jenny and Kelso (2007) outline some of the techniques that cartographers can use to generate maps that are easy to read for individuals with color vision impairments as well as those with normal color vision. In another study, Nguyen, Do, Chia, Wang & Duh (2014) evaluate “DoDo’s Catching Adventure”, a new color vision deficient screening test used among the children. The game has embedded game elements into a color vision screening test and it is currently being used to enable the early detection of color blindness among children. Both the Color Oracle software tool and the DoDo’s Catching Adventure test have proved to be a beneficial tool in education, research, and healthcare by aiding in the understanding individual needs and overcoming hardships associated with color vision impairment (Nguyen, Do, Chia, Wang & Duh, 2014).
The lighting design used in computer graphics is borrowed from cinematography and many designers utilize conventional wisdom to convey appeal, emotion and drama using lighting (Wisessing, Dingliana, & McDonnell, 2016). Most of the knowledge used in the selection of color in game design is gained through experience. However, as Szwoch (2015) states, “individual development of affect-aware video games generates individual solutions which are not compatible with each other” (p.1). There is a need to come up with a universally accepted solution that can be employed by cartographers from around the world in the selection of colors during game design (Szwoch, 2015).
Understanding Human Facial Expressions Emotions
The development of complex computer graphics is on the rise. Consequently, there is an incessant expansion of the variety and authenticity of virtual characters used in games and movies. It is widely accepted in the field of social psychology that the dimensions of human facial features can directly influence the perception of their personality. Some of the traits directly associated with facial features include dominance, trustworthiness, and aggressiveness. If this correlation holds true for virtual faces, this could make valuable contribution to game design principles. Fesrtl, Kokkinara & McDonnell (2016) carried out an exploratory study to gain insight into how facial features affected the perceived personality. This study findings show that there was not correlation between the perceptions of real human faces and abstract faces, and in some instances it was the complete opposite. These results are significant and they enhance our understanding of the perception of virtual faces and they can be used in the creation of principles for portraying personality with minimal facial cues (Ferstl, Kokkinara & McDonnell, 2016)
Virtual characters are expected to fulfill a range of social roles across a number of domains (Hyde, Carter, Kiesler, & Hodgins, 2016). To achieve success in game design, these characters must have the ability to exhibit a variety of personalities. Currently, designers develop characters with appropriate personalities by utilizing their artistic expertise and intuition. According to Hyde, Carter, Kiesler, & Hodgins (2016) there is a need to provide evidence-based principles for the creation of social characters. In their article, they provide an account two experiments to display how damped and exaggerated face motion impacted impressions of cartoon and more realistic animated virtual characters. The research findings showed that facial motion magnitude influence the social traits of warmth, extroversion, and competence, which are applicable in therapy, entertainment, and education. These results provide a starting point for game designers in their quest for fine-tuning facial motion to control perception of the personalities of virtual characters Hyde, Carter, Kiesler, & Hodgins, 2016)
In another study by Reategui and Fruet (2006) they investigate the impact of different virtual characters in product recommendations. This is done through an experiment to evaluate the effect that unique interface characters could have on the user’s perception of their interaction with an e-commerce website in making personalized recommendations. The characters used the same language and strategy but they had different features including being a known or unknown person, representation by an anthropomorphic figure or non-representation. The experiment involved 180 students who completed purchases in the website and the virtual characters provided tips and made suggestions based on the user’s perceived interests. In relation to the ease of interaction, students who used the website with a virtual character felt that this interactive experience was more straightforward. This is attributed to the uniqueness of the virtual characters showing that facial features can stimulate the users and elicit positive experiences during the interaction. These results confirm the conclusions of past research in this area of study (Retegui & Fruet, 2006).
Fertl, Kokkinara & McDonnell (2017) argue that computer graphic design has been mostly led by the preferences of artists as opposed to perceptual studies. Fertl, Kokkinara & McDonnell (2017) carried out a study to investigate the effect of using effective non-player character design on gameplay. The study focused on abstract virtual characters with limited facial features. The study findings showed that the appearance of the character interacting with the subject moderated aggressive responses toward a non-present person. According to Ochs and Pelachaud (2012), a smile may convey a range of communicative intentions based on subtle characteristics of the facial appearance. They argue that during interactions, the appearance of a smile influences the observer’s perception of the social bearing of the orator and the content of the interactive process. In their study, they explored the perception of smiling in virtual character by displaying varying types of smiles when interacting with the user. Ochs and Pelachaud (2012) propose the use of a model to automatically detect the user’s potential perception of the virtual character’s social bearing based on its gender and smiling behavior.
Developments in modern game design have made it more economical to assemble virtual characters from various sources of motions and appearances. However, the creation of virtual characters from varying sources of motions and appearances may have unintended effects in relation to how individuals perceive their expressions. Palmber, Peters, and Qureshi (2017) present a study exploring the impact of facial expressions and full body motions from varying sources on the perception of emotional expressions (positive, neutral or negative) in small groups of virtual characters. The study found out a huge influence of the valence of facial expressions on the perceptions of emotions in the group (Palmber, Peters, & Qureshi, 2017).
Understanding Body Motion Perception in Virtual Characters
One of the most popular and widely used techniques for animating virtual characters is motion capture (Normoyle, Liu, Kapadia, Badler, Jorg, 2013). All practical applications of motion capture depend on methods used in motion editing to increase the flexibility and reusability of capture motions. Therefore, it is important to understand the perceptual consequences of motion editing because humans have shown proficiency in detecting and understanding minor details in human motion. Normoyle et al., (2013) conducted a study to enhance the comprehension of how motion editing may impact the emotional content of a capture performance, with special interests in changes in dynamics and posture (Normoyle, et al., 2013).
The study showed that changes in dynamics and posture are viewed as significant perceptual indicators of human emotions. Some of the key elements that should be considered in motion editing of virtual are angles, velocities, recognition rates and perceived intensities of a wide-ranging series of full body motion clips. In their study, Normoyle et al., (2013) analyze these properties and perceptions across a range of motion clips representing six emotions namely; happiness, anger, disgust, fear, surprise and sadness. The study findings make a significant addition to literature on motion editing as they found that emotions are mostly conveyed through the upper part of the body. Further, their study revealed that the perceived intensity of a motion can be minimized by blending with a neutral position and that posture changes can modify the perceived emotion (Normoyle, Liu, Kapadia, Badler, Jorg, 2013).
In another study by McDonnell, Jorg, McHugh, Newell & O’Sullivan (2008), they created virtual and real replicas of actors displaying different emotions to analyze the emotional content of motions depicted by different characters. In this study, they evaluate the emotions of happiness, anger, disgust, fear, surprise and sadness. The research incorporated a video of the real actor and his actions were applied to varying virtual body shapes. McDonnell et al., (2008) argue that the perception of complex emotional actions are highly robust and to a large extent independent of the virtual character’s body. “When realistic human body motion is used, it is the motion and not the body representation that dominates our perception of portrayed emotion.” (McDonnell et al., 2008 p. 70).
According to Harris et al., (2014) the process of building affect driven adaptive environments is geared toward creating environments that can change based on the affective state of a target user. In their research on including affect-driven adaptation to the Pac-Man video game, Harris et al used various sensors to collect the physiological data of the user and an emotion recognition model was utilized to connect the sensed data and affective states. Game changes driven by affective states can be used to enhance the user experience by increasing or maintaining the player’s engagement (Harris et al., (2014). “Building affect-driven adaptive environments comprises reading a user’s physiological information through sensors, inferring the user’s affect, and then using this information to create a feedback loop” (Harris et al., 2014).
McDonnell, Jorg, Hodgins, Newell (2009) conducted an experiment to establish factors that influence the perceived sex of virtual characters. They used four different types of models and different forms of motion were applied to these models. These models were: highly realistic male and female models, an androgynous character, and a point light walker. The results showed that both form and motion influence the sex perception for these characters. McDonnell et al., (2009) state that, “people expect a virtual character to behave in a manner befitting its appearance and they find discrepancies in behavior disturbing” (p. 22). McDonnell et al., (2009) explains that studies on people’s perceptions of the gender of a virtual character have been conducted in the recent past in experimental psychology, with regards to human motion. However, information on the impact of body shape is scanty and there is need for more research.
In another study McDonnell, Jorg, Hodgins, Newell, & O’Sullivan (2017) assess the emotional content of motions portrayed by different characters. They used the same technique by creating virtual and real replicas of an actor exhibiting the six basic emotions in a video. Additionally, the actions of the actor were applied to diverse virtual body shapes. They also used point light conditions to test the impact of the absence of a body on the perceived emotion of the movements. The study findings showed that the perception of emotional actions is full bodied and to a large extent independent of the character’s body, as long as form is present. The presence of motion alone elicited less intense emotions when compared to instances when the form was present (McDonnell, et al., 2017).
According to Jain, Anthony, Aloba, Castonguay, Cuba, Shaw, & Woodward (2016) children’s movements and adult’s movement patterns differ when the two performed similar actions. Past research on the relationship between computer graphic design and human motion has primarily focused on adult motion. There is a need to understand how the two movements differ and whether these variations will affect animation and interaction (Jain et al., 2016). In their exploratory study by Jain et al., (2016) they use non-marked motion capture to gather data on child and adult motion and carried out a perceptual study with point light displays to find out whether observers could identify differentiate between child and adult motion. The study’s results showed that the observers were generally successful the detection of body motion. These observations can be integrated in game design to create more realistic and engaging avatars for games, movies, online social media, and animated videos.
Bartram, L., Patra, A., & Stone, M. (2017, May). Affective Color in Visualization. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (pp. 1364-1374). ACM.
Bartram, L., Cheung, B., & Stone, M. (2011). The effect of colour and transparency on the perception of overlaid grids. IEEE Transactions on Visualization and Computer Graphics, 17(12), 1942-1948.
Birk, M. V., Iacovides, I., Johnson, D., & Mandryk, R. L. (2015, October). The false dichotomy between positive and negative affect in game play. In Proceedings of the 2015 Annual Symposium on Computer-Human Interaction in Play (pp. 799-804). ACM.
Bernays, R., Mone, J., Yau, P., Murcia, M., Gonzalez-Sanchez, J., Chavez-Echeagaray, M. E., … & Atkinson, R. (2012, October). Lost in the dark: emotion adaption. In Adjunct proceedings of the 25th annual ACM symposium on User interface software and technology (pp. 79-80). ACM.
Dormann, C., & Biddle, R. (2008, November). Understanding game design for affective learning. In Proceedings of the 2008 Conference on Future Play: Research, Play, Share (pp. 41-48). ACM.
Ferstl, Y., Kokkinara, E., & Mcdonnell, R. (2017). Facial features of non-player creatures can influence moral decisions in video games. ACM Transactions on Applied Perception (TAP), 15(1), 4.
Ferstl, Y., Kokkinara, E., & McDonnell, R. (2016, July). Do I trust you, abstract creature?: a study on personality perception of abstract virtual faces. In Proceedings of the ACM Symposium on Applied Perception (pp. 39-43). ACM.
Harris, A., Hoch, A., Kral, R., Teposte, M., Villa, A., Chavez-Echeagaray, M. E., … & Atkinson, R. K. (2014, September). Including affect-driven adaptation to the Pac-Man video game. In Proceedings of the 2014 ACM International Symposium on Wearable Computers: Adjunct Program (pp. 1-4). ACM.
Hyde, J., Carter, E. J., Kiesler, S., & Hodgins, J. K. (2016). Evaluating animated characters: facial motion magnitude influences personality perceptions. ACM Transactions on Applied Perception (TAP), 13(2), 8.
Jain, E., Anthony, L., Aloba, A., Castonguay, A., Cuba, I., Shaw, A., & Woodward, J. (2016). Is the Motion of a Child Perceivably Different from the Motion of an Adult?. ACM Transactions on Applied Perception (TAP), 13(4), 22.
Jenny, B. & Kelso, N.V. (2007). Color Design for the Color Vision Impaired. Cartographic perspectives. Pp 61-68
Kim, E., & Suk, H. J. (2016, October). Key Color generation for affective multimedia production: an initial method and its application. In Proceedings of the 2016 ACM on Multimedia Conference (pp. 1316-1325). ACM.
Kao, D., & Harrell, D. F. (2016, May). Exploring the Impact of Avatar Color on Game Experience in Educational Games. In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems (pp. 1896-1905). ACM.
Lambrant, A., Luro, F. L., & Sundstedt, V. (2015, September). Avatar preference selection in game design based on color theory. In Proceedings of the ACM SIGGRAPH Symposium on Applied Perception (pp. 15-18). ACM.
Mauderer, M., Flatla, D. R., & Nacenta, M. A. (2016, May). Gaze-Contingent Manipulation of Color Perception. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (pp. 5191-5202). ACM.
McDonnell, R., Jörg, S., McHugh, J., Newell, F. N., & O’Sullivan, C. (2009). Investigating the role of body shape on the perception of emotion. ACM Transactions on Applied Perception (TAP), 6(3), 14.
McDonnell, R., Jörg, S., McHugh, J., Newell, F., & O’Sullivan, C. (2008, August). Evaluating the emotional content of human motions on real and virtual characters. In Proceedings of the 5th symposium on Applied perception in graphics and visualization (pp. 67-74). ACM.
McDonnell, R., Jörg, S., Hodgins, J. K., Newell, F., & O’sullivan, C. (2009). Evaluating the effect of motion and body shape on the perceived sex of virtual characters. ACM Transactions on Applied Perception (TAP), 5(4), 20.
McDonnell, R., Jörg, S., Hodgins, J. K., Newell, F., & O’Sullivan, C. (2007, July). Virtual shapers & movers: form and motion affect sex perception. In Proceedings of the 4th symposium on Applied perception in graphics and visualization (pp. 7-10). ACM.
McDonnell, R., Jörg, S., McHugh, J., Newell, F., & O’Sullivan, C. (2008, August). Evaluating the emotional content of human motions on real and virtual characters. In Proceedings of the 5th symposium on Applied perception in graphics and visualization (pp. 67-74). ACM.
Mehta, R. and Zhu, R. J. (2009) Blue or Red? Exploring the Effect of Color on Cognitive Task Performances. Science. Vol 323. pp. 1226-1231
Normoyle, A., Liu, F., Kapadia, M., Badler, N. I., & Jörg, S. (2013, August). The effect of posture and dynamics on the perception of emotion. In Proceedings of the ACM Symposium on Applied Perception (pp. 91-98). ACM.
Nguyen, L. C., Do, E. Y. L., Chia, A., Wang, Y., & Duh, H. B. L. (2014, April). DoDo game, a color vision deficiency screening test for young children. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 2289-2292). ACM.
Ochs, M., & Pelachaud, C. (2012, June). Model of the perception of smiling virtual character. In Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems-Volume 1 (pp. 87-94). International Foundation for Autonomous Agents and Multiagent Systems.
Palmberg, R., Peters, C., & Qureshi, A. (2017, September). When facial expressions dominate emotion perception in groups of virtual characters. In Virtual Worlds and Games for Serious Applications (VS-Games), 2017 9th International Conference on (pp. 157-160). IEEE.
Reategui, E., & Fruet, C. (2006, November). Evaluating the impact of different virtual characters in product recommendations. In Proceedings of VII Brazilian symposium on Human factors in computing systems (pp. 73-80). ACM.
Sun, E., Matthews, M., Gay, G., Morris, M. E., & Carmean, D. (2014, October). Playing with emotions: sentiment design for public space. In Proceedings of the first ACM SIGCHI annual symposium on Computer-human interaction in play (pp. 439-440). ACM.
Szwoch, M. (2015, June). Design elements of affect aware video games. In Proceedings of the Mulitimedia, Interaction, Design and Innnovation (p. 18). ACM.
Wisessing, P., Dingliana, J., & McDonnell, R. (2016, July). Perception of lighting and shading for animated virtual characters. In Proceedings of the ACM Symposium on Applied Perception (pp. 25-29). ACM.