The Electronic Gaming industry has been growing ever fast. The innovations and virtual realities clubbed together are building a new horizon of games ever built before. Lot of research is encouraged to apply artificial intelligence to gaming applications. One of latest technologies in building intelligent games is the application of Affective Computing in Gaming environment.
Affective Computing is an interdisciplinary field which includes Engineering, Neuroscience, Psychology, Cognitive science, computer science, ethics and more. Advances in wearable technology lead to non-intrusive measurement of required physiological parameters. Efficient machine learning algorithms have been developed to identify patterns in the obtained data. Though huge leaps of innovation have been made, understanding and classifying human emotions is still a complex process. But, Affective Computing is already being applied in various fields like personal health, machines with empathy, assessing stress, haptic technology and health technology.
A person’s emotions is one of the important factors that plays a major role in making a decision. To map the decision making abilities into a computer it is necessary for the computer to identify the emotions of a person and respond to them. There has been a lot of research to determine the physiological parameters that can help to estimate an individual’s emotional state. Some of the commonly used parameters are Skin Conductance, Heart Beat, and Electroencephalogram. Use of the emotional state in computing lead to the field of Affective Computing.
Determining human emotions can play a vital role in many fields. It’s importance has already been recognized in the fields of psychology, intelligent system design, biomedical systems and entertainment industry.
In this project, Affective Computing is applied in the video games. This project was intended to estimate the gamer’s affect and use it to modify the gaming environment making it a better and more interesting experience for each gamer. The project was carried out in three stages. In the first stage, all the necessary sensors were designed to read the biological parameters of the gamer. All these sensors were then induced in the wearable cap that can read the various sensor values and send it to the Arduino. Arduino then samples these values and sends it to the PC where the data is processed
In the second stage, the sensor data so read are processed to estimate the emotional of the gamer. In this stage, an SVM classifier was trained to provide an estimate of the emotional state of the person.
In the third stage, the estimate of the emotional state of the gamer is used in the video game to control the game environment. If the gamer was relaxed, the game environment changed the centre of gravity of the car making it difficult to play the game. If he is stressed, the game environment will change by reducing the suspension and providing a good control of the car. Also, if the gamer is stressed, the racing track will provide lot of bonus points which the gamer can grab thus giving him additional points and a sense of victory. Thus, by adapting rapidly to the gamer’s emotional status, the game provides a really enhanced experience for the gamer.
These affect based games keeps learning on its own. In other words, these games will customize themselves dynamically to the gamer’s needs and feelings thus providing the best possible experience to him.
The project implemented each stage and tested for the real game play. The system monitored the biological parameters of the gamer, determined gamer’s emotional state and controlled the game environment. However, there is still a tremendous scope to improve in each stage with better mapping of sensors to emotions and emotions to game control in real time.
This project aims enhancing the gaming experience of the gamer. With this project enables the game environment to change itself according to the emotional state of the gamer. Thus, it provides an enhanced interactions between the game and the gamer. The gamer will no longer have to play easier levels before he reaches his challenges. The game quickly adapts itself to his gaming capabilities. This project employs the novel idea of Affective Computing in gaming to estimate his emotional state for the game.
Interest in determining human emotions dates back many centuries. It encompasses many disciplines- neuroscience, psychology, sociology, cognitive science and computer science. From this a new branch of study called affective computing emerged. It strives to build computing systems that can interpret human emotions.
Human emotions can be related to numerous biological parameters. Research in neuroscience helped to determine the structures in brain that can be related to human emotions. The electrical charge stored on neurons in these structures were studied by using Electroencephalogram. Another biological parameter of importance is skin conductance. Discovery of skin conductance in 1889 by Ivane Tarkhnishvili lead to a flood of experiments to determine its relation to external stimuli. These two parameters seem to be the most responsive to human emotions. Other parameters include blood oxygen content, facial expressions, posture, Electrooculogram.
Another challenge after determining the biological parameters is to classify them. Determining accurate classification from various modes of data requires a machine learning algorithm with high accuracy. Some of the approaches include Support Vector Machines, Bayesian Approach, Neural Network and Probabilistic Graphical Models.
The emotional state of the person affects his behavior in many ways ranging from his reactions to the external environment to more internal aspects such as his biological parameters like pulse rate, skin conductance etc. This project makes use of this fact that the emotional state of the person affects his biological parameters. With this, this project tries to reverse engineer to develop an estimate of the person’s emotional state by monitoring his biological parameters. This section talks about how some of the biological parameters of the person is affected by his emotional state.
Sensor 1: Electroencephalogram (EEG):
Cerebrum of Human brain can be classified into a total of 6 lobes, each of which is related to each of which are related to different brain functions. Lobe and their corresponding functions are shown below
|Frontal Lobe||Determines future consequences based on present input and output. It retains memories for a long time that are not tasked based. They are usually based on emotion input from the limbic system. They adapt the emotions to fit socially acceptable norm.|
|Parietal Lobe||This part of the brain processes information various senses including the sense of touch. It plays an important role in determining relationship between numbers and is essential in manipulating objects.|
|Occipital Lobe||As the name indicates this lobe is the visual processing center of the brain. It determines orientation, color properties and frequency. Damage in this lobe can lead to hallucinations|
|Temporal Lobe||This lobe is involved In high level auditory processing and also in processing visual stimuli. This lobe establishes object recognition based on visual input. It is an important region for language/speech recognition|
|Limbic lobe||This lobe is visible in medial cross section of brain. This lobes determines the emotion. A direct relationship between emotional responses and limbic lobe can be established|
|Insular Lobe||This part of brain recognizes pain and few other social emotions. It is also responsible in motor control of hand and eye. It is a factor in establishing self-awareness.|
An electroencephalogram can be used to read signals from any one of these lobes and those signals can be classified into emotions. As you can see, each lobe performs only specific set of functions so it is essential to choose the right lobes to study when determining emotional response to a set of stimuli. An internationally recognized system called 10-20 system helps in determining the placement of electrodes on scalp for the study of any of the six lobes
This project mainly consists of visual and auditory stimuli. Since the main objective is to redesign the gaming console this project has chosen to study the frontal and occipital lobes of the brain because they are easily accessible.
Sensor 2: Galvanic Skin Response (GSR)
Study of Galvanic Skin Response and its response to external stimulus began in early 1900’s. It was observed that in presence of external stimuli there is a detectable change in the galvanic skin response. Galvanic Skin response encompasses all the electrical properties of skin. It was determined that skin conductance is an objective index of emotional responses. Galvanic skin response consist of five principal measures:
|Skin Conductance Level(SCL)||Base level of electrical conductivity of skin|
|Skin Conductance Response(SCR)||Phasic change in electrical conductivity of skin|
|Non-Specific SCR’s(NS-SCR)||Skin conductance response without any stimuli|
|Frequency of NS-SCR’s||Rate at which skin conductance response changes in absence of stimuli|
|Event Specific SCR||Skin conductance response In presence of external stimuli|
Recent advances in technology lead to better design of circuits to measure skin conductance and improvements in the way skin conductance values are analyzed. Since skin conductance response has proven to respond to stimuli in a reasonable amount of time we included it as a parameter.
Sensor 3: Pulse Sensor
Studies show that, heart beat variability can be associated with change in emotional state of a person. It is very well known that heart beat rises when doing a strenuous task or when excited. Pulse sensor is usually used in fitness monitor to make sure that the heart rate doesn’t cross specified levels.
Based on our research, we believe pulse sensor can prove to be another objective measure of emotional state so we included it as one of our parameters.
Sensor 4: Body Temperature
The fourth parameter we choose to study is Body temperature. Although this parameter doesn’t change very fast and doesn’t help much in determining the emotional status of the person, the body temperature can affect other parameters being used. Thus, the temperature variation can be used to dynamically calibrate other parameters to estimate the emotional status of the person. So it is essential to include body temperature as a factor in the emotion classification algorithms.
This project aims at enhancing the gaming experience for the gamer by developing the game that can adapt to the gamer’s abilities quickly.
This is achieved by developing a game that can take his stress levels as inputs and changing the game environment. If the gamer is relaxed and bored, the game should quickly detect this and provide better challenges to him to make him interested. On the other hand, if the gamer is stressed, the game should decrease his difficulty levels thus keeping him interested.
This solution of feeding the stress levels to the game so that it can adapt itself to gamer’s stress levels is inspired by the novel idea of Affective Computing. Affective Computing helps to estimate the emotional status of the person and the same is applied to control the video game environment.
The block diagram of the complete system is shown below.
As can be seen from the above block diagram, the system is developed in three stages as mentioned below.
- Stage 1: Sensor circuits design and interface with Arduino.
- Stage 2: Emotion detection
- Stage 3: Video game development
The video game receives the game control inputs from the game controller operated by the gamer. Also, various sensor circuits’ reads gamer’s biological parameters and sends it to Arduino. The emotion processor reads the various biological parameters of the gamer and estimates the emotions. Once the emotion is estimated, the same is provided as another input to the video game to change the game environment.
Stage 1: Sensor circuits design and interface with Arduino
In this section, the circuits for various sensor interface are designed. As mentioned above, there are biological parameters used in this project to estimate the emotional status of the person. Each of these parameters are read through corresponding sensor circuits and supplied to the Arduino. Arduino reads these parameters and processes them before sending these data to the Emotion Processor. The hardware circuit design and software processing on Arduino for each parameter are explained below.
Kindly refer to the circuit diagram below used for sensing the pulse rate of the person.
This circuit is designed in two stages. The first stage corresponds to reading the pulse rate. To determine the pulse, an IR light from IR transmitter is passed through the tip of the figure and the same is read by the IR receiver. Whenever there is a pulse, the intensity of the IR light refracted changes. This change in intensity is converted into change in the voltage levels across the IR receiver. This change in the voltage levels determines the pulse.
In the second stage, the pulse so determined as a voltage change, is then amplified and filtered by a low pass filter. This amplification and filtering is done in two stages. Each stage is designed for cut off frequency of around 3 Hz and gain of around 100. The actual calculations for each parameters are shown below with actual values.
Gain = 1 + = 1 + = 1 + 100 = 101
Cut-off frequency = = = 3.12 Hz.
The Arduino Due has an 84 MHz main clock and the analogRead() function takes about 40us. At this rate, Arduino can read 25K samples in 1 second. However, the sampling is limited by the serial communications between the Arduino and the PC. This is because, in each iterations of the loop, the Arduino samples the analog input channel and sends the data serially. Since serial communication is slow, this acts as the bottle neck to number of samples read each second. Arduino sends the data at 9600 baud rate. Thus, the sampling rate will also be limited by this baud rate. The baud rate can be increased to read more samples if needed.
The Arduino samples the pulse values and determines the peaks corresponding to each pulse. Finally, it counts the number of pulses and sends the data to the Emotion Processor on the PC serially.
To determine the pulse, the program sets a threshold on the voltage level to determine when the sensor data obtained peaks. Each peak is associated with one beat of the heart. To determine heart beats per minute i.e. heart rate, number peaks in a 10 second interval are counted and used interpolation to determine the number of peaks in 60 seconds. Then, the number of peaks and thus the heart rate count is determined.
Galvanic Skin Resistance:
Kindly refer to the interface circuit for the galvanic skin resistance shown below.
As explained for the pulse sensor, the GSR interface circuit is also designed in the similar two stages. However, unlike pulse sensor, in the first stage, the GSR value is determined by the pair of electrodes stuck at the middle of the two fingers. The voltage difference in both the electrodes is then amplified and filtered in the second stage. The amplification and filtering employed in the second stage of GSR is same as that in Pulse. In each stage, the gain is 101 and cut-off frequency is 3.12Hz as shown below.
Gain = 1 + = 1 + = 1 + 100 = 101
Cut-off frequency = = = 3.12 Hz.
As described in the previous section, the Arduino Due has an 84 MHz main clock and can read 25K samples in 1 second for default settings. However, the sampling is limited by the serial communications between the Arduino and the PC due to the sequential execution in each iteration. Since the serial communication is set at 9600 baud rate the sampling is limited to this rate. The baud rate can be increased to read more samples if required.
The skin conductance response is plotted with respect to time in Processing. It was observed that the baseline in skin conductance response changes rapidly. As such, instead of referring to the actual values of the skin conductance, the project referred to the first derivative of the skin conductance values thus referring to the changes in the skin conductance values for different events.
The EEG sensor interface circuit is shown below.
This circuit has three parts. In the first part, a neutral reference ground (or a floating ground) is generated half way between the power supply to provide a positive and negative voltage levels to the amplifier. In the second stage, the EEG ways are measured using three electrodes. The first electrode is attached to the A1 point as per the 10-20 system (behind the left ear). This electrode connects to the floating ground. The second and third electrodes attaches to O2 and FP2 points respectively on the 10-20 system. Both, second and third pins will be connected to the input terminals of the Instrumentation Amplifier namely Pins 2 and 3 respectively.
The EEG waves picked up by these electrodes is then amplified by the amplification stage. The trimmer in the amplification stage is calibrated to bias the output of the amplifier in the middle of the required output voltage range. The, overall amplification of the system is around 1500.
As described in the previous section, the Arduino Due has an 84 MHz main clock and can read 25K samples in 1 second for default settings. However, the sampling is limited by the serial communications between the Arduino and the PC due to the sequential execution in each iteration. Since the serial communication is set at 9600 baud rate the sampling is limited to this rate. The EEG values are read by the Arduino and transmitted serially to the Emotion Processor on PC at 9600 baud rate.
Unlike rest of the parameters, information regarding the emotional responses is obtained from frequency spectrum. To observe this, the Fast Fourier transform (FFT) of the brainwaves is plotted using a library in Arduino and displayed it using Processing. The magnitude of frequency response changes in the frequency band around 4-30 Hz. The frequency spectrum of the EEG brainwaves is characterized by different brainwave components namely alpha (8-12 Hz), beta (12 – 25 Hz) and delta (4-8 Hz) waves. These brainwaves corresponds to different emotional state of a person. The EEG waves were then processed to calculate the contribution of various brainwaves. These values are then mapped to the emotional state of the person based on the experiments. During the experiments, we observed that the beta waves of the person are stronger than alpha and delta waves when he is stressed and they are comparable when he is relaxed. This fact is further used in the training of the SVM classifier in emotion processor. This is described in Emotion detection section later.
The body temperature changes very slowly and can be read using a thermistor. This sensor doesn’t require any amplification. The temperature sensor circuit is a voltage divider as shown below.
The thermistor is denoted by R Thermistor in the above circuit. The lower resistor R1 is a known constant resistor across which the voltage is measured. Based on this value the body temperature is mapped in the software.
As described in the previous section, the Arduino Due has an 84 MHz main clock and can read 25K samples in 1 second for default settings. However, the sampling is limited by the serial communications between the Arduino and the PC due to the sequential execution in each iteration. Since the serial communication is set at 9600 baud rate the sampling is limited to this rate. The body temperature values are read by the Arduino and sent to the emotion processor at 9600 baud rate. We observed that body temperature doesn’t settle down at one level instead it seems to vary sporadically. So we took an average of 10 values to obtain a more smoothed level of temperature.
Stage 2: Emotion Detection
Emotion detection is the most challenging part in the whole of the project mainly due to its inherent research requirements. Thus, we dedicated a separate stage for this task. The emotion detection task involved identifying the parameters that can be used to uniquely estimate emotions. This task involves lot of experiment based analysis to estimate emotional status of the person. Various sensors designed in the above section are used to determine various features that can be used for emotion detection. The emotion detection employed SVM classifier for emotion detection. SVM is a supervised learning model that takes labeled data to train the model. It classifies the training data by means of hyper planes in the multi-dimension space. Based on experiment results, we narrowed down on following set of features to train SVM model.
- Energy in Delta Brainwaves
- Difference in energies between Alpha and Beta Brainwaves
- 1st Derivative of Skin Conductance
These features requires only EEG waves and skin conductance values. Thus the remaining two parameters namely body temperature and pulse rate are omitted for the emotion detection. The block diagram for the emotion detection task is shown below.
Figure 7 Emotion detection block diagram.
As explained above, SVM is a supervised learning model i.e. the SVM classifier is trained based on the labeled datasets and then builds a model based on these labeled datasets. Then, new data sets are predicted by the model thus trained.
To train the SVM model, system allows the gamer to play for the first lap. During this period all the required sensors parameters are read and the corresponding feature set are extracted. These feature sets are then labeled. Finally, these labeled training data are passed to the SVM model and the model is trained.
The labels are applied to the training datasets in two ways. In the first approach, a key “K” is provided for the user to input his stress levels. If he is stressed, he can press on the key “K” which labels the datasets corresponding to that time. This approach gives a more user controlled approach of labelling the datasets for his stress levels. The datasets labelled this way cannot be overwritten. In the second approach, the difference in the beta and alpha brainwaves is calculated for all the datasets. Amongst these values for all the datasets are the top 20% of the range is used to label stressed state whereas rest values corresponds to relaxed state. This labelling is based on several experiments. Based on the experiments, it was found that the difference between beta and alpha energy levels increases tremendously when stressed out and is less for relaxed state.
Once the SVM model is trained, the sensor values are read and features are extracted. These feature set are then passed to the trained SVM model to estimate the emotional status of the gamer. This emotional status is then passed to the video game to control the game environment.
Stage 3: Video Game development
We used Unity Game Engine for the video game development. The project focused on developing a first person shooter game from scratch. We tested the results for the same with some free and open racing games developed in Unity. We found that the racing game was more stressful than the first person shooter game as the racing game needed full attention throughout the game. Thus, we decided to enhance the racing game to control the game environment.
There were two levels developed for the video game. The first level was the basic game which was used to train the SVM model. This level did not have any game environment changes. The second level was used for the testing. In this level based on the biological parameters of the gamer his emotional status is estimated and the same is then passed to the video game. Based on the emotional status, the video game environment changed. The various parameters of the video game that was controlled are divided into two sections as mentioned below.
- Car related parameters such as wheel suspension and the center of gravity of the car etc.
- Race track parameters such as obstructions on the track and special points at the curves
As described above, this project is carried out in three stages mentioned below.
- Stage 1: Sensor circuits design and interface with Arduino.
- Stage 2: Emotion detection
- Stage 3: Video game development
In the first stage, the various sensor circuits are designed along with the interface software program for Arduino. One team member, Deepak, was assigned with circuit design tasks while the other team member, Mavitha, developed the software interface programs. All sensor circuits and interface programs were developed in the first semester.
In the second stage, various feature set are determined for the emotion detection based on the various set of experiments. Once the feature set were determined, the same were used to train the SVM model to predict the emotional status.
In the third stage, the video game was developed and tested for game environment control based various inputs. Finally, the video game was integrated with the emotion detection program to develop the flow of the game including laps for training the SVM model and further laps for controlling the video game environment based emotional status inputs.
Both emotion detection and video game development tasks were carried out in the second semester. Although both team members worked together to discuss about the approaches and perform the tasks, Manvitha took over the video game development while Deepak designed the SVM model for emotion detection.
The project achieved various objectives and milestones. Various sensors were interfaced successfully and studied their behavior with respect the emotional status of the person. Then the project narrowed down the sensors to select the features for training SVM model to predict the emotional status of the gamer. Finally, the emotional status is used to control the video game environment.
As mentioned earlier, the training data for the SVM classifier was labeled based on the difference in the beta and alpha brainwaves. A sample of the training labels classified are plotted as shown below. A sample of the EEG and Skin Conductance values recorded during game play are listed in Appendix A and the corresponding SVM training datasets generated for the same data is listed in Appendix B. Label the axis
Figure 8 Training data classification based on threshold levels for SVM Model training.
In the above figure, the dataset are plotted which are used for training the SVM classifier. The X axis corresponds to the time axis and the Y-axis corresponds to difference in beta and alpha waves. The dataset corresponding to point above the threshold line corresponds to Stressed State and ones below the threshold corresponds to the Relaxed state. There is no clear demarcation between the relaxed and stressed out states. Thus, it was decided to consider the datasets with highest (beta-alpha) values to represent stressed state. Thus, the threshold was chosen to be 80% of the range of beta-alpha values. The actual value of threshold itself will be different for each use as it depends on many factors such as the emotional state of the person, environmental factors, how well are the electrodes connected etc.
The EEG brainwaves for Relaxed and Stressed status are shown below.
In the above figures, the sharp peaks observed at around 41Hz and 18Hz are expected to be the 60Hz noise aliased into the lower frequencies due to lower sampling rates. These frequencies are expected to correspond to first and second harmonics of the 60Hz respectively.
The Skin conductance waveforms for Relaxed and Stressed status are shown below.
In the above figures, there are two kinds of events. One due to physical movements of the fingers to which the electrodes are connected to. These events corresponds to the ones with wider widths and marked in blue. Another kind of events are due to emotional state changes. These are shown by frequent changes in the skin conductance values with narrow ‘V’ shaped widths and are marked in red in the above figures. In the relaxed state, events due to emotional state change will be less and it’ll be more in stressed state as represented in the above figures.
The feature used for SVM training is the first derivative of the skin conductance values. Since these two events couldn’t be differentiated by the system, both these events will be considered during the SVM training. However, the SVM training should only consider the events due to emotional changes and not the ones due to finger movements, the latter was suppressed by suggesting the user not to move his fingers on which electrodes were attached.
The game environment for Relaxed and Stressed status are shown below.
As can be seen from the above figures, in the relaxed state, the car throws a green light showing the relaxed state. In this state, the car controls will be changed by shifting the center of gravity of the car thus making it more difficult to play. This gives a sense of challenge to the gamer.
In the stressed state, the car emits red light showing the signs of stress. Also, there will be many bonus points you can get when you are stressed. Thus giving the gamer sense of victory when he is stressed.
Recognition of Human emotions have applications in many fields. Some of the domains where human emotion detection can be used are explained here.
In psychology, affective computing is being used to monitor mental health and improve emotional wellbeing. It is used to assist in learning needs and social skills of children diagnosed with Autism and ADHD. It helps monitor depression, sleep and stress patterns. Another important application of this technology in this field is in the case of Epilepsy. It helps recognize the anomalies in the nervous system activity that precede seizures.
In intelligent system design, affective computing is being put to use in helping systems understand emotions and respond accordingly. Few examples include engaging concerts to determine reactions to musical expressions, gesture based guitar, system to practice social interactions in face-to-face scenarios.
In biomedical system, affective computing is being used to remotely monitor vital signs like heart rate, blood pressure and breathing rate thereby improving the delivery of primary healthcare. It also increases the amount of interaction between doctor and patient and helps reduce the need for trained personnel during diagnostic tests.
In entertainment industry, affective computing is being used to enhance user experience. It is being used to set just the right amount of challenge in games and toys. It is also being used to reinvent user interactions in the retail industry.
The emotion detection can also be applied in customer experience determination by estimating the customer’s emotional responses to a product. Thus it helps in redesigning the brand perception providing a greater satisfaction for the customers.
We have successfully designed systems to measure and monitor four biological parameters- Skin conductance response, Electroencephalogram, Pulse sensor and Body temperature. Our next steps would be as follows
- Determining response of each parameter to different kinds of visual and audio stimuli
- Classifying the response into Excited, sad and normal using an efficient machine learning algorithm
- Building an interactive video game which changes difficulty level, graphics and audio based on output from the machine learning algorithm
We would like to express our deepest appreciation for all those who provided us encouragement and guidance to complete this project. A special gratitude to our project advisor Dr. Bruce Land for his infinite knowledge and continuous guidance. His stimulating suggestions and encouragement helped us to plan and coordinate this project successfully.
- Picard, R.W. (1997) Affective computing. MIT Press, Cambridge.
- Vladimir Vapnik, Statistical Learning Theory, 2nd Edition, Springer Science and Business Media.
- You-Yun Lee, Shulan Hsieh, Classifying Different Emotional States by Means of EEG-Based Functional Connectivity Patterns, 2014.
- Charles Moyes, Mengxiang Jiang, Brain Computer Interface,