A probabilistic framework modeling relationships between student mental states and body gestures | |
| Author | Abbasi, Abdul Rehman |
| Call Number | AIT Diss. no.ISE-09-08 |
| Subject(s) | Emotions Human engineering Pattern recognition systems |
| Note | A dissertation submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy in Mechatronics Engineering, School of Engineering and Technology |
| Publisher | Asian Institute of Technology |
| Series Statement | Dissertation ; no. ISE-09-08 |
| Abstract | Do facial expressions always convey the actual emotion or affect of a person? Are there other visual clues that may indicate the affect or mental state of a person? How can computers know the affect or mental state of their users on reading these clues? How can these clues be utilized towards development of context-specific applications such as an affect-sensitive intelligent tutoring system (ITS)? This thesis investigates these research questions. Many psychologists have studied human visual clues such as facial expressions or body movements, and attempted to interpret affect or mental state from these clues. Based on their interpretations, computer scientists have proposed, and evaluated many models that could upgrade many existing man-machine systems (e.g. human-computer or human-robot applications) with human-like capabilities of knowing their users’ af- fect or mental state. However, it is clear that a deterministic interpretation can not hold in all situations or contexts; for example, a smile does not show happiness under all situations. Towards investigating this issue, in our first human study, we analyze the facial expressions of 13 human subjects watching emotion-evoking video clips. First, we extracted their facial response and got it labeled by three non-expert human label- ers. To know the actual emotion or affect, we then obtained subjects’ self reports in post-experiment interviews. In addition, we used the commercial facial expression analysis software “Face Reader” to automatically label the facial response of our four subjects. This subset of data was also manually labeled by 20 under-graduate students. In our first analysis of this study, we compared subjects’ self reports to their manually-labeled emotion categories, and found significant disagreement between the labeled and reported emotions. In the second analysis of the same study, we compared the machine-labeled emotion categories with the self reports and the manual labels, and we found that the machine categorization was even worse than the manually-labeled categorization. Higher agreement was found among the labelers for positively-valenced emotions such as “Happiness” while negatively-valenced emotions were either misclassi- fied or isolated as “Neutral”. Results from both of these analyses suggest that gestures such as facial expressions give little information about a human’s internal emotion perse in the absence of any context, and manual and machine recognition of visual clues needs more careful treatment than the basic or conventional expression-emotion inter- pretation can provide. Towards investigating this issue, in our second human study, we recorded the ac- tivities of a total of 11 students attending a class lecture, in five separate sessions with different teachers. We extracted all the gestures that they performed during those ses- sions, and to know their affect or mental state, we conducted self-reporting interviews. We found that gestures such as a “Chin Rest” or an “Eye Rub” probabilistically in- dicate “Thinking” or “Tired” state, respectively; however, the relationships between these “unintentional” gestures and reported mental states were uncertain. The study gives us correlations between eight gestures and six identifiable mental states. Since probabilistic models are known to perform well with little or incomplete knowledge and uncertain data, we propose using these models to predict student men-tal states from their unintentional yet visually observable gestures. First, we construct a simple probabilistic framework, modeling context and statistical data using static Bayesian networks (SBNs). The model performs well in a leave-one out cross valida- tion evaluation scheme and predicts reported mental states with an accuracy of above 96% when the student reported an identifiable mental state and above 84% when un- certain reports were included. SBNs are limited to performing inference at a single time instant; they lack the ability to capture temporal relationships. In contrast, dynamic Bayesian networks (DBNs) provide a powerful way to represent the causal as well as temporal dependen- cies; hence we construct DBNs. With same evaluation criteria as in the SBN experi- ments, we obtained a generalization accuracy of above 97% where students reported a definite mental state, and above 83% when we included uncertain cases. Although the results from the SBNs and DBNs are similar, the DBN provides a complete picture of evolving mental states at all time steps. The core contribution of this work is that it is the first to use unintentional ges- tures in context to infer mental states. It is also the first to estimate the parameters of the probabilistic model directly from spontaneous training data acquired during real classroom situations, without presupposing any relationship between gestures and mental state. The results are promising on a small but novel data set. The model could be integrated with existing systems to develop affect-sensitive ITSs |
| Year | 2009 |
| Corresponding Series Added Entry | Asian Institute of Technology. Dissertation ; no. ISE-09-08 |
| Type | Dissertation |
| School | School of Engineering and Technology (SET) |
| Department | Department of Industrial Systems Engineering (DISE) |
| Academic Program/FoS | Industrial Systems Engineering (ISE) |
| Chairperson(s) | Afzulpurkar, Nitin V. ;Dailey, Matthew N. (Co-Chairperson); |
| Examination Committee(s) | Manukid Parnichkun ;Bohez, Erik L. J.; |
| Scholarship Donor(s) | Higher Education Commission, Pakistan ;Asian Institute of Technology Fellowship; |
| Degree | Thesis (Ph.D.) - Asian Institute of Technology, 2009 |