Our facial expressions constitute a powerful non-verbal communication system used in our social interactions to communicate our feelings. The movements of our facial muscles produce expressions that are usually linked to some specific emotional arousal. And these movements can be interpreted or decoded either by a human or computer-based system. The computer-based decoding process works by identifying key points on our faces and decoding the feeling associated with expression generated by the activity of facial muscles.

The basic unit of measure in facial expression analysis is called the action unit (AU). Each AU corresponds to the movement of a specific muscle or group of muscles. By combining a series of different AUs, the system decodes the expression and the emotional arousal associated with it. Some systems use information encoded directly into their sources, and others use machine learning techniques. The evolution of computational technologies has increased the precision in decoding facial expressions, turning them into a valuable data collection technique. Furthermore, computer-based facial expression analysis can be applied online, in real-time, or offline to video-recorded materials. These features give you, as a researcher, more possibilities to conduct research using these innovative technologies.

It is important to emphasise that not all emotional excitement translates into the movement of facial muscles. In other words, emotions are not just made up of physical elements. Furthermore, there are still doubts about the extent to which these morphological changes and affective feelings are related. That is why it is essential to combine different methods in your research to get a more accurate measurement.

There are also discussions about the universal aspect of facial expression. Different cultures may express emotions differently. However, some studies suggest a kind of universality in the facial expressions of emotion. At least for some basic emotions, there is a cross-cultural aspect. In simple, certain basic emotions are perceived and expressed in the same way, regardless of cultural context or background.

Conducting studies using computer-based facial expression analysis may also have some challenges or limitations, e.g. lighting variation. Differences in lighting can make it challenging for the system to capture the key points needed to decode facial expressions. Therefore, it requires attention controlling the research setting to ensure measurement accuracy.

However, even considering the mentioned technological limitations and the discussions around some aspects of facial expression and emotions, this is still a valuable technique you should consider in your investigation. Supposing human emotions are part of your research interest, this innovative data collection technique can help you answer some of your research questions.

The primary metric of facial expression analysis is the percentage of time participants had a specific emotional reaction (e.g., fear, anger, happiness, disgust) during the event/experiment. It is also possible to have just the engagement level of the participants, which is just the change in their natural facial expressions. All these data can be obtained in time percentage or number of frames.

Depending on your research design, you can obtain some more specific information. It is possible to collect data on the percentage of time or number of frames that respondents smile, close their eyes, open their mouths, lift their chin, raise their eyebrows, wrinkle their nose, etc. And, like eye tracking and Galvanic skin responses, you can export data into a file format supported in Excel and analyse it using an appropriate statistical program.

For what can you use this data? Well, that depends on your research investigation problem. Broadly speaking, if understanding human behaviour while interacting with a specific event is relevant to your research topic, you should use it. For example, how people react while watching a marketing campaign video, looking at a beautiful touristic landscape, reading instructions on a computer system, tasting food, etc.

As we know, it is not the research problem that should fit the research method, but in fact, it is the other way around. So, if what you have read about this data collection technique matches your research project, it is time to use it in your data collection process. And as you know, the DigiHealth & Smart Tourism Lab at the European Campus Rottal-Inn has this technology to help you conduct your research.

Celso Caciano Brito

Celso Brito is a Brazilian master student of International Tourism Development at Technische Hochschule Deggendorf (THD-ECRI). Experienced travel manager and tour leader, he has visited many countries around the world, mainly in Europe, South and North America. He loves literature, art, music, wine, gastronomy and is always ready for a trekking adventure.