INTERVIEW with Shunji Mitsuyoshi from University of Tokyo

Unveiled by Japanese technology firm SoftBank in 2014 as the world’s first robot capable of reading human emotions, Pepper is fast becoming a familiar sight at mobile phone shops, malls and bank branches across Japan. Pepper stands 121centimetres tall, weighs 29 kilograms and has four in-built microphones, two cameras and a depth sensor that enable it to capture human voices and facial expressions. It displays its own “emotions” through colours and gestures. With its ability to analyse voice tones and interact, Pepper symbolizes a new direction for the robotics industry. Its developers claim that “empathetic” robots may revolutionize medical applications, information services and entertainment. Shunji Mitsuyoshi, the man who designed the “emotional engine” driving Pepper’s
“heart”, explains what makes Pepper unique, what led to his collaboration with SoftBank and how robotics may continue to change the way we live.

What originally led to your work with SoftBank and the development of Pepper?

It all began by collaborating with my friend Kiyoshi Oura, who presented our research on emotion recognition to Masayoshi Son, the founder and chief executive of SoftBank. Our fi rst breakthrough was the development of Sensibility Technology (ST) in 1999; we developed an algorithm for voice recognition, which led to further research on how the amygdala — the part of the brain involved in processing emotions — is connected to the central nervous system, the heart and vocal
cords. Th e next step was the development of our Mind Monitoring System (MIMOSYS) technology in 2014, which enabled us to monitor changes in mood through voice recognition applications. Th en in 2015, I developed an “emotional map” based on my research looking at how hormones aff ect emotions. We were able to identify 223 emotions, which we classifi ed into four colours.
The development of Pepper was made possible through a combination of all three – ST, MIMOSYS and the emotional map.

What was the goal of creating Pepper?

Th e idea was to develop a user-friendly robot capable of responding to human emotions. Mr Son will often talk about the story of Astro Boy (Tetsuwan Atomu), a popular manga hero in Japan. Astro Boy is a humanoid robot capable of empathy. It can cheer us up when we are sad. It can help us by understanding human needs. In a similar vein, we wanted to create a human-like robot, not a mindless or heartless machine

Can you describe what Pepper can do?

Currently, Pepper is capable of voice and face recognition and of interacting by providing information and playing games. It has some ability to pick out human emotions —happiness, frustration, anger and sadness–and to respond accordingly. We want people to think of Pepper as a friend rather than just a robot.
One of our long-term goals is to use Pepper in health and social care. So, it may be possible for Pepper to assist in the early detection of illnesses or disorders such as dementia, heart attacks and Post-Traumatic Stress Disorder (PTSD) through voice recognition.

In what way can Pepper “learn”?

The next generation of humanoid robots will focus on incorporating the concept of a “virtual ego”. If Pepper has its own sense of identity, then it can really become a member of the family. People will be able to “raise” Pepper like a child. It can then “grow” or “learn” based on communication. It’s important to note that this is not the same as deep learning, an altogether diff erent goal that other companies are pursuing. Instead, we envision cases where Pepper can “like” or “dislike” someone based on its own “reasoning”. Th e question of building in “morals” then becomes critical. Working with researchers in South Korea, my team at the University of Tokyo are now trying to uncover the physiological basis of morality, reasoning and deduction in the human brain.

There has already been more than 10,000 Pepper robots sold in Japan, both for corporate and home use. Are there plans for Pepper to be available in other languages and in other countries?

We are focusing on making Pepper universal. In order to do so, I am exploring languages other than Japanese. For example, in 2015 my team at the University of Tokyo started collaborating with a university in Romania. We wanted to learn more about how the nervous system connects to voice and emotion in diff erent languages. Romania is an interesting place to collect voice samples because people in that country speak not only Romanian but also Hungarian, German, Russian, Spanish and Turkish. We are studying how the human voice and intonation may be analysed to diagnose conditions such as dementia, Alzheimer’s disease and Parkinson’s disease. In total, we have collected around 7,000 voice samples so far.

Romania is an interesting place to collect voice samples because people in that country speak not only Romanian but also Hungarian, German, Russian, Spanish and Turkish.

What other products are you now working on?

Based on the “emotional map” I have developed, I am looking at developing other emotion-based technologies that can be used in cars or other everyday items and devices

Can you tell us more about SoftBank World, to be held on 21–22 July 2016 in Tokyo, Japan?

Th is is an event for corporate users of SoftBank in Japan and is made up of lectures from sponsorship partners and exhibitions. Th ere are plans to unveil new technologies based on emotion recognition that will have an impact on people’s daily lives. We encourage everybody to see and experience these new robot technologies for themselves. Visitors to SoftBank World are in for a treat.