Case Study: What I learned from design multimodal UX on Pepper Robot
Say Hi to Pepper!
Pepper is the world’s first social humanoid robot able to recognize faces and basic human emotions manufactured by SoftBank Robotics. Pepper was optimized for human interaction and is able to engage with people through conversation and touch screen.
What‘s my task?
When I first joined this project, the team already finished the development on Pepper and is ready to take Pepper out of the lab to real-world user testings. We conducted user testings at a bunch of public events and observe people’s behaviour when they interact with Pepper. As a ui/ux designer, I also take the initiative to go through semi-structured interviews after people finish interacting with Pepper.
User Testing
The outcome from the first user testing session is quite frustrated as most people find it hard to perform tasks on Pepper, even though we do have those features built on Pepper.
Well, the overall feedback on Pepper is really positive, and people think she is friendly and would like to spend time interacting with her, which gave us more motivation to improve the experience on Pepper.
Testing Analysis
1. Let user feel understood
Let users know when to start the interaction with Pepper
Due to technical restriction, Pepper is not always in the listening mode, users will have to wait her eyes are blue (the indicator of listening mode) in order to initial the conversation. Although Pepper would always say “talk to me when my eyes are blue“, most people ignore this when they are super excited or when the environment is loud. Therefore the interaction is not well connected, and makes user think Pepper couldn’t understand their instruction.
Confirm that Pepper understood users’ instruction
Provide more visual/audio cue to confirm that users’ instruction has been received and understood: highlight active buttons/status, let Pepper repeat part of users’ instruction to confirm that she understands.
Make sure there is always a fallback in case Pepper doesn’t have the answer or interaction designed. Hearing “Sorry, I don’t understand“ all the time can be the most frustrating experience, thus we designed a couple of alternatives and also randomized the answer, which also includes instructions like “Would you like to take a selfie with me“ to redirect users to the right flow.
2. Encourage users to start
In user testing, we found that most people would observe afar before they move closer to interact with Pepper. Some of them are hesitate to start the interaction especially when there is a crowd: shy of interacting with unfamiliar technology in public and afraid of the awkwardness/mistakes. People are more likely to start the interaction when a facilitator take the initiative to introduce Pepper.
Content retrieval is not as robust as it looks. We used a quote-to-quote approach and develop answers and conversations based on keywords. In this case, guiding users to say those keywords becomes the key for a successful Interaction.
To encourage the interaction, I designed a signage showing some conversation starter for our end users. In this way, users are able to understand how to start the interaction without feeling that they have no idea what to say.
3. Environment / Noise Issue
Pepper can achieve the best performance in a 1 on 1 conversation in quiet environment which is hard to guarantee in a branch. When people stand behind Pepper, she gets confused and might lock her eyes on the wrong user.
To solve this problem, I designed some signage and marks on the floor which provide a step-by-step guide and let users understand where they should stand and what to expect when they interact with Pepper.
Other Interesting Findings
Adding a sense of humour makes the user experience more engaging (just like talk to a funny person).
Understanding the personality of Pepper helps to set the tone for all the interaction design on Pepper.
Robotics could help with child education as nowadays children are more likely to search questions on digital media rather than asking their parents.
Although for most people, this is their first time interacting with a robot, but some of them have experience using Siri/Alexa and other voice tech assistant, so they expect the same interaction pattern: saying “hi Pepper“ to wake her up and start the interaction.
Key Takeaways
I really appreciate that I’ve got the opportunity to work on project like Pepper: there is no standardized process or formula that guide you though the design process. Keep testing, keep iterating, this is the only way to make the experience more intuitive and robust.
This project bridges the gap between knowledge I gained from books and real-world scenario where we are also facing technical challenges and marketing requirements. It also allows me to transfer my design skills from 2d screen based design to a multimodal user experience challenge where I bring some design techniques such as paper prototyping, user interview to emerging tech like this, which motivates me to work at the intersection of design and technology.