Engineers at Columbia University have developed a robot capable of learning and recreating facial lip motions for speech and singing by watching hours of YouTube videos and practicing in front of a mirror. The robot uses 26 facial motors to articulate words in multiple languages and even sing a song from its AI-generated debut album, as reported in the journal Science Robotics.
The research, led by Yuhang Hu for his PhD, aims to bridge the 'uncanny valley' effect where robots appear almost human but not quite right. The robot's lip-syncing ability, when combined with conversational AI like ChatGPT or Gemini, could enhance human-robot interactions significantly.
Lead researcher Hod Lipson acknowledged current limitations, particularly with hard sounds like 'B' and lip-puckering sounds such as 'W', but expressed confidence that these will improve with practice. The team believes that facial expression is as crucial as leg and hand motion for humanoid robots.
Potential applications include entertainment, education, medicine, and elder care, with predictions of over a billion humanoids being manufactured in the next decade. Lipson emphasised that robots must learn these abilities rather than being programmed with rigid rules, stating, 'Something magical happens when a robot learns to smile or speak just by watching and listening to humans.'
