What we want you to know is
The research team implanted a paper-thin rectangular electrode sheet containing 253 electrodes on the cortical surface of the language area of Ann’s brain, and set up a port on her head to connect to the computer with a cable. In order to create a digital avatar, the research team developed a synthetic voice algorithm and trained it with image files of Ann’s speech at a wedding before her stroke, creating a personalized synthetic voice that sounded like her.
Text: Liu Xinxiang
On August 23, U.S. time, a research team from the University of California, San Francisco (UCSF) and the University of California, Berkeley (UC Berkeley) used brain-computer interface (BCI) combined with artificial intelligence (AI) technology to convert brain signals into synthetic speech for the first time. and facial expression animation, successfully allowing a woman who lost the ability to speak due to a stroke to communicate naturally as a digital avatar and speak 78 words per minute, faster than any similar technology in the past. This result was published in the top scientific journal “Nature”.
Leading this research is Edward Chang, a well-known Taiwanese neuroscientist and director of neurosurgery at UCSF School of Medicine. He has been involved in brain-computer interface research for more than ten years. His team published in “NEJM” in July 2021 that for the first time, the brain activity signals of paralyzed people when they tried to speak were translated into single words and sentences and presented on the screen. At that time, they could type 15 words per minute.
Zhang Fulun said, “In this new study, we accelerated the speed of translating brain activity signals when trying to speak into text to about 78 words per minute; not only that, we can also directly convert this brain signal into words that can be listened to synthesized speech and rendered accurate facial movements on the digital avatar.”
Participating in this study is a 48-year-old woman, Ann. She suffered a brainstem stroke 18 years ago, resulting in severe paralysis. After years of physical therapy, she was able to move her head and control facial muscle movements such as laughing or crying. However, , the muscles that allowed her to speak remained motionless.
The research team implanted a paper-thin rectangular electrode sheet containing 253 electrodes on the cortical surface of the language area of Ann’s brain, and set up a port on her head to connect to the computer with a cable. This technique, called electrocorticography (ECoG), can record the combined activity of thousands of neurons simultaneously. If Ann had not had a stroke, the nerve signals picked up by these electrodes might have been sent to the muscles of her tongue, jaw, throat and face.
Photo Credit: UCSF official website The team implanted a paper-thin rectangular electrode sheet containing 253 electrodes on the cortical surface of the language area of Ann’s brain.
Ann worked with the research team to train the system’s artificial intelligence algorithm for several weeks, repeatedly trying different phrases from a conversational vocabulary of 1,024 words until the computer could recognize the uniqueness of each of her phonemes. neural activity patterns. Phoneme refers to the smallest sound unit in speech. For example, “Hello” contains four phonemes: HH, AH, L and OW.
Using this method, the computer only needs to learn 39 phonemes to decipher any word in English, which not only improves the system’s accuracy but also makes it three times faster.
In order to create a digital avatar, the research team developed a synthetic voice algorithm and trained it with image files of Ann’s speech at a wedding before her stroke, creating a personalized synthetic voice that sounded like her.
The research team also used a set of facial animation software developed by Speech Graphics and used machine learning to convert Ann’s brain signals into muscle movements on the virtual character’s face, including movements of the jaw, lips, and tongue, as well as expressions of happiness, sadness, and Facial expressions of emotions such as surprise.
Regarding hearing the digital clone successfully speak for her, Ann said, “It felt very interesting. It was like hearing the voice of an old friend.” She also said, “Participating in this study gave me a sense of mission. I feel that I I am contributing to society and feel like I have a job again. This study makes me feel truly alive!”
Zhang Fulun said that our goal is to allow paralyzed people to restore a complete and authentic way of communication, which is the most natural way for people to talk to others.
The research team said that the key to the next stage is to create a brain-computer interface that does not require wire connections.
References
This articleGlobal Biotechnology MonthlyAuthorized to publish, original textPublished here
Original title: “Nature” brain-computer interface breaks through again!Aphasic woman uses digital avatar to speak at record 78 words per minute
Further reading
[Join as a member of Key Comment Network]Excellent articles will be sent directly to your mailbox every day, and you will receive exclusive weekly editorial selections, current affairs selections, art weekly and other special e-newsletters. You can also leave a message to discuss the content of the article with the author, reporter, or editor. Click now to become a member for free!
Editor in charge: Zhu Jiayi
Verification editor: Weng Shihang
2023-10-14 08:00:01
#Taiwanese #neuroscientist #braincomputer #interface #combined #technology #aphasic #woman #speak #digital #avatar #breaking #record #words #minute #News #Lens #Critical #Comment #Network
