MADRID, 5 Feb. (EUROPA PRESS) –
A team at the University of Washington wondered if artificial intelligence could recreate the performance experience of musicians and their instruments using only visual cues.
The researchers used machine learning to create a system, called Audeo, that creates audio from silent piano performances.
When the team tested the music that Audeo created with music recognition apps, such as SoundHound, applications correctly identified the piece that Audeo reproduced approximately 86% of the time. For comparison, these apps identified the part in the audio tracks of the source videos 93% of the time.
The researchers presented Audeo at the NeurIPS 2020 conference, in December.
“Creating music that sounds like it could be played in a musical performance was previously believed impossible,” he said. it’s a statement lead author Eli Shlizerman, assistant professor in the departments of applied mathematics and electrical and computer engineering.
“An algorithm needs to discover the signals, or ‘characteristics’, in the video frames that are related to the generation of music, and it needs to ‘imagine’ the sound that is happening between the video frames. It requires a system that is to the once precise and imaginative. The fact that we got music that sounded pretty good was a surprise. “
Audeo uses a series of steps to decode what is happening in the video and then translate it to music. First you have to detect which keys are pressed in each frame of video to create a diagram over time. Then you need to translate that diagram into something that a music synthesizer actually recognizes as a sound that a piano would make. This second step cleans up the data and adds more information, such as how hard each key is pressed and for how long.
“If we tried to synthesize music from the first step, we would find that the quality of the music is unsatisfactory”Shlizerman said. “The second step is like how a teacher reviews the music of a student composer and helps improve it.”
Researchers trained and tested the system using YouTube videos of pianist Paul Barton. The line-up consisted of some 172,000 video frames of Barton playing music by well-known classical composers such as Bach and Mozart. They then tested Audeo with nearly 19,000 frames of Barton playing different music by these composers and others, like Scott Joplin.
Once Audeo has generated a transcript of the music, it’s time to feed it to a synthesizer that can translate it into sound. Each synthesizer will make the music sound a little different; this is similar to changing the “instrument” setting on an electric keyboard. For this study, the researchers used two different synthesizers.
“Fluidsynth produces synthesizer piano sounds that we are familiar with. They are somewhat mechanical but fairly accurate,” said Shlizerman. “We also use PerfNet, a new AI synthesizer that generates richer and more expressive music. But it also generates more noise.”
Audeo was trained and tested only on Paul Barton piano videos. Future research is needed to see how well you can transcribe music for any musician or pianoShlizerman said.
“The objective of this study was to see if artificial intelligence could generate music played by a pianist in a video recording, although we did not mean to replicate Paul Barton because he is a virtuoso “, Shlizerman said. “We hope that our studio will allow for novel ways of interacting with music. For example, a future application is that Audeo can be extended to a virtual piano with a camera that records only a person’s hands. Furthermore, by placing a camera on top of a piano, Audeo could potentially help in new ways of teaching students to play. “
–
