What Descript did with Lyrebird AI might be the most innovative use of voice technology I’ve ever seen.
Descript is a transcription service. A very good transcription service, but a transcription service nonetheless.
You provide it with a spoken word audio sample: a podcast recording, a video, a meeting, and it’ll transcribe it for you pretty accurately.
You’ve always been able to edit the transcript and see those edits effect the audio sample. If you remove a few words here and there, it’ll delete them from the audio and vice versa if you edit the audio.
But editing is always subtractive. Taking bits out is easy. What about if you wanted to add something in?
Lyrebird rose to fame by being able to accurately clone people’s voices and generate a text to speech synthetic voice from as little as 1 minute of sample audio.
Descript acquired Lyrebird in 2019 and has now rolled out Overdub, a feature that let’s you type edits into your transcript and, using Lyrebird tech, it’ll generate a synthetic voice of the person speaking, then add the words you’ve added into the original audio sample.
That means that not only can you create transcripts from audio, you can also generate audio from transcripts!
What did Descript do with Lyrebird? And is it one of the best applications of voice technology yet?
To answer that question, we need to take a long walk to the Whiteboard.
For those of you that don’t know, Descript acquired Lyrebird in 2019. And what Lyrebird did was enable you to create clones of your voice, but synthetic clones, so, like, you could create a text-to-speech synthetic voice, based on your actual voice!
They were behind some of the things that you might have seen like the clone of Barack Obama, which sounded like this…
“They launched today their website where you can create a digital copy of your voice. They only need you to record one minute of audio. This is just the beginningand they are working hard to improve the results”.
And the clone of Donald Trump, which sounded like this…
“South Korea is finding, as I have told them, that their talk of appeasement with NorthKorea will not work. They only understand one thing”
And so Descript acquired Lyrebird in 2019, but what exactly did they do with the technology? To answer that question, you kind of got a look at what Descript does.
Now, fundamentally, Descript is a transcribing solution.
It takes audio, transcribes it into text and then provides you with both audio and the text. And it’s fantastic for doing things like captioning on videos or transcripts for podcasts or transcripts for meetings or anything like that.
It does get fairly sophisticated and what it does is it will tell you, based on the transcript, you’ll recognize who’s speaking and it will then be able to tell you when different people are speaking, which is pretty cool.
It also lets you edit some of the copy and whatever you edit is reflected in the audio. So if you had a phrase that was ‘VUX World rocks’, which it does, and you took out the word ‘world’ from the transcript, it would then remove that from the audio and vice versa. If you remove it from the audio, it would remove from the transcript, which is pretty cool.
But audio editing like this is subtractive and most of the time it is subtractive because you have an audio file and that’s all you have. You can’t do anything with it. You can put stuff over the top of it and underneath it like if you have a guitar sound you can put a piano over the top and a drum beat underneath, but you can’t change the guitar sound.
You can take clips from one guitar from over here and put it, insert it into the middle of this file, but it’s not going to sound very natural. You can’t generate more sound from one sound file and that’s always been the limitation of Descript is that it only goes one way. It only allows you to remove stuff, but you can probably see where I’m heading with this.
What they did with Lyrebird is they’ve created a new function called Overdub, a new feature, and what that lets you do is add phrases and words into the transcript. So you could change the phrase from ‘VUX World rocks’ to ‘VUX World rocks, subscribe at VUX.World/subscribe and what it will do, using the Lyrebird technology, is it will clone the voice of the person speaking when you add the text in, it will then generate synthetic speech based on that voice and insert it into the audio.
And so the end result is you have an additive audio process as well as a subtractive audio and transcription editing process.
That means that, in the end, you have an audio file full of the things that you’ve removed and the things that you’ve added in, which isn’t going to blend together totally seamlessly.
If you listen to a voice clone versus the real voices, you’re not going to get away with it entirely, but it will certainly work for things like podcasts where it’s a long form audio, like videos, animated videos and even for articles.
You could dictate an article in Descript, edit the article in Descript, have some of the synthetic audio and some of your live audio mixed together. And so the output will be a written article as well as the audio file.
The potential of this is absolutely huge and I’m looking forward to seeing what happens to it because I do think that this is one of the most innovative uses of voice technology that I’ve seen.
Let me know if you would agree, if you’ve got any thoughts on this.I think it’s fantastic.
Looking at the Apple and Google partnership that could enable tracking of coronavirus cases and shares why you should think about voice early in projects. read more
‘Voice’ is forecast to be the next technology platform for customer engagement and the next interface to the web and computing, but what actually is it?
What is voice? So most people, most normal people, don’t know what ‘voice’ is and they don’t use the term ‘voice’.
It’s just, like, an industry term that we’ve picked up because it’s short and it’s easy to describe.
Most people, like my dad, for example, he just talks to his phone. He doesn’t know what Google Assistant is,he doesn’t know what ‘voice’ is.
He doesn’t really care. He just talks to his phone. And so it’s easier usually to explain to people that it’s just like what we’re doing is we’re creating experiences for, and establishing a presence for brands on, the kind of interfaces or the kind of surfaces that people just talk to and that’s the basic definition of what voice is. It’s anything; software, devices or any kind of computer type program that you can talk to, and it understands you and can then either do something for you, like turn on your light or take out an insurance policy – that was a bit of a, two random examples, from light switch into insurance! – But but there is a German company that is doing that but it can respond to you in spoken language as well.
So the most common examples that people will have is probably Siri on the iPhone, Alexa, the Echo smart speakers, Google Assistant on Android and also the smart speakers. Those are examples of what we would call voice platforms.
So in this definition of platform, we’re talking about something that has baseline, a base level of functionality or value and that you can build on top of to increase the value that the platform has so it’s the same as think of it as Windows or iOS.
Yeah, so Windows has a base level of functionality. You can build applications on top of the Windows operating system to increase the functionality that the
Windows operating system has in its entirety. Alexa is kind of like that.
So Alexa comes with a base level of functionality: timers and calendar integrations and things like that: music, but then you can build on top of that with customized third-party bespoke applications. They call them skills in the Alexa world or actions in the Google world. They increase the functionality and capability of the entire thing.
So it breaks down into two things one is the platform
and the other is an interface or voice user interface or VUI as it’s often called, and that is anything that it might not have a platform, might not have a voice assistant in there, but it might just be something that you can talk to and make something happen.
So for example, you might be able to say to your coffee machine “make me a coffee” and it will just perform the action of making you a coffee and so voice breaks down into those two areas; platforms and assistants, and interfaces and the interface can be an interface to any kind of technology in any kind of Hardware or even an interface to the to the web.
Taken from Kane’s interview on the 414 Podcast discussing Voice strategy: how to create a voice search strategy for your brand.
Everyone’s asking whether 2020 will be the year of voice. Here’s our take on it. read more
Why has Apple acquired Voysis and what does that mean for Siri?
To answer that, we need to take a long walk over to the Whiteboard. read more
Will the current cover situation change our behaviour towards touching screens outside of the home? And can voice technologies play a role in facilitating changing behaviour? read more
Apple acquired voice-specialist conversational AI company, Voysis, recently and how we discovered it was, well timed and kinda funny. read more