OpenAI Cautiously Debuts Multilingual Generative AI Voice Engine

OpenAI Cautiously Debuts Multilingual Generative Voice Engine

On March 29, 2024, OpenAI debuted its generative AI model for speech synthesis, Voice Engine. 

According to an OpenAI blog post, Voice Engine has been in development since late 2022, and the tech behind the product has already been put to use in existing offerings, such as OpenAI’s text-to-speech API and ChatGPT Voice and Read Aloud.

But OpenAI is trying to tread carefully beyond these applications, citing the risk of “potential synthetic voice misuse.” 

To that end, OpenAI has not yet made Voice Engine available to the public, and the company is currently testing the model with a small group of “trusted partners.” (Beyond the responsibilities of individual companies such as OpenAI, the US Federal Communications Commission made AI-generated voices in robocalls illegal in February 2024.)

These include a few users whose work involves translation. Using just a 15-second audio clip, Voice Engine can produce a synthetic voice speaking in another language. 

Dimagi, which builds tools for frontline healthcare workers in underserved communities in different countries, uses Voice Engine to provide interactive feedback in workers’ native languages.

“When used for translation, Voice Engine preserves the native accent of the original speaker: for example generating English with an audio sample from a French speaker would produce speech with a French accent,” OpenAI explained in the blog post.

Case in point: An English source, or reference, audio clip is accompanied by snippets of generated versions in Spanish, Mandarin, German, French, and Japanese (each of which features an identifiably American accent). 

Standing Out from the Crowd

So-called “voice cloning” technology is the cornerstone of a number of startups, such as ElevenLabs, Papercup, Deepdub, and Respeecher, and a major interest for some of the biggest names in tech, among them Amazon, Microsoft, and Google

The source of Voice Engine’s training data is a touchy subject for OpenAI, which was sued by The New York Times for copyright infringement related to its text-generation tools. 

Jeff Harris, a member of OpenAI Product Staff, told TechCrunch that the Voice Engine model was trained “on a mix of licensed and publicly available data,” adding that “the audio that’s used (i.e., 15-second clips from users) is dropped after the request is complete.”

TechCrunch also reported price estimates of approximately USD 1 per hour — less expensive than certain competitors, such as ElevenLabs, which charges USD 11/100,000 characters per month.

“OpenAI developed Voice Engine in 2022… and you still think AGI [artificial general intelligence] hasn’t been achieved internally?”

The proliferation of speech synthesis technology has already caused controversy in the entertainment industry. In particular, concerns that performers might lose out on work in dubbing contributed, in part, to the months-long actors strike in 2023. Impressions outside Hollywood have been more mixed.

Despite The New York Times’ ongoing case against OpenAI, the paper wrote up Voice Engine with a kind of breathless wonder, writing that OpenAI “has unveiled technology that can recreate someone’s voice.”

“OpenAI developed Voice Engine in 2022…” one impressed commentator noted, adding rhetorically “…and you still think AGI [artificial general intelligence] hasn’t been achieved internally?”

Another observer on X had a decidedly less positive take: “Now, with only 15 seconds of audio, OpenAI can totally mimic your voice, like a robot parrot from hell.”