If any company is going to win an award for the best-named AI initiatives, it will be Adobe. At the Adobe MAX creativity conference in October 2023, Adobe gave a sneak peek at its latest early-stage AI innovations.
Project Stardust lets users edit elements in an image with a click. Project See Through removes reflections from photos. Project Primrose turns textiles into creative canvases. Project Neo helps incorporate 3D shapes into 2D designs. Project Poseable dynamically adapts human poses inside images.
Most significant for the language industry, however, is Project Dub Dub Dub, which automates video dubbing while preserving the voice of the original speaker. The move to add a dubbing feature to the Adobe suite is part of a broader trend among big tech companies of injecting language AI capabilities into their products.
Demoing the dubbing feature in the “Sneaks” phase of the conference, Adobe Audio Research Scientist Zeyu Jin showed how automated dubbing can be applied to audio and video “while keeping the voice the same.”
Jin shared two “stress tests” with the audience. First, the Adobe researcher dubbed a colleague’s voice into six languages. Then, Jin auto-dubbed a clip of the fast-talking classic movie “His Girl Friday” into Spanish and German, while retaining the voices of original actors Rosalind Russell and Cary Grant.
Intended to offer just a “sneak peek” at an early-stage innovation, the demo was light on technical details. The feature uses “speech-to-speech translation,” but it’s not clear whether Adobe is leveraging its own family of creative generative AI models (“FireFly”) or — like Spotify’s Voice Translation feature — is drawing on translation and voice synthesis capabilities from leading large language model (LLM) providers.
Initial feedback on the quality of the dubbing was underwhelming, with some YouTube users questioning the intelligibility and timing of the Spanish and German dubs. Other users, however, pointed out that the feature was simply a prototype and shared their expectation that quality would inevitably improve over time.
Adobe has framed the feature as enabling “brands to be able to build trust with international audiences”, with Jin describing the auto-dubbing option as a means for content creators to get more views, by “getting content to people who do not speak the language of the video.”
It’s a goal shared by an ever-more crowded landscape of AI-enabled localization solutions and providers — from popular YouTuber Mr Beast to well-funded AI dubbing startups (such as VOISEED, Neural Garage, Dubverse, Blanc, Dubdub, and Deepdub), and more established players (such as PaperCup, Synthesia and AppTek), as well as traditional media localization companies who are also now leveraging AI to complement human dubbing services.
By weaving together AI-enabled creative and multilingual features within a leading technology portfolio, Adobe joins other big tech companies like Microsoft (with its AI co-pilot offering) in presenting a new and compelling proposition for solving localization needs.
However, as Adobe explained, “Project Dub Dub Dub is still under development and is not available yet. Please stay tuned for updates.”
Your information will not be shared with third parties. No Spam.