logo image
  • News
    • People Moves
    • Deal Wins
    • Demand Drivers
    • M&A and Funding
    • Financial Results
    • Technology
    • Academia
    • Industry News
    • Features
    • Machine Translation
    • — Divider —
    • Slator Pro
    • — Divider —
    • Press Releases
    • Sponsored Content
  • Data & Research
    • Research Reports & Pro Guides
    • Language Industry Investor Map
    • Real-Time Charts of Listed LSPs
    • Language Service Provider Index
  • Podcasts & Videos
  • Events
    • Design Thinking – February 2021
    • — Divider —
    • SlatorCon Coverage
    • Other Events
  • Directory
  • RFP Center
  • Jobs
MENU
  • News
    • People Moves
    • Deal Wins
    • Demand Drivers
    • M&A and Funding
    • Financial Results
    • Technology
    • Academia
    • Industry News
    • Features
    • Machine Translation
    • — Divider —
    • Slator Pro
    • — Divider —
    • Press Releases
    • Sponsored Content
  • Data & Research
    • Research Reports & Pro Guides
    • Language Industry Investor Map
    • Real-Time Charts of Listed LSPs
    • Language Service Provider Index
  • Podcasts & Videos
  • Events
    • Design Thinking – February 2021
    • — Divider —
    • SlatorCon Coverage
    • Other Events
  • Directory
  • RFP Center
  • Jobs

Advertise on Slator! Download the 2021 Online Media Kit Now

  • Slator Market Intelligence
  • Slator Advertising Services
  • Slator Advisory
  • Login
Search
Generic filters
Exact matches only
Advertisement
Good, Bad or Loose: Amazon’s New Subtitle Quality Estimation System

9 months ago

May 4, 2020

Good, Bad or Loose: Amazon’s New Subtitle Quality Estimation System

Machine Translation ·

by Esther Bond

On May 4, 2020

9 months ago
Machine Translation ·

by Esther Bond

On May 4, 2020

Good, Bad or Loose: Amazon’s New Subtitle Quality Estimation System

What are you doing to stay occupied during the lockdown? Baking, DIY, yoga, Xbox? For many, devouring new movies and TV programs is now more than just an enjoyable way to unwind in an evening — streaming content has become an essential mechanism for coping during the weeks and months spent confined at home during the lockdowns.

Streaming platforms such as Amazon Prime Video, Netflix, Hulu and others are well aware that delivering a good viewing experience is essential to keeping their audiences captive. For foreign-language content, the quality of subtitles can be the difference between viewers staying tuned or switching off.

Two researchers from Amazon Prime Video, International Expansion, Prabhakar Gupta and Anil Nelakanti, observed in a recent paper that “providers increase their viewership by enabling content in local languages.” At the same time, they said, “low translation quality can cause increased usage drop-off and hurt content viewership for audience of target language.” So getting the quality right is crucial. 

Advertisement

A new research paper by Amazon explores the topic of quality estimation (QE) in translated subtitles. The paper, entitled “DeepSubQE: Quality estimation for subtitle translations,” presents a new system for estimating the quality of translated subtitles, whether human- or machine-generated. The DeepSubQE system reduces both cost and time in subtitle translation while assuring quality, the researchers said.

“Low translation quality can cause increased usage drop-off and hurt content viewership for audience of target language” — Prabhakar Gupta and Anil Nelakanti, Amazon Prime Video, International Expansion

Subtitling and dubbing are indeed topical for Amazon; As the researchers explained, the “digital entertainment industry is growing multifold with ease of internet access and numerous options for on-demand streaming platforms.” And, although a separate paper published by Amazon in January 2020 looked into machine dubbing, Gupta and Nelakanti pointed out that “translation of subtitles across languages is a preferred cost effective industry practice to maximize content reach.” 

Subtitling may be a more cost-efficient approach than dubbing, but the researchers also noted that using human translators to localize subtitles is expensive, and the “man-power cost” increases significantly in the case of low-resource target languages. Since a second person normally checks the translated subtitles to improve the quality of the translation where needed, quality evaluation is “as expensive as generating the translation itself,” they said.

Slator 2020 Language Industry Market Report

Data and Research, Slator reports
55 pages. Total market size, biz dev and sales insights, TMS & MT review, buyer segment analysis, M&A, Covid impact & outlook.
$480 BUY NOW

Exploring an automated approach for estimating the quality of translated subtitles is therefore logical, and all the more so for Amazon, who is among the biggest players in the on-demand streaming space.

Good, Bad or Loose

Clearly, quality estimation for any kind of translation is challenging because there is more than one way to translate a given sentence into a specific target language. Legitimate translation techniques like paraphrasing, rephrasing and idiom, which are frequently used in subtitling translation, complicate the matter further since they confound binary methods of quality estimation. 

Most QE methods are binary methods, however, and simply classify translations as “good” or “bad.” They fail to deal with ““loosely” translated samples that often occur due to human judgment.” 

Under Amazon’s DeepSubQE model, which introduces a “loose” translation category, a good translation is one that retains “all meaning from [the] source and reads fluently;” A bad translation is one that bears no resemblance to the meaning of the original and is “disconnected from the context in the video;” A loose translation is one that uses paraphrasing, colloquialism, idiom, or “contains some contextual information not available in [the] source text.” 

Slator RFP Service - Request for Proposal

RFP Center

Business Development, Market Intelligence
Receive daily email alerts of tenders and RFPs issued by governments, NGOs and private entities from across the world.
BUY NOW

The researchers also noted how categorizing subtitle translations into the three categories helps to indicate what further work the translations require, if any. Good translations are fine as is and need no further improvement, while loose translations may require human post-edits and bad translations need a “complete rewrite,” they said.  

The researchers worked with 30,000 video subtitle files in English and their corresponding translations in French, German, Italian, Portuguese and Spanish for their experiments. They found that the DeepSubQE model was accurate in its estimations more than 91% of the time for all five languages. The system performed slightly better for longer sentences.

One area of improvement they noted for DeepSubQE was related to the operational load. Currently, the system relies on training one model per language pair, which requires “considerable operational load,” the researchers said. However, future work could explore using a multilingual model, which would help to reduce the load and also be of benefit to “resource starved languages,” they added.

TAGS

AmazonAmazon Prime Videodubbingmachine translationmachine translation quality estimationmedia entertainmentMedia Localizationsubtitling
SHARE
Esther Bond

By Esther Bond

Research Director at Slator. Localization enthusiast, linguist and inquisitor. London native.

Advertisement

SUBSCRIBE TO THE SLATOR WEEKLY

Language Industry Intelligence
In Your Inbox. Every Friday

SUBSCRIBE

SlatorSweepSlatorPro
ResearchRFP CENTER

PUBLISH

PRESS RELEASEDIRECTORY LISTING
JOB ADEVENT LISTING

Bespoke advisory including speaking, briefings and M&A

SLATOR ADVISORY
Advertisement

Featured Reports

See all
Slator 2020 Language Industry M&A and Funding Report

Slator 2020 Language Industry M&A and Funding Report

by Slator

Slator 2021 Data-for-AI Market Report

Slator 2021 Data-for-AI Market Report

by Slator

Slator 2020 Medtech Translation and Localization Report

Slator 2020 Medtech Translation and Localization Report

by Slator

Pro Guide: Sales and Marketing for Language Service Providers

Pro Guide: Sales and Marketing for Language Service Providers

by Slator

Press Releases

See all
Rheinschrift Language Services – Strategic Improvements and Workforce Expansion in 2021

Rheinschrift Language Services – Strategic Improvements and Workforce Expansion in 2021

by Rheinschrift Language Services

Memsource Acquires Phrase

Memsource Acquires Phrase

by Memsource

Across Systems will be part of the Volaris Group

Across Systems will be part of the Volaris Group

by Across Systems GmbH

Upcoming Events

See All
  1. Memsource MT Post-Editing Pricing Models Webinar

    Pricing Models for MT Post-Editing Workshop

    by Memsource

    · February 3

    Hear a panel of innovative localization professionals share different approaches for MT post-editing pricing.

    More info FREE

Featured Companies

See all
Text United

Text United

Memsource

Memsource

Wordbank

Wordbank

Protranslating

Protranslating

Seprotec

Seprotec

Versacom

Versacom

SDL

SDL

Smartling

Smartling

Lingotek

Lingotek

XTM International

XTM International

Smartcat

Smartcat

Translators without Borders

Translators without Borders

STAR Group

STAR Group

memoQ Translation Technologies

memoQ Translation Technologies

Advertisement

Popular articles

Why Netflix Shut Down Its Translation Portal Hermes

Why Netflix Shut Down Its Translation Portal Hermes

by Esther Bond

Top Language Industry Quotes of 2020

Top Language Industry Quotes of 2020

by Monica Jamieson

The Slator 2020 Language Service Provider Index

The Slator 2020 Language Service Provider Index

by Slator

SlatorPod: The Weekly Language Industry Podcast

connect with us

footer logo

Slator makes business sense of the language services and technology market.

Our Company

  • Support
  • About us
  • Terms & Conditions
  • Privacy Policy

Subscribe to the Slator Weekly

Language Industry Intelligence
In Your Inbox. Every Friday

© 2021 Slator. All rights reserved.

Sign up to the Slator Weekly

Join over 13,000 subscribers and get the latest language industry intelligence every Friday

Your information will never be shared with third parties. No Spam.