logo image
  • News
    • People Moves
    • Deal Wins
    • Demand Drivers
    • M&A and Funding
    • Financial Results
    • Technology
    • Academia
    • Industry News
    • Features
    • Machine Translation
    • — Divider —
    • Slator Pro
    • — Divider —
    • Press Releases
    • Sponsored Content
  • Data & Research
    • Research Reports & Pro Guides
    • Language Industry Investor Map
    • Real-Time Charts of Listed LSPs
    • Language Service Provider Index
  • Podcasts & Videos
  • Events
    • SlatorCon Remote May 2021
    • Localizing at Scale for International Growth
    • Design Thinking May 2021
    • — Divider —
    • SlatorCon Coverage
    • Other Events
  • Directory
  • RFP Center
  • Jobs
MENU
  • News
    • People Moves
    • Deal Wins
    • Demand Drivers
    • M&A and Funding
    • Financial Results
    • Technology
    • Academia
    • Industry News
    • Features
    • Machine Translation
    • — Divider —
    • Slator Pro
    • — Divider —
    • Press Releases
    • Sponsored Content
  • Data & Research
    • Research Reports & Pro Guides
    • Language Industry Investor Map
    • Real-Time Charts of Listed LSPs
    • Language Service Provider Index
  • Podcasts & Videos
  • Events
    • SlatorCon Remote May 2021
    • Localizing at Scale for International Growth
    • Design Thinking May 2021
    • — Divider —
    • SlatorCon Coverage
    • Other Events
  • Directory
  • RFP Center
  • Jobs

Register Now for SlatorCon Remote on May 13th!

  • Slator Market Intelligence
  • Slator Advertising Services
  • Slator Advisory
  • Login
Search
Generic filters
Exact matches only
Advertisement
Facebook Patents Alternative to ‘Expensive’ BLEU

2 years ago

August 22, 2019

Facebook Patents Alternative to ‘Expensive’ BLEU

Machine Translation ·

by Esther Bond

On August 22, 2019

2 years ago
Machine Translation ·

by Esther Bond

On August 22, 2019

Facebook Patents Alternative to ‘Expensive’ BLEU

Facebook recently patented its process for measuring machine translation quality. The patent was granted on July 30, 2019, almost one year after the application date in September 2018.

Four inventors are listed on the patent, which is entitled “Optimizing Machine Translations for User Engagement”: Ying Zhang, Fei Hung, Kay Rottmann and Necip Fazil Ayan. Ayan is Research Lead at Facebook AI, and heads up the Language and Translation Technologies team.

The patent relates to Facebook’s method of gathering user engagement data on machine translation output and using this data to improve the quality of its machine translations. Translations appear in a news feed, in a banner or in an ad, for example. They may be automatically displayed or hidden until the user requests a translation. 

Advertisement

The idea hinges on the premise that the more people like, share and comment on a machine translated post, the better the translation is assessed to be. If you then add weightings and normalize the results, user engagement can serve as a proxy by which to evaluate machine translation quality. 

Slator 2020 Language Industry Market Report

Data and Research, Slator reports
55 pages. Total market size, biz dev and sales insights, TMS & MT review, buyer segment analysis, M&A, Covid impact & outlook.
$480 BUY NOW

Inside the Translation System

Facebook says that suitable options for underlying translation systems include the “PanDoRA” system developed at Mobile Technologies, LLC, as well as machine translation systems developed by IBM Corporation, SRI, BBN or at Aachen University.” The same companies also offer automatic speech recognition (ASR), which Facebook may use to convert audio input to text.

Facebook said in the patent that it displays different translations to different groups of users, called candidate translations. How one candidate translation performs relative to another tells Facebook which one is preferred by users, and therefore better.

From here, Facebook can then tweak its models to ensure that the preferred translation is favored, making it more likely to be used in future. Facebook explains the iterative process as being “repeatedly applied in order to create a feedback system in which multiple candidate translations are generated using a model, the translations are evaluated for user engagement, the model is modified to favor the translation having greater positive engagement, the updated model generates multiple candidate translations, and the process repeats.”

Beyond assessing which candidate translations are preferred, Facebook may also be able to tell which groups of people prefer which translations. For example, since Facebook often holds information about a user’s age, gender and nationality, it may calculate engagement scores on this basis. According to Facebook, “different translations may be generated based on the language patterns of different demographic groups, and an appropriate translation may be provided based upon an identity of a user requesting the translation, or a target group identified in the translation request.”

Usability Not Ratings

In some cases, Facebook shows prompts to users, asking them to say whether a translation is usable or understandable. The emphasis on usability rather than ratings is intentional: the patent inventors found that “asking a user to ‘rate’ a translation often yields inconsistent results, because a user may not know on what basis they should be rating the translation.” By contrast, “asking a user whether a translation was ‘useable’ or ‘understandable’ produced more consistent and more useful results.”

With its method, Facebook is aiming to simplify the usually difficult and time-consuming process of “identifying which translations are favored and communicating this information in a way that a machine translation system can consistently apply.” It is an alternative to the BLEU score, which the Facebook patent points out has “several problems.”

Slator 2019 Neural Machine Translation Report: Deploying NMT in Operations

Data and Research
32 pages, NMT state-of-the-art, 5 case studies, 30 commentaries, NMT in day-to-day operations
$85 BUY NOW

While Facebook acknowledges that BLEU remains the “industry standard in evaluating machine translations,” it is problematic for a number of reasons: BLEU is expensive because it relies on human-produced reference translations and “there are questions as to how well the BLEU score measures translation quality,” Facebook said. For example, “the BLEU score may not accurately capture whole sentence-level meaning, does not address grammatical correctness, and has difficulty evaluating translations involving languages that lack clear word-level boundaries.”

Skeptical About This Direction

Slator reached out to prolific machine translation researcher Rico Sennrich, Lecturer in Machine Learning at the University of Edinburgh, for his assessment of Facebook’s newly patented quality evaluation system. 

Sennrich said that he was skeptical about using big data and user engagement to optimize MT. “Social media platforms and search engines have managed to show users more relevant content by optimizing for user engagement. I understand why there’s interest in using user engagement also to optimize MT – big platforms get this data essentially for free, and it aligns with their business objectives, but I’m skeptical about this direction,” he commented.

Explaining his position, Sennrich added that “I’m happy to believe that improving the translation quality will lead to higher user engagement on the platform. I’m less inclined to believe that optimizing user engagement directly will lead to better translation quality.”

Slator 2019 Neural Machine Translation Report: Deploying NMT in Operations

Data and Research
32 pages, NMT state-of-the-art, 5 case studies, 30 commentaries, NMT in day-to-day operations
$85 BUY NOW

Moreover, he said, “with user engagement as its main objective, there is a risk that translation systems will learn to produce text that maximizes user engagement while sacrificing translation accuracy. To give an example, when translating product descriptions in an online marketplace, naively using sales as the optimization criterion for an MT system could reward the system for embellishing a product and misleading users, rather than translating the description accurately.”

TAGS

Aachen Universityartificial intelligenceASRautomatic speech recognitionBBNBLEUBLEU scorecrowdsourcingFacebookFei HungIBM CorporationKay Rottmannmachine learningmachine translationMobile TechnologiesNecip Fazil Ayanneural machine translationPanDoRApatentsRico SennrichSRIUniversity of Edinburghuser engagementYing Zhang
SHARE
Esther Bond

By Esther Bond

Research Director at Slator. Localization enthusiast, linguist and inquisitor. London native.

Advertisement

SUBSCRIBE TO THE SLATOR WEEKLY

Language Industry Intelligence
In Your Inbox. Every Friday

SUBSCRIBE

SlatorSweepSlatorPro
ResearchRFP CENTER

PUBLISH

PRESS RELEASEDIRECTORY LISTING
JOB ADEVENT LISTING

Bespoke advisory including speaking, briefings and M&A

SLATOR ADVISORY
Advertisement

Featured Reports

See all
Pro Guide: Translation Pricing and Procurement

Pro Guide: Translation Pricing and Procurement

by Slator

Slator 2020 Language Industry M&A and Funding Report

Slator 2020 Language Industry M&A and Funding Report

by Slator

Slator 2021 Data-for-AI Market Report

Slator 2021 Data-for-AI Market Report

by Slator

Slator 2020 Medtech Translation and Localization Report

Slator 2020 Medtech Translation and Localization Report

by Slator

Press Releases

See all
MasterWord Services Inc. Names Jeanette Stewart as Vice President of Operations

MasterWord Services Inc. Names Jeanette Stewart as Vice President of Operations

by MasterWord

XTRF Welcomes Roberto Ganzerli to Its Advisory Board

XTRF Welcomes Roberto Ganzerli to Its Advisory Board

by XTRF

Venga Reshapes Language Review with InQA Cloud Application

Venga Reshapes Language Review with InQA Cloud Application

by Venga Global

Upcoming Events

See All
  1. SlatorCon Remote May 2021

    by Slator

    · May 13 @ 3:00 pm - 8:00 pm

    A rich online conference which brings together our research and network of industry leaders.

    More info $110

Featured Companies

See all
Sunyu Transphere

Sunyu Transphere

Text United

Text United

Memsource

Memsource

Wordbank

Wordbank

Protranslating

Protranslating

SeproTec

SeproTec

Versacom

Versacom

Smartling

Smartling

XTM International

XTM International

Translators without Borders

Translators without Borders

STAR Group

STAR Group

memoQ Translation Technologies

memoQ Translation Technologies

Advertisement

Popular articles

Google Translate Not Ready for Use in Medical Emergencies But Improving Fast — Study

Google Translate Not Ready for Use in Medical Emergencies But Improving Fast — Study

by Seyma Albarino

The Slator 2021 Language Service Provider Index

The Slator 2021 Language Service Provider Index

by Slator

Why Netflix Shut Down Its Translation Portal Hermes

Why Netflix Shut Down Its Translation Portal Hermes

by Esther Bond

SlatorPod: The Weekly Language Industry Podcast

connect with us

footer logo

Slator makes business sense of the language services and technology market.

Our Company

  • Support
  • About us
  • Terms & Conditions
  • Privacy Policy

Subscribe to the Slator Weekly

Language Industry Intelligence
In Your Inbox. Every Friday

© 2021 Slator. All rights reserved.

Sign up to the Slator Weekly

Join over 13,800 subscribers and get the latest language industry intelligence every Friday

Your information will never be shared with third parties. No Spam.