logo image
  • News
    • People Moves
    • Deal Wins
    • Demand Drivers
    • M&A and Funding
    • Financial Results
    • Technology
    • Academia
    • Industry News
    • Features
    • Machine Translation
    • — Divider —
    • Slator Pro
    • — Divider —
    • Press Releases
    • Sponsored Content
  • Data & Research
    • Research Reports & Pro Guides
    • Language Industry Investor Map
    • Real-Time Charts of Listed LSPs
    • Language Service Provider Index
  • Podcasts & Videos
  • Events
    • SlatorCon Remote May 2021
    • Email Marketing for Freelance Linguists
    • Preparing for the Critical Google Update Coming in May 2021
    • — Divider —
    • SlatorCon Coverage
    • Other Events
  • Directory
  • RFP Center
  • Jobs
MENU
  • News
    • People Moves
    • Deal Wins
    • Demand Drivers
    • M&A and Funding
    • Financial Results
    • Technology
    • Academia
    • Industry News
    • Features
    • Machine Translation
    • — Divider —
    • Slator Pro
    • — Divider —
    • Press Releases
    • Sponsored Content
  • Data & Research
    • Research Reports & Pro Guides
    • Language Industry Investor Map
    • Real-Time Charts of Listed LSPs
    • Language Service Provider Index
  • Podcasts & Videos
  • Events
    • SlatorCon Remote May 2021
    • Email Marketing for Freelance Linguists
    • Preparing for the Critical Google Update Coming in May 2021
    • — Divider —
    • SlatorCon Coverage
    • Other Events
  • Directory
  • RFP Center
  • Jobs

Register For Email Marketing for Freelance Linguists and Learn How To Win New Clients.

  • Slator Market Intelligence
  • Slator Advertising Services
  • Slator Advisory
  • Login
Search
Generic filters
Exact matches only
Advertisement
Yes, Now They Claim Machines ‘Outperform’ Human Translation in Adequacy

6 months ago

September 10, 2020

Yes, Now They Claim Machines ‘Outperform’ Human Translation in Adequacy

Machine Translation ·

by Seyma Albarino

On September 10, 2020

6 months ago
Machine Translation ·

by Seyma Albarino

On September 10, 2020

Yes, Now They Claim Machines ‘Outperform’ Human Translation in Adequacy

After machine translation became neural machine translation around 2016, academia and big tech research groups began publishing papers implying machine translation had reached human-level quality (whatever that means). Simplified headlines in tech publications followed, researchers argued journalists cherry-picked quotes, and Slator tried to make sense of it all by asking the experts.

“Achieving human-level translation” is so 2018, though. In 2020, it has become “outperforming human-level translation.” That claim was made in a paper published on September 1, 2020, about CUBBITT, a new Transformer-based deep-learning system. The authors include Google’s Łukasz Kaiser, Jakob Uszkoreit of Google Brain Berlin, and Ondřej Bojar at Charles University in Prague.

Okay, let’s start with the caveats. The paper’s title only mentions reaching translation quality “comparable” to human professionals in the domain of “news” translation. And the claim of outperforming humans is reserved for the “adequacy” metric. Still, the claim of “outperforming humans” on any metric other than speed seems new.

Advertisement

Furthermore, the authors conceded that “highly qualified human translators with [an] infinite amount of time and resources will likely produce better translations than any MT system.”

“The quality of professional-agency translations is not unreachable by MT”

They added, however, that “many clients cannot afford the costs of such translators and instead use services of professional translation agencies, where the translators are under certain time pressure. Our results show that the quality of professional-agency translations is not unreachable by MT, at least in certain aspects, domains, and languages.” An interesting take on the professionalism of linguists who earn their living from translation.

Sentence-level Translation Turing Test

So how did CUBBITT’s supposed outperformance come about? The study defined adequacy as “adequately expressing [the source text’s] intended meaning in the target language.” The assertion that CUBBITT outperformed human translation, therefore, means that human evaluators rated CUBBITT’s translations as representing the source text’s meaning better than the human reference translations: 52% of CUBBITT’s sentences scored higher than the human translations; 26% of CUBBITT translations were scored lower than human translations.

Machine Translation Outperforms Human Translation Claim
Screenshot from paper

Using the same source documents and translations from CUBBITT’s winning performance on the WMT18 news translation task, 15 human evaluators rated the quality of almost 8,000 sentences across 53 documents. Unlike the news translation task, however, evaluators were provided document-level context for the translations. This allowed evaluators to catch errors that might not have been evident without context, such as a gender mismatch or the incorrect translation of an ambiguous expression.

Slator 2021 Data-for-AI Market Report

Slator 2021 Data-for-AI Market Report

Data and Research, Slator reports
44-pages on how LSPs enter and scale in AI Data-as-a-service. Market overview, AI use cases, platforms, case studies, sales insights.
$380 BUY NOW

Compared to the human reference translations, the authors observed that “CUBBITT made significantly fewer errors in addition of meaning, omission of meaning, shift of meaning, other adequacy errors, grammar, and spelling.” On the other hand, CUBBITT made significtantly more errors due to cross-sentence context (as the researchers anticipated), and human translation was still rated as more fluent.

“CUBBITT made significantly fewer errors in addition of meaning, omission of meaning, shift of meaning, other adequacy errors, grammar, and spelling.”

The group also conducted a “sentence-level translation Turing test” by showing evaluators 100 pairs of sentences, each consisting of a source sentence and a translation. Participants then identified each translation as produced by either a human or by MT. CUBBITT translations were less likely to be identified as MT than translations produced by Google Translate.

Contributing to Human-Likeness

“One potential contributor to human-likeness of CUBBITT could be the fact that it is capable of restructuring translated sentences where the English structure would sound unnatural in Czech,” the authors posited, crediting CUBBITT’s training on back-translation data.

To overcome the lack of English–Czech parallel data for training, the researchers used back-translation, translating more widely available monolingual target language data into the source language. The resulting sentence pairs comprise additional synthetic parallel training data, which are traditionally mixed together with authentic sentences in random order.

Slator 2020 Language Industry Market Report

Data and Research, Slator reports
55 pages. Total market size, biz dev and sales insights, TMS & MT review, buyer segment analysis, M&A, Covid impact & outlook.
$480 BUY NOW

CUBBITT is “trained with back-translation data in a novel block regime (block-BT), where the training data are presented to the neural network in blocks of authentic parallel data alternated with blocks of synthetic data.”

The authors noted that back-translation can sometimes have the inadvertent benefit of improving the fluency (and sometimes adequacy) of the final translations, “since the target side in back-translation are authentic sentences originally written in the target language.”

The English–French and English–Polish versions of CUBBITT attained BLEU results consistent with those of the English–Czech version. Document-level evaluations suggest that CUBBITT performs best on articles related to business and politics, and performs the worst on articles about art, entertainment, and sports.

TAGS

back-translationCUBBITTGoogleGoogle TranslateJakob UszkoreitŁukasz KaiserOndřej BojarTransformerWMT
SHARE
Seyma Albarino

By Seyma Albarino

Staff Writer at Slator. Linguist, music blogger and reader of all things dystopian. Based in Chicago after adventures on three continents.

Advertisement

SUBSCRIBE TO THE SLATOR WEEKLY

Language Industry Intelligence
In Your Inbox. Every Friday

SUBSCRIBE

SlatorSweepSlatorPro
ResearchRFP CENTER

PUBLISH

PRESS RELEASEDIRECTORY LISTING
JOB ADEVENT LISTING

Bespoke advisory including speaking, briefings and M&A

SLATOR ADVISORY
Advertisement

Featured Reports

See all
Slator 2020 Language Industry M&A and Funding Report

Slator 2020 Language Industry M&A and Funding Report

by Slator

Slator 2021 Data-for-AI Market Report

Slator 2021 Data-for-AI Market Report

by Slator

Slator 2020 Medtech Translation and Localization Report

Slator 2020 Medtech Translation and Localization Report

by Slator

Pro Guide: Sales and Marketing for Language Service Providers

Pro Guide: Sales and Marketing for Language Service Providers

by Slator

Press Releases

See all
BLEND Raises $10m to Fuel Global Growth with End-to-end Localization Services

BLEND Raises $10m to Fuel Global Growth with End-to-end Localization Services

by BLEND

Iconic Launches INTRA Translation Platform

Iconic Launches INTRA Translation Platform

by Iconic

Pangeanic Is Now Certified to ISO 27001 Information Security

Pangeanic Is Now Certified to ISO 27001 Information Security

by Pangeanic

Upcoming Events

See All
  1. Smartling - Global Ready Conference 2021

    Global Ready Conference

    by Smartling

    · April 14

    When you can't traverse the world, let the world come to you. Join our annual global event from home.

    More info FREE

Featured Companies

See all
Sunyu Transphere

Sunyu Transphere

Text United

Text United

Memsource

Memsource

Wordbank

Wordbank

Protranslating

Protranslating

Seprotec

Seprotec

Versacom

Versacom

Smartling

Smartling

XTM International

XTM International

Translators without Borders

Translators without Borders

STAR Group

STAR Group

memoQ Translation Technologies

memoQ Translation Technologies

Advertisement

Popular articles

Poland Rules on LSP Using Google Translate; Defines ‘Professional Translator’

Poland Rules on LSP Using Google Translate; Defines ‘Professional Translator’

by Marion Marking

The Slator 2021 Language Service Provider Index

The Slator 2021 Language Service Provider Index

by Slator

Behind the Scenes of the European Parliament’s Pivot to Remote Interpreting

Behind the Scenes of the European Parliament’s Pivot to Remote Interpreting

by Seyma Albarino

SlatorPod: The Weekly Language Industry Podcast

connect with us

footer logo

Slator makes business sense of the language services and technology market.

Our Company

  • Support
  • About us
  • Terms & Conditions
  • Privacy Policy

Subscribe to the Slator Weekly

Language Industry Intelligence
In Your Inbox. Every Friday

© 2021 Slator. All rights reserved.

Sign up to the Slator Weekly

Join over 13,500 subscribers and get the latest language industry intelligence every Friday

Your information will never be shared with third parties. No Spam.