logo image
  • News
    • People Moves
    • Deal Wins
    • Demand Drivers
    • M&A and Funding
    • Financial Results
    • Technology
    • Academia
    • Industry News
    • Features
    • Machine Translation
    • — Divider —
    • Slator Pro
    • — Divider —
    • Press Releases
    • Sponsored Content
  • Data & Research
    • Research Reports & Pro Guides
    • Language Industry Investor Map
    • Real-Time Charts of Listed LSPs
    • Language Service Provider Index
  • Podcasts & Videos
  • Events
    • Design Thinking – February 2021
    • — Divider —
    • SlatorCon Coverage
    • Other Events
  • Directory
  • RFP Center
  • Jobs
MENU
  • News
    • People Moves
    • Deal Wins
    • Demand Drivers
    • M&A and Funding
    • Financial Results
    • Technology
    • Academia
    • Industry News
    • Features
    • Machine Translation
    • — Divider —
    • Slator Pro
    • — Divider —
    • Press Releases
    • Sponsored Content
  • Data & Research
    • Research Reports & Pro Guides
    • Language Industry Investor Map
    • Real-Time Charts of Listed LSPs
    • Language Service Provider Index
  • Podcasts & Videos
  • Events
    • Design Thinking – February 2021
    • — Divider —
    • SlatorCon Coverage
    • Other Events
  • Directory
  • RFP Center
  • Jobs

Advertise on Slator! Download the 2021 Online Media Kit Now

  • Slator Market Intelligence
  • Slator Advertising Services
  • Slator Advisory
  • Login
Search
Generic filters
Exact matches only
Advertisement
Researchers Explore Surprising Behavior of Machine Translation

8 months ago

June 3, 2020

Researchers Explore Surprising Behavior of Machine Translation

Machine Translation ·

by Esther Bond

On June 3, 2020

8 months ago
Machine Translation ·

by Esther Bond

On June 3, 2020

Researchers Explore Surprising Behavior of Machine Translation

Claims that machine translation had achieved near-human parity (however it is defined) back in 2016 were met with disbelief. The technology was still far from being able to produce quality equivalent to that of human translators and it was the metrics that were flawed, people were quick to point out.

Love it or hate it, neural machine translation (NMT) became widely adopted across the language industry in the years that followed. It has fundamentally changed the supply chain and disrupted the way humans interact with translation technology, generating significant productivity gains for users.

NMT now underpins parts of the translation workflow, but relatively little is known about how the machine actually understands content or generates output, and why some of the residing quality issues persist.

Advertisement

Two researchers have now shone a light on some of the oddities found in NMT output, exploring unexpected behavior in RNN and Transformer NMT models. In a paper published on pre-print platform arXiv on May 25, 2020, Marzieh Fadaee and Christof Monz from the University of Amsterdam looked into “The Unreasonable Volatility of Neural Machine Translation Models.”

RNNs (Recurrent Neural Networks) are a type of artificial neural network, while Transformer is a deep machine learning model that was introduced by Google researchers in 2017. The latter is the newer and now more prevalent architecture used in machine translation and speech processing.

Fadaee was a PhD candidate at the university and has since become an NLP / ML Research Engineer at deep learning R&D lab Zeta Alpha Vector. Monz, who remains Associate Professor, describes his research interests as covering “information retrieval, document summarization and machine translation” on his LinkedIn page.

Slator 2020 Language Industry Market Report

Data and Research, Slator reports
55 pages. Total market size, biz dev and sales insights, TMS & MT review, buyer segment analysis, M&A, Covid impact & outlook.
$480 BUY NOW

The basis for their research is that, although NMT performs well, it is not generally understood how the models behave. Examining the unexpected behavior of NMT could reveal more about its capabilities as well as shortcomings.

During their research, Fadaee and Monz observed that minor changes to the source sentences sometimes resulted in an “unexpected change in the translation,” which in some cases constituted a translation error. Since the models behaved inconsistently when confronted with similar source sentences, they are considered “volatile,” the two explained.

Important to note is that all source sentences, including modified ones, were semantically correct and plausible for the purposes of their experiments.

The researchers performed a series of tests to analyze the translations of the modified source sentences and the types of changes that occurred.

Important to note is that all source sentences, including modified ones, were semantically correct and plausible for the purposes of their experiments. The changes the researchers made to source sentences were minor and limited to the following: removing adverbs, changing numbers (by a maximum of plus five), and inserting common words. They also changed gender pronouns, having been inspired by prior work on gender bias.

One test applied only to changes to numbers in source sentences. For this category of change, it was possible to have multiple variations of the original source sentence (e.g., +1, +2, +3, +4 and +5). Logically, the translations of the changed sentences should only differ to account for the change in number, but researchers found examples of “unexpectedly large oscillations” for both models.

They also looked at deviations from the original translation and classified them as major or minor deviations. The results showed major differences in 18% of RNN translations and 13% of Transformer translations.

Most of the deviations (ca. 70%) were “as expected,” meaning that they were justified by the change to the original source sentence, while unexpected changes included different verb tenses, reordered phrases, paraphrasing, preposition changes, and more. “The vast majority of changes are due to paraphrasing and dropping of words,” the researchers found. Unexpected changes did not necessarily impact translation quality.

SlatorSweep - Daily Market Intelligence

SlatorSweep

Data and Research, Market Intelligence
Curated news from thousands of sources, SlatorSweep’s daily news service gives you a competitive edge on time sensitive market intelligence.
BUY NOW

Translation quality was tested separately through a manual evaluation by human annotators. Overall, 26% of changes observed for the RNN model impacted translation quality, compared to 19% of those observed for the Transformer model.

In conclusion, the researchers said, “even with trivial linguistic modifications of source sentences, we can effectively identify a surprising number of cases where the translations of extremely similar sentences are surprisingly different.” This means that NMT models are vulnerable to the slightest change in the source sentence, which points to two other potential shortcomings: generalization and compositionality.

Generalization refers to an MT system being able to translate long source sentences that it has not previously encountered. Compositionality is where an MT system combines multiple, simple sentence parts to build a longer, more complex string.

In their view, “the volatile behavior of the MT systems in this paper is a side effect of the current models not being compositional” because the systems clearly do not demonstrate a good understanding of the underlying sentence parts — if they did, they would not generate the inconsistencies observed.

Moreover, Fadaee and Monz said, while NMT models are capable of generalization, they do so without compositionality. As such, the researchers argued that NMT models “lack robustness” and hoped that their “insights will be useful for developing more robust NMT models.”

TAGS

Christof Monzmachine translationMarzieh Fadaeeneural machine translationRNNTransformer
SHARE
Esther Bond

By Esther Bond

Research Director at Slator. Localization enthusiast, linguist and inquisitor. London native.

Advertisement

SUBSCRIBE TO THE SLATOR WEEKLY

Language Industry Intelligence
In Your Inbox. Every Friday

SUBSCRIBE

SlatorSweepSlatorPro
ResearchRFP CENTER

PUBLISH

PRESS RELEASEDIRECTORY LISTING
JOB ADEVENT LISTING

Bespoke advisory including speaking, briefings and M&A

SLATOR ADVISORY
Advertisement

Featured Reports

See all
Slator 2020 Language Industry M&A and Funding Report

Slator 2020 Language Industry M&A and Funding Report

by Slator

Slator 2021 Data-for-AI Market Report

Slator 2021 Data-for-AI Market Report

by Slator

Slator 2020 Medtech Translation and Localization Report

Slator 2020 Medtech Translation and Localization Report

by Slator

Pro Guide: Sales and Marketing for Language Service Providers

Pro Guide: Sales and Marketing for Language Service Providers

by Slator

Press Releases

See all
Across Systems will be part of the Volaris Group

Across Systems will be part of the Volaris Group

by Across Systems GmbH

How Localex Made It Through the Pandemic

How Localex Made It Through the Pandemic

by Localex

Join Us for the First Virtual Together 2021 Next Month!

Join Us for the First Virtual Together 2021 Next Month!

by Elia

Upcoming Events

See All
  1. Handling Sensitive Information Webinar

    Handling Sensitive Calls with Limited English Proficient Consumers

    by Lionbridge

    · February 10

    Learn more about how Lionbridge Over-the-Phone Interpretation Services can help bridge communication gaps with limited...

    More info FREE

Featured Companies

See all
Text United

Text United

Memsource

Memsource

Wordbank

Wordbank

Protranslating

Protranslating

Seprotec

Seprotec

Versacom

Versacom

SDL

SDL

Smartling

Smartling

Lingotek

Lingotek

XTM International

XTM International

Smartcat

Smartcat

Translators without Borders

Translators without Borders

STAR Group

STAR Group

memoQ Translation Technologies

memoQ Translation Technologies

Advertisement

Popular articles

Why Netflix Shut Down Its Translation Portal Hermes

Why Netflix Shut Down Its Translation Portal Hermes

by Esther Bond

Top Language Industry Quotes of 2020

Top Language Industry Quotes of 2020

by Monica Jamieson

The Most Popular Language Industry Stories of 2020

The Most Popular Language Industry Stories of 2020

by Seyma Albarino

SlatorPod: The Weekly Language Industry Podcast

connect with us

footer logo

Slator makes business sense of the language services and technology market.

Our Company

  • Support
  • About us
  • Terms & Conditions
  • Privacy Policy

Subscribe to the Slator Weekly

Language Industry Intelligence
In Your Inbox. Every Friday

© 2021 Slator. All rights reserved.

Sign up to the Slator Weekly

Join over 13,000 subscribers and get the latest language industry intelligence every Friday

Your information will never be shared with third parties. No Spam.