logo image
  • News
    • People Moves
    • Deal Wins
    • Demand Drivers
    • M&A and Funding
    • Financial Results
    • Technology
    • Academia
    • Industry News
    • Features
    • Machine Translation
    • — Divider —
    • Slator Pro
    • — Divider —
    • Press Releases
    • Sponsored Content
  • Data & Research
    • Research Reports & Pro Guides
    • Language Industry Investor Map
    • Real-Time Charts of Listed LSPs
    • Language Service Provider Index
  • Podcasts & Videos
  • Events
    • SlatorCon Remote May 2021
    • Email Marketing for Freelance Linguists
    • Preparing for the Critical Google Update Coming in May 2021
    • — Divider —
    • SlatorCon Coverage
    • Other Events
  • Directory
  • RFP Center
  • Jobs
MENU
  • News
    • People Moves
    • Deal Wins
    • Demand Drivers
    • M&A and Funding
    • Financial Results
    • Technology
    • Academia
    • Industry News
    • Features
    • Machine Translation
    • — Divider —
    • Slator Pro
    • — Divider —
    • Press Releases
    • Sponsored Content
  • Data & Research
    • Research Reports & Pro Guides
    • Language Industry Investor Map
    • Real-Time Charts of Listed LSPs
    • Language Service Provider Index
  • Podcasts & Videos
  • Events
    • SlatorCon Remote May 2021
    • Email Marketing for Freelance Linguists
    • Preparing for the Critical Google Update Coming in May 2021
    • — Divider —
    • SlatorCon Coverage
    • Other Events
  • Directory
  • RFP Center
  • Jobs

Register For Email Marketing for Freelance Linguists and Learn How To Win New Clients.

  • Slator Market Intelligence
  • Slator Advertising Services
  • Slator Advisory
  • Login
Search
Generic filters
Exact matches only
Advertisement
Amazon Pits Neural Machine Translation Framework Against Google, Facebook, Others

3 years ago

December 20, 2017

Amazon Pits Neural Machine Translation Framework Against Google, Facebook, Others

Technology ·

by Gino Diño

On December 20, 2017

3 years ago
Technology ·

by Gino Diño

On December 20, 2017

Amazon Pits Neural Machine Translation Framework Against Google, Facebook, Others

Following Amazon’s announcement of its own neural machine translation (NMT) offering, the company’s machine learning scientists published a paper on research repository Arxiv.org that details the inner workings of Sockeye, their open-source, sequence-to-sequence toolkit for NMT.

They also showed off benchmark performance comparisons between Sockeye and popular, open-source NMT toolkits using the Conference on Machine Translation 2017 datasets (WMT17) and results.

Amazon unexpectedly singled out language service providers (LSPs) as potential buyers during its late-November launch announcement. Sockeye, however, was introduced several months prior, on July 2017.

Advertisement

Open Sourcing Neural

The Sockeye announcement in July came on the heels of Google’s own July 2017 pitch for TensorFlow, the advanced machine learning model the search giant uses for Google Translate.

In a blog post, Google recognized MT as an active area of research where there was “a lack of material that teaches people both the knowledge and the skills to easily build high-quality translation systems.”

Amazon’s own open-source pitch for Sockeye is pretty much the same, and in its blog post AWS gave readers an idea idea how Sockeye works and how to install, train, and translate right away given a parallel corpus of one million pairs of translated sentences from WMT2017.

And what about the actual paper submitted to Arxiv months after the fact showing off Sockeye’s performance compared to currently popular NMT systems?

“Such papers are necessary when releasing a toolkit that you want to gain wider adoption within the community,” said Dr. John Tinsley, MT expert and CEO of Iconic Translation Machines.

Amazon Flexing Its Muscles

The December 2017 paper reported Sockeye’s performance as a production-ready framework that can train and apply any of the three most-used NMT models currently. These are recurrent neural nets with attention mechanisms (e.g. Harvard and Systran’s system used by Booking.com), self-attentional transformers (such as Google’s TensorFlow), and convolutional neural networks (Facebook’s Fairseq).

In a sort of delayed entry to WMT2017, the paper compares Sockeye’s performance against the WMT 2017 newstest evaluation set results on two language directions: English to German and Latvian to English. The researchers used BLEU (Bilingual Evaluation Understudy) to score performance.

Tuning Sockeye to use recurrent neural nets with an attention mechanism, Amazon researchers claim the system beat both the Lua and Python implementations of OpenNMT, the Torch-based NMT toolkit developed jointly by Systran and Harvard University.

However, Sockeye could not displace top scorers Nematus and Marian, NMT systems both developed by the University of Edinburgh.

Next, the researchers put a self-attentional transformer model on top of the Sockeye framework. They reportedly managed to barely edge out two other transformer systems: Tensor2Tensor, a sequence-to-sequence model based on Google’s Tensorflow, and Marian, again from the University of Edinburgh.

Finally, the researchers also used convolutional neural nets on Sockeye, reportedly besting the framework used by the Facebook Artificial Intelligence Research (FAIR) team, Fairseq.

“What this paper shows is that Amazon have certainly developed a very competitive framework, which is not surprising given the breadth of MT talent they have in-house,” Tinsley said.

He noted, however, that at the pace NMT has developed over the past year, “the dust hasn’t yet settled on which tool might ultimately become the standard (a la Moses in SMT). We’re some time from that point yet.”

Scored with a Questionable Yardstick

Amazon’s researchers might claim Sockeye is competitive to current NMT models, but it is worth noting that their basis—BLEU scores—offer a limited view of actual translation fluency.

“Human assessments are the only reliable yardstick,” said Tinsley. “People continue to report BLEU scores nearly because they have to, despite mounting evidence that they’re less effective with NMT.”

In an December 2016 article about BLEU as a metric, Tinsley suggested other existing scoring systems better suited for NMT, such as METEOR, TER (Translation Edit Rate), and GTM (General Text Matcher).

He acknowledged that while it is best that everything undergo human assessment, “that doesn’t work / scale in the research scenario.” He did remain hopeful that new, more applicable metrics will emerge out of research soon.

“At the end of the day, something more akin to confidence estimation might be ultimately more effective for NMT as opposed to comparison against reference translations,” Tinsley said.

Regarding his overall impression of Amazon’s push into neural, Tinsley said it did not come as a surprise given Amazon’s 2015 Safaba acquisition and their move to build a Pittsburgh office.

“While initial focus might have been on their internal translation needs, it’s certainly interesting to see how they’re positioning the service,” Tinsley said. “It appears to be closer to Microsoft Translator rather than Google, in that they’re aiming at business use cases, but it’s still an out-of-the-box offering with a limited number of languages and no ability to customise.”

“It remains to be seen how far they go with it,” he concluded.

Download the Slator 2019 Neural Machine Translation Report for the latest insights on the state-of-the art in neural machine translation and its deployment.

Slator 2019 Neural Machine Translation Report: Deploying NMT in Operations

Data and Research
32 pages, NMT state-of-the-art, 5 case studies, 30 commentaries, NMT in day-to-day operations
$85 BUY NOW

TAGS

AmazonFacebookGoogleneural machine translationOpenNMTSockeyeTensorFlowWMT
SHARE
Gino Diño

By Gino Diño

Content strategy expert and Online Editor for Slator; father, husband, gamer, writer―not necessarily in that order.

Advertisement

SUBSCRIBE TO THE SLATOR WEEKLY

Language Industry Intelligence
In Your Inbox. Every Friday

SUBSCRIBE

SlatorSweepSlatorPro
ResearchRFP CENTER

PUBLISH

PRESS RELEASEDIRECTORY LISTING
JOB ADEVENT LISTING

Bespoke advisory including speaking, briefings and M&A

SLATOR ADVISORY
Advertisement

Featured Reports

See all
Slator 2020 Language Industry M&A and Funding Report

Slator 2020 Language Industry M&A and Funding Report

by Slator

Slator 2021 Data-for-AI Market Report

Slator 2021 Data-for-AI Market Report

by Slator

Slator 2020 Medtech Translation and Localization Report

Slator 2020 Medtech Translation and Localization Report

by Slator

Pro Guide: Sales and Marketing for Language Service Providers

Pro Guide: Sales and Marketing for Language Service Providers

by Slator

Press Releases

See all
Iconic Launches INTRA Translation Platform

Iconic Launches INTRA Translation Platform

by Iconic

Pangeanic Is Now Certified to ISO 27001 Information Security

Pangeanic Is Now Certified to ISO 27001 Information Security

by Pangeanic

VSI Acquires Leading Brazilian Dubbing Studio, Vox Mundi

VSI Acquires Leading Brazilian Dubbing Studio, Vox Mundi

by VSI

Upcoming Events

See All
  1. Smartling - Global Ready Conference 2021

    Global Ready Conference

    by Smartling

    · April 14

    When you can't traverse the world, let the world come to you. Join our annual global event from home.

    More info FREE

Featured Companies

See all
Sunyu Transphere

Sunyu Transphere

Text United

Text United

Memsource

Memsource

Wordbank

Wordbank

Protranslating

Protranslating

Seprotec

Seprotec

Versacom

Versacom

Smartling

Smartling

XTM International

XTM International

Translators without Borders

Translators without Borders

STAR Group

STAR Group

memoQ Translation Technologies

memoQ Translation Technologies

Advertisement

Popular articles

Poland Rules on LSP Using Google Translate; Defines ‘Professional Translator’

Poland Rules on LSP Using Google Translate; Defines ‘Professional Translator’

by Marion Marking

Behind the Scenes of the European Parliament’s Pivot to Remote Interpreting

Behind the Scenes of the European Parliament’s Pivot to Remote Interpreting

by Seyma Albarino

Why Netflix Shut Down Its Translation Portal Hermes

Why Netflix Shut Down Its Translation Portal Hermes

by Esther Bond

SlatorPod: The Weekly Language Industry Podcast

connect with us

footer logo

Slator makes business sense of the language services and technology market.

Our Company

  • Support
  • About us
  • Terms & Conditions
  • Privacy Policy

Subscribe to the Slator Weekly

Language Industry Intelligence
In Your Inbox. Every Friday

© 2021 Slator. All rights reserved.

Sign up to the Slator Weekly

Join over 13,500 subscribers and get the latest language industry intelligence every Friday

Your information will never be shared with third parties. No Spam.