logo image
  • News
    • People Moves
    • Deal Wins
    • Demand Drivers
    • M&A and Funding
    • Financial Results
    • Technology
    • Academia
    • Industry News
    • Features
    • Machine Translation
    • — Divider —
    • Slator Pro
    • — Divider —
    • Press Releases
    • Sponsored Content
  • Data & Research
    • Research Reports & Pro Guides
    • Language Industry Investor Map
    • Real-Time Charts of Listed LSPs
    • Language Service Provider Index
  • Podcasts & Videos
  • Events
    • Design Thinking – February 2021
    • — Divider —
    • SlatorCon Coverage
    • Other Events
  • Directory
  • RFP Center
  • Jobs
MENU
  • News
    • People Moves
    • Deal Wins
    • Demand Drivers
    • M&A and Funding
    • Financial Results
    • Technology
    • Academia
    • Industry News
    • Features
    • Machine Translation
    • — Divider —
    • Slator Pro
    • — Divider —
    • Press Releases
    • Sponsored Content
  • Data & Research
    • Research Reports & Pro Guides
    • Language Industry Investor Map
    • Real-Time Charts of Listed LSPs
    • Language Service Provider Index
  • Podcasts & Videos
  • Events
    • Design Thinking – February 2021
    • — Divider —
    • SlatorCon Coverage
    • Other Events
  • Directory
  • RFP Center
  • Jobs

Advertise on Slator! Download the 2021 Online Media Kit Now

  • Slator Market Intelligence
  • Slator Advertising Services
  • Slator Advisory
  • Login
Search
Generic filters
Exact matches only
Advertisement
And the Winner Is…ACL 2020 Announces Best Paper Awards

7 months ago

July 9, 2020

And the Winner Is…ACL 2020 Announces Best Paper Awards

Academia ·

by Marion Marking

On July 9, 2020

7 months ago
Academia ·

by Marion Marking

On July 9, 2020

And the Winner Is…ACL 2020 Announces Best Paper Awards

The Annual Meeting of the Association for Computational Linguistics (ACL) announced the Best Paper Awards of 2020 via Twitter on July 8.

ACL is also the organizer of the world’s largest natural language processing conference, EMNLP. ACL’s Annual Meeting covers a range of research areas on computational approaches to natural language, drawing thousands of research papers from across the globe. And like all conferences, ACL, has also gone remote in 2020.

The ACL 2020 conference committee narrowed down this year’s 3,088 submissions, selecting 779 papers (571 of them long, 208, short); an acceptance rate of 25.2%.

Advertisement

Best Overall Paper went to Beyond Accuracy: Behavioral Testing of NLP Models with CheckList by Marco Tulio Ribeiro of Microsoft Research; Tongshuang Wu of the University of Washington; Carlos Guestrin and Sameer Singh of the University of California, Irvine.

Among the two Honorable Mentions for Best Overall Paper was one that, in the words of the paper’s authors, “adds to the case for retiring BLEU as the de facto standard metric.” This sentiment echoes that of other experts who, as previously mentioned, believe that BLEU is fast becoming useless.

Slator 2020 How to Run a Translation and Localization RFP - Procurement

Pro Guide: How to Run a Translation and Localization RFP

Data and Research, Slator reports
25 pages. Actionable guidance for translation and localization buyers on how to qualify vendors and streamline procurement.
$375 BUY NOW

In fact, although still widely used in academia, BLEU’s reliance on reference text makes it impractical for industry use, as NVIDIA’s Senior Deep Learning Engineer, Chip Huyen told the SlatorCon audience last fall.

In their ACL 2020 winning paper, Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metrics, University of Melbourne researchers Nitika Mathur, Timothy Baldwin, and Trevor Cohn show that “current methods for judging metrics are highly sensitive to the translations used for assessment, particularly the presence of outliers, which often leads to falsely confident conclusions about a metric’s efficacy.”

The Unimelb trio found that outlier systems (or those whose quality is much higher or lower than the rest of the systems), can have “a disproportionate effect on the computed correlation of metrics,” such that “the resulting high values of correlation can […] lead to false confidence in the reliability of metrics.”

When the outliers are removed, they said, “the gap between correlation of BLEU and other ‘more powerful’ metrics (e.g., CHRF, YISI-1, and ESIM) becomes wider. In the worst case scenario, outliers introduce a high correlation when there is no association between metric and human scores for the rest of the systems. Thus, future evaluations should also measure correlations after removing outlier systems.”

Slator 2020 Language Industry Market Report

Data and Research, Slator reports
55 pages. Total market size, biz dev and sales insights, TMS & MT review, buyer segment analysis, M&A, Covid impact & outlook.
$480 BUY NOW

It is time to retire BLEU, the authors concluded, and use instead other metrics such as CHRF, YISI-1, or ESIM because “they are more powerful in assessing empirical improvements.”

They end by saying that “human evaluation must always be the gold standard, and for continuing improvement in translation, to establish significant improvements over prior work, all automatic metrics make for inadequate substitutes.”

The other Honorable Mention for Best Overall Paper was Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks by researchers from the Allen Institute for Artificial Intelligence and the University of Washington.After holding its 57th annual meeting in Florence, Italy, last year, ACL originally planned to hold this year’s conference in Seattle, Washington, but moved online due to the Covid-19 pandemic. ACL’s Twitter page (@aclmeeting) was abuzz on the days of the virtual conference, July 5 – 10, 2020, with live tweeters providing updates in many languages.

TAGS

ACLACL 2020ACL2020BLEUBLEU scoreCHRFESIMNitika MathurTimothy BaldwinTrevor CohnUniversity of MelbourneYISI-1
SHARE
Marion Marking

By Marion Marking

Slator consultant and corporate communications professional who enjoys exploring Asian cities.

Advertisement

SUBSCRIBE TO THE SLATOR WEEKLY

Language Industry Intelligence
In Your Inbox. Every Friday

SUBSCRIBE

SlatorSweepSlatorPro
ResearchRFP CENTER

PUBLISH

PRESS RELEASEDIRECTORY LISTING
JOB ADEVENT LISTING

Bespoke advisory including speaking, briefings and M&A

SLATOR ADVISORY
Advertisement

Featured Reports

See all
Slator 2020 Language Industry M&A and Funding Report

Slator 2020 Language Industry M&A and Funding Report

by Slator

Slator 2021 Data-for-AI Market Report

Slator 2021 Data-for-AI Market Report

by Slator

Slator 2020 Medtech Translation and Localization Report

Slator 2020 Medtech Translation and Localization Report

by Slator

Pro Guide: Sales and Marketing for Language Service Providers

Pro Guide: Sales and Marketing for Language Service Providers

by Slator

Press Releases

See all
Donna Thomas Joins Visual Data Media Services as Senior Vice President of Sales, Americas

Donna Thomas Joins Visual Data Media Services as Senior Vice President of Sales, Americas

by Visual Data Media Services

iDISC Awarded ISO 27001 Information Security Management Certification

iDISC Awarded ISO 27001 Information Security Management Certification

by iDISC

XTRF Launches a Bi-Monthly Free Networking Event for Localization Professionals

XTRF Launches a Bi-Monthly Free Networking Event for Localization Professionals

by XTRF

Upcoming Events

See All
  1. Memsource MT Post-Editing Pricing Models Webinar

    Pricing Models for MT Post-Editing Workshop

    by Memsource

    · February 3

    Hear a panel of innovative localization professionals share different approaches for MT post-editing pricing.

    More info FREE

Featured Companies

See all
Text United

Text United

Memsource

Memsource

Wordbank

Wordbank

Protranslating

Protranslating

Seprotec

Seprotec

Versacom

Versacom

SDL

SDL

Smartling

Smartling

Lingotek

Lingotek

XTM International

XTM International

Smartcat

Smartcat

Translators without Borders

Translators without Borders

STAR Group

STAR Group

memoQ Translation Technologies

memoQ Translation Technologies

Advertisement

Popular articles

Why Netflix Shut Down Its Translation Portal Hermes

Why Netflix Shut Down Its Translation Portal Hermes

by Esther Bond

The Slator 2020 Language Service Provider Index

The Slator 2020 Language Service Provider Index

by Slator

Top Language Industry Quotes of 2020

Top Language Industry Quotes of 2020

by Monica Jamieson

SlatorPod: The Weekly Language Industry Podcast

connect with us

footer logo

Slator makes business sense of the language services and technology market.

Our Company

  • Support
  • About us
  • Terms & Conditions
  • Privacy Policy

Subscribe to the Slator Weekly

Language Industry Intelligence
In Your Inbox. Every Friday

© 2021 Slator. All rights reserved.

Sign up to the Slator Weekly

Join over 13,000 subscribers and get the latest language industry intelligence every Friday

Your information will never be shared with third parties. No Spam.