Slator attended this year’s EMNLP conference, held in Brussels, Belgium between October 31 and November 4, 2018. EMNLP stands for Empirical Methods in Natural Language Processing and is organised by the Association for Computational Linguistics (ACL).
The conference has been running annually for over 20 years and attracts participants across the natural language processing research space. On the back of a boom in anything related to artificial intelligence the numbers of participants ballooned to 2,500, more than double the number from 2017. Researchers from across the globe gathered to share insights from the latest research spanning the breadth of NLP.
Each year in the run up to the conference, researchers are asked to submit papers to the conference reviewers for consideration. EMNLP 2018 solicited papers on a range of topics, among them language models, spoken language processing, text mining, natural language generation, machine translation and machine learning for NLP.
Slator 2021 Data-for-AI Market Report
Google AI posterOf the 2,231 long and short papers submitted in 2018, just under a quarter (549) were accepted. The authors of accepted papers were invited to showcase their research as either a talk (221) or a poster (328). Posters, containing research highlights from a chosen paper, were displayed on a collection of stands in the main hall. Conference attendees were able to browse the selection and discuss the topic of the poster with its author(s).
The geographic spread of conference submissions showed strong participation from Asia (42.5%). 32.1% of submissions came from North America and 20.9% from Europe, with other submissions coming from Latin America, Africa, the Middle East and Oceania.
Asia and the US also had a hefty sponsorship presence at the EMNLP 2018 conference, while European sponsors were in noticeably short supply.
The many conference sponsors included Google, Facebook, Apple, Amazon, Microsoft, Bloomberg, Baidu, Grammarly, ebay, Oracle, Yitu, Sogou, Duolingo and more. There were, in fact, two European companies on the rosta. For the big tech companies, EMNLP is not only an opportunity to share their own latest NLP research findings, but also to actively scout talent from among the conference attendees: research and development professionals from the academic research community, AI startups and big tech.
A Problem Shared…
Machine translation is not a small part of the EMNLP mega conference and has its own conference stream, called the Conference on Machine Translation (WMT). WMT started out in 2006 as a series of workshops offered at EMNLP and became a full blown conference in its own right in 2016. EMNLP Organizer ACL was in fact originally named the Association for Machine Translation and Computational Linguistics (AMTCL) and changed name in 1962, six years after its founding.
Alibaba posterEMNLP 2018 hosted the Third Conference of Machine Translation (WMT18). All accepted papers are included in the thousand-page conference proceedings.
Many of the talks and posters focused on reporting results from a “shared task,” a predefined problem that a group of researchers attempt to solve by approaching the task in different ways. There were seven shared tasks announced in preparation for WMT18:
- a news translation task
- a biomedical translation task
- a multimodal translation task
- a metrics task (assess MT quality given reference translation)
- a quality estimation task (assess MT quality without access to any reference)
- an automatic post-editing task
- a parallel corpus filtering task
The headline task, a news translation shared task, received 103 submissions from 32 institutions. Given its popularity and status as the “premier shared task,” the news translation task was covered in the Findings of the 2018 Conference on Machine Translation paper. For the purposes of the shared task, the 32 institutions were organized into 35 different teams.
The task involved building machine translation systems between English and any of seven languages in both directions (Chinese, Czech, Estonian, German, Finnish, Russian, and Turkish). The machine translation output was later evaluated by humans against a test set that had been translated by native, professional translators, selected for their knowledge of the domain.
The test set was made up of around 3,000 sentences per language pair. 1,500 English source sentences were translated into each of the other languages and 1,500 sentences were translated into English from each of the other languages. The test set was larger for Estonian since it was a new language pair for 2018. For each of the 14 language pairs, one translator translated the sentences while a second translator evaluated a sample of the work and scored the first translator.
The machine translation engine output was also evaluated by humans, 915 crowd-sourced workers and 584 researchers, through direct assessment (DA) of the translation quality.The evaluators were asked to indicate to what extent, out of 100, the sentence translated by the machine translation system expressed the meaning of the sentence translated by the human translator.
Among the named participants were mystery contributions from five online machine translation services, covering 39 language pairs. Three of these anonymous online services featured in the top three rankings. ONLINE-B scooped the most top three rankings of any participating MT system, coming in first, second or third for seven of the fourteen language pairs. Each system did not necessarily appear in all translation tasks. The top three ranked teams per language pair were as follows:
One MT system that did not quite make the list (it placed fourth on the metric used below but technically tied for second place according to another metric) warrants honorable mention is ModernMT’s production engine, the machine translation engine co-developed by Translated.net, which made it into the second tier for the English to German language pair. Translated.net CEO Marco Trombetti posted on LinkedIn in celebration of the team’s achievement: “What for me is impressive is that 1) this was the first time MMT participated. 2) MMT did not submit a research prototype but its current enterprise product based on software already available on MMT github to everyone.”
Not only did Translated.net submit its enterprise machine translation product for the shared task, but the company also translated the Czech and German test sets to and from English.
The paper concludes that, “in addition to highlighting the best-performing systems in each of the 14 examined translation directions, the results indicate that for some language pairs, the state of the art in machine translation is very close to the performance of human translators.” However, the paper continues, “the style of evaluation (DA for individual sentences) has to be carefully considered before making any strong claims.”
The News Translation shared task was part-funded by the European Union’s Horizon 2020 research and innovation programme and the Connecting Europe Facility under grant agreements, according to the paper covering the Findings of the News Translation shared task.
NMT Research on the Up
Natural language processing has become an increasingly active area of research, and machine translation research within the wider NLP field is booming. Slator routinely monitors neural machine translation (NMT) research activity by tracking the number of NMT papers submitted to research portal arXiv. There is an unmistakable upward trend, with Microsoft, Google, Amazon and Facebook all avid contributors.
As neural machine translation research shows no signs of slowing and attracts more interest from big tech names in 2018, adoption and application of NMT throughout the localization supply chain is also becoming more mature, and will continue to impact on productivity and pricing, changing the nature of the language industry landscape.
Download the Slator 2019 Neural Machine Translation Report for the latest insights on the state-of-the art in neural machine translation and its deployment.
Your information will not be shared with third parties. No Spam.