Machine Translation Researchers Set to Find Out Whose System Is Best

In early September 2017, hundreds of machine translation researchers will descend on the Danish capital of Copenhagen. They come to find out whose machine translation system is best at the Second Conference on Machine Translation (WMT17).

Competition is fierce. In 2016, nearly 200 MT systems from two dozen institutions were submitted and evaluated in 10 different so-called tasks.

WMT has been held annually since 2006, first in the form of annual workshops and, with the recent boom in MT research, the 2016 forum was upgraded to a conference.

Advertisement

WMT17 is piggybacking on the larger Conference on Empirical Methods in Natural Language Processing (EMNLP 2017), which is held from September 7-11, 2017, and is organized by the Association for Computational Linguistics.

The surge in interest in artificial intelligence and sub-domains such as natural language processing is proving to be a boon for the organizers, who say registrations are already 150% over their original estimate.

To find out whose system is best, WMT17 features eight shared tasks for researchers to work on, including tasks in news translation, biomedical translation, automatic post editing, and training of neural MT (NMT) systems. In fact, 2017 is the first year researchers submit an NMT task. In 2016, NMT was still only mentioned as a “topic of interest.”

Preliminary results of the news translation shared task are already available, both for manual evaluation and automatic evaluation. Results on automatic evaluations performed by some of the organizers have also been released. Furthermore, a draft version of the conference proceedings has been made available online. Final results for all shared tasks will be announced during the conference.

Among the lead organizers of WMT 2017 are Philipp Koehn, Professor at John Hopkins University’s Language & Speech Processing Group and Chief Scientist for Omniscien Technologies; and Barry Haddow, Senior Research Fellow at the University of Edinburgh.

Among the 21 organizers listed are representatives from Heidelberg University, the University of Edinburgh, and the Fondazione Bruno Kessler.

Roughly three-quarters of the 115 researchers on WMT’s program committee hail from academia with the University of Edinburgh sending seven, the University of Amsterdam sending six, and the Karlsruhe Institute of Technology four researchers.

University of Zurich’s Martin Volk, who recently secured a research grant for his NMT project, will be present, as will New York University’s Kyunghyun Cho, one of the speakers at the upcoming SlatorCon New York.

Big technology companies have participated at the WMT for years and in 2017 companies like IBM, Google, Amazon, Microsoft are well represented on the program committee. Conspicuously absent is Facebook, which in early August 2017 announced it had fully transitioned to neural MT.

The results published from WMT have become a crucial benchmark for academic researchers and research teams of large technology companies to evaluate their systems.

Commercial machine translation providers tend to stay away from competing in WMT, however, because it involves months of preparation and the outcome is uncertain. It would not make for good marketing if your system is ranked in the bottom half.

Still, commercial providers would use WMT to benchmark their systems and use the resulting BLEU scores to do a sanity check when developing a system for a new language combination, for example.

In the main EMNLP conference that follows the WMT, conference sessions on machine translation, especially neural machine translation will figure prominently. These sessions will run alongside discussions on machine learning, multilingual natural language processing (NLP), NLP applications, and linguistic theory, to name only a few.

Editor’s Note: This article has been updated to include preliminary results of the shared task on news translation. Hat tip to Tilde Researcher Mārcis Pinnis for bringing this to our attention.