How Reliable is Google Translate’s ‘Reviewed’ Badge

How Reliable is Google Translate's Reviewed Badge

To help support coverage of its 133 languages, Google Translate appeals to users and polyglot volunteers to provide review and validation services of the machine translation (MT) engine’s output as well as to translate words and phrases themselves.

The Translate Community first appeared in 2014 when Google announced in a blog post that users could rate existing translations, generate new translations, and send feedback. “We plan to incorporate your corrections and over time learn your language a little bit better”, Google stated.

Before 2014, it was possible to rate translations as ‘Helpful’, ‘Unhelpful’, or ‘Offensive’, but now, there are three ways to “Contribute to improve translation quality for your language”. And it could not be easier to do so.

Once on the Google Translate webpage, users simply click on the ‘Contribute’ icon, enter the languages they know or would like to work with, and select which of the three contribution tasks they want to do.

How Does it Work?

For each of the three ways to contribute, there is a ‘Flag’ button, presumably to flag the source, although this is unclear. The first way to contribute is to ‘Rate formality for words and phrases’. There are five formality levels to choose from for each phrase: very informal, informal, neutral, formal, or very formal. 

The second option, ‘Validate translations for accuracy’, requires users to simply state whether it is correct or not.

Google Translate - Validate translations for accuracy

Finally, users can ‘Translate words and phrases’. Google offers no information other than the source language phrase or word and users input their target language translation.

Google Translate - Translate words and phrases

Google states, “Other users who contribute to Google Translate may review your suggested translation. If your review is marked as correct, Google Translate may show your translation with a badge. This indicates that the translation has been “Reviewed by Contributors”.

Google Translate - Reviewed by Contributors

According to a Google spokesperson, “We mark translations as ‘verified’ when they’ve been reviewed by several volunteers in the Google Translate Community”. Exactly how many reviewers must provide a positive verification for a translation to qualify for the badge is unknown.

Users can also click to assess the translation, or choose ‘Suggest an edit’ to enter a new translation.

Quantity over Quality for Better or for Worse

The system lends itself to a high volume of translations and validations which is exactly what Google Translate needs to do to continuously improve its MT engine. Crowdsourcing is extremely cheap, yet allows access to as many users as possible. The hope is that the more contributors, the higher the quality of the output. 

“Human translators are concerned with meaningful outputs, so that everything makes sense and is true to the source language”, The Translation Company observes. This should improve the quality of translations that go through the review process.

However, there are limitations to this method. Although the fast set-up process makes for an attractive user experience, it means that volunteers are not provided with any introduction to the process, rules for reviewing or translating, or a style guide to follow. 

One user questioned what sort of quality was expected: “Flawless human translation quality? Or translations that correctly convey the meaning?”. Consequently, users individually make a judgment call; every reviewer has their own criteria and rules for deciding what is correct, and where to be lenient.

The same volunteer highlights that sometimes the translation selected as “correct” by the reviewer may not actually be 100% accurate. For one example she explains, “most of these [translations] get the point across but fail to reflect the politeness in the original Japanese. I choose the one closest, despite its lack of capitalization and punctuation.”

This is likely one of the reasons why there are still complaints about the quality. The questionable accuracy of Google Translate is not a new discussion, with high-resource languages faring better than low-resource languages. 

There may still be grammatical errors in some languages (e.g., recognition of the perfect vs imperfect tenses in Romance languages), failure to distinguish between the formal and informal forms of address (‘vous’ vs ‘tu’ in French, ‘Sie’ vs ‘du’ in German for ‘you’., etc.), and gendering (e.g., in languages in which nouns indicating profession are gendered). Whereas polysemy is a struggle for Google Translate – when words have multiple meanings – human translators can use context to decide on the most appropriate translation.

Of course, an obvious issue with crowdsourcing is that contributors can make mistakes. Google also requires no proof that reviewers are “qualified” (know the language well enough) to make these verification judgments or to translate. It would be very easy for anyone to participate and troll the internet.