Survey Examines Machine Translation Post-Editing Among Freelancers and LSPs

Two recent surveys conducted by PhD student Clara Ginovart have focused on practitioners’ views of post-editing of machine translation (PEMT). Supervised by Marina Frattino, Ginovart’s PhD at the Pompeu Fabra University, Spain, is in Training in Post Editing Machine Translation. Ginovart’s university directors are Carme Colominas from Pompeu Fabra University, and Antoni Oliver from Universitat Oberta de Catalunya. The granting institution is Agaur.

The first survey aimed to look at the state of PEMT from a business perspective. The 66 respondents to this survey each had experience in outsourcing machine translation post-editing, and were mainly working in small to mid-sized LSPs. The survey opened in December 2018 and closed in February 2019.

LSP respondents were mostly based in Spain (17), while the remainder were based elsewhere in Europe (42) and in Russia, Turkey, or other (7). LSP respondents said the most common source language for post-editing was English. Spanish, English, French, German, Italian, and Dutch were the most common target languages.

The second survey aimed to look at the state of PEMT from a post-editor’s perspective. The 142 respondents were mainly freelance or independent translators (84%). The survey opened in January 2019 and closed in April 2019.

Top locations for post-editor respondents were Spain, Italy, UK, Germany, and France. Top target languages represented in the survey were English, Spanish, Italian, German, French, and Dutch. English was, by far, the most common source language for post-editors, with French, Germany, Spanish, Italian, and others also represented.

Slator reviewed the survey results and analyzed the findings, comparing the similarities and differences between answers from the two groups of respondents where relevant.

Resourcing Models, Volumes, Content Types

LSPs had a variety of resourcing models for the post-editing task: 85% said they rely on a pool of freelance post-editors to some degree, while a small number (15%) said they do not outsource any post-editing tasks. Over half (58%) the LSP respondents said they do some form of post-editing in-house.

Most LSP respondents (56%) said their end customers decide whether or not a post-editing workflow is used, while 41% said the decision is made internally. Of 58 LSP respondents, 60% said they inform the client when using MT workflows.

Of 58 LSP respondents, 60% said they inform the client when using MT workflows.

Of 96 freelance respondents, the majority (78%) said that most of the post-editing work they do is for LSPs, while only 20% said that most of the post-editing work they do is for direct customers. Three quarters (75%) said the requester is responsible for deciding whether it is appropriate to use PEMT, while 18% said they themselves choose whether or not to use PEMT.

The majority of freelancers (78%) said that most of the post-editing work they do is for LSPs, while only 20% said that most of the post-editing work they do is for direct customers.

73% of LSP respondents and 62% of post-editors said that post-editing accounts for 25% or less of all translation work. A similar number of LSP respondents (21%) and post-editors (18%) said that post-editing accounts for between 26% and 50% of all translation work.

100 post-editors said they post-edit high-visibility content (for public consumption) and 67 post-editors said they post-edit low-visibility content (for limited dissemination).

Output Quality & Quality Management

The majority of LSP respondents (73%) said post-editing to a human-professional standard is the most commonly used form of PEMT, while 21% said the service most commonly required is light or “good enough” PEMT.

There was a discrepancy between the level of output quality observed by LSP respondents and post-editors: 73% of LSP respondents and just 42% of post-editors said the task of post-editing involves “improving medium (acceptable) quality raw output to publishable quality,” while 56% of post-editors said they are asked to improve poor quality output to either a publishable or acceptable quality.

There was also a discrepancy around how much feedback post-editors were asked to provide on the quality of the MT output: 70% of post-editors said they are not asked for feedback on MT output quality, yet 73% of all LSP respondents said they ask the post-editor for feedback.

70% of post-editors said they are not asked for feedback on MT output quality, yet 73% of all LSP respondents said they ask the post-editor for feedback.

Another discrepancy between the two groups was in the level of instructions respondents said were provided: 46% of post-editors said they are not provided specific guidelines for PEMT, while most LSP respondents (87% of 45) said they give detailed instructions to the post-editor, whether standardized for the company or tailored to the content type or language.

Productivity & Metrics

Productivity levels observed by LSPs and post-editors were similar, although post-editors generally reported higher levels of productivity based on words post-edited per hour than LSP respondents.

55% of post-editors said that they track their PEMT productivity. Of 64 post-editors, around half use Excel to track their productivity, while around 10% use proprietary tools. Less than 10% use third-party project management software. Of 44 LSP respondents, some use internal productivity tracking tools (13) and others use Excel (10). Still others use project management software (9) and some use no tools at all (7).

Both groups reported that the dominant payment model was per source word. 57% of post-editors and around 40 out of 56 LSP respondents said they use this model. Less than 10 LSP respondents said they pay per hour based on editor-reported time spent, while 25% of post-editors said they charge per hour.

Tools & Training

Post-editors were asked what MT systems they used most. Popular MT systems for post-editors included Google, DeepL, SDL (Adaptive MT, Language Cloud, or ETS), Amazon Translate, and SYSTRAN.

The most commonly used productivity tool (CAT) for post-editors and LSPs is SDL Trados Studio. Other popular productivity tools for both LSPs and post-editors include memoQ, Memsource, Wordfast, MateCat, Smartcat, Across, Transifex, Localize, and GlobalSight, as well as proprietary solutions. Post-editors also used other tools in addition to those mentioned.

The top QA tool for post-editors and LSPs is Xbench followed by “none.” Next preferred options for LSPs are Verifika and QA Distiller, while post-editors employ a wider range of QA tools.

Both LSPs and post-editors had mixed opinions on the quality of existing PEMT training courses: 41 post-editors felt they are not adequate, 45 said they are adequate, and 56 do not know. Some post-editors (23%) had a training course provided by their company, a few had one provided externally (12%), or said they had done one at university (8%). 42% of LSP respondents felt that current PEMT training courses are not adequate, while 35% said they are.

53% of post-editors said they had never attended a PEMT training course, while two-thirds (67%) of LSPs said they had not yet organized specific training on PEMT.
For complete survey results and additional information, contact Clara Ginovart at clara.ginovart@upf.edu.