Real-time machine translation (MT) for travelers is a classic use case for translation technology — one that has seen a number of companies pitch gadgets to help tourists overcome language barriers, with varying degrees of success. (Apple and Google, of course, stand apart as household names whose applications are regularly used in just these situations.)
Now, researchers have put ChatGPT — OpenAI’s multi-purpose, generative AI model, introduced in November 2022 — to the test, a peek into the possible productization of ChatGPT.
Co-authors Rohit Kumar, founder of AI consulting firm VoiceThesis, and high school Voice Thesis intern (and nonprofit founder) Sanjana Kolar tested ChatGPT’s ability to provide English translation into three Indian languages for 50 questions related to general knowledge, food, or travel.
The scenario that gave rise to their July 2023 paper, Multilingual Tourist Assistance using ChatGPT: Comparing Capabilities in Hindi, Telugu, and Kannada, is increased tourism to India, a country well-known for its multilingualism. Beyond international tourism, translation from English into local Indian languages can also be helpful for Indian travelers visiting different parts of their own country.
In a conversational interaction with the system, users would inform ChatGPT that it was “a helpful assistant that translates English to [target language],” and instruct it to “Translate the following English text to [target language].” Evaluators assessed the translated text’s accuracy and quality, rating each on a scale of 1-5. These ratings were then converted to BLEU scores.
As might be expected, translations into Hindi received the highest scores, “indicating a high resemblance to reference translations and effective preservation of meaning in phrases.” One reviewer said they were “pleasantly surprised” with some translations, and praised ChatGPT’s rich vocabulary and solid grammar. Still, the system is not perfect, as another evaluator pointed out: “Some [translations] did not make much sense and had [the] opposite meaning. But a good start :)”
Kannada earned “decent” ratings on accuracy and fluency. Reviewers described it as a “work in progress” that needs more input and refinement to improve translation accuracy. “Colloquial and spoken Kannada is quite different,” one wrote. “It has work to do there.”
Telugu, meanwhile, had the lowest average scores across all three themes, with ChatGPT “struggling with maintaining meaning in longer phrases.” Evaluators said although some parts of translations were correct, others were “ambiguous” and “not usable.”
“Based on this, the study advised enhancing Telugu translations through exploring advanced models, diversifying training data, or employing techniques like transfer learning,” Kolar and Kumar wrote, adding that Kannada could be improved with expanded training data, fine-tuning methods, or leveraging sophisticated translation models.
Ongoing evaluation and training with refreshed data sets could maintain the high performance in Hindi translation.
Kolar and Kumar suggested that future studies in this area might include a wider range of participants (such as non-Indian tourists); additional language pairs (English translation into regional languages of other countries); and a comparative study of the effectiveness of ChatGPT compared to other AI language models used for tourism communication.
Your information will not be shared with third parties. No Spam.