The Good, the Bad, and the Unexpected of Machine Translation in BLOOMChat

Machine Translation in BloomChat

Call it an upgrade: HuggingFace’s large language model BLOOM got a chatbot makeover.

BLOOMChat, a 176-billion-parameter multilingual chat model, appeared on French open source AI model platform HuggingFace in May 2023, less than a year after volunteer research collective BigScience open-sourced BLOOM, in July 2022.

Hyped as a gamechanger, BLOOM is architecturally similar to its LLM competitors, including OpenAI’s GPT and Meta’s OPT-175B. But BLOOM differentiated itself from the competition with an intentionally multilingual design and the goal of keeping the model open to the public but safe from misuse. 

“BLOOM is the seed of a living family of models that we intend to grow, not just a one-and-done model,” HuggingFace explained in an official July 2022 blog post. “We’re ready to support community efforts to expand it.”

So, while HuggingFace may have built — and hosted — BLOOM, having another player reconfigure BLOOM to power a multilingual chatbot seems to fit in with the platform’s long-term plans. The force behind BLOOMChat is Palo Alto, California-headquartered AI startup SambaNova.

The company’s flagship products include DataScale, a platform for generative AI development, and SambaNova Suite, a group of generative AI models for enterprise and government. Among verticals served, SambaNova’s website highlights financial services and insurance; healthcare and life sciences; manufacturing; and the energy sector. 

In other words, SambaNova was not necessarily the most obvious candidate to take BLOOM in this direction.

You Never Know What You’re Gonna Get

SambaNova reportedly trained BLOOMChat “on assistant-style conversation datasets” and described the bot as “good at multilingual instruction following, summarizing, classification, and dialog.”

As might be expected, the quality of BLOOM’s own machine translation seems to impact BLOOMChat’s capabilities. As of March 2023, BLOOM was able to produce “adequate” MT results — provided it received adequate training in advance. Researchers found BLOOM’s MT quality to be, overall, inconsistent. (HuggingFace’s own HuggingChat seems to produce similar results.)

This squares with Slator’s own, admittedly limited, experience with BLOOMChat. While BLOOM can generate text in 46 natural languages, BLOOMChat explains that “currently, it can translate from and to English, Spanish, Portuguese, French, German, Dutch, Japanese, and Korean.”

And that was the case — sometimes. When first prompted to translate from French to English an excerpt from Amazon’s careers page, the chatbot responded with more than two dozen lines of prayer, in English, asking Allah to “cleanse me from my sins with the water of forgiveness.” 

Prompted a second time, however, BLOOMChat translated the text without any issues. Later on BLOOMChat simply regurgitated the input, regardless of whether it was prompted in French or in English.

For certain prompts, such as text about SambaNova, the chatbot not only translated from English to French and to German, but also added more information.

How about languages outside BLOOMChat’s stated repertoire? BLOOMChat translated text from Russian social network VKontakte into English without a problem. But translating the original Russian into another “unlisted” language was a no-go — not surprising, considering that researchers’ experiments with BLOOM and zero-shot translation resulted in numerous errors.

Asked to translate an English headline, “The Vatican Has Told Nuns Not to Overdo It on Social Media,” into Russian, BLOOMChat restated it in English and provided an additional, hallucinated paragraph specifying possible consequences for nuns and priests who fail to comply with the Pope’s orders. 

In its limited linguistic abilities and unexpected content generation, BLOOMChat calls to mind Google’s Bard, currently available in English, Japanese, and Korean. BLOOMChat, however, takes pride in its apparently altruistic origin story.

“We put BLOOMChat on HuggingFace as a contribution to the open source community,” BLOOMChat stated. “By putting the model on HuggingFace, we make it easier for others to build on top of our work and for researchers to study and improve large language models. We hope that BLOOMChat can serve as a starting point for building more inclusive and accurate conversational agents.”