Phillinda.space, the brainchild of a team of four Boston-area students and winner of HackHarvard 2019, is truly a sign of the times.
The concept behind the app is relatively simple: The app uses Google Translate and newer Google Cloud products to translate forms from English into another language, and to translate a user’s responses back into English.
What makes Phillinda.space an interesting case study is that it is powered by machine learning products that barely existed two years ago — products that are easy enough to use that savvy students were able to build a functional app within the hackathon’s 36-hour time limit.
HackHarvard, hosted by Harvard University, has many big-name sponsors, including Google Cloud. Team member Rishab Nayak told Slator that although the affiliation played a role in the team’s choice to use Google Translate, the final decision came down to the number of languages Google Translate supports.
“Even though it’s not the most accurate in certain language pairs, it still works for a lot more languages than some [other translation engines] you would find, which only work for specific pairs,” Nayak said.
The team built the app using Flutter, an open-source UI software development framework officially released by Google in December 2018.
Offline Machine Translation
What differentiates Phillinda.space from similar applications is that it performs translations on a user’s Android or iOS device, as opposed to on the cloud. Although Flutter does not officially support this type of offline work, another Google offering, Natural Language, does, and one can use the app even without an Internet connection. This combination of Flutter and Natural Language, Nayak said, is what enabled Phillinda.space to offer on-device translation.
However, extracting fields from a PDF file needs to be done on the cloud. The app’s API is currently hosted on Google App Engine. The user supplies the original PDF and the API on Google App Engine returns the PDF’s fields to the user in their language. At this point, the user fills in the form in their own language, and their responses are sent back to the API. The user can then download the form with their responses in the PDF’s original language.
Slator 2020 Language Industry Market Report
During the hackathon, the multilingual team was able to check the quality of their app in seven languages: Dutch, English, French, German, Hindi, Portuguese, and Spanish. The app can now translate between 59 different languages. The team acknowledges that Google Translate does not always provide perfect translations, and that the quality of the content varies depending on the language pair.
“I wouldn’t say it was completely accurate, but it worked well enough that a native speaker could understand it was a translation, and understand it with a pinch of salt,” Nayak said.
The app provides more than just text translations. It also has speech-to-text and text-to-speech capabilities to accommodate users with disabilities.
People with visual impairments often find it difficult to navigate PDFs, Nayak explained, but Phillinda.space can turn a PDF into a fully annotated list of fields that the user can scroll through. A voiceover function can then read the fields aloud in any language supported by Google Translate and by the user’s device.
For the voiceover feature, the team initially tried to use Google’s Cloud Text-to-Speech API, but due to the hackathon’s time constraints the students opted instead to write code in Flutter to use the sets of voices provided by Android and iOS.
OCR Dilemma
Languages that do not have a corresponding voice on Android or iOS present a challenge. The current fix uses an English voice to sound out non-English words. But the team understands this will not work long-term since the English voice’s default pronunciation can interfere with a native speaker’s understanding of the translation. Nayak said that a better solution, which the team is currently working on, would be to generate a voice and stream it to the device.
Phillinda.space was designed to handle computer-generated forms (specifically visa applications, although it can now work with any computer-generated form) and, for the time being, it cannot extract fields from forms that have been photographed. Nayak described the relationship between a form’s text and fields as “a very complicated OCR [optical character recognition] problem, which a lot of companies are trying to solve and no one really has a solution.”
Nayak also cautioned against using the app for medical forms due to the risk that mistranslations could pose to a patient’s well-being.
“Some medical terms are very specialized and basic translation systems are incapable of translating these,” Nayak said. “Until we come up with a custom representation of our own, we would not recommend using this for medical forms.”
Building on the momentum of the HackHarvard win, the team attended a pitch event at Boston University’s BUild Lab on November 14, 2019, with the goal of finding like-minded students to contribute to the project. Another incentive? The possibility of attracting some funding, which could help the team present at other events, recruit talent, and, hopefully, release the app before the end of 2019.
Your information will not be shared with third parties. No Spam.