The Meta AI research team has announced that it would make large language models (LLMs) more accessible to researchers. In early May 2022, the social media giant released what it called Open Pre-trained Transformers (OPT), a suite of decoder-only and, well, pre-trained transformers, ranging from 125 million to 175 billion parameters.
The launch of OPT-175B comes shortly after Google unveiled its Pathways Language Model (PaLM) in early April. Meta follows in the footsteps of other big tech companies by building LLMs that have shown remarkable capabilities for zero- and few-shot learning.
Given their computational cost, these models are hard to replicate without significant capital. As for those made available through APIs, no access is granted to full models, making them difficult to study.
Various Twitter users shared their excitement about OPT being released under an open-source license, with Mike Schroepfer, Meta’s former CTO commenting, “Up till now only researchers at a handful of large companies could actually see what was behind the curtain. Now Meta AI has released a fully trained model for researchers…”
Percy Liang, Associate Professor at the Stanford University Computer Science Department, tweeted, “[This] is an exciting step towards opening new opportunities in research. In general, we can think of stronger release as enabling researchers to tackle deeper questions.”
Joelle Pineau, a long-time advocate for transparency in tech development (and also Co-Managing Director at Meta AI), noted how “the ability for others to scrutinize your work is an important part of research. We really invite that collaboration.”
Apart from making OPT available for non-commercial use, Meta also released its code with the logbook documenting the training process. The logbook contains daily updates from team members regarding training data: how and when it was added to the model, what worked, and what didn’t.
Meta Agenda
So why is Meta doing this? According to Pineau, “That commitment to open science is why I’m here, I wouldn’t be here on any other terms.” Ultimately, Pineau wants to change how the market judges AI. “What we call state-of-the-art nowadays can’t just be about performance,” she said. “It has to be state-of-the-art in terms of responsibility as well.”
Of course, this “open-sourcing” bears some risk too. As Margaret Mitchell, Researcher and Chief Ethics Scientist at Hugging Face pointed out, “This model will be able to generate harmful content not only by itself, but through downstream applications that researchers build on top of it.”
Some people also raised concerns about how potential risks would be handled. According to Pineau, she has had countless conversations regarding the best ways to release OPT-175B, knowing that there is “a non-zero risk in terms of reputation and a non-zero risk in terms of harm.”
“We have different opinions around the world about what speech is appropriate, and AI is a part of that conversation. But how do we grapple with that? You need many voices in that discussion” — Joelle Pineau, Managing Director, Meta AI
She shut down the idea of not releasing a model just because it could, potentially, be dangerous. This is “not a research mindset,” Pineau said.
The Meta AI Co-Managing Director does not expect language models to say things that everyone will be happy with — but bringing in numerous voices and building trust through transparency would definitely help mitigate the uncertainties surrounding AI.
Your information will not be shared with third parties. No Spam.