Many NLP tasks benefit from using large language models (LLMs) that often have more than 100 billion parameters. With the release of BLOOM-176B and OPT-175B, everyone can download pretrained models of this scale. Still, using these models requires high-end hardware unavailable to many researchers. In some cases, LLMs can be used more affordably via RAM offloading or hosted APIs. However, these techniques have innate limitations: offloading is too slow for interactive inference, while APIs are not flexible enough for research. In this work, we propose Petals - a system for inference and fine-tuning of large models collaboratively by joining the resources of multiple parties. We demonstrate that this strategy significantly outperforms offloading for very large models, running inference of BLOOM-176B on consumer GPUs with $\approx$ 1 step per second. Unlike most inference APIs, Petals also natively exposes the hidden states of served models, allowing its users to train and share custom model extensions based on efficient fine-tuning methods.

Over the past two years, EleutherAI has established itself as a radical initiative aimed at both promoting open-source research and conducting research in a transparent, openly accessible and collaborative manner. EleutherAI's approach to research goes beyond transparency; by doing research entirely in public, anyone in the world can observe and contribute at every stage. Our work has been received positively and resulted in several high-impact projects in Natural Language Processing and other fields. In this paper, we describe our experience doing public-facing machine learning research, the benefits we believe this approach brings, and the pitfalls we have encountered.

Open Artificial Intelligence (Open AI) collaboratives offer alternative pathways for how AI can be developed beyond well-resourced technology companies and who can be a part of the process. To understand how and why they work and what additionality they bring to the landscape, we focus on three such communities, each focused on a different kind of activity around AI: building models (BigScience workshop), tools/ways of working (The Turing Way), and ecosystems (Mozilla Festival’s Building Trustworthy AI Working Group). First, we document the community structures that facilitate these distributed, volunteer-led teams, comparing the collaboration styles that drive each group towards their specific goals. Through interviews with community leaders, we map user journeys for how members discover, join, contribute, and participate. Ultimately, this paper aims to highlight the diversity of AI work and workers that have come forth through these collaborations and how they offer a broader practice of openness to the AI space.

As the frontiers of machine learning (ML) continue to expand, the gap between the public understanding of ML and state-of-the-art research widens. While laboratory researchers benefit from easily accessible and encouraged collaboration with domain experts, the same cannot be said of newcomers to the field. At the undergraduate level, where socioeconomic inequality means some students have stronger backgrounds than their peers, increasing the accessibility of practical, hands-on opportunities in machine learning is essential to narrowing this gap. In this paper, we detail the approach of Machine Learning @ Berkeley (ML@B), a university-based undergraduate student organization aimed at bridging this gap by encouraging collaboration with established figures in the field as well as within the organization itself. We hope that our perspectives gained from ML@B provide insights into successfully integrating undergraduates into a technical environment and fostering an academic culture that encourages collaboration.

This paper aims to survey the problem space around cultural barriers in research collaboration, specifically for Machine Learning (ML). We review (1) unequal representation in ML/AI and STEM, (2) socioeconomic influences on retention of scientists and researchers, and (3) existing educational opportunity programs for people from disadvantaged backgrounds, with emphasis on Post-Baccalaureate support. We provide evidence that students from disadvantaged backgrounds not only experience barriers to gaining intellectual and technical expertise, they often experience cultural gaps that impede their participation in graduate programs and inclusion in research collaborations. We discuss relevant research on culture differences and the ways that some U.S. Federal TRIO programs explicitly address them, highlighting standardization as one means of demystifying academic and research cultures. We conclude with recommendations toward understanding post-education culture gaps with the goal of finding better solutions for increasing diversity in research collaborations.

We present a model of a student-led community of researchers to highlight the impact of pursuing collaborative machine learning research on the group’s members individually as well as towards achieving shared goals. We provide concrete examples of the guiding principles that led to the evolution of the collaborative from a reading group into a research group and eventually launching a non-profit software product to help non-technical stakeholders leverage artificial intelligence (AI), improving access to advanced technologies, and promoting open science. Our goal is to lay out a template to launch similar small-scale collaborative organisations at different institutes around the world.

Many NLP tasks benefit from using large language models (LLMs) that often have more than 100 billion parameters. With the release of BLOOM-176B and OPT-175B, everyone can download pretrained models of this scale. Still, using these models requires high-end hardware unavailable to many researchers. In some cases, LLMs can be used more affordably via RAM offloading or hosted APIs. However, these techniques have innate limitations: offloading is too slow for interactive inference, while APIs are not flexible enough for research. In this work, we propose Petals - a system for inference and fine-tuning of large models collaboratively by joining the resources of multiple parties. We demonstrate that this strategy significantly outperforms offloading for very large models, running inference of BLOOM-176B on consumer GPUs with $\approx$ 1 step per second. Unlike most inference APIs, Petals also natively exposes the hidden states of served models, allowing its users to train and share custom model extensions based on efficient fine-tuning methods.

Data Science Network (DSN) is a non-profit based out of Nigeria that focuses its efforts on high impact research and capacity building of the next generation of African data scientists. The organization harnesses trans-disciplinary and transnational partnerships to create data driven solutions for public health, agriculture, public safety, education, inequality, among others. DSN’s unique organizational structure hinges on the flow of connections and resources among its Knowledge Centres, Enabling Centres, Supporting Centres, and Catalyzing Centres. The results from the organization’s unique model has won global awards from UNESCO, the Economic Computation Conference, and by the African Union as a reference point on home-grown African-centric approach to sustainable application of research practice to address social problems. The organization's structure and key performance indicators of DSN are discussed in this position paper.

Over the past two years, EleutherAI has established itself as a radical initiative aimed at both promoting open-source research and conducting research in a transparent, openly accessible and collaborative manner. EleutherAI's approach to research goes beyond transparency; by doing research entirely in public, anyone in the world can observe and contribute at every stage. Our work has been received positively and resulted in several high-impact projects in Natural Language Processing and other fields. In this paper, we describe our experience doing public-facing machine learning research, the benefits we believe this approach brings, and the pitfalls we have encountered.

Learning tabula rasa, that is without any prior knowledge, is the prevalent workflow in reinforcement learning (RL) research. Unfortunately, the inefficiency of deep RL typically excludes researchers without access to industrial-scale resources from tackling computationally-demanding problems. Furthermore, as RL research moves toward more complex benchmarks, the computational barrier to entry would further increase. To address these issues, we present reincarnating RL as an alternative workflow or class of problem settings, where prior computational work (e.g., learned policies) is reused or transferred between design iterations of an RL agent, or from one RL agent to another. RRL can democratize research by allowing the broader community to tackle complex RL problems without requiring excessive computational resources. To demonstrate this, we present a case study on Atari games showing how superhuman Atari agents can be trained using only a few hours, as opposed to few days on a single GPU. Finally, we address reproducibility and generalizability concerns in this research workflow. Overall, this work argues for an alternate approach to RL research, which we believe could significantly improve real-world RL adoption and help democratize it further.

Extracting insights from text documents and developing predictive models for analytics is of critical importance in several domains. However, it is a challenging task owing to the diversity in linguistic characteristics of large scale text corpora, exacerbated by a lack of labeled data. We present here a case-study on extracting semantics from complex legal and regulatory documents and applying them to perform analytical tasks such as violation detection and penalty estimation. Our system was developed in a joint academic-industry collaboration effort and benefited from their complementary research strengths. Specifically, the domain expertise and problem formulation process in the industrial setting were combined with the exploratory research and experimentation rigor of the educational world to develop a system that can help legal actors improve their productivity. We outline our collaboration mechanism, detail the techniques used and functionalities developed, and also discuss the key take-aways that can benefit the research community.

Open Artificial Intelligence (Open AI) collaboratives offer alternative pathways for how AI can be developed beyond well-resourced technology companies and who can be a part of the process. To understand how and why they work and what additionality they bring to the landscape, we focus on three such communities, each focused on a different kind of activity around AI: building models (BigScience workshop), tools/ways of working (The Turing Way), and ecosystems (Mozilla Festival’s Building Trustworthy AI Working Group). First, we document the community structures that facilitate these distributed, volunteer-led teams, comparing the collaboration styles that drive each group towards their specific goals. Through interviews with community leaders, we map user journeys for how members discover, join, contribute, and participate. Ultimately, this paper aims to highlight the diversity of AI work and workers that have come forth through these collaborations and how they offer a broader practice of openness to the AI space.

Three challenges limit the progress of robot learning research: robots are expensive (few labs can participate), everyone uses different robots (findings do not generalize across labs), and we lack internet-scale robotics data. We take on these challenges via a new benchmark: Train Offline, Test Online (TOTO). TOTO provides remote users with access to shared robots for evaluating methods on common tasks and an open-source dataset of these tasks for offline training. Its manipulation task suite requires challenging generalization to unseen objects, positions, and lighting. We present initial results on TOTO comparing five pretrained visual representations and four offline policy learning baselines, remotely contributed by five institutions. The real promise of TOTO, however, lies in the future: we release the benchmark for additional submissions from any user, enabling easy, direct comparison to several methods without the need to obtain hardware or collect data.

MIT wanted to commission a large scale artwork that would serve to “illuminate a new campus gateway, inaugurate a space of exchange between MIT and Cambridge, and inspire our students, faculty, visitors, and the surrounding community to engage with art in new ways and to have art be part of their daily lives.” Among other things, the art was to reflect the fact that scientific discovery is often the result of many individual contributions, both acknowledged and unacknowledged. This paper details a collaboration between a widely-exhibited artist, computer scientists, and the broader community to produce a set of collective signatures. After collecting signatures from two communities - the university, and the surrounding city - the computer scientists developed generative models and a human-in-the-loop feedback process to work with the artist to create an original signature-like structure representative of each community. These signatures are now large-scale steel, LED, and neon light sculptures that appear to sign two new buildings in Cambridge, MA.

This is a position paper proposing the idea of separating research data from its presentation, or view. Today, researchers must not only produce the results of their research but must also think about how to present them. Moreover, in many cases, it is a matter of determining how well the results are displayed within the display format of a single paper. This has caused research results to be unnecessarily difficult to read and research to be unacceptable due to poor presentation. Therefore, we propose to allow separate creators of research data and creators of its display. This can lead to the following benefits: adoption of display formats for different purposes, utilization of buried data, improved reproducibility of studies, and gradual transition to better display formats.

The BigScience Workshop was a value-driven initiative that spanned one and half years of interdisciplinary research and culminated in the creation of ROOTS, a 1.6TB multilingual dataset that was used to train BLOOM, one of the largest multilingual language models to date.In addition to the technical outcomes and artifacts, the workshop fostered multidisciplinary collaborations around large models, datasets, and their analysis. This in turn led to a wide range of research publications spanning topics from ethics to law, data governance, modeling choices and distributed training. This paper focuses on the collaborative research aspects of BigScience and takes a step back to look at the challenges of large-scale participatory research, with respect to participant diversity and the tasks required to successfully carry out such a project. Our main goal is to share the lessons we learned from this experience, what we could have done better and what we did well. We show how the impact of such a social approach to scientific research goes well beyond the technical artifacts that were the basis of its inception.

This is a position paper proposing the idea of managing the entire research process on GitHub. The current machine learning research community faces a variety of problems, such as poor quality and low reproducibility of peer review at international conferences. These problems are caused by a lack of transparency in the research process and a lack of accessibility, where not everyone can participate in any given process of research. Thus, we propose that any information that arises in the research process be posted on GitHub and that contributions to the research be managed like those in an open-source software project. This could provide a springboard for solving the challenges of machine learning through clarifying contributors, allowing fine-grained contributions, improving reproducibility, enabling post-publication peer review, enhancing diversity, and protecting ideas.

Expansive, informative datasets are vital in providing foundations and possibilities for scientific research and development across many fields of study. Assembly of grand datasets, however, frequently poses difficulty for the author and stakeholders alike, with a variety of considerations required throughout the collaboration efforts and development lifecycle. In this work, we discuss and analyse the challenges and opportunities we faced throughout the creation of a transnational, European agricultural dataset containing reference labels of cultivated crops. Together, this forms a succinct framework of important elements one should consider when forging a dataset of their own.

Three challenges limit the progress of robot learning research: robots are expensive (few labs can participate), everyone uses different robots (findings do not generalize across labs), and we lack internet-scale robotics data. We take on these challenges via a new benchmark: Train Offline, Test Online (TOTO). TOTO provides remote users with access to shared robots for evaluating methods on common tasks and an open-source dataset of these tasks for offline training. Its manipulation task suite requires challenging generalization to unseen objects, positions, and lighting. We present initial results on TOTO comparing five pretrained visual representations and four offline policy learning baselines, remotely contributed by five institutions. The real promise of TOTO, however, lies in the future: we release the benchmark for additional submissions from any user, enabling easy, direct comparison to several methods without the need to obtain hardware or collect data.

The BigScience Workshop was a value-driven initiative that spanned one and half years of interdisciplinary research and culminated in the creation of ROOTS, a 1.6TB multilingual dataset that was used to train BLOOM, one of the largest multilingual language models to date.In addition to the technical outcomes and artifacts, the workshop fostered multidisciplinary collaborations around large models, datasets, and their analysis. This in turn led to a wide range of research publications spanning topics from ethics to law, data governance, modeling choices and distributed training. This paper focuses on the collaborative research aspects of BigScience and takes a step back to look at the challenges of large-scale participatory research, with respect to participant diversity and the tasks required to successfully carry out such a project. Our main goal is to share the lessons we learned from this experience, what we could have done better and what we did well. We show how the impact of such a social approach to scientific research goes well beyond the technical artifacts that were the basis of its inception.

This is a position paper proposing the idea of managing the entire research process on GitHub. The current machine learning research community faces a variety of problems, such as poor quality and low reproducibility of peer review at international conferences. These problems are caused by a lack of transparency in the research process and a lack of accessibility, where not everyone can participate in any given process of research. Thus, we propose that any information that arises in the research process be posted on GitHub and that contributions to the research be managed like those in an open-source software project. This could provide a springboard for solving the challenges of machine learning through clarifying contributors, allowing fine-grained contributions, improving reproducibility, enabling post-publication peer review, enhancing diversity, and protecting ideas.