Physical sciences and machine learning: more than the sum of their parts. Join us to discuss research at the convergence of these fields!

Machine learning is emerging as a valuable tool in mitigating and adapting to climate change, while climate change has been noted as a valuable area for inspiring cutting-edge algorithms in machine learning. This workshop is intended to form connections and foster cross-pollination between researchers in machine learning and experts in complementary climate-relevant fields, in addition to providing a forum for those in the machine learning community who wish to tackle climate change. This workshop distinguishes itself from previous editions of the popular ‘Tackling Climate Change with Machine Learning’ workshop series by focusing on a key challenge: questioning common machine learning assumptions in the context of climate impact. Specifically, we will concentrate on two questions that are very timely for the machine learning community: (i) the various climate-related benefits and costs of large vs small models, (ii) the design of effective benchmarks for climate-related applications.

This workshop brings together ML and policy experts to identify and address various technical, policy, and fair use challenges that arise when regulating ML models.

The rise of foundation models (FMs) amplifies the importance and relevance of federated learning (FL) as a crucial research direction. With FMs becoming the norm in machine learning development, the focus shifts from model architecture design to tackling the issues surrounding privacy-preserving and distributed learning. Advancements in FL methods have the potential to unlock the use of FMs, enabling efficient and scalable training while safeguarding sensitive data. With this in mind, we invite original research contributions, position papers, and work-in-progress reports on various aspects of federated learning in the era of foundation models. Since the emergence of foundation models has been a relatively recent phenomenon, their full impact on federated learning has not yet been well explored or understood. We hope to provide a platform to facilitate interaction among students, scholars, and industry professionals from around the world to discuss the latest advancements, share insights, and identify future directions in this exciting field. This workshop aims to bring together academic researchers and industry practitioners to address open issues in this interdisciplinary research area. For industry participants, we intend to create a forum to communicate problems are practically relevant. For academic participants, we hope to make it easier to become productive in this area. The workshop will focus on the theme of combining FL with FM to open up opportunities to address new challenges.

Advancing machine learning research to aid simulations (e.g., improve accuracy, speed) in various domains (e.g., physics, graphics).

Machine Learning (ML) for Systems describes the application of machine learning techniques to problems related to computer systems. By leveraging supervised learning and reinforcement learning (RL) approaches, machine learning can replace longstanding heuristics that currently drive many of these systems. This includes a wide range of topics, including multi-objective tasks such as designing new data structures, integrated circuits, or design verification, as well as implementing control algorithms for applications such as compilers, databases, memory management, or ML frameworks. While the systems community increasingly recognizes the importance of ML in solving a variety of different systems problems, ML for Systems remains an emerging area without widely established best practices, methods and strategies for the application of state-of-the-art machine learning techniques. The goal of this workshop is to provide an interdisciplinary venue for ML and Systems experts to push this boundary and start new directions within the ML for Systems area. This year, we will encourage work in key emerging areas such as Large Language Model (LLM) training and serving.

This workshop will delve into aspects of time series prediction and analysis in the age of large models, focusing on the topics of building foundation models for time series, leveraging pretrained models of other modalities for time series, multimodal time series models and time series evaluation and real-world applications.

Compositional learning, inspired by the human ability to derive complex ideas from simpler constituents, seeks to equip machines with analogous capabilities for understanding, reasoning, and adaptive learning. This methodology bolsters machines' ability to generalize to out-of-distribution samples through the recombination of learned components, proving effective across diverse tasks such as machine translation, visual reasoning, image generation, reinforcement learning, and more. Despite notable advancements, persistent challenges remain in achieving robust compositional generalization and reasoning within even the most advanced foundation models. Our workshop aims to discuss these challenges as well as untapped opportunities ahead from the following four aspects: exploring the capacity for compositionality in foundation models and dissecting the underlying mechanisms of their compositional learning; devising reliable and model-agnostic strategies for constructing compositional systems; establishing theoretical and empirical connections between modular architectures and compositional generalization; and extending compositional learning principles to continual learning contexts. By confronting these themes, we aim to foster a collaborative exploration of theoretical and empirical dimensions of compositional learning, thus advancing understanding and practical applications of compositional learning.

In this workshop, we plan to gather researchers working in neural algorithmic learning, multimodal reasoning, and cognitive models of intelligence to showcase their cutting-edge research, discuss the latest challenges, as well as bring to the forefront problems in perception and language modeling that are often overlooked but are pivotal in achieving true artificial general intelligence. An emphasis of this workshop is on the emerging topic of multimodal algorithmic reasoning, where a reasoning agent is required to automatically deduce new algorithms/procedures for solving real-world tasks, e.g., algorithms that use multimodal foundational models for analysis, synthesis, and planning, new approaches towards solving challenging vision-and-language mathematical (Olympiad type) reasoning problems, deriving winning strategies in multimodal games, procedures for using tools in robotic manipulation, etc. We hope to deep dive into this exciting topic at the intersection of multimodal learning and cognitive science to understand what we have achieved thus far in machine intelligence and what we are lacking in relation to the human way of thinking -- through talks from outstanding researchers and faculty that could inspire the audience to search for the missing rungs on the ladder to true intelligence.

Structural biology is the study of biological function with the awareness that molecules exist in four dimensions. AlphaFold2 demonstrated deep learning’s capability to solve one low-hanging problem in this field: predicting a single protein structure from its sequence, trained from the ~180,000 structures standardized and collected in the Protein Data Bank. However, there remain many harder unsolved challenges that need progress if we wish to understand and design functional biomolecules. There is a strong need to gather deep learning experts and biologists together to address these challenges. We have assembled and confirmed a set of diverse speakers who are world leaders in current challenges, including how to incorporate large-scale stability datasets, dynamics, ligand binding, into the fold of modern deep learning for structural biology. One of the biggest bottlenecks for all of these problems is the data available for training and how to create clear and stringent tests to evaluate progress. Our workshop will highlight two new benchmarks in a special track of our call for papers, and create a platform for open-sourced models, in collaboration with HuggingFace. We anticipate this workshop to be of great interest to many NeurIPS attendees, and to create lasting impact in establishing benchmarks and accessible modelling resources for deep learning and structural biology communities.

The integration of artificial intelligence (AI) and machine learning (ML) into scientific discovery represents a pivotal shift in traditional methodologies. Historically, scientific exploration has been systematic and logical, but AI and ML promise to transform fundamental discoveries. This shift enhances interdisciplinary dialogue and stimulates innovative problem-solving, enriching the scientific community's ability to tackle complex problems. Foundation models, such as GPT-3 and CLIP, have revolutionized computer vision and natural language processing, providing versatile, pre-trained bases for various applications. Leveraging these models addresses critical challenges like long-term planning and multi-modal reasoning, essential for applications in robotics and dialogue systems. The integration of AI-for-Science and foundation models offers a transformative force in scientific domains, solving complex problems and enabling domain-specific adaptations. This synergy is poised to radically improve the modeling of complex phenomena, making it a crucial investment for future scientific advancements. This workshop aims to bring together experts to discuss and collaborate on transformative questions and challenges in advancing scientific problems through foundation models.

How do humans learn to master an ever-expanding repertoire of skills without being told what to learn? The Intrinsically Motivated Open-ended Learning (IMOL) workshop tackles this fundamental question at the intersection of artificial intelligence and cognitive science. While recent breakthroughs in curiosity-driven learning have shown promise, artificial agents still lack the remarkable flexibility of human learners who continuously generate their own goals, explore efficiently, and build upon previous knowledge. By bringing together perspectives from reinforcement learning, developmental psychology, and robotics, IMOL aims to uncover the key mechanisms – from autonomous goal generation to incremental skill building – that enable truly open-ended learning in both natural and artificial systems.

Advanced Artificial Intelligence (AI) techniques based on deep representations, such as GPT and Stable Diffusion, have demonstrated exceptional capabilities in analyzing vast amounts of data and generating coherent responses from unstructured data. They achieve this through sophisticated architectures that capture subtle relationships and dependencies. However, these models predominantly identify dependencies rather than establishing and making use of causal relationships. This can lead to potential spurious correlations and algorithmic bias, limiting the models’ interpretability and trustworthiness.In contrast, traditional causal discovery methods aim to identify causal relationships within observed data in an unsupervised manner. While these methods show promising results in scenarios with fully observed data, they struggle to handle complex real-world situations where causal effects occur in latent spaces when handling images, videos, and possibly text.Recently, causal representation learning (CRL) has made significant progress in addressing the aforementioned challenges, demonstrating great potential in understanding the causal relationships underlying observed data. These techniques are expected to enable researchers to identify latent causal variables and discern the relationships among them, which provides an efficient way to disentangle representations and enhance the reliability and interpretability of models.The goal of this workshop is to explore the challenges and opportunities in this field, discuss recent progress, and identify open questions, and provide a platform to inpire cross-disciplinary collaborations.

The increasing capabilities of foundation models have raised concerns about their potential to generate undesirable content, perpetuate biases, and promote harmful behaviors. To address these issues, we propose a workshop that focuses on understanding the inner workings of foundation models and identifying actionable mechanisms involved in generation. Recent studies have shown promise in directly intervening on model activations or a low-rank subset of the weights to provide fine-grained control over model generation to mitigate the generation of harmful and toxic content. This workshop aims to bring together researchers to explore methods for improving the controllability of foundation models and developing a better understanding of their behavior and potential misuse.

This workshop is the second in a series focused on interpretability and explainability. The first workshop, titled "XAI in Action: Past, Present, and Future Applications," was held at NeurIPS 2023. In this edition, we aim to bridge classical interpretability and modern methods for foundation models. We retain the core organizing team from the previous workshop while welcoming three new members. Additionally, we have introduced research roundtables to support community building.

While deep learning continues to achieve impressive results on an ever-growing range of tasks, our understanding of the principles underlying these successes remains largely limited. This problem is usually tackled from a mathematical point of view, aiming to prove rigorous theorems about optimization or generalization errors of standard algorithms, but so far they have been limited to overly-simplified settings. The main goal of this workshop is to promote a complementary approach that is centered on the use of the scientific method, which forms hypotheses and designs controlled experiments to test them. More specifically, it focuses on empirical analyses of deep networks that can validate or falsify existing theories and assumptions, or answer questions about the success or failure of these models. This approach has been largely underexplored, but has great potential to further our understanding of deep learning and to lead to significant progress in both theory and practice. The secondary goal of this workshop is to build a community of researchers, currently scattered in several subfields, around the common goal of understanding deep learning through a scientific lens.

The primary objective of this workshop is to bridge the gap between AI and emerging drug modalities, such as gene and cell therapies, and RNA-based drugs. These modalities are important for disease mechanisms which were previously considered difficult, if not impossible, to target. They offer novel mechanisms of action, target previously undruggable proteins, and can address unmet medical needs with higher precision and efficacy. Traditional modalities such as small-molecule drugs, recombinant proteins, and monoclonal antibodies have formed the basis of numerous existing treatments. Recently, there has been growing recognition that AI methods have the potential to expedite breakthroughs in these established modalities. This same potential extends to the emerging field of new drug modalities, where AI can play a transformative role in accelerating the discovery and development process, overcoming challenges, and ultimately bringing innovative therapies to patients more efficiently and effectively. We aim to bring specialists in these modalities and ML communities together to discuss and investigate how AI can accelerate drug development in these emerging modalities.

Our workshop focuses on improving reasoning in neural networks, particularly the challenges and strategies for achieving System-2 reasoning in transformer-like models. The workshop addresses issues like distinguishing memorization from rule-based learning, understanding, syntactic generalization, and compositionality. The workshop also covers the importance of understanding how systematic models are in their decisions for AI safety, integrating neural networks with symbolic reasoning, and developing new architectures for enhanced reasoning capabilities. We have (tentatively) confirmed a distinguished group of speakers and panelists who are some of the most influential figures in recent literature on reasoning. Considering how important these topics are today and our distinguished lineup of speakers, we expect \textbf{more than 500 participants to the workshop}.

The advanced generative artificial intelligence (AI) techniques, such as large language models and large multimodal models, are transforming many aspects of educational assessment. The integration of AI into education has the potential to revolutionize not only test development and evaluation but also the way students can learn. Over the past years, some successful adoptions of machine learning in this area are using natural language processing for automated scoring, or applying collaborative filtering to predict student responses. The rapid advances of large foundation models (e.g., ChatGPT, GPT-4, Llama, Gemini) demonstrate the potential of intelligent assessment with data-driven AI systems. These models could potentially benefit test construct identification, automatic item generation, multimodal item design, automated scoring, and assessment administration. Meanwhile, new research challenges arise in the intersection of AI and educational assessments. For instance, the explainability and accountability of current large foundations models are still inadequate to convince the stakeholders in the educational ecosystem, which limits the adoption of AI techniques in large-scale assessments. Also, it is still unclear whether the large foundation models are capable of assisting complex assessment tasks that involve creative thinking or high-order reasoning. Tackling these research challenges would require collaborative efforts from researchers and practitioners in both AI and educational assessment. This one-day workshop provides a forum for researchers from AI and educational assessment to review and discuss the recent advances of applying large foundation models for educational assessment.

Foundation models are increasingly being augmented with new modalities and access to a variety of tools and software. Systems that can take action in a more autonomous manner have been created by assembling agent architectures or scaffolds that include basic forms of planning and memory or multi-agent architectures. As these systems are made more agentic, this could unlock a wider range of beneficial use-cases, but also introduces new challenges in ensuring that such systems are trustworthy. Interactions between different autonomous systems create a further set of issues around multi-agent safety. The scope and complexity of potential impacts from agentic systems means that there is a need for proactive approaches to identifying and managing their risks. Our workshop will surface and operationalize these questions into concrete research agendas.

Touch is a crucial sensor modality for both humans and robots, as it allows us to directly sense object properties and interactions with the environment. Recently, touch sensing has become more prevalent in robotic systems, thanks to the increased accessibility of inexpensive, reliable, and high-resolution tactile sensors and skins. Just as the widespread availability of digital cameras accelerated the development of computer vision, we believe that we are rapidly approaching a new era of computational science dedicated to touch processing. We believe that AI/ML will play a critical role in successfully processing touch as a sensing modality. However, this raises important questions regarding which computational models are best suited to leverage the unique structure of touch, similar to how convolutional neural networks leverage spatial structure in images. The development and advancement of touch processing will greatly benefit a wide range of fields, including tactile and haptic use cases. For instance, advancements in tactile processing (from the environment to the system) will enable robotic applications in unstructured environments, such as agricultural robotics and telemedicine. Understanding touch will also facilitate providing sensory feedback to amputees through sensorized prostheses and enhance future AR/VR systems.

Digital computing is approaching fundamental limits and faces serious challenges in terms of scalability, performance, and sustainability. At the same time, generative AI is fuelling an explosion in compute demand. There is, thus, a growing need to explore non-traditional computing paradigms, such as (opto-)analog, neuromorphic hardware, and physical systems.Expanding on last year's successful NeurIPS workshop, which was the first of its kind in this community, we aim to bring together researchers from machine learning and alternative computation fields to establish new synergies between ML models and non-traditional hardware. Co-designing models with specialized hardware, a feature that has also been key to the synergy of digital chips like GPUs and deep learning, has the potential to offer a step change in the efficiency and sustainability of machine learning at scale. Beyond speeding up standard deep learning, new hardware may open the door for efficient inference and training of model classes that have been limited by compute resources, such as energy-based models and deep equilibrium models. So far, however, these hardware technologies have fallen short due to inherent noise, device mismatch, a limited set of compute operations, and reduced bit-depth. As a community, we need to develop new models and algorithms that can embrace and, in fact, exploit these characteristics. This workshop aims to encourage cross-disciplinary collaboration to exploit the opportunities offered by emerging AI accelerators both at training and at inference.

The NeurIPS 2024 LLM Privacy Challenge is designed to address the critical issue of privacy in the use of Large Language Models (LLMs), which have become fundamental in a wide array of artificial intelligence applications. This competition acknowledges the potential privacy risks posed by the extensive datasets used to train these models, including the inadvertent leakage of sensitive information. To mitigate these risks, the challenge is structured around two main tracks: the Red Team, focusing on identifying and exploiting privacy vulnerabilities, and the Blue Team, dedicated to developing defenses against such vulnerabilities. Participants will have the option to work with LLMs fine-tuned on synthetic private data or LLMs interacting with private system/user prompts, thus offering a versatile approach to tackling privacy concerns. The competition will provide participants with access to a toolkit designed to facilitate the development of privacy-enhancing methods, alongside baselines for comparison. Submissions will be evaluated based on attack accuracy, efficiency, and the effectiveness of defensive strategies, with prizes awarded to the most innovative and impactful contributions. By fostering a collaborative environment for exploring privacy-preserving techniques, the NeurIPS 2024 LLM Privacy Challenge aims to catalyze advancements in the secure and ethical deployment of LLMs, ensuring their continued utility in sensitive applications without compromising user privacy.

Training high-performing large language models (LLMs) from scratch is a notoriously expensive and difficult task, costing hundreds of millions of dollars in compute alone. These pretrained LLMs, however, can cheaply and easily be adapted to new tasks via fine-tuning, leading to a proliferation of models that suit specific use cases. Recent work has shown that specialized fine-tuned models can be rapidly merged to combine capabilities and generalize to new skills. This raises the question: given a new suite of desired skills and design parameters, is it necessary to fine-tune or train yet another LLM from scratch, or can similar existing models be re-purposed for a new task with the right selection or merging procedure? The LLM Merging challenge aims to spur the development and evaluation of methods for merging and reusing existing models to form stronger new models without needing additional training. Specifically, the competition focuses on merging existing publicly-released expert models from Hugging Face, using only minimal compute and additional parameters. The goal will be to develop merged models that outperform existing models and existing merging baselines. Submissions will be judged based on the average accuracy on a set of held-out multiple-choice evaluation tasks and their efficiency. To make the competition as accessible as possible and ensure that the merging procedures are more efficient than fine-tuning, we will enforce a compute budget and focus on merging models with fewer than 8B parameters. A starter kit with all necessary materials (baseline implementations, requirements, the evaluation script, etc.) will be released on May 1st.

In recent years, AI has made significant strides in achieving success across various domains, demonstrating capabilities that often surpass human performance in specific tasks. However, the real world presents challenges that go beyond single tasks, objectives, or predefined, static environments. We propose to consider open-world environments as the new habitat for AI agents: highly diverse and dynamic, fully interactive, teaming up with infinite and creative tasks, and requiring continuing learning and growth. Therefore, open-world agents, are expected to possess remarkable problem-solving capabilities across all cognitive functions, notably, reasoning and decision-making compared to specialized AI agents.

This workshop aims to bring together researchers from various fields to discuss emerging topics about reasoning and decision-making in open-world environments. This topic can be overly broad, but we are particularly interested in synergizing reasoning and decision-making, i.e., open-world agents that can simultaneously perform reasoning (e.g., QA, dialogue) and decision-making (e.g., planning and control), and how such unification helps tackle the challenges brought by the open world to both parties. To this end, the related fields are not limited to interleaved reasoning with decision-making, reasoning in embodied learning agents, LLM tool usage, reinforcement learning in open-world environments, open vocabulary learning, continued learning, multi-agent learning, and emerging ethical considerations in open-world environments. Our objective is to foster collaboration and insights into addressing the scientific questions about developing open-world reasoning and decision-making agents.

Building on the success of the Melting Pot contest at NeurIPS 2023, which challenged participants to develop multi-agent reinforcement learning agents capable of cooperation in groups, we are excited to propose a new contest centered on cooperation between language model (LM) agents in intricate, text-mediated environments. Our goal is to advance research on the cooperative intelligence of such LM agents. Of particular interest are the agents capable of using natural language to effectively cooperate with each other in complex environments, even in the face of challenges such as competing interests, differing values, and potential miscommunication. To this end, we will leverage the recently released Concordia framework, an open-source library for defining open-ended environments where LM agents like those of Park et al. (2023) can interact with one another by generating free-form natural text describing what they intend to do or say. Concordia provides a suite of mixed-motive social dilemma scenarios where cooperation is valuable but hard to achieve. The proposed contest will challenge the participants to develop LM agents that exhibit cooperative intelligence in a variety of Concordia scenarios designed to assess multiple distinct skills of cooperation, including promise-keeping, negotiation, reciprocity, reputation, partner choice, compromise, and sanctioning. Participants will be scored based on the ability of their trained agents in executing skillful cooperation, particularly in the presence of new co-players in unforeseen (held-out) scenarios. Given the rapid development of LMs and the anticipated increase in the use of personalised LM agents, we contend that their propensity and ability to cooperate well with a diverse array of other actors (human or machine) will soon be of critical importance.

In the past two years, generative AI has been the major driving force behind the development of advanced AI productssuch as ChatGPT4, AlphaFold, and StableDiffusion. These technologies, while significantly improving productivity for many, have raised significant safety concerns. However, there has been no workshop focusing on this topic in the past two years. This workshop, emphasizing AI safety concerns related to the use of generative AI, is very needed for the community. Generative AI, including large language models, vision-language models, diffusion models, and many more, has significantly aided various aspects of both academia and industry. In scientific discovery, these aspects encompass experimental design, hypothesis formulation, theoretical reasoning, and observation organization. In commercial applications, generative models such as large language models and diffusion algorithms have changed the lifestyles and workflows of billions around the world. This workshop aims to convene experts from various fields to address these challenges and explore potential solutions.

Machine learning and compression have been described as "two sides of the same coin", and the exponential amounts of data being generated in diverse domains underscores the need for improved compression as well as efficient AI systems. Leveraging deep generative models, recent machine learning-based methods have set new standards for compressing images, videos, and audio. Despite these strides, significant challenges, such as computational efficiency and theoretical limitations, remain. Parallel advances in large-scale foundation models further requires research in efficient AI techniques such as model compression and distillation. This workshop aims to unite researchers from machine learning, data/model compression, and information theory. It will focus on enhancing compression techniques, accelerating large model training and inference, exploring theoretical limits, and integrating information-theoretic principles to improve learning and generalization. By bridging disciplines, we seek to catalyze the next generation of scalable, efficient information-processing systems.

Optimization lies at the heart of many machine learning algorithms and enjoys great interest in our community. Indeed, this intimate relation of optimization with ML is the key motivation for the OPT series of workshops. We aim to foster discussion, discovery, and dissemination of state-of-the-art research in optimization relevant to ML.

The focus of OPT 2024 is on "Scaling up optimization". The advent of large language models (LLMs) has changed our perceptions of the landscape of optimization and is resulting in the emergence of new interesting questions related to scaling. For instance, we can view optimization as a sequence of problems parameterized by the size of the model. Questions naturally arise around scaling and optimization. Are there natural model size dependent learning rates that allow extrapolation from smaller models to large ones, and therefore facilitating fine-tuning? Or given a fixed compute budget, how should one choose the hyper-parameters of the model (e.g., width size, depth size, architecture, batch) so as to minimize the loss function? How dependent are these scaling laws on the optimization algorithm? Answers to these questions would have a huge impact in AI – saving time and millions of dollars in training, plus helping reduce AI’s environmental impact through reducing energy costs. The new area of scaling laws and its deep ties to the optimization community warrants a necessary discussion.

The development and proliferation of modern generative AI models has introduced valuable capabilities, but these models and their applications also introduce risks to human safety. How do we identify risks in new systems before they cause harm during deployment? This workshop focuses on red teaming, an emergent adversarial approach to probing model behaviors, and its applications towards making modern generative AI safe for humans.

The competition will advance modern algorithms in AI and machine learning through a highly topical interdisciplinary competition challenge: The prediction of hi-res rain radar movies from multi-band satellite sensors requires data fusion of complementary signal sources, multi-channel video frame prediction, as well as super-resolution techniques. To reward models that extract relevant mechanistic patterns reflecting the underlying complex weather systems our evaluation incorporates spatio-temporal shifts: Specifically, algorithms need to forecast several hours of ground-based hi-res precipitation radar from lo-res satellite spectral images in a unique cross-sensor prediction challenge. Models are evaluated within and across regions on Earth with diverse climate and different distributions of heavy precipitation events. Conversely, robustness over time is achieved by testing predictions on data one year after the training period.Now, in its third year, Weather4acst 2024 aims to improve rain forecasts world-wide on an expansive data set with over a magnitude more hi-res rain radar data, allowing a move towards Foundation Models through multi-modality, multi-scale, multi-task challenges. Accurate rain predictions are becoming ever more critical for everyone, with climate change increasing the frequency of extreme precipitation events. Notably, the new models and insights will have a particular impact for the many regions on Earth where costly weather radar data are not available. Join us on www.weather4cast.net!

Generative AI systems are becoming increasingly prevalent in society across modalities, producing content such as text, images, audio, and video, with far-reaching implications. The NeurIPS Broader Impact statement has notably shifted norms for AI publications to consider negative societal impact. However, no standard exists for how to approach these impact assessments. While new methods for evaluation of social impact are being developed, including notably through the NeurIPS Datasets and Benchmarks track, the lack of standard for documenting their applicability, utility, and disparate coverage of different social impact categories stand in the way of broad adoption by developers and researchers of generative AI systems. By bringing together experts on the science and context of evaluation and practitioners who develop and analyze technical systems, we aim to help address this issue through the work of the NeurIPS community.

"Erasing the Invisible" is a pioneering competition designed to rigorously stress-test image watermarks, aiming to enhance their robustness significantly. Its standout feature is the introduction of dual tracks for black-box and beige-box attacks, providing a nuanced approach to validate the reliability and robustness of watermarks under varied conditions of visibility and knowledge. The competition spans from July 18 to October 31, inviting individuals and teams to register and participate in a dynamic challenge. Throughout the competition, employing a dataset of 10k images accessed through the Hugging Face API, competitors will receive updated evaluation results on a rolling basis and submit their refined techniques for the final evaluation, which will be conducted on an extensive dataset of 50k images. The evaluation process of this competition not only emphasizes the effectiveness of watermark removal but also highlights the critical importance of maintaining image quality, with results reflected on a continuously updated leaderboard. "Erasing the Invisible" promises to elevate watermarking technology to new heights of resilience, setting a precedent for future research and application in digital content security and safeguarding against unauthorized use and misinformation in the digital age.

The Edge-Device Large Language Model Competition seeks to explore the capabilities and potential of large language models (LLMs) deployed directly on edge devices. The incredible capacity of LLMs makes it extremely tantalizing to be applied to practical edge devices to enable wide applications of LLMs in various disciplines. However, the massive size of LLMs poses significant challenges for edge devices where the computing resources and memory are strictly limited. For instance, deploying a small-scale 10B LLM could require up to 20GB of main memory (DRAM) even after adopting INT8 quantization, which unfortunately has exceeded the memory of most commodity smartphones. Besides, the high energy consumption of LLMs will drain smartphones' battery quickly. To facilitate applications of LLMs in a wide range of practical scenarios, we propose this timely competition to encourage practitioners in both academia and industry to come up with effective solutions for this pressing need. By challenging participants to develop efficient and optimized models that can run on resource-constrained edge devices, the competition aims to address critical economic and environmental issues related to LLMs, foster interdisciplinary research collaborations, and enhance the privacy and security of AI systems.

Small molecule drugs are often discovered using a brute force physical search,wherein scientists test for interactions between candidate drugs and their proteintargets in a laboratory setting. As druglike chemical space is large (10^60), moreefficient methods to search through this space are desirable. To enable the discoveryand application of such methods, we generated the Big Encoded Library forChemical Assessment (BELKA), roughly 3.6B physical binding measurementsbetween 133M small molecules and 3 protein targets using DNA-encoded chemicallibrary technology. We hope this dataset encourages the community to exploremethods to represent small molecule chemistry and predict likely binders usingchemical and protein target structure.

Ensuring safety emerges as a pivotal objective in developing large language models(LLMs) and LLM-powered agents. The Competition for LLM and Agent Safety(CLAS) aims to advance the understanding of the vulnerabilities in LLMs andLLM-powered agents and to encourage methods for improving their safety. Thecompetition features three main tracks linked through the methodology of promptinjection, with tasks designed to amplify societal impact by involving practicaladversarial objectives for different domains. In the Jailbreaking Attack track,participants are challenged to elicit harmful outputs in guardrail LLMs via promptinjection. In the Backdoor Trigger Recovery for Models track, participants aregiven a CodeGen LLM embedded with hundreds of domain-specific backdoors.They are asked to reverse-engineer the trigger for each given target. In the Back-door Trigger Recovery for Agents track, trigger reverse engineering will befocused on eliciting specific backdoor targets based on malicious agent actions. Asthe first competition addressing the safety of both LLMs and LLM agents, CLAS2024 aims to foster collaboration between various communities promoting researchand tools for enhancing the safety of LLMs and real-world AI systems.