Title: Towards AI 4 all: From AutoML to AutoML+

Abstract: Automated Machine Learning (AutoML) has long been a "dream" pursued by ML researchers. In this talk, we share our journey in advancing the frontiers of AutoML through the organization of machine learning challenges. We propose a classification of challenge types based on the information available to participants or organizers, highlighting how, over time, increasing autonomy has been delegated to autonomous agents. Additionally, greater rigor in evaluation has been achieved by providing organizers with more control to train and test agents on challenge platforms. However, as models continue to grow in size, resource constraints have made it increasingly difficult to train and test agents in controlled environments. Does this compromise the rigor of evaluations?