Timezone: »

Poster
Benchopt: Reproducible, efficient and collaborative optimization benchmarks
Thomas Moreau · Mathurin Massias · Alexandre Gramfort · Pierre Ablin · Pierre-Antoine Bannier · Benjamin Charlier · Mathieu Dagréou · Tom Dupre la Tour · Ghislain DURIF · Cassio F. Dantas · Quentin Klopfenstein · Johan Larsson · En Lai · Tanguy Lefort · Benoît Malézieux · Badr MOUFAD · Binh T. Nguyen · Alain Rakotomamonjy · Zaccharie Ramzi · Joseph Salmon · Samuel Vaiter

Thu Dec 01 02:00 PM -- 04:00 PM (PST) @ Hall J #915
Numerical validation is at the core of machine learning research as it allows us to assess the actual impact of new methods, and to confirm the agreement between theory and practice. Yet, the rapid development of the field poses several challenges: researchers are confronted with a profusion of methods to compare, limited transparency and consensus on best practices, as well as tedious re-implementation work. As a result, validation is often very partial, which can lead to wrong conclusions that slow down the progress of research. We propose Benchopt, a collaborative framework to automatize, publish and reproduce optimization benchmarks in machine learning across programming languages and hardware architectures. Benchopt simplifies benchmarking for the community by providing an off-the-shelf tool for running, sharing and extending experiments. To demonstrate its broad usability, we showcase benchmarks on three standard ML tasks: $\ell_2$-regularized logistic regression, Lasso and ResNet18 training for image classification. These benchmarks highlight key practical findings that give a more nuanced view of state-of-the-art for these problems, showing that for practical evaluation, the devil is in the details.

#### Author Information

##### Cassio F. Dantas (INRAE, TETIS, Montpellier)

**Cassio Fraga Dantas** is a research scientist at [INRAE](https://www.inrae.fr/), UMR TETIS, Montpellier, France. Previously, he was a postdoctoral researcher at [IMAG](https://imag.edu.umontpellier.fr/) mathematics laboratory of Montpellier, in the probability and statistics team (EPS) and, prior to that, he was a 2-year postdoctoral researcher at [IRIT](https://www.irit.fr/) computer science laboratory of Toulouse, within the ERC project [FACTORY](http://projectfactory.irit.fr/) in 2020 and 2021. He performed his Ph.D studies at [Inria Rennes](https://www.inria.fr/fr/centre-inria-rennes-bretagne-atlantique), in the PANAMA group, and received his degree on signal, image and vision in 2019 from University of Rennes 1. Before that, in 2014, he obtained an engineering degree from the [Ecole Polytechnique](https://www.polytechnique.edu/) of Paris with a double degree and M.Sc in Electrical Engineering from [University of Campinas](https://www.fee.unicamp.br/?language=en), Brazil. He also has over two years of R\&D experience as an engineer at Idea Electronic Systems, LIP6 laboratory and Schneider Electric prior to his Ph.D studies. His recent research activities lie on the frontier between signal processing, machine learning and convex optimization, including contributions on: sparse inverse problem for image processing; matrix and tensor decomposition; and multi-dimensional data modeling.

##### Tanguy Lefort (University of Montpellier France)

- :seedling: Currently in a PhD thesis in Statistics at the University of Montpellier under the supervision of [Joseph Salmon](http://josephsalmon.eu/), [Benjamin Charlier](https://imag.umontpellier.fr/~charlier/index.php?page=index) and [Alexis Joly](http://www-sop.inria.fr/members/Alexis.Joly/wiki/pmwiki.php) (Inria) - :telescope: I am working on image classification and crowdsourced data, take into account label uncertainty and tasks difficulty (more soon) - :superhero: Also a comic books fan