`

Timezone: »

 
Poster
Algorithmic Assurance: An Active Approach to Algorithmic Testing using Bayesian Optimisation
Shivapratap Gopakumar · Sunil Gupta · Santu Rana · Vu Nguyen · Svetha Venkatesh

Thu Dec 06 02:00 PM -- 04:00 PM (PST) @ Room 517 AB #127

We introduce algorithmic assurance, the problem of testing whether machine learning algorithms are conforming to their intended design goal. We address this problem by proposing an efficient framework for algorithmic testing. To provide assurance, we need to efficiently discover scenarios where an algorithm decision deviates maximally from its intended gold standard. We mathematically formulate this task as an optimisation problem of an expensive, black-box function. We use an active learning approach based on Bayesian optimisation to solve this optimisation problem. We extend this framework to algorithms with vector-valued outputs by making appropriate modification in Bayesian optimisation via the EXP3 algorithm. We theoretically analyse our methods for convergence. Using two real-world applications, we demonstrate the efficiency of our methods. The significance of our problem formulation and initial solutions is that it will serve as the foundation in assuring humans about machines making complex decisions.

Author Information

Shivapratap Gopakumar (Deakin University)
Sunil Gupta (Deakin University)
Santu Rana (Deakin University)
Vu Nguyen (Deakin University)
Svetha Venkatesh (Deakin University)

More from the Same Authors

  • 2021 : Offline neural contextual bandits: Pessimism, Optimization and Generalization »
    Thanh Nguyen-Tang · Sunil Gupta · A. Tuan Nguyen · Svetha Venkatesh
  • 2021 Poster: Model-Based Episodic Memory Induces Dynamic Hybrid Controls »
    Hung Le · Thommen Karimpanal George · Majid Abdolshah · Truyen Tran · Svetha Venkatesh
  • 2021 Poster: Kernel Functional Optimisation »
    Arun Kumar Anjanapura Venkatesh · Alistair Shilton · Santu Rana · Sunil Gupta · Svetha Venkatesh
  • 2021 Poster: Tuning Mixed Input Hyperparameters on the Fly for Efficient Population Based AutoRL »
    Jack Parker-Holder · Vu Nguyen · Shaan Desai · Stephen J Roberts
  • 2020 Poster: Sub-linear Regret Bounds for Bayesian Optimisation in Unknown Search Spaces »
    Hung Tran-The · Sunil Gupta · Santu Rana · Huong Ha · Svetha Venkatesh
  • 2019 : Poster Session »
    Matthia Sabatelli · Adam Stooke · Amir Abdi · Paulo Rauber · Leonard Adolphs · Ian Osband · Hardik Meisheri · Karol Kurach · Johannes Ackermann · Matt Benatan · GUO ZHANG · Chen Tessler · Dinghan Shen · Mikayel Samvelyan · Riashat Islam · Murtaza Dalal · Luke Harries · Andrey Kurenkov · Konrad Żołna · Sudeep Dasari · Kristian Hartikainen · Ofir Nachum · Kimin Lee · Markus Holzleitner · Vu Nguyen · Francis Song · Christopher Grimm · Felipe Leno da Silva · Yuping Luo · Yifan Wu · Alex Lee · Thomas Paine · Wei-Yang Qu · Daniel Graves · Yannis Flet-Berliac · Yunhao Tang · Suraj Nair · Matthew Hausknecht · Akhil Bagaria · Simon Schmitt · Bowen Baker · Paavo Parmas · Benjamin Eysenbach · Lisa Lee · Siyu Lin · Daniel Seita · Abhishek Gupta · Riley Simmons-Edler · Yijie Guo · Kevin Corder · Vikash Kumar · Scott Fujimoto · Adam Lerer · Ignasi Clavera Gilaberte · Nicholas Rhinehart · Ashvin Nair · Ge Yang · Lingxiao Wang · Sungryull Sohn · J. Fernando Hernandez-Garcia · Xian Yeow Lee · Rupesh Srivastava · Khimya Khetarpal · Chenjun Xiao · Luckeciano Carvalho Melo · Rishabh Agarwal · Tianhe Yu · Glen Berseth · Devendra Singh Chaplot · Jie Tang · Anirudh Srinivasan · Tharun Kumar Reddy Medini · Aaron Havens · Misha Laskin · Asier Mujika · Rohan Saphal · Joseph Marino · Alex Ray · Joshua Achiam · Ajay Mandlekar · Zhuang Liu · Danijar Hafner · Zhiwen Tang · Ted Xiao · Michael Walton · Jeff Druce · Ferran Alet · Zhang-Wei Hong · Stephanie Chan · Anusha Nagabandi · Hao Liu · Hao Sun · Ge Liu · Dinesh Jayaraman · John Co-Reyes · Sophia Sanborn
  • 2019 Poster: Bayesian Optimization with Unknown Search Space »
    Huong Ha · Santu Rana · Sunil Gupta · Thanh Nguyen-Tang · Hung Tran-The · Svetha Venkatesh
  • 2019 Poster: Multi-objective Bayesian optimisation with preferences over objectives »
    Majid Abdolshah · Alistair Shilton · Santu Rana · Sunil Gupta · Svetha Venkatesh
  • 2018 Poster: Variational Memory Encoder-Decoder »
    Hung Le · Truyen Tran · Thin Nguyen · Svetha Venkatesh
  • 2017 Poster: Process-constrained batch Bayesian optimisation »
    Pratibha Vellanki · Santu Rana · Sunil Gupta · David Rubin · Alessandra Sutti · Thomas Dorin · Murray Height · Paul Sanders · Svetha Venkatesh
  • 2017 Spotlight: Process-constrained batch Bayesian optimisation »
    Pratibha Vellanki · Santu Rana · Sunil Gupta · David Rubin · Alessandra Sutti · Thomas Dorin · Murray Height · Paul Sanders · Svetha Venkatesh