TinyML aims to enhance paradigms like healthcare, surveillance, and activity detection, etc. by scaling down Machine Learning (ML) algorithms to the level of resource-constrained devices such as microcontrollers (MCUs). MLOps practices for deploying, monitoring, and updating ML models in production, which can be challenging due to the limitations of MCU devices. Therefore, this paper highlights various key challenges for the successful training, deployment, and monitoring of ML models on MCUs and their limitations from both hardware and software perspectives. Such difficulties have an impact on the productivity, dependability, and scalability of TinyML systems.

Imitation learning is emerging as one of the promising approaches for enabling robots to acquire abilities. Since imitation learning provides methods of learning policies through imitation of an expert’s behavior, it requires sophisticated and sufficient expert behavior trajectories. However, current interfaces for imitation such as kinesthetic teaching or remote operation considerably restrict the ability to efficiently collect diverse demonstration data. To address this challenge, this work proposes an alternative interface for imitation, which can easily transfer human motions to robots while simplifying the demo collection process using a VR-based hand tracking solution. In addition, a novel method that performs data augmentation on expert trajectories is proposed to improve imitation performance. Experimental results showed that the proposed method is effective in collecting expert demonstrations and augmenting the expert trajectories and successfully completing robot manipulation tasks.

Enterprise adoption of AI/ML services has significantly accelerated in the last few years. However, the majority of ML models are still developed with the goal of solving a single task, e.g., prediction, classification. In this context, Compositional AI envisions seamless composition of existing AI/ML services, to provide a new (composite) AI/ML service, capable of addressing complex multi-domain use-cases. In this work, we consider two MLOps aspects that need to be enabled to realize Composable AI scenarios: (i) integration of DataOps and MLOps, and (ii) extension of the integrated DataOps-MLOps pipeline such that inferences made by a deployed ML model can be provided as training dataset for a new model. In an enterprise AI/ML environment, this enables reuse, agility, and efficiency in development and maintenance efforts.

Multi-Task Learning (MTL) has shown its importance at user products for fasttraining, data efficiency, reduced overfitting etc. MTL achieves it by sharing  the network parameters and training a network for multiple tasks simultaneously.  However, MTL does not provide the solution, if each task needs training from  a different dataset. In order to solve the stated problem, we have proposed an  architecture named TreeDNN along with it’s training methodology. TreeDNN  helps in training the model with multiple datasets simultaneously, where each  branch of the tree may need a different training dataset. We have shown in the  results that TreeDNN provides competitive performance with the advantage of  reduced ROM requirement for parameter storage and increased responsiveness of  the system by loading only specific branch at inference time.

Building reliable AI decision support systems requires a robust set of data on which to train models; both with respect to quantity and diversity. Obtaining such datasets can be difficult in resource limited settings, or for applications in early stages of deployment. Sample rejection is one way to work around this challenge, however much of the existing work in this area is ill-suited for such scenarios. This paper substantiates that position and proposes a simple solution as a proof of concept baseline.

Neural network compression can open up new deployment schemes for deep learning models by making it feasible to ship deep neural networks with millions of parameters directly within a mobile or web app rather than running them on a remote data center, thus reducing server costs, network usage, latency, and privacy concerns. In this paper, we propose and empirically evaluate a simple and generic compression method for trained neural networks that builds on variational inference and on the Variational Bayesian Quantization algorithm [Yang et al., 2020]. We find that the proposed method achieves significantly lower bit rates than existing post-training compression methods at comparable model performance. The proposed method demonstrates a new use case of Bayesian neural networks (BNNs), and we analyze how compression performance depends on the temperature of a BNN.

Following the development of digitization, a growing number of large Original Equipment Manufacturers (OEMs) are adapting computer vision or natural language processing in a wide range of applications such as anomaly detection and quality inspection in plants. Deployment of such system is becoming a critical topic. Our work starts with the least-automated deployment technologies of machine learning systems, includes several iterations of updates, and ends with a comparison of automated deployment techniques. The objective is, on the one hand, to compare the pros and cons of various technologies in theory and practice so as to facilitate later adopters to avoid making generalized mistakes when implementing actual use cases and thereby choose a better strategy for their own enterprises. On the other hand, to raise the awareness of the evaluation framework for the deployment of machine learning systems, to have more comprehensive and valuable evaluation metrics rather than only focusing on a single factor (e.g., company's cost). This is especially important for decision-makers in the industry.

Inference is a significant part of ML software infrastructure. Despite the variety of inference frameworks available, the field as a whole can be considered in its early days. This paper puts forth a range of important qualities that next generation of inference platforms should be aiming for. We present our rationale for the importance of each quality, and discuss ways to achieve it in practice.

ML models often operate within the context of a larger system that can adapt its response when the ML model is uncertain, such as falling back on safe defaults or a human in the loop. This commonly encountered operational context calls for principled techniques for training ML models with the option to abstain from predicting when uncertain. Selective neural networks are trained with an integrated option to abstain, allowing them to learn to recognize and optimize for the subset of the data distribution for which confident predictions can be made. However, optimizing selective networks is challenging due to the non-differentiability of the binary selection function (the discrete decision of whether to predict or abstain). This paper presents a general method for training selective networks that leverages the Gumbel-softmax reparameterization trick to enable selection within an end-to-end differentiable training framework. Experiments on public datasets demonstrate the potential of Gumbel-softmax selective networks for selective regression and classification.

The rapid and growing popularity of machine learning (ML) applications has led to an increasing interest in MLOps, i.e., the practice of continuous integration and deployment (CI/CD) of ML-enabled systems. Since changes may affect not only the code but also the ML model parameters and the data themselves, the automation of traditional CI/CD needs to be extended to manage model retraining in production.Here we present an initial investigation of the MLOps practices implemented in a set of ML-enabled systems retrieved from GitHub. Our preliminary results suggest that the current adoption of MLOps workflows in open-source GitHub projects is rather limited. Issues are also identified, which can guide future research work.

Panel Chair: Prof Stephen Roberts

Predictive models are increasingly used for consequential decisions in high-stakes domains such as healthcare, finance, and policy. It becomes critical to ensure that these models make accurate predictions, are robust to shifts in the data, do not rely on spurious features, and do not unduly discriminate against minority groups. To this end, several approaches spanning various areas such as explainability, fairness, and robustness have been proposed in recent literature. Such approaches need to be human-centered as they cater to the understanding of the models to their users. However, there is a research gap in understanding the human-centric needs and challenges of monitoring machine learning (ML) models once they are deployed. To fill this gap, we conducted an interview study with 13 practitioners who have experience at the intersection of deploying ML models and engaging with customers spanning domains such as financial services, healthcare, hiring, online retail, computational advertising, and conversational assistants. We identified various human-centric challenges and requirements for model monitoring in real-world applications. Specifically, we found the need and the challenge for the model monitoring systems to clarify the impact of the monitoring observations on outcomes. Further, such insights must be actionable, customizable for domain-specific use cases, and cognitively considerate to avoid information overload.

Automated slicing aims to identify subsets of evaluation data where a trained model performs anomalously. This is an important problem for machine learning pipelines in production since it plays a key role in model debugging and comparison, as well as the diagnosis of fairness issues. Scalability has become a critical requirement for any automated slicing system due to the large search space of possible slices and the growing scale of data. We present AutoSlicer, a scalable system that searches for problematic slices through distributed metric computation and hypothesis testing. We develop an efficient strategy that reduces the search space through pruning and prioritization. In the experiments, we show that our search strategy finds most of the anomalous slices by inspecting a small portion of the search space.

ML models often operate within the context of a larger system that can adapt its response when the ML model is uncertain, such as falling back on safe defaults or a human in the loop. This commonly encountered operational context calls for principled techniques for training ML models with the option to abstain from predicting when uncertain. Selective neural networks are trained with an integrated option to abstain, allowing them to learn to recognize and optimize for the subset of the data distribution for which confident predictions can be made. However, optimizing selective networks is challenging due to the non-differentiability of the binary selection function (the discrete decision of whether to predict or abstain). This paper presents a general method for training selective networks that leverages the Gumbel-softmax reparameterization trick to enable selection within an end-to-end differentiable training framework. Experiments on public datasets demonstrate the potential of Gumbel-softmax selective networks for selective regression and classification.

We describe the current content moderation strategy employed by Meta to remove policy-violating content from its platforms. Meta relies on both handcrafted and learned risk models to flag potentially violating content for human review. Our approach aggregates these risk models into a single ranking score, calibrating them to prioritize more reliable risk models. A key challenge is that violation trends change over time, affecting which risk models are most reliable. Our system additionally handles production challenges such as changing risk models and novel risk models. We use a contextual bandit to update the calibration in response to such trends. Our approach increases Meta's top-line metric for measuring the effectiveness of its content moderation strategy by 13%.

Prior research has shown that artificial intelligence (AI) systems often encode biases against minority subgroups; however, little work has focused on ways to mitigate the harm discriminatory algorithms can cause in high-stakes settings such as medicine. In this study, we experimentally evaluated the impact biased AI recommendations have on emergency decisions, where participants respond to mental health crises by calling for either medical or police assistance. We found that although respondent decisions were not biased without advice, both clinicians and non-experts were influenced by prescriptive recommendations from a biased algorithm, choosing police help more often in emergencies involving African-American or Muslim men. Crucially, we also found that using descriptive flags rather than prescriptive recommendations allowed respondents to retain their original, unbiased decision-making. Our work demonstrates the practical danger of using biased models in health contexts, and suggests that appropriately framing decision support can mitigate the effects of AI bias. These findings must be carefully considered in the many real-world clinical scenarios where inaccurate or biased models may be used to inform important decisions.

As machine learning techniques are increasingly used to make societal-scale decisions, model performance issues stemming from data-drift can result in costly consequences. While methods exist to quantify data-drift, a further classification of drifted points into groups of similarly anomalous points can be helpful for practitioners as a means to combating drift (e.g. by providing context about how/where in the data pipeline shift might be introduced). We show how such characterization is possible by making use of tools from the model explainability literature. We also show how simple rules can be extracted to generate database queries for anomalous data and detect anomalous data in the future.

Edge inference is becoming ever prevalent through its applications from retail to wearable technology. Clusters of networked resource-constrained edge devices are becoming common, yet there is no production-ready orchestration system for deploying deep learning models over such edge networks which adopts the robustness and scalability of the cloud. We present SEIFER, a framework utilizing a standalone Kubernetes cluster to partition a given DNN and place these partitions in a distributed manner across an edge network, with the goal of maximizing inference throughput. The system is node fault-tolerant and automatically updates deployments based on updates to the model's version. We provide a preliminary evaluation of a partitioning and placement algorithm that works within this framework, and show that we can improve the inference pipeline throughput by 200% by utilizing sufficient numbers of resource-constrained nodes. We have implemented SEIFER in open-source software that is publicly available to the research community.

Reinforcement learning-based controllers (RL-controllers) in self-driving datacenters have evolved into complex dynamic systems that require continuous tuning to achieve higher performance than hand-crafted expert heuristics. The operating environment of these controllers poses additional challenges as it can often change significantly, thus the controllers must be adapted to new external conditions. To obtain trustworthy RL-controllers for self-driving datacenters, it is essential to guarantee that RL-controllers that are trained continuously in these changing environments behave according to the designer’s notions of reliability and correctness. Traditionally, RL-controllers are evaluated by comparing their reward function statistics. However, that does not capture all desired properties, e.g., stability, of the controller. In this work, we propose enhancing the evaluation criteria for RL-controllers with a set of novel metrics that quantify how well the controller performs with respect to user-defined properties. We leverage formal methods for computing our novel metrics. Thus, our work makes a step forward toward improving trustworthiness of RL-controllers. We show that these metrics are useful in evaluating a standalone controller or in comparing multiple controllers that achieve the same reward.

Panel discussion with the invited speakers.

Automated slicing aims to identify subsets of evaluation data where a trained model performs anomalously. This is an important problem for machine learning pipelines in production since it plays a key role in model debugging and comparison, as well as the diagnosis of fairness issues. Scalability has become a critical requirement for any automated slicing system due to the large search space of possible slices and the growing scale of data. We present AutoSlicer, a scalable system that searches for problematic slices through distributed metric computation and hypothesis testing. We develop an efficient strategy that reduces the search space through pruning and prioritization. In the experiments, we show that our search strategy finds most of the anomalous slices by inspecting a small portion of the search space.

Imitation learning is emerging as one of the promising approaches for enabling robots to acquire abilities. Since imitation learning provides methods of learning policies through imitation of an expert’s behavior, it requires sophisticated and sufficient expert behavior trajectories. However, current interfaces for imitation such as kinesthetic teaching or remote operation considerably restrict the ability to efficiently collect diverse demonstration data. To address this challenge, this work proposes an alternative interface for imitation, which can easily transfer human motions to robots while simplifying the demo collection process using a VR-based hand tracking solution. In addition, a novel method that performs data augmentation on expert trajectories is proposed to improve imitation performance. Experimental results showed that the proposed method is effective in collecting expert demonstrations and augmenting the expert trajectories and successfully completing robot manipulation tasks.

Multi-Task Learning (MTL) has shown its importance at user products for fasttraining, data efficiency, reduced overfitting etc. MTL achieves it by sharing  the network parameters and training a network for multiple tasks simultaneously.  However, MTL does not provide the solution, if each task needs training from  a different dataset. In order to solve the stated problem, we have proposed an  architecture named TreeDNN along with it’s training methodology. TreeDNN  helps in training the model with multiple datasets simultaneously, where each  branch of the tree may need a different training dataset. We have shown in the  results that TreeDNN provides competitive performance with the advantage of  reduced ROM requirement for parameter storage and increased responsiveness of  the system by loading only specific branch at inference time.

We describe the current content moderation strategy employed by Meta to remove policy-violating content from its platforms. Meta relies on both handcrafted and learned risk models to flag potentially violating content for human review. Our approach aggregates these risk models into a single ranking score, calibrating them to prioritize more reliable risk models. A key challenge is that violation trends change over time, affecting which risk models are most reliable. Our system additionally handles production challenges such as changing risk models and novel risk models. We use a contextual bandit to update the calibration in response to such trends. Our approach increases Meta's top-line metric for measuring the effectiveness of its content moderation strategy by 13%.

Neural network compression can open up new deployment schemes for deep learning models by making it feasible to ship deep neural networks with millions of parameters directly within a mobile or web app rather than running them on a remote data center, thus reducing server costs, network usage, latency, and privacy concerns. In this paper, we propose and empirically evaluate a simple and generic compression method for trained neural networks that builds on variational inference and on the Variational Bayesian Quantization algorithm [Yang et al., 2020]. We find that the proposed method achieves significantly lower bit rates than existing post-training compression methods at comparable model performance. The proposed method demonstrates a new use case of Bayesian neural networks (BNNs), and we analyze how compression performance depends on the temperature of a BNN.

Enterprise adoption of AI/ML services has significantly accelerated in the last few years. However, the majority of ML models are still developed with the goal of solving a single task, e.g., prediction, classification. In this context, Compositional AI envisions seamless composition of existing AI/ML services, to provide a new (composite) AI/ML service, capable of addressing complex multi-domain use-cases. In this work, we consider two MLOps aspects that need to be enabled to realize Composable AI scenarios: (i) integration of DataOps and MLOps, and (ii) extension of the integrated DataOps-MLOps pipeline such that inferences made by a deployed ML model can be provided as training dataset for a new model. In an enterprise AI/ML environment, this enables reuse, agility, and efficiency in development and maintenance efforts.

The rapid and growing popularity of machine learning (ML) applications has led to an increasing interest in MLOps, i.e., the practice of continuous integration and deployment (CI/CD) of ML-enabled systems. Since changes may affect not only the code but also the ML model parameters and the data themselves, the automation of traditional CI/CD needs to be extended to manage model retraining in production.Here we present an initial investigation of the MLOps practices implemented in a set of ML-enabled systems retrieved from GitHub. Our preliminary results suggest that the current adoption of MLOps workflows in open-source GitHub projects is rather limited. Issues are also identified, which can guide future research work.