Case Study: Applying Decision Focused Learning in the Real World

Many real world optimization problems with unknown parameters are solved using the predict-then-optimize framework where a learnt model predicts the parameters of an optimization problem which is subsequently solved using an optimization algorithm.However, this approach maximises for the predictive accuracy rather than the quality of the final solution. Decision Focused Learning (DFL) solves this objective mismatch by integrating the optimization problem in the learning pipeline. Previous works have only shown the applicability of DFL in simulation setting. In our work, we consider the optimization problem of scheduling limited live service calls in Maternal and Child Health Awareness Programs and model it using Restless Multi-Armed Bandits (RMAB).We present results from a large-scale field study consisting of 9000 beneficiaries and demonstrate that DFL cuts $\sim 200\%$ more call engagement drops as compared to previous methods. Through detailed post-hoc analysis, we show that high predictive accuracy of problem parameters is not sufficient to ensure a well-performing system. We also demonstrate that DFL makes optimal decision choices by learning a better decision boundary between the RMAB actions, and by correctly predicting parameterswhich contribute most to the final decision outcome.