We consider prediction problems where outcomes in the training data are missing not at random (MNAR). MNAR outcomes can induce arbitrary levels of bias in downstream prediction tasks. To counteract this bias, one may (1) incorporate additional information that is external to the data, or (2) collect additional data under a different policy from the policy that generated the original dataset. For (1), we consider making models robust to MNAR via distributionally robust optimization. For (2), we develop an active learning approach in which the model training procedure and the acquisition function are attuned to the MNAR setting. Experiments demonstrate the benefits of this approach over standard active and passive learning approaches.