We look at the problem of learning causal structure for a fixed downstream causal effect optimization task. In contrast to previous work which often focuses on running interventional experiments, we consider an often overlooked source of information - a domain expert. In the Bayesian setting this amounts to augmenting the likelihood with a user model whose parameters account for possible biases of the expert. Such a model allows for active elicitation in a manner that is most informative to the optimization task at hand.