In the recent years, we have observed an explosion of research approaches at the intersection of causality and fairness in machine learning (ML). These works are often motivated by the promise that causality allows us to reason about the causes of unfairness both in the data and in the ML algorithm. However, the promises of existing causal fair approaches require strong assumptions, which hinder their practical application. In this talk, I will provide a quick overview of both the promises and the technical challenges of causal fair ML frameworks from a theoretical perspective. Finally, I will show how to leverage probabilistic ML to partially relax causal assumptions in order to develop more practical solutions to causal fair ML.