Though research in algorithmic fairness is rapidly expanding, most of the existing work are tailored towards and bench-marked against social datasets. There is limited work which takes into holistic account the specific challenges unique to affect recognition. We outline some key specific challenges unique to affective computing and highlight how existing causal fairness methods and mechanisms are insufficient to fully address them.