FairMT: Fairness for Heterogeneous Multi-Task Learning

Fairness in multi-task learning (MTL) is challenging when heterogeneous output spaces must be controlled through a shared multi-head model. Existing fairness methods are often tied to a single prediction type, while MTL optimizers coordinate task utility without specifying how heterogeneous group disparities should be represented, aggregated, or propagated through task-specific heads. We introduce FairMT, which frames fair heterogeneous MTL through a coordinate-allocation view. FairMT builds a Fairness-Coordinate Interface for binary detection, one-vs-rest multi-class classification, and scalar regression, and instantiates these coordinates asymmetrically using the current best-performing valid group as a detached reference. This directs correction toward groups falling behind the reference, reducing disparities while better preserving task utility. Asymmetric Heterogeneous Fairness Disparity Aggregation (AHFDA) then adaptively allocates constraint pressure over normalized coordinates and converts heterogeneous disparities into a single differentiable constraint signal. Finally, a Head-Aware Optimization Proxy maps the allocated constraint through task-head geometry into task-combination weights for primal--dual training, reducing routing bias caused by heterogeneous head scales, sensitivities, and decision geometries. Experiments on visual and language MTL benchmarks show that FairMT reduces group-level disparities across heterogeneous output types while maintaining utility close to utility-oriented MTL optimizers.