Curvature-Guided Parameter Initialization for Multi-Task Learning

Multi-task learning (MTL) aims to jointly optimize multiple related tasks to obtain shared representations within a single model. Recent research has focused on developing MTL algorithms to alter optimization dynamics through task re-weighting or gradient manipulation. However, in this paper, we first empirically identify that current MTL approaches are highly sensitive to the model initialization, complicating empirical evaluation and limiting practical reliability, which has been overlooked but appears to be critical. To better understand this phenomenon, we provide a local theoretical analysis that motivates the connection between early-stage MTL optimization behavior and curvature around initialization. Motivated by this analysis, we propose a Curvature-guided Parameter Initialization (CPI) approach tailored for MTL. Specifically, CPI performs a short warm-up phase to probe local curvature around the initial parameters and heuristically biases the initialization toward empirically favorable curvature profiles for subsequent multi-task optimization. The proposed approach is optimizer-agnostic, requires no modification to existing MTL algorithms, and can be seamlessly integrated as a plug-in prior to standard training. Extensive experiments on standard MTL benchmarks show that CPI improves the aggregate multi-task trade-off while remaining compatible with mainstream MTL methods.