Simultaneous Multi-objective Alignment Across Verifiable and Non-verifiable Rewards

Aligning large language models to human preferences is inherently multidimensional, yet most large language model alignment pipelines reduce diverse human preferences to a single objective. We address simultaneous alignment across domains with verifiable rewards, non-verifiable subjective preferences, and complex interactions. Our framework unifies process reward model (PRM) training across these settings, applies $\textbf{M}$ulti-$\textbf{A}$ction-$\textbf{H}$ead $\textbf{DPO}$ (MAH-DPO) with a vectorized reward to capture multiple objectives, and enables fine-grained user control at inference time. Our experiments show improved multi-objective performance with reduced trade-offs and enhanced controllability.