Finding specific preference-guided Pareto solutions that represent different trade-offs among multiple objectives is critical yet challenging in multi-objective problems. Existing approaches to tackling this problem have limitations in restrictive preference definitions and/or their theoretical guarantees. In this work, we propose a Flexible framEwork for pREfeRence-guided multi-Objective learning (FERERO) by casting it as a constrained vector optimization problem. Specifically, two preferences are incorporated into this formulation -- the relative preference defined by the partial ordering induced by a general polyhedral cone, and the absolute preference defined by constraints that are linear functions of the objectives. To solve this problem, algorithms are developed with stochastic variants and finite-time convergence guarantees. With the flexible definitions of the preferences, the algorithms are adaptive in the sense that they do not require solving different subprograms at different stages depending on whether the constraints are satisfied, but automatically adapt to the constraint function values. Experiments on multiple benchmarks demonstrate the proposed method is capable of finding preference-guided Pareto optimal solutions.