Value alignment has been widely argued to be one of the central safety problems in AI. While the problem itself arises from the way humans interact with the AI systems, most current solutions to value alignment tend to sideline the human or make unrealistic assumptions about possible human interactions. In this position paper, we propose a human-centered formalization of the value alignment problem that generalizes human-AI interaction frameworks that were originally developed for explainable AI. We see how such a human-aware formulation of the problem provides us with novel ways of addressing and understanding the problem.