As reinforcement learning continues to advance, the integration of efficient planning algorithms with powerful representation learning becomes crucial for solving long-horizon tasks. We address key challenges in planning, reward learning, and representation learning through the objective of learning value-based abstractions. We explore this idea via goal-conditioned reinforcement learning to learn generalizable value functions and action-free pre-training. By leveraging self-supervised reinforcement learning and efficient planning algorithms, these approaches collectively contribute to the advancement of decision-making systems capable of learning and adapting to diverse tasks in real-world environments.