Physics Informed Neural Networks (PINNs) promise performance gains in solving Partial Differential Equations related to diverse applications. Yet, their training can be challenging, attributed in part to their unique loss function components. This study examines the optimization trajectory of PINNs for the heat equation, comparing it to a similarly-architected regression model. Our initial findings suggest that PINNs experience prolonged plateaus and unstable training behaviors predominantly due to misaligned update step. This research shines a light on underlying training dynamics, paving the way for improved PINN training methods.