We study dynamic finite-player and mean-field stochastic games within the framework of Markov perfect equilibria (MPE). Unlike their continuous-time analogues, discrete-time finite-player games generally do not admit unique MPE. However, we show that uniqueness is remarkably recovered when the time steps are sufficiently small, and we provide examples demonstrating the necessity of this assumption. This result, established without relying on any monotonicity conditions, underscores the importance of inertia in dynamic games. Furthermore, we discuss different learning algorithms and prove their convergence to the unique MPE.