We consider one-dimensional excited random walks (ERWs) with i.i.d. Markovian cookie stacks in the non-boundary recurrent regime. We prove that under diffusive scaling such an ERW converges in the standard Skorokhod topology to a multiple of Brownian motion perturbed at its extrema (BMPE). All parameters of the limiting process are given explicitly in terms of those of the cookie Markov chain at a single site. While our results extend the results in Dolgopyat and Kosygina (Electron Commun Probab 17:1-14, 2012) (ERWs with boundedly many cookies per stack) and Kosygina and Peterson (Electron J Probab 21:1-24, 2016) (ERWs with periodic cookie stacks), the approach taken is very different and involves coarse graining of both the ERW and the random environment changed by the walk. Through a careful analysis of the environment left by the walk after each "mesoscopic" step, we are able to construct a coupling of the ERW at this "mesoscopic" scale with a suitable discretization of the limiting BMPE. The analysis is based on generalized Ray-Knight theorems for the directed edge local times of the ERW stopped at certain stopping times and evolving in both the original random cookie environment and (which is much more challenging) in the environment created by the walk after each "mesoscopic" step.