Journal of Applied Mathematics
Volume 2012 (2012), Article ID 103698, 16 pages
http://dx.doi.org/10.1155/2012/103698
Research Article

A Version of the Euler Equation in Discounted Markov Decision Processes

H. Cruz-Suárez, G. Zacarías-Espinoza, and V. Vázquez-Guevara

Facultad de Ciencias Físico Matemáticas, Benemérita Universidad Autónoma de Puebla, Avenida San Claudio y Río Verde, Col. San Manuel, CU, 72570 Puebla, PUE, Mexico

Received 21 July 2012; Revised 29 August 2012; Accepted 6 September 2012

Academic Editor: Vu Phat

Copyright © 2012 H. Cruz-Suárez et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract

This paper deals with Markov decision processes (MDPs) on Euclidean spaces with an infinite horizon. An approach to study this kind of MDPs is using the dynamic programming technique (DP). Then the optimal value function is characterized through the value iteration functions. The paper provides conditions that guarantee the convergence of maximizers of the value iteration functions to the optimal policy. Then, using the Euler equation and an envelope formula, the optimal solution of the optimal control problem is obtained. Finally, this theory is applied to a linear-quadratic control problem in order to find its optimal policy.