Approximate dynamic programming via direct search in the space of value function approximations
Approximate dynamic programming via direct search in the space of value function approximations
This paper deals with approximate value iteration (AVI) algorithms applied to discounted dynamic programming (DP) problems. For a fixed control policy, the span semi-norm of the so-called Bellman residual is shown to be convex in the Banach space of candidate solutions to the DP problem. This fact motivates the introduction of an AVI algorithm with local search that seeks to minimize the span semi-norm of the Bellman residual in a convex value function approximation space. The novelty here is that the optimality of a point in the approximation architecture is characterized by means of convex optimization concepts and necessary and sufficient conditions to local optimality are derived. The procedure employs the classical AVI algorithm direction (Bellman residual) combined with a set of independent search directions, to improve the convergence rate. It has guaranteed convergence and satisfies, at least, the necessary optimality conditions over a prescribed set of directions. To illustrate the method, examples are presented that deal with a class of problems from the literature and a large state space queueing problem setting.
Convex optimization, Direct search methods, Dynamic programming, Markov decision processes
343-351
Arruda, E.F.
8eb3bd83-e883-4bf3-bfbc-7887c5daa911
Fragoso, M.D.
7f484139-de97-4458-aa6b-dc3249811a08
Do Val, J.B.R.
4139d2f5-1439-45d9-a77e-8e7e20ec98b8
1 June 2011
Arruda, E.F.
8eb3bd83-e883-4bf3-bfbc-7887c5daa911
Fragoso, M.D.
7f484139-de97-4458-aa6b-dc3249811a08
Do Val, J.B.R.
4139d2f5-1439-45d9-a77e-8e7e20ec98b8
Arruda, E.F., Fragoso, M.D. and Do Val, J.B.R.
(2011)
Approximate dynamic programming via direct search in the space of value function approximations.
European Journal of Operational Research, 211 (2), .
(doi:10.1016/j.ejor.2010.11.019).
Abstract
This paper deals with approximate value iteration (AVI) algorithms applied to discounted dynamic programming (DP) problems. For a fixed control policy, the span semi-norm of the so-called Bellman residual is shown to be convex in the Banach space of candidate solutions to the DP problem. This fact motivates the introduction of an AVI algorithm with local search that seeks to minimize the span semi-norm of the Bellman residual in a convex value function approximation space. The novelty here is that the optimality of a point in the approximation architecture is characterized by means of convex optimization concepts and necessary and sufficient conditions to local optimality are derived. The procedure employs the classical AVI algorithm direction (Bellman residual) combined with a set of independent search directions, to improve the convergence rate. It has guaranteed convergence and satisfies, at least, the necessary optimality conditions over a prescribed set of directions. To illustrate the method, examples are presented that deal with a class of problems from the literature and a large state space queueing problem setting.
This record has no associated files available for download.
More information
Accepted/In Press date: 13 November 2010
e-pub ahead of print date: 13 January 2011
Published date: 1 June 2011
Keywords:
Convex optimization, Direct search methods, Dynamic programming, Markov decision processes
Identifiers
Local EPrints ID: 444753
URI: http://eprints.soton.ac.uk/id/eprint/444753
ISSN: 0377-2217
PURE UUID: 3731aebb-6f72-4aa4-802e-cd03db44cc46
Catalogue record
Date deposited: 03 Nov 2020 17:32
Last modified: 06 Jun 2024 02:09
Export record
Altmetrics
Contributors
Author:
E.F. Arruda
Author:
M.D. Fragoso
Author:
J.B.R. Do Val
Download statistics
Downloads from ePrints over the past year. Other digital versions may also be available to download e.g. from the publisher's website.
View more statistics