The University of Southampton
University of Southampton Institutional Repository

Thompson sampling based Monte-Carlo planning in POMDPs

Bai, Aijun, Wu, Feng, Zhang, Zongzhang and Chen, Xiaoping (2014) Thompson sampling based Monte-Carlo planning in POMDPs At Proceedings of the 24th International Conference on Automated Planning and Scheduling (ICAPS-14), United States. 21 - 26 Jun 2014.

Record type: Conference or Workshop Item (Paper)


Monte-Carlo tree search (MCTS) has been drawing
great interest in recent years for planning under uncertainty. One of the key challenges is the tradeoff
between exploration and exploitation. To address
this, we introduce a novel online planning algorithm
for large POMDPs using Thompson sampling based
MCTS that balances between cumulative and simple regrets.
The proposed algorithm — Dirichlet-Dirichlet-
NormalGamma based Partially Observable Monte-
Carlo Planning (D2NG-POMCP) — treats the accumulated
reward of performing an action from a belief
state in the MCTS search tree as a random variable following
an unknown distribution with hidden parameters.
Bayesian method is used to model and infer the
posterior distribution of these parameters by choosing
the conjugate prior in the form of a combination of two
Dirichlet and one NormalGamma distributions. Thompson
sampling is exploited to guide the action selection in
the search tree. Experimental results confirmed that our
algorithm outperforms the state-of-the-art approaches
on several common benchmark problems.

PDF full.pdf - Author's Original
Download (361kB)

More information

Accepted/In Press date: June 2014
Venue - Dates: Proceedings of the 24th International Conference on Automated Planning and Scheduling (ICAPS-14), United States, 2014-06-21 - 2014-06-26
Organisations: Agents, Interactions & Complexity


Local EPrints ID: 360985
PURE UUID: 1054b540-c04e-4424-833d-a2949c9e365f

Catalogue record

Date deposited: 14 Jan 2014 10:39
Last modified: 18 Jul 2017 03:08

Export record


Author: Aijun Bai
Author: Feng Wu
Author: Zongzhang Zhang
Author: Xiaoping Chen

University divisions

Download statistics

Downloads from ePrints over the past year. Other digital versions may also be available to download e.g. from the publisher's website.

View more statistics

Atom RSS 1.0 RSS 2.0

Contact ePrints Soton:

ePrints Soton supports OAI 2.0 with a base URL of

This repository has been built using EPrints software, developed at the University of Southampton, but available to everyone to use.

We use cookies to ensure that we give you the best experience on our website. If you continue without changing your settings, we will assume that you are happy to receive cookies on the University of Southampton website.