A Heuristic Variable Grid Solution Method for POMDPs

Abstract

Partially observable Markov decision processes (POMDPs) are an appealing tool for modeling planning problems under uncertainty. They incorporate stochastic action and sensor descriptions and easily capture goal oriented and process oriented tasks. Unfortunately, POMDPs are very difficult to solve. Exact methods cannot handle problems with much more than 10 states, so approximate methods must be used. In this paper, we describe a simple variable-grid solution method which yields good results on relatively large problems with modest computational effort. Introduction Markov decision processes (MDPs) (Bellman 1962) provide a mathematically elegant model of planning problems where actions have stochastic effects and tasks can be process oriented or have a more complex, graded notion of goal state. Partially observable MDPs (POMDPs) enhance this model, allowing for noisy and imperfect sensing, as well. Unfortunately, solving POMDPs, i.e., obtaining the optimal prescription for action choic...