Predictive State Representations with Options

Abstract

Recent work on predictive state representation (PSR) models has focused on using predictions of the outcomes of open-loop action sequences as state. These predictions answer questions of the form What is the probability of seeing observation sequence o1, o2, ..., oN if the agent takes action sequence a1, a2, ..., aN from some given history? We would like to ask more expressive questions in our representation of state, such as If I behave according to some policy until I terminate, what will be my last observation? We extend the linear PSR framework to answer questions like these about options -- temporally extended, closed-loop courses of action -- bounding the size of the linear PSR needed to model questions about a certain class of options. We introduce a hierarchical PSR (HPSR) that can make predictions about both options and primitive action sequences and show empirical results from learning HPSRs in simple domains.