This Article Full Text (PDF) References Alert me when this article is cited Alert me if a correction is posted Services Email this article to a friend Similar articles in this journal Similar articles in Web of Science Alert me to new issues of the journal Add to Saved Citations Download to citation manager Request Permissions Request Reprints Add to My Marked Citations Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Citing Articles via Scopus Google Scholar Articles by Whiteson, S. Articles by Stone, P. Search for Related Content Social Bookmarking                         What's this?

Empirical Studies in Action Selection with Reinforcement Learning

Department of Computer Sciences, University of Texas, Austin, USA, shimon{at}cs.utexas.edu

Matthew E. Taylor

Department of Computer Sciences, University of Texas, Austin, USA

Peter Stone

Department of Computer Sciences, University of Texas, Austin, USA

To excel in challenging tasks, intelligent agents need sophisticated mechanisms for action selection: they need policies that dictate what action to take in each situation. Reinforcement learning (RL) algorithms are designed to learn such policies given only positive and negative rewards. Two contrasting approaches to RL that are currently in popular use are temporal difference (TD) methods, which learn value functions, and evolutionary methods, which optimize populations of candidate policies. Both approaches have had practical successes but few studies have directly compared them. Hence, there are no general guidelines describing their relative strengths and weaknesses. In addition, there has been little cross-collaboration, with few attempts to make them work together or to apply ideas from one to the other. In this article we aim to address these shortcomings via three empirical studies that compare these methods and investigate new ways of making them work together.

First, we compare the two approaches in a benchmark task and identify variations of the task that isolate factors critical to the performance of each method. Second, we investigate ways to make evolutionary algorithms excel at on-line tasks by borrowing exploratory mechanisms traditionally used by TD methods. We present empirical results demonstrating a dramatic performance improvement. Third, we explore a novel way of making evolutionary and TD methods work together by using evolution to automatically discover good representations for TD function approximators. We present results demonstrating that this novel approach can outperform both TD and evolutionary methods alone.

Key Words: reinforcement learning • temporal difference methods • evolutionary computation • neural networks • robot soccer • autonomic computing

Adaptive Behavior, Vol. 15, No. 1, 33-50 (2007)
DOI: 10.1177/1059712306076253


CiteULike    Complore    Connotea    Del.icio.us    Digg    Reddit    Technorati    Twitter    What's this?

This article has been cited by other articles:

				P. Ozturk Levels and Types of Action Selection: The Action Selection Soup Adaptive Behavior, December 1, 2009; 17(6): 537 - 554. [Abstract] [PDF]