Advanced Search

Journal Navigation

Journal Home

Subscriptions

Archive

Contact Us

Table of Contents

Click here to sign up for SAGE Journal Email Alerts today!

Sign In to gain access to subscriptions and/or personal tools.
Adaptive Behavior
This Article
Right arrow Abstract Freely available
Right arrow Free Full Text (Free PDF) Free
Right arrow Alert me when this article is cited
Right arrow Alert me if a correction is posted
Services
Right arrow Email this article to a friend
Right arrow Similar articles in this journal
Right arrow Similar articles in Web of Science
Right arrow Alert me to new issues of the journal
Right arrow Add to Saved Citations
Right arrow Download to citation manager
Right arrowRequest Permissions
Right arrow Request Reprints
Right arrow Add to My Marked Citations
Citing Articles
Right arrow Citing Articles via HighWire
Right arrow Citing Articles via Web of Science (2)
Right arrow Citing Articles via Google Scholar
Right arrow Citing Articles via Scopus
Google Scholar
Right arrow Articles by Broekens, J.
Right arrow Articles by Verbeek, F. J.
Right arrow Search for Related Content
Social Bookmarking
Add to CiteULike   Add to Complore   Add to Connotea   Add to Del.icio.us   Add to Digg   Add to Reddit   Add to Technorati   Add to Twitter  
What's this?

Affect, Anticipation, and Adaptation: Affect-Controlled Selection of Anticipatory Simulation in Artificial Adaptive Agents

Joost Broekens

Leiden Institute of Advanced Computer Science (LIACS), Leiden University, Leiden, The Netherlands, broekens{at}liacs.nl

Walter A. Kosters

Leiden Institute of Advanced Computer Science (LIACS), Leiden University, Leiden, The Netherlands, kosters{at}liacs.nl

Fons J. Verbeek

Leiden Institute of Advanced Computer Science (LIACS), Leiden University, Leiden, The Netherlands, fverbeek{at}liacs.nl

Emotion plays an important role in thinking. In this article we study affective control of the amount of simulated anticipatory behavior in adaptive agents using a computational model. Our approach is based on model-based reinforcement learning (RL) and inspired by the simulation hypothesis (Cotterill, 2001; Hesslow, 2002). The simulation hypothesis states that thinking is internal simulation of behavior using the same sensory-motor systems as those used for overt behavior. Here, we study the adaptiveness of an artificial agent, when action-selection bias is induced by an affect-controlled amount of simulated anticipatory behavior . To this end, we introduce an affect-controlled simulation-selection mechanism that uses the predictions of the agent's RL model to select anticipatory behaviors for simulation. Based on eXperiments with adaptive agents in two nondeterministic partially observable grid-worlds we conclude that (1) internal simulation has an adaptive benefit and (2) affective control can reduce the amount of simulation needed for this benefit. This is specifically the case if the following relation holds: positive affect decreases the amount of simulation towards simulating the best potential neXt action, while negative affect increases the amount of simulation towards simulating all potential neXt actions. In essence we use artificial affect to control mental eXploration versus eXploitations. Thus, agents "feeling positive" can think ahead in a narrow sense and free up working memory resources, while agents "feeling negative" must think ahead in a broad sense and maXimize usage of working memory. Our results are consistent with several psychological findings on the relation between affect and learning, and contribute to answering the question of when positive versus negative affect is useful during adaptation.

Key Words: affect • action selection • anticipatory simulation • simulation selection • working memory • simulated adaptive agents

References

  • Ashby, F.G., Isen, A.M., & Turken, U. (1999). A neuropsychological theory of positive affect and its influence on cognition. Psychological Review, 106, 529—550.[CrossRef][Web of Science][Medline] [Order article via Infotrieve]
  • Avila-Garcia, O., & Cañamero, L. (2004). Using hormonal feedback to modulate action selection in a competitive scenario. From Animals to Animats 8: Proceedings of the 8th International Conference on Simulation of Adaptive Behavior (pp. 243—252). Cambridge, MA: MIT Press.
  • Aylett, R. (2006). Emotion as an integrative process between non-symbolic and symbolic systems in intelligent agents. Proceedings of the AISB'06 Symposium on Architecture of Brain and Mind (pp. 43—47). AISB Press.
  • Baddeley, A. (2000). The episodic buffer: A new component of working memory? Trends in Cognitive Sciences, 4, 417—423.[CrossRef][Medline] [Order article via Infotrieve]
  • Belavkin, R.V. (2004). On relation between emotion and entropy. Proceedings of the AISB'04 Symposium on Emotion, Cognition and A fective Computing (pp. 1—8). AISB Press.
  • Berridge, K.C. (2003). Pleasures of the brain. Brain and Cognition, 52, 106—128.
  • Bickhard, M.H. (1998). Levels of representationality. Journal of Experimental and Theoretical Artificial Intelligence, 10, 179—215.[CrossRef]
  • Blanchard, A.J., & Cañamero, L. (2006). Modulation of exploratory behavior for adaptation to the context. Proceedings of the AISB'06 Symposium on Biologically Inspired Robotics (Biro-net) (pp 131—137). AISB Press.
  • Botelho, L.M., & Coelho, H. (1998). Information processing, motivation and decision making. Proceedings of the 4th International Workshop on Artificial Intelligence in Economics and Management, (pp. 233—250).
  • Broekens, J. (2005). Internal simulation of behavior has an adaptive advantage. Proceedings of the CogSci'05 Conference (pp. 342—347). Mahwah, NJ: Lawrence Erlbaum Associates.
  • Broekens, J., & DeGroot, D. (2004). Emergent representations and reasoning in adaptive agents. Proceedings of the Third International Conference on Machine Learning and Applications (pp. 207—214). Los Alamitos, CA: IEEE Press.
  • Broekens, J., Kosters, W.A., & Verbeek, F.J. (2007). On affect and self-adaptation: Potential benefits of valence-controlled action-selection. In: J. Mira and J.R. Álvarez (Eds.), IWINAC 2007, Part I, LNCS 4527 (pp. 357—366). Berlin: Springer.
  • Broekens, J., & Verbeek, F.J. (2005). Simulation, emotion and information processing: Computational investigations of the regulative role of pleasure in adaptive behavior. Proceedings of the Workshop on Modeling Natural Action Selection (pp. 166—173). AISB Press.
  • Butz, M.V., Sigaud, O., & Gerard, P. (2003). Internal models and anticipations in adaptive learning systems. In: LNAI 2684: Anticipatory behavior in adaptive learning systems (pp. 86—109). Berlin: Springer.
  • Cañamero, D. (2000). Designing emotions for activity selection. Department of Computer Science Technical Report DAIMI PB 545. University of Aarhus, Denmark.
  • Clore, G.L., & Gasper, K. (2000). Feeling is believing: Some affective influences on belief. In: N. Frijda, A. S. R. Manstead, & S. Bem (Eds.), Emotions and beliefs. Cambridge, UK: Cambridge University Press.
  • Cohen, J.D., & Blum, K.I. (2002) Reward and decision. Neuron, 36, 193—198.[CrossRef][Web of Science][Medline] [Order article via Infotrieve]
  • Cos-Aguilera, I., Cañamero, L., Hayes, G.M., & Gillies, A. (2005). Ecological integration of affordances and drives for behaviour selection. Proceedings of the Workshop on Modeling Natural Action Selection, Edinburgh: AISB Press, (pp. 225—228).
  • Cotterill, R.M.J. (2001). Cooperation of the basal ganglia, cerebellum, sensory cerebrum and hippocampus: Possible implications for cognition, consciousness, intelligence and creativity. Progress in Neurobiology, 64, 1—33.[CrossRef]
  • Craig, S.D., Graesser, A.C., Sullins, J., & Gholson, B. (2004). Affect and learning: An exploratory look into the role of affect in learning with AutoTutor. Journal of Educational Media, 29, 241—250.
  • Custers, R., & Aarts, H. (2005). Positive affect as implicit motivator: On the nonconscious operation of behavioral goals. Journal of Personality and Social Psychology, 89, 129—142.[CrossRef][Web of Science][Medline] [Order article via Infotrieve]
  • Damasio, A.R. (1994). Descartes' error: Emotion, reason, and the human brain. New York: Penguin Putnam.
  • Davidson, R.J. (2000). Cognitive neuroscience needs affective neuroscience (and vice versa). Brain and Cognition, 42, 89—92.
  • Dayan, P., & Balleine, B.W. (2002) Reward, motivation, and reinforcement learning. Neuron 36, 285—298.[CrossRef][Web of Science][Medline] [Order article via Infotrieve]
  • Deheane, S., Sergent, C., & Changeux, J-P. (2003). A neuronal network model linking subjective reports and objective physiological data during conscious perception. Proceedings of the Natural Academy of Sciences, 100, 8520—8525.[CrossRef]
  • Demiris, Y., & Johnson, M. (2003). Distributed, predictive perception of actions: A biologically inspired robotics architecture for imitation and learning. Connection Science, 15, 231—243.
  • Doya, K. (2002). Metalearning and neuromodulation. Neural Networks, 15, 495—506.[CrossRef][Web of Science][Medline] [Order article via Infotrieve]
  • Dreisbach, G., & Goschke, K. (2004). How positive affect modulates cognitive control: Reduced perseveration at the cost of increased distractibility. Journal of Experimental Psychology: Learning, Memory, and Cognition, 30, 343—353.[CrossRef][Web of Science][Medline] [Order article via Infotrieve]
  • Forgas, J.P. (2000). Feeling is believing? The role of processing strategies in mediating affective influences in beliefs. In: N. Frijda, A. S. R. Manstead, & S. Bem (Eds.), Emotions and beliefs. Cambridge, UK: Cambridge University Press.
  • Foster, D.J., & Wilson, M.A. (2006). Reverse replay of behavioural sequences in hippocampal place cells during the awake state. Nature, 440, 680—683.[CrossRef][Medline] [Order article via Infotrieve]
  • Frijda, N.H., Manstead, A.S.R., & Bem. S. (2000). The influence of emotions on beliefs. In: N. Frijda, A. S. R. Manstead, & S. Bem (Eds.), Emotions and beliefs. Cambridge, UK: Cambridge University Press.
  • Frijda, N.H., & Mesquita, B. (2000). Beliefs through Emotions. In: N. Frijda, A. S. R. Manstead, & S., Bem, (Eds.), Emotions and beliefs. Cambridge, UK: Cambridge University Press.
  • Gadanho, S.C. (2003). Learning behavior-selection by emotions and cognition in a multi-goal robot task. Journal of Machine Learning Research, 4, 385—412.[CrossRef][Web of Science]
  • Griffith, P.E. (1999). Modularity & the psychoevolutionary theory of emotion. Mind and Cognition: An Anthology. Oxford: Blackwell.
  • Hesslow, G. (2002). Conscious thought as simulation of behaviour and perception. Trends in Cognitive Sciences, 6, 242—247.[CrossRef][Medline] [Order article via Infotrieve]
  • Hoffmann, H., & Möller, R. (2004). Action selection and mental transformation based on a chain of forward models. Proceedings of the 8th International Conference on the Simulation of Adaptive Behavior (pp. 213—222). Cambridge, MA: MIT Press.
  • Hogewoning, E., Broekens, J., Eggermont, J., & Bovenkamp, E.G.P. (2007). Strategies for affect-controlled action-selection in Soar-RL. In: J. Mira and J.R. Álvarez (Eds.), IWINAC 2007, Part II, LNCS 4528 (pp. 501—510). Berlin: Springer.
  • Kaelbling, L.P., Littman, M.L., & Moore, A.W. (1996). Reinforcement learning: A survey. Journal of Artificial Intelligence Research, 4, 237—285.[Web of Science]
  • Lahnstein, M. (2005). The emotive episode is a composition of anticipatory and reactive evaluations. Proceedings of the AISB'05 Symposium on Agents that Want and Like (pp. 62—69). AISB Press.
  • McCallum, A. (1995). Instance-based utile distinctions for reinforcement learning with hidden state. Proceedings of the Twelfth International Conference on Machine Learning, San Francisco: Morgan Kaufmann, (pp. 387—395).
  • McMahon, A., Scott, D., Baxter, P., & Browne, W. (2006). An autonomous explore/exploit strategy. Proceedings of the AISB'06 Symposium on Nature Inspired Systems (pp. 192— 201). AISB Press.
  • Montague, P.R., Hyman S.E., & Cohen, J.D. (2004). Computational roles for dopamine in behavioural control. Nature, 431, 760—767.[CrossRef][Medline] [Order article via Infotrieve]
  • Nason, S., & Laird, J.E. (2005). Soar-RL, integrating reinforcement learning with soar. Cognitive Systems Research, 6, 51—59.
  • Phaf, R.H., & Rotteveel, M. (2005). Affective modulation of recognition bias. Emotion, 5, 309—318.[CrossRef][Web of Science][Medline] [Order article via Infotrieve]
  • Picard, R.W., Papert, S., Bender, W., Blumberg, B., Breazeal, C. Cavallo, D., Machover, T., Resnick, M., Roy, D., & Strohecker, C. (2004). Affective learning—A manifesto. BT Technology Journal, 22, 253—269.[CrossRef][Web of Science]
  • Rolls, E.T. (2000). Précis of The brain and emotion. Behavioral and Brain Sciences, 23, 177—191.[CrossRef][Web of Science][Medline] [Order article via Infotrieve]
  • Rose, S.A., Futterweit, L.R., & Jankowski, J.J. (1999). The relation of affect to attention and learning in infancy. Child Development, 70, 549—559.[CrossRef][Web of Science][Medline] [Order article via Infotrieve]
  • Russell, J.A. (2003). Core affect and the psychological construction of emotion. Psychological Review, 110, 145—172.[CrossRef][Web of Science][Medline] [Order article via Infotrieve]
  • Salichs, M.A., & Malfaz, M. (2006). Using emotions on autonomous agents. The role of happiness, sadness and fear. Proceedings of the AISB'06 Symposium on Integrative Approaches to Machine Consciousness (pp. 157—164). AISB Press.
  • Scherer, K.R. (2001). Appraisal considered as a process of multilevel sequential checking. In: K. R. Scherer, A. Schott, & T. Johnstone (Eds.) Appraisal processes in emotion: Theory, Methods, Research. New York: Oxford Univ. Press,.
  • Schultz, W., Dayan, P., & Montague, P.R. (1997). A neural substrate of prediction and reward. Science, 275, 1593—1599.[Abstract/Free Full Text]
  • Schweighofer, N., & Doya, K. (2003). Meta-learning in reinforcement learning. Neural Networks, 16, 5—9.[CrossRef][Web of Science][Medline] [Order article via Infotrieve]
  • Shanahan, M. (2006). A cognitive architecture that combines internal simulation with a global workspace. Consciousness and Cognition, 15, 433—449.[CrossRef][Web of Science][Medline] [Order article via Infotrieve]
  • Sutton, R.S. (1990). Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. Proceedings of the Seventh International Conference on Machine Learning (pp. 216—224). Morgan Kaufmann.
  • Sutton, R.S., & Barto, A.G. (1998). Reinforcement Learning: An Introduction. Cambridge, MA: MIT Press.
  • Tyrell, T. (1993). Computational mechanisms for action selection. Ph.D. Thesis, University of Edinburgh.
  • van Dartel, M.F., & Postma, E.O. (2005) Symbol manipulation by internal simulation of perception and behaviour. Proceedings of the 5th International Workshop on Epigenetic Robotics. Lund University Cognitive Studies Vol. 123 (pp. 121—124). Lund: Lund University.
  • van Dartel, M., Postma, E. & van den Herik, J. (2004) Categorisation through internal simulation of perception and behaviour. Proceedings of the 16th Belgium-Netherlands Conference on Artificial Intelligence (pp. 187—194).
  • von Hecker, U., & Meiser, T., (2005). Defocused attention in depressed mood: Evidence from source monitoring. Emotion, 5, 456—463.[CrossRef][Web of Science][Medline] [Order article via Infotrieve]
  • Velasquez, J.D. (1998). A computational framework for emotion-based control. In: SAB'98 Work-shop on Grounding Emotions in Adaptive Systems, 5th International Conference of the Society for Adaptive Behavior, Zurich.
  • Ziemke, T., Jirenhed, D., & Hesslow, G. (2005). Internal simulation of perception: A minimal neuro-robotic model. Neurocomputing, 68, 85—104.[CrossRef][Web of Science]

Adaptive Behavior, Vol. 15, No. 4, 397-422 (2007)
DOI: 10.1177/1059712307084686
Add to CiteULike CiteULike   Add to Complore Complore   Add to Connotea Connotea   Add to Del.icio.us Del.icio.us   Add to Digg Digg   Add to Reddit Reddit   Add to Technorati Technorati   Add to Twitter Twitter    What's this?


This article has been cited by other articles:


Home page
 Adaptive BehaviorHome page
J. L. Krichmar
The Neuromodulatory System: A Framework for Survival and Adaptive Behavior in a Challenging World
Adaptive Behavior, December 1, 2008; 16(6): 385 - 399.
[Abstract] [PDF]

This Article
Right arrow Abstract Freely available
Right arrow Free Full Text (Free PDF) Free
Right arrow Alert me when this article is cited
Right arrow Alert me if a correction is posted
Services
Right arrow Email this article to a friend
Right arrow Similar articles in this journal
Right arrow Similar articles in Web of Science
Right arrow Alert me to new issues of the journal
Right arrow Add to Saved Citations
Right arrow Download to citation manager
Right arrowRequest Permissions
Right arrow Request Reprints
Right arrow Add to My Marked Citations
Citing Articles
Right arrow Citing Articles via HighWire
Right arrow Citing Articles via Web of Science (2)
Right arrow Citing Articles via Google Scholar
Right arrow Citing Articles via Scopus
Google Scholar
Right arrow Articles by Broekens, J.
Right arrow Articles by Verbeek, F. J.
Right arrow Search for Related Content
Social Bookmarking
Add to CiteULike   Add to Complore   Add to Connotea   Add to Del.icio.us   Add to Digg   Add to Reddit   Add to Technorati   Add to Twitter  
What's this?