Advanced Search

Journal Navigation

Journal Home

Subscriptions

Archive

Contact Us

Table of Contents

Sign In to gain access to subscriptions and/or personal tools.
Adaptive Behavior
This Article
Right arrow Abstract Freely available
Right arrow Free Full Text (Free PDF) Free
Right arrow Alert me when this article is cited
Right arrow Alert me if a correction is posted
Right arrow Citation Map
Services
Right arrow Email this article to a friend
Right arrow Similar articles in this journal
Right arrow Similar articles in Web of Science
Right arrow Alert me to new issues of the journal
Right arrow Add to Saved Citations
Right arrow Download to citation manager
Right arrowRequest Permissions
Right arrow Request Reprints
Right arrow Add to My Marked Citations
Citing Articles
Right arrow Citing Articles via HighWire
Right arrow Citing Articles via Web of Science (18)
Right arrow Citing Articles via Google Scholar
Right arrow Citing Articles via Scopus
Google Scholar
Right arrow Articles by Paine, R. W.
Right arrow Articles by Tani, J.
Right arrow Search for Related Content
Social Bookmarking
Add to CiteULike   Add to Complore   Add to Connotea   Add to Del.icio.us   Add to Digg   Add to Reddit   Add to Technorati   Add to Twitter  
What's this?

How Hierarchical Control Self-organizes in Artificial Adaptive Systems

Rainer W. Paine

Laboratory for Behavior and Dynamic Cognition, RIKEN Brain Science Institute, rpainemail-riken{at}yahoo.com

Jun Tani

Laboratory for Behavior and Dynamic Cognition, RIKEN Brain Science Institute, tani{at}brain.riken.jp

Diverse, complex, and adaptive animal behaviors are achieved by organizing hierarchically structured controllers in motor systems. The levels of control progress from simple spinal reflexes and central pattern generators through to executive cognitive control in the frontal cortex. Various types of hierarchical control structures have been introduced and shown to be effective in past artificial agent models, but few studies have shown how such structures can self-organize. This study describes how such hierarchical control may evolve in a simple recurrent neural network model implemented in a mobile robot. Topological constraints on information flow are found to improve system performance by decreasing interference between different parts of the network. One part becomes responsible for generating lower behavior primitives while another part evolves top-down sequencing of the primitives for achieving global goals. Fast and slow neuronal response dynamics are automatically generated in specific neurons of the lower and the higher levels, respectively. A hierarchical neural network is shown to outperform a comparable single-level network in controlling a mobile robot.

Key Words: neural network • hierarchy • self-organization • initial sensitivity • genetic algorithm • evolutionary robotics

References

  • Aharonov-Barki, R., Beker, T., & Ruppin, E. (2001). Emergence of memory-driven command neurons in evolved artificial agents . Neural Computation, 13, 691-716 .[CrossRef][Web of Science][Medline] [Order article via Infotrieve]
  • Albert, R., & Barabasi, A. (2002). Statistical mechanism of complex networks . Reviews of Modern Physics, 74(1), 47-97 .[CrossRef][Web of Science]
  • Arbib, M. A. (1981). Perceptual structures and distributed motor control. In Brooks, V. B. (Ed.), Handbook of physiology (Vol. II, pp. 1449-1480). Bethesda, MD: American Physiological Society .
  • Blynel, J. (2003). Evolving reinforcement learning-like abilities for robots . In A. Tyrrell, P. C. Haddow, & J. Torresen (Eds.), Evolvable systems: From biology to hardware: Fifth International Conference, ICES 2003,Lecture Notes in Computer Science (pp. 320-331 ), Vol. 2606 Berlin: Springer Verlag.
  • Blynel, J., & Floreano, D. (2002). Levels of dynamics and adaptive behavior in evolutionary neural controllers . In B. Hallam, D. Floreano, J. Hallam, G. Hayes, & J. A. Meyer (Eds.), From animals to animats 7: Proceedings of the Seventh International Conference on Simulation of Adaptive Behavior (pp. 272-281 ). Cambridge, MA: MIT Press, Bradford Books.
  • Crutcher, M. D., & Alexander, G. E. (1990). Movement-related neuronal activity selectively coding either direction or muscle pattern in 3 motor areas of the monkey . Journal of Neurophysiology, 64(1), 151-163 .[Abstract/Free Full Text]
  • Cyberbotics, Ltd. http://www.cyberbotics.com, Webots Commercial Mobile Robot Simulation Software.
  • Dietterich, T. (2000). Hierarchical reinforcement learning with the MAXQ value function decomposition . Journal of Artificial Intelligence Research, 13, 227-303 .
  • Doya K., Samejima K., Katagiri K., & Kawato M. (2002). Multiple model-based reinforcement learning . Neural Computation, 14, 1347-1369 .[CrossRef][Web of Science][Medline] [Order article via Infotrieve]
  • Eck, D., & Schmidhuber, J. (2002). Learning the long-term structure of the blues . In J. R. Dorronsoro (Ed.), Proceedings of the International Conference on Artificial Neural Networks, Lecture Notes in Computer Science (Vol. 2415). Berlin: Springer.
  • Fan, J., Yao, Q., & Tong, H. (1996). Estimation of densities and sensitivity measures in nonlinear dynamical systems . Biometrika, 83(1), 189-206 .[Abstract/Free Full Text]
  • Fikes, R. E., Hart, P. E., & Nilsson, N. J. (1972). Learning and executing generalized robot plans . Artificial Intelligence, 3, 251-288 .[CrossRef]
  • Floreano, D., & Mondada, F. (1994). Automatic creation of an autonomous agent: Genetic evolution of a neural-network driven robot . In D. Cliff, P. Husbands, J. Meyer, & S. W. Wilson (Eds.), From animals to animats 3: Proceedings of the Third Conference on Simulation of Adaptive Behavior. Cambridge, MA: The MIT Press, Bradford Books.
  • French, R. M. (1991). Using semi-distributed representation to overcome catastrophic forgetting in connectionist networks . In K. J. Hammond & K. J. Genther (Eds.) Proceedings of the 13th Annual Cognitive Science Society Conference (pp. 173-178 ). Hillsdale, NJ: Lawrence Erlbaum Publishers.
  • Fujimoto, K., & Kaneko, K. (2003). How fast elements can affect slow dynamics . Physica D: Nonlinear Phenomena, 180, 1-2 .[CrossRef]
  • Gers, F. A., Schraudolph, N. N., & Schmidhuber, J. (2002). Learning precise timing with LSTM recurrent networks . Journal of Machine Learning Research, 3, 115-143 .[CrossRef]
  • Ghez, C., & Krakauer, J. (2000). The organization of movement. In E. R. Kandel, J. H. Schwartz, T. M. Jessell (Eds.), Principles of neural science, (4th ed.), (pp. 653-673). New York: McGraw-Hill .
  • Giszter, S. F., Mussa-Ivaldi, F. A., & Bizzi, E. (1993). Convergent force fields organized in the frog’s spinal cord . Journal of Neuroscience, 13(2), 467-491 .[Abstract]
  • Goldberg, D. E. (2002). The design of innovation: Lessons from and for competent genetic algorithms. Boston, MA: Kluwer .
  • Grossberg, S. (1982). Studies of mind and brain: Neural principles of learning, perception, development, cognition, and motor control. Boston Studies in the Philosophy of Science, Vol. 70. Dordrecht: Reidel .
  • Holland, J. (1975). Adaptation in natural and artificial systems. Ann Arbor, MI: University of Michigan Press .
  • Ito, M., & Tani, J. (2004). On-line imitative interaction with a humanoid robot using a dynamic neural network model of a mirror system . Adaptive Behavior, 12(2), 93-115 .[Abstract]
  • Kelly, J. P. (1991). The neural basis of perception and movement. In E. R. Kandel, J. H. Schwartz, & T. M. Jessell (Eds.), Principles of neural science, (3rd ed.) (pp. 283-295). Norwalk, CN: Appleton and Lange .
  • Lipson, H., & Pollack, J. B. (2000). Automatic design and manufacture of robotic lifeforms . Nature, 406, 974-978 .[CrossRef][Medline] [Order article via Infotrieve]
  • Lu, X., & Ashe, J. (2005). Anticipatory activity in primary motor cortex codes memorized movement sequences . Neuron, 45, 967-973 .[CrossRef][Web of Science][Medline] [Order article via Infotrieve]
  • McCloskey, M., & Cohen, N. J. (1989). Catastrophic interference in connectionist networks: The sequential learning problem . The Psychology of Learning and Motivation, 24, 109-165 .
  • McGovern, A., & Barto, A. G. (2001). Accelerating reinforcement learning through the discovery of useful subgoals . In Proceedings of the Sixth International Symposium on Artificial Intelligence, Robotics, and Automation in Space: i-SAIRAS, Montreal, Canada, June 2001, Electronically Published.
  • McGovern, A., Sutton, R. S., & Fagg, A. H. (1997). Roles of macro-actions in accelerating reinforcement learning . Proceedings of the 1997 Grace Hopper Celebration of Women in Computing (pp. 13-17 ). San Jose, CA: Anita Borg Institute for Women and Technology.
  • Miller, G. F., Todd, P. M., & Hegde, S. U. (1989). Designing neural networks using genetic algorithms . In J. D. Schaffer (Ed.), Proceedings of the Third International Conference on Genetic Algorithms (pp. 379-384 ). San Mateo, CA: Morgan Kaufmann.
  • Mitchell, M. (1998). An introduction to genetic algorithms. Cambridge, MA: The MIT Press .
  • Mozer, M. C. (1992). Induction of multiscale temporal structure. In J. E. Moody, S. J. Hanson, & R. P. Lippmann (Eds.), Advances in neural information processing systems IV (pp. 275-282). San Mateo, CA: Morgan Kaufmann .
  • Ninokura, Y., Mushiake, H., & Tanji, J. (2003). Integration of temporal order and object information in the monkey lateral prefrontal cortex . Journal of Neurophysiology, 10, 1152 .
  • Nishimoto, R., & Tani, J. (2004). Learning to generate combinatorial action sequences utilizing the initial sensitivity of deterministic dynamical systems . Neural Networks, 17(7), 925-933 .[CrossRef][Web of Science][Medline] [Order article via Infotrieve]
  • Nolfi, S. (2002). Evolving robots able to self-localize in the environment: The importance of viewing cognition as the result of processes occurring at different time scales . Connection Science, 14(3), 231-244 .
  • Ogata, T., Sugano, S., & Tani, J. (2004). Open-end human robot interaction from the dynamical systems perspective: Mutual adaptation and incremental learning . Lecture Notes in Artificial Intelligence (Vol. 3029, pp. 435-444 ), Berlin: Springer.
  • Paine, R. W., & Tani, J. (2004). Motor primitive and sequence self-organization in a hierarchical recurrent neural network . Neural Networks, 17, 1291-1309 .[CrossRef][Web of Science][Medline] [Order article via Infotrieve]
  • Parr, R., & Russell, S. (1998). Reinforcement learning with hierarchies of machines. In M. I. Jordan, M. J. Kearns, & S. A. Solla (Eds.), Advances in neural information processing systems (Vol. 10, pp. 1043-1049). Cambridge, MA: The MIT Press .
  • Precup, D., & Sutton, R. S. (1997). Multi-time models for temporally abstract planning. In M. I. Jordan, M. J. Kearns, & S. A. Solla (Eds.), Advances in neural information processing systems (Vol. 10, pp. 1050-1056). Cambridge, MA: The MIT Press .
  • Ruppin, E. (2002). Evolutionary autonomous agents: A neuroscience perspective . Nature Reviews Neuroscience, 3, 132-141 .[CrossRef][Web of Science][Medline] [Order article via Infotrieve]
  • Sakai, K., Kitaguchi, K., & Hikosaka, O. (2003). Chunking during human visuomotor sequence learning . Experimental Brain Research, 152(2), 229-242 .[CrossRef][Web of Science][Medline] [Order article via Infotrieve]
  • Siegelmann, H. T., & Sontag, E. D. (1995). On the computational power of neural nets . Journal of Computer and System Sciences, 50(1), 132-150 .[CrossRef]
  • Stanley, K. O., & Mikkulainen, R. (2004). Competitive coevolution through evolutionary complexification . Journal of Artificial Intelligence Research, 21, 63-100 .
  • Sugita, Y., & Tani, J. (2005). Learning semantic combinatoriality from the interaction between linguistic and behavioral processes . Adaptive Behavior, 13, 33-52 .[Abstract]
  • Sutton, R., Precup, D., & Singh, S. (1999). Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning . Artificial Intelligence, 112, 181-211 .[CrossRef]
  • Tani, J. (2003). Learning to generate articulated behavior through the bottom-up and the top-down interaction processes . Neural Networks, 16(1), 11-23 .[CrossRef][Web of Science][Medline] [Order article via Infotrieve]
  • Tani, J., & Nolfi, S. (1999). Learning to perceive the world as articulated: An approach for hierarchical learning in sensory-motor systems . Neural Networks 12, 1131-1141 .[CrossRef][Web of Science][Medline] [Order article via Infotrieve]
  • Tanji, J., & Kurata, K. (1982). Comparison of movement-related activity in 2 cortical motor areas of primates . Journal of Neurophysiology, 48(3), 633-653 .[Free Full Text]
  • Tanji, J., & Shima, K. (1994). Role for supplementary motor area cells in planning several movements ahead . Nature, 371, 413-416 .[CrossRef][Medline] [Order article via Infotrieve]
  • Thoroughman, K. A., & Shadmehr, R. (2000). Learning of action through combination of motor primitives . Nature, 407, 742-747 .[CrossRef][Web of Science][Medline] [Order article via Infotrieve]
  • Todd, P. M. (1991). A connectionist approach to algorithmic composition. In P. M. Todd & D. G. Loy (Eds.), Music and connectionism (pp. 173-193). Cambridge, MA: MIT Press .
  • Torres, J. J., Munoz, M. A., Marro, J, & Garrido, P. L. (2004). Influence of topology on the performance of a neural network . Neurocomputing, 58-60, 229-234 .[CrossRef]
  • Tucci, E., Quinn, M., & Harvey, I. (2002). An evolutionary ecological approach to the study of learning behavior using a robot-based model . Adaptive Behavior, 10(3/4), 201-222 .[Abstract/Free Full Text]
  • Wolpert, D. M., & Kawato, M. (1998). Multiple paired forward and inverse models for motor control . Neural Networks, 11, 1317-1329 .[CrossRef][Web of Science][Medline] [Order article via Infotrieve]
  • Yamauchi, B., & Beer, R. D. (1994). Sequential behavior and learning in evolved dynamical neural networks . Adaptive Behavior, 2(3), 219-246 .[Abstract/Free Full Text]
  • Ziemke, T., & Thieme, M. (2002). Neuromodulation of reactive sensorimotor mappings as a short-term memory mechanism in delayed response tasks . Adaptive Behavior, 10(3/4), 185-199 .[Abstract/Free Full Text]

Adaptive Behavior, Vol. 13, No. 3, 211-225 (2005)
DOI: 10.1177/105971230501300303
Add to CiteULike CiteULike   Add to Complore Complore   Add to Connotea Connotea   Add to Del.icio.us Del.icio.us   Add to Digg Digg   Add to Reddit Reddit   Add to Technorati Technorati   Add to Twitter Twitter    What's this?


This article has been cited by other articles:


Home page
 Adaptive BehaviorHome page
J. Tani
Autonomy of Self at Criticality: The Perspective from Synthetic Neuro-Robotics
Adaptive Behavior, October 1, 2009; 17(5): 421 - 443.
[Abstract] [PDF]

Home page
 Adaptive BehaviorHome page
F. Alnajjar, I. Bin Mohd Zin, and K. Murase
A Hierarchical Autonomous Robot Controller for Learning and Memory: Adaptation in a Dynamic Environment
Adaptive Behavior, June 1, 2009; 17(3): 179 - 196.
[Abstract] [PDF]

Home page
 Adaptive BehaviorHome page
M. Maniadakis and J. Tani
Acquiring Rules for Rules: Neuro-Dynamical Systems Account for Meta-Cognition
Adaptive Behavior, February 1, 2009; 17(1): 58 - 80.
[Abstract] [PDF]

Home page
 Adaptive BehaviorHome page
S. Nolfi, T. Ikegami, and J. Tani
Editorial: Behavior and Mind as a Complex Adaptive System
Adaptive Behavior, April 1, 2008; 16(2-3): 101 - 103.
[PDF]

Home page
 Adaptive BehaviorHome page
R. Nishimoto, J. Namikawa, and J. Tani
Learning Multiple Goal-Directed Actions Through Self-Organization of a Dynamic Neural Network Model: A Humanoid Robot Experiment
Adaptive Behavior, April 1, 2008; 16(2-3): 166 - 181.
[Abstract] [PDF]

Home page
 Adaptive BehaviorHome page
K. Davids, C. Button, D. Araujo, I. Renshaw, and R. Hristovski
Movement Models from Sports Provide Representative Task Constraints for Studying Adaptive Behavior in Human Movement Systems
Adaptive Behavior, March 1, 2006; 14(1): 73 - 95.
[Abstract] [PDF]

This Article
Right arrow Abstract Freely available
Right arrow Free Full Text (Free PDF) Free
Right arrow Alert me when this article is cited
Right arrow Alert me if a correction is posted
Right arrow Citation Map
Services
Right arrow Email this article to a friend
Right arrow Similar articles in this journal
Right arrow Similar articles in Web of Science
Right arrow Alert me to new issues of the journal
Right arrow Add to Saved Citations
Right arrow Download to citation manager
Right arrowRequest Permissions
Right arrow Request Reprints
Right arrow Add to My Marked Citations
Citing Articles
Right arrow Citing Articles via HighWire
Right arrow Citing Articles via Web of Science (18)
Right arrow Citing Articles via Google Scholar
Right arrow Citing Articles via Scopus
Google Scholar
Right arrow Articles by Paine, R. W.
Right arrow Articles by Tani, J.
Right arrow Search for Related Content
Social Bookmarking
Add to CiteULike   Add to Complore   Add to Connotea   Add to Del.icio.us   Add to Digg   Add to Reddit   Add to Technorati   Add to Twitter  
What's this?