Department of Electrical and Computer Engineering
and Beckman Institute for Advanced Science and Technology
Contents
Introduction to Our Laboratory
Here at the Language Acquistion and Robotics Laboratory, we are aiming to develop intelligent robots with the ability to learn natural language. While there are many aspects of this research, the most important are the following ideas:- There is no such thing as a disembodied mind.
- There are no isolated perceptual or cognitive functions.
- Language is acquired through interaction with the real world. Sensory-motor function is essential.
- Mental processes are largely based on associative memory
- The Language Engine is primarily semantic, not syntactic.
Back to Contents...
Our Robots
The iCub, the robot we are getting this Fall. More details to come, see videos section in the meantime.
Alan |
Illy |
Norbert |
The three robots we work with are all from Arrick Robotics. Alan is a first-generation Trilobot, with a monocular camera, microphones, and a microwave transmitter and wireless modem for control from a remote workstation. Illy and Norbert are both second-generation, heavily modified Trilobots, with capabilities for binaural hearing, stereo vision, tactile sense, and basic proprioceptive control.
Back to Contents...
Demonstration Videos
Object Tracking and Recognition in iCub
The following videos are from the 2009 RobotCub summer school, a 10 day long open lab for the iCub robot which we are receiving in Fall 2009. This is a demonstration of a simple autoassociative memory based on color histogram matching working in conjunction with depth-based object tracking.
Semantic Based Learning of Syntax
In these videos, Illy shows its knowledge of two-word sentences. The words that Illy knows are "kitten", "puppy", "can", "stay", "move", and "gone", which it learned using its associative memory - see the "Associative Learning" video and/or Kevin Squire's Ph.D. thesis below. The meanings of these words should be obvious, except for "gone", which to Illy means that the object has moved far away.
Illy has learned about syntax by hearing example two-word sentences (from an experimenter) that describe events in its environment. Illy then used its knowledge of the words to deduce the syntactic information. It is important to note that in these videos, Illy produces two-word sentences that it has never heard before - while it was trained with the single word "puppy", it did not hear the word "puppy" during the syntax training.
In the "Syntax Demo" video, Illy searches for objects to play with. Illy also demonstrates its sound source localization ability by turning toward the experimenter when called. Once the experimenter gets Illy's attention in this manner, Illy is directed to look for a specific object when it hears the object's name. In the "Syntax Test" video, the experimenter places objects in front of Illy for it to play with.
Videos:
Syntax Demo
- Formats: mpeg (~32 MB). Video length is about 4 min 25 seconds.
Syntax Test
- Formats: mpeg (~10 MB), QuickTime movie (~13 MB). Video length is about 58 seconds.
Vision-Based Localization and Map Learning
In this experiment, our robot Illy demonstrates its ability to acquire the mental map of its environment and use the mental map for localization. During the learning phase, Illy is put in an unknown environment and memorizes its navigational experience while exploring. This experience, which is recorded as sequences of images collected from Illy’s camera, is then consolidated to form a map of the environment using our proposed Learning Nonlinear Manifolds from Time Series (Poster Presentation, 314) algorithm. Once Illy has acquired the mental map, it can accurately locate itself in the environment.The video shows an episode of the localization experiment. The image on the left is Illy’s visual perception, and figure on the right shows Illy’s mental map. There are nine probabilistic maps shown in a 3x3 grid. The eight maps which form the border of the grid show the conditional probability distribution of Illy’s x-y position given a particular (discretized) direction that it is facing. The map on the center is the final probability distribution of Illy's x-y position with the direction parameter marginalized out. As can been from the video, Illy can accurately infer it position and orientation from the visual input.
Video and Figures:
- Localization Video (3.3 MB)
- Illy's Trajectory
Associative Learning
In this demonstration, our robot Illy is wandering around in her pen. For some time now, we've been teaching her the names of some objects, and in this video, I call to her, and tell her to play with the cat. After she approaches the cat, I tell her the word for cat a few times to help strengthen the association between that word and the object she sees in front of her. After this she plays with the cat briefly, then looks around for her other toys.Video:
Food Hunting by Illy (2001 Beckman Open House)
Illy has to save enough food before another snow storm comes to Urbana-Champaign. Illy’s favorite food is in some specially designed cans. She needs to find those cans and move them to her nest. The demonstrator eagerly helps Illy with her food hunting. He calls Illy every time he finds a can. When Illy hears the call, she instinctively turns toward the demonstrator and looks for her food. This demonstration shows the following capabilities:- Sound Source Localization: Illy is curious about loud sounds. When she hears a loud sound, she will localize the direction and turn toward the sound source.
- Object Recognition: Illy can recognize her food cans based on what she learned before the demo.
- Hand Eye Coordination: Illy can pick up her food cans based on her visual input.
- Sound Mimicking: Illy has the ability to mimic sounds she hears, which is an important step in speech acquisition. After she learns a sufficient number of sounds, she should be able to communicate with us.
Videos:
First Attempts (2000 Beckman Open House)
- Instinctual behaviors: Illy knows how to move around her environment, avoiding obstacles by using her sonar range finder, her whiskers, her light level detector, and other sensors. By watching where she's going, she tries to stay out of trouble!
- Object Recognition: Illy uses her stereo camera to recognize a number of common objects she encounters in her lab, including her remote control and joystick. Using edge detection and pattern recognition algorithms, she can identify objects from many different angles.
- Language Acquisition: Illy learns to associate spoken words with actions and can then respond to simple requests such as moving forward or backwards or turning.
Videos:
Back to Contents...
People
Advisor:Stephen E. Levinson
Current Students:
Lydia Majure
Logan Niehaus
Aaron Silver
Luke Wendt
Former Students:
Tuna Oezer (PhD '07)
Matthew McClain (PhD. '06)
Ruei-sung Lin (PhD '05) Kevin Squire (PhD '04) Ryan Rivera (MS '04)
Danfeng Li (PhD '03) Weiyu Zhu (PhD '03) Christopher Dodsworth (MS '01)
Josh Horvath (MS '01)
Qiong Liu (PhD '00)
Back to Contents...
Publications
Books
- S.E. Levinson, Mathematical Models for Speech Technology. London, UK: John Wiley, 2005.
Theses
- L. Majure, Unsupervised phoneme acquisition using hierarchical temporal models, M.S. 2009.
- T. Oezer, Discovering audio-visual associations in narrated videos of human activities. Ph.D. 2008.
- R. S. Lin, Learning vision-based robot navigation using memory-based reinforcement learning. MS 2002.
- W. Zhu, Vision-based behavior learning and automatic scene understanding by an autonomous robot. Ph.D. 2003
- Q. Liu, Interactive and incremental learning via a multisensory robot. Ph.D. 2001
- D. Li, Computational models for binaural sound source location and sound understanding. Ph.D. 2003.
- R. Rivera, 360 degree object recognition using SIFT features with autonomous model building. MS 2004.
Journal Publications
- M. McClain, Semantic Based Learning of Syntax in an Autonomous Robot. Int. J. of Humanoid Robotics, Vol. 4, No. 2, pp. 321 - 346, 2007.
- K. Squire, and S.E. Levinson, HMM-Based Semantic Learning for a Mobile Robot. IEEE Transactions on Evolutionary Computation Vol. 11, No. 2, pp 199-212, April 2007.
Technical Reports
- K. Squire, S.E. Levinson, and P.G. Xavier, "A Robotic Framework for Semantic Concept Learning". Technical Report SAND2004-4518, Sandia National Laboratories, 2004.
Conference Publications
- R. S. Lin, C-B Liu, M-H Yang, N. Ahuja, and S.E. Levinson. Learning Nonlinear Manifolds from Time Series. 9th European Conference on Computer Vision, Proceedings, p 245-256. 2006.
- M. McClain and S.E. Levinson, "Semantic Based Learning of Syntax in an Autonomous Robot: Preliminary Results," presented at Int'l Conference on Development and Learning, 2006.
- S.E. Levinson, W. Zhu, D. Li, K. Squire, R.S. Lin, M. Kleffner, M. McClain, and J. Lee, "Automatic Language Acquisition by Autonomous Robot," in Proc. Int. Joint Conference on Neural Networks, 2003. (Invited paper)
- W. Zhu, S. Wang, R-S Lin, and S.E. Levinson. Tracking of object with SVM Regression. Proc. of 2001 IEEE Comp. Soc. Conference on Computer Vision and Pattern Recognition. 2001.
- W. Zhu, S.E. Levinson. PQ-Learning: An Efficient Robot Learning Method for Intelligent Behavior Acquisition. Proc. of the Seventh International Conference on Intelligent Autonomous Systems, 2002.
- W. Zhu and S.E. Levinson, Vision-based Reinforcement Learning for Robot Navigation. Int. Joint Conference on Neural Networks. Vol. 2, pp 1025-1030. 2001.
Workshop Publications
- S.E. Levinson, K. Squire, R.S. Lin, and M. McClain, Automatic Language Acquisition by an Autonomous Robot. AAAI Spring Symposium on Developmental Robotics, March 21-23, 2005.
- S.E. Levinson, Q. Liu, K. Squire, C. Dodsworth, R.S. Lin, W. Zhu, and M. Kleffner, The Role of Sensorimotor Function, Associative Memory and Reinforcement Learning in Automatic Acquisition of Spoken Language by an Autonomous Robot. In Proc. Joint NSF/DARPA Workshop on Development and Learning, East Lansing, MI, April 5-7, 2000.
Back to Contents...
Copyright Notice
The following notice is required by IEEE:This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and conditions invoked by each authors copyright. In most cases, these works may not be reposted without explicit permission from the copyright holder