|
An introductory course for the IEEE Robotics and Automation Society Technical Committee for Cognitive Robotics |
|
Instructors: send an email to request the complete set of PowerPoint slides. Example code is available on Github.
If you already have a copy, check the Version History to make sure you have the most recent version. If you don't, please send an email to request it.
A PR2 robot pours popcorn from a saucepan (left) and sets a table (right) during demonstrations of cognitively-enabled robot manipulation using CRAM.
Image courtesy of the Everyday Activity Science and Engineering (EASE) interdisciplinary research center at the University of Bremen, Germany.
Course Description | Learning Objectives | Content | Lecture Notes | Course Textbook | Recommended Reading | Software | Resources | Acknowledgements
Cognitive Robotics
"The word cognition derives from the Latin verb cognosco, a composition of con (meaning related to) and gnosco (to know). Cognitive robotics, then, is the branch of robotics where knowledge plays a central role in supporting action selection, execution, and understanding.
It focuses on designing and building robots that have the ability to learn from experience and from others, commit relevant knowledge and skills to memory, retrieve them as the context requires, and flexibly use this knowledge to select appropriate actions in the pursuit of their goals, while anticipating the outcome of those actions when doing so.
Cognitive robots can use their knowledge to reason about their actions and the actions of those with whom they are interacting, and thereby modify their behavior to improve their overall long-term effectiveness.
In short, cognitive robots are capable of flexible, context-sensitive action, knowing what they are doing and why they are doing it."
Sandini, G., Sciutti, A., and Vernon, D. (2021) "Cognitive Robotics", in Ang M., Khatib O., Siciliano B. (eds), Encyclopedia of Robotics . Springer, Berlin, Heidelberg.
Course Description
This course provides an introduction to cognitive robotics, a branch of robotics in which knowledge plays a central role in supporting action selection, planning, and execution. Cognition is essential for robots to be able to perform tasks in a response to a request by a human, but without the human having to specify explicitly everything that is needed to fulfil the task. Many everyday activities fall into this category. For example, when we ask someone to fetch something for us, we don't have to say how they are to fetch it. The goal of the course is to give students an understanding of what is involved in the design a cognitive robot and give them the knowledge and skills to produce working implementations for simple instances of cognitive fetch and place tasks. Students will learn through a combination of classroom lectures and laboratory assignments that consolidate their understanding through practical exercises using both robot simulators and physical robots. Student progress is assessed by a series of multiple choice tests and individual & group assignments. There are no prerequisites for taking this course, although it would be an advantage to have taken Robotics: Principles and Practice and Artificial Cognitive Systems.
Learning Objectives
Students will be introduced to the general area of robotics. They will learn how to develop software using ROS (Robot Operating System) and they will learn the principles of robot manipulation and task level robot programming, including the mathematical tools required to specify the position and orientation of robots and objects in the robot environment. Students will be introduced to the main topics in artificial cognitive systems, including the different paradigms of cognitive science and cognitive architectures. These components form the foundation for the remainder of the course, involving a detailed study of the CRAM (Cognitive Robot Abstract Machine) cognitive architecture, building on ROS, and exploiting functional programming in Lisp to reason about and execute under-determined tasks in everyday activities. Students will learn how to write CRAM plans in the Lisp-based CRAM plan language for the PR2 humanoid mobile robot in a simulation environment and the Lynxmotion AL5D robot manipulator, both simulated and real.
Outcomes
After completing this course, students should be able to:
Lecture Notes
Module 1: Overview of Cognitive Robotics
Module 2: The Robot Operating System (ROS)
Module 3: Mobile Robots (optional)
Module 4: Robot Manipulators
Module 5: Robot Vision (optional)
Module 6: Artificial Cognitive Systems
Module 7: Cognitive Architectures
Module 8: An Introduction to Functional Programming with Lisp
Module 9: The CRAM Plan Language
Module 10: Using Turtlesim with CRAM
Module 11: Mobile Manipulation using the PR2 Robot with CRAM
Module 12: Using the Lynxmotion AL5D Robot Manipulator with CRAM
Module 13: Using Pepper with CRAM
Instructors: send an email to request the complete set of PowerPoint slides.
If you already have a copy, check the Version History to make sure you have the most recent version. If you don't, please send an email to request it.
At present, there is no textbook that covers all the material in this course. The recommended reading below provides partial coverage.
Recommended Reading
Cangelosi, A. and Asada, M. (2021), Eds. , Cognitive Robotics, MIT Press, in press.
Corke, P. (2017). Robotics, Vision and Control, 2nd Edition, Springer.
O'Kane, J. M. (2018). A Gentle Introduction to ROS.
Paul, R. (1981). Robot Manipulators: Mathematics, Programming, and Control. MIT Press.
Szeliski, R. (2010). Computer Vision: Algorithms and Applications, Springer.
Vernon, D. (1991). Machine Vision: Automated Visual Inspection and Robot Vision, Prentice-Hall.
Vernon, D. Artificial Cognitive Systems, MIT Press, 2014.
Software Development Environment
Lecture 4 in Module 1 has detailed instructions for installing the software required for the various exercises in the course.
The simulator for the Lynxmotion AL5D robot manipulator for Module 4 and the ROS example programs for Modules 2, 3, 4, and 5 are available on the course GitHub repository github.com/cognitive-robotics-course.
Resources
Additional material can be found on the Resources page of the IEEE Technical Committee for Cognitive Robotics website.
Acknowledgements
I wish to acknowledge with thanks the support I received from the IEEE Robotics and Automation Society under the program Creation of Educational Material in Robotics and Automation (CEMRA) 2020.
This course was developed over a four year period leading up to, during, and directly after the time I spent working at Carnegie Mellon University Africa in Rwanda. My thanks go to the students I taught there, several of whom have contributed directly or indirectly to the material. Their deep interest and searching questions made all the difference.
Special thanks go to Vinny Adjibe, Abrham Gebreselasie, Innocent Mukoki, Ribeus Munezero, and Timothy Odonga for their work developing material and tools for the course during their Summer 2020 CMU-Africa internships, and to Derrick Odonkor, who validated the material and developed the Pepper tutorial for CRAM during his internship at the Institute for Artificial Intelligence, University of Bremen, in the Spring of 2020. Derrick Odonkor's intership was mostly funded by the IEEE Robotics and Automation Society under the program Creation of Educational Material in Robotics and Automation (CEMRA) 2020.
My thanks go to Carnegie Mellon University Africa for its generous support in sponsoring the internships and teaching assistantships, and supplementing the funds provided by the IEEE Robotics and Automation Society to support Derek Odonkor's visit to the University of Bremen.
The module on mobile robots benefitted greatly from a course developed by Alessandro Saffiotti, Örebro University, Sweden, on Artificial Intelligence Techniques for Mobile Robots. I borrowed heavily from this material, whilst creating my own illustrations and diagrams.
The module on robot vision is a very short version of my course on applied computer vision which, in turn, drew inspiration from several sources, including courses given by Kenneth Dawson-Howeat Trinity College Dublin, Kris Kitani at Carnegie Mellon University, Francesca Odone at University of Genova, and Markus Vincze at Technische Universitat Wien. Many of the OpenCV examples are taken from Kenneth Dawson-Howe's book
A Practical Introduction to Computer Vision with OpenCV and the code samples.
The material on CRAM (Cognitive Robot Abstract Machine) was derived from tutorials on the CRAM website. I am indebted to Michael Beetz and Gayane Kazhoyan for the time and effort they invested explaining CRAM and teaching me how to write CRAM Plan Language programs during my summer visits to the Institute for Artificial Intelligence, University of Bremen, and since joining Prof. Beetz's team in August 2020. Thank you to the Institute for Artificial Intelligence, University of Bremen, for hosting the three-month visit by Derrick Odonkor.
All images and diagrams are either original or have their source credited. My apologies in advance for any unintended omissions. Technical drawings were produced in LaTeX using TikZ and the 3D Plot package.
David Vernon, Carnegie Mellon University Africa, Rwanda, and Institute for Artificial Intelligence, University of Bremen, Germany.
Lecture 1. Component disciplines; the nature of cognition; definition of cognitive robotics.
Lecture 2. Operation of a cognitive robot; reasons for studying cognitive robotics.
Lecture 3. Industrial requirements; resources.
Lecture 4. Installation of software development environments for exercises.
Lecture 1. Introduction to ROS (Robot Operating System); the Turtlesim turtlebot simulator.
Lecture 2. Writing ROS software in C++: publishers.
Lecture 3. Writing ROS software in C++: subscribers,
Lecture 4. Writing ROS software in C++: services.
Lecture 1. Types of mobile robots; locomotion vs navigation; challenges of navigation: localization, search, path planning, planning, coverage, SLAM.
Lecture 2. Absolute position estimation.
Lecture 3. Relative position estimation using inertial sensors.
Lecture 4. Relative position estimation using odometry.
Lecture 5. Kinematics of a two-wheel differential drive robot.
Lecture 6. The go-to-position problem; divide-and-conquer controller.
Lecture 7. The go-to-position and go-to-pose problems; MIMO controllers.
Lecture 8. Path planning: the wavefront algorithm to find a shortest path on a map using breadth-first search for unweighted graphs.
Lecture 9. Path planning: Dijkstra's algorithm for weighted graphs.
Lecture 10. Path planning: A* algorithm; other search approaches.
Lecture 1. Robot programming; coordinate frames of reference and homogenous transformations.
Lecture 2. Object pose specification with homogenous transformations and vectors & quaternions.
Lecture 3. Robot programming by frame-based task specification.
Lecture 4. Pick-and-place example of task-level robot programming.
Lecture 5. Implementation of the pick-and-place example for a Lynxmotion AL5D robot arm using the Frame class in C++.
Lecture 6. Kinematics; Denavit-Hartenberg representation; kinematics and inverse kinematics of the LynxMotion AL5D arm.
Lecture 1. Computer vision; optics and sensors; image acquisition; image representation; image processing.
Lecture 2. Introduction to OpenCV.
Lecture 3. Segmentation; region-based approaches; feature-based thresholding; graph cuts.
Lecture 4. Segmentation; boundary-based approaches; edge detection.
Lecture 5. Image analysis; feature extraction.
Lecture 6. K-nearest neighbour, minimum distance, linear, maximum likelihood and Bayes classifiers.
Lecture 7. Perspective transformation; camera model; camera calibration.
Lecture 8. Inverse perspective transformation; stereo vision; epipolar geometry.
Lecture 1. The paradigms of cognitive science: the cognitivist paradigm.
Lecture 2. The paradigms of cognitive science: the emergent and hybrid paradigms.
Lecture 3. Learning and development.
Lecture 4. Memory and prospection.
Lecture 5. Internal simulation; the symbol grounding problem.
Lecture 6. Interaction, social cognition; theory of mind; instrumental helping; collaboration.
Lecture 1. Role and requirements; desirable characteristics; core cognitive abilities.
Lecture 2. Example cognitive architectures: Soar, ACT-R, CLARION, BBD.
Lecture 3. Example cognitive architectures: ISAC
Lecture 4. The CRAM cognitive architecture: design principles.
Lecture 5. The CRAM cognitive architecture: structure.
Lecture 6. The CRAM cognitive architecture: operation.
Lecture 1. Common Lisp - REPL, lists, structures, equality, conditionals, CONS, CAR, CDR, dotted and assoc-lists.
Lecture 2. Common Lisp - functions, I/O, recursion, iteration, lambda and mapping functions, CLOS, inference.
Lecture 3. Emacs.
Lecture 1. Fluents, concurrency, reasoning, exception handling.
Lecture 2. Designators, process modules.
Lecture 1. Pose specification in ROS; Transform Library (TF).
Lecture 2. Creating a CRAM package; controlling a turtle.
Lecture 3. Implementing plans to move a turtle.
Lecture 4. Using Prolog for reasoning.
Lecture 5. Creating motion designators for the TurtleSim.
Lecture 6. Creating process modules.
Lecture 7. Automatically choosing a process module for a motion.
Lecture 8. Using location designators with TurtleSim.
Lecture 9. Writing high-level plans for TurtleSim.
Lecture 10. Implementing failure handling for TurtleSim.
Lecture 1. Fetch-and-place CRAM plan with the Bullet real-time physics simulator.
Lecture 2. Error handling and recovery looking in different places.
Lecture 3. Error handling and recovery using different arms.
Lecture 4. Defining a new grasp pose.
Coming soon ...
Coming later ...
November 2021.