Introduction to Cognitive Robotics

David Vernon
Institute for Artificial Intelligence
University of Bremen
Germany



A PR2 robot pours popcorn from a saucepan during a demonstration of cognitively-enabled robot manipulation using CRAM.
Image courtesy of the EASE interdisciplinary research center at the University of Bremen, Germany.

 

"Cognitive Robotics is the field that combines insights and methods from robotics, AI and cognitive and biological sciences to design an integrated cognitive system combining the sensorimotor behavior and higher-level functions and social capabilities of an intelligent robot"

Cangelosi, A. and Asada, M. (eds) Cognitive Robotics, MIT Press, in press.


Course Description  |  Learning Objectives  |  Content  |  Lecture Notes  |  Course Textbook  |  Recommended Reading |  Software |  Resources  |  Acknowledgements

Cognitive Robotics

"The word cognition derives from the Latin verb cognosco, a composition of con (meaning related to) and gnosco (to know). Cognitive robotics, then, is the branch of robotics where knowledge plays a central role in supporting action selection, execution, and understanding.

It focuses on designing and building robots that have the ability to learn from experience and from others, commit relevant knowledge and skills to memory, retrieve them as the context requires, and flexibly use this knowledge to select appropriate actions in the pursuit of their goals, while anticipating the outcome of those actions when doing so.

Cognitive robots can use their knowledge to reason about their actions and the actions of those with whom they are interacting, and thereby modify their behavior to improve their overall long-term effectiveness.

In short, cognitive robots are capable of flexible, context-sensitive action, knowing what they are doing and why they are doing it."

Sandini, G., Sciutti, A., and Vernon, D. (2021) "Cognitive Robotics", in Ang M., Khatib O., Siciliano B. (eds) Encyclopedia of Robotics. Springer, Berlin, Heidelberg.


Course Description

Since this course does not assume prior knowledge of robotics, it first covers the essentials of classical robotics and builds on these when covering the core topics in cognitive robotics.

After a general overview of the field, the course begins with the key elements of mobile robots, robot manipulators, and robot vision, using ROS (Robot Operating System), C/C++, and OpenCV. It then progresses to the main topics in artificial cognitive systems, including the different paradigms of cognitive science and cognitive architectures. These components form the foundation for the remainder of the course, involving a detailed study of the CRAM (Cognitive Robot Abstract Machine) cognitive architecture, building on ROS, and exploiting functional programming in Lisp to reason about and execute under-determined tasks in everyday activities.

The course covers both theory and practice, using robot simulators as well as low-cost robots and cameras for practical examples and exercises.


Learning Objectives

After completing this course, students should be able to:

  1. Apply their knowledge of machine vision and robot kinematics to create computer programs that control mobile robots and robot arms, enabling the robots to recognize and manipulate objects and navigate their environments.

  2. Explain how a robot can be designed to exhibit cognitive goal-directed behaviour through the integration of computer models of vision, reasoning, learning, prospection, and social interaction.

  3. Use computer programs that realize limited instances of these faculties.


Course Content

Module 1: Overview of Cognitive Robotics

  • The interdisciplinary nature of cognitive robotics.
  • The nature of cognition.
  • Definition of cognitive robotics.
  • The need for cognition in robotics.
  • Operation of a cognitive robot.
  • Reasons for studying cognitive robotics.
  • Industrial requirements for cognitive robots.
  • Resources.
  • Installation of software development environments for exercises.

Module 2: Mobile Robots

  • Introduction to ROS (Robot Operating System).
  • Using the ROS Turtlebot simulator.
  • Writing ROS software.
  • Types of mobile robots.
  • The challenge of robot navigation
  • Relative inertial position estimation.
  • Kinematics of a two-wheel differential drive robot.
  • Relative odometry-based position estimation.
  • Absolute position estimation.
  • Closed-loop feedback control.
  • PID control.
  • Go-to-position and go-to-pose problems.
  • Divide and conquer controller.
  • MIMO controller.
  • Path tracking through waypoints
  • Finding a shortest path in a map.
  • Breadth-First Search (BFS).
  • Other search strategies.

Module 3: Robot Manipulators

  • Robot programming.
  • Description of object pose with homogenous transformations.
  • Robot programming by frame-based task specification.
  • Convention for embedding T6 and end-effector frames.
  • Specifying orientation: roll, pitch, yaw; Euler angles.
  • Pick-and-place example of task-level robot programming.
  • Implementation for a Lynxmotion AL5D robot arm using Frame classes in C++.
  • Forward and inverse kinematics of a manipulator.
  • Denavit-Hartenberg reporesentation.
  • Kinematic specification of the LynxMotion AL5D arm.
  • Inverse kinematics of the LynxMotion AL5D arm.

Module 4: Robot Vision

  • Computer vision.
  • Optics and sensors.
  • Image acquisition and image representation.
  • Image processing.
  • Introduction to OpenCV.
  • Segmentation.
  • Region-based approaches: feature-based thresholding, connected component analysis, graph cuts.
  • Boundary-based approaches: edge detection, boundary representations.
  • Image Analysis. Inspection, location, identification.
  • Feature extraction.
  • Classification: k-nearest neighbour, minimum distance, linear, maximum likelihood and Bayes classifiers.
  • Homogeneous coordinates and transformations.
  • Perspective transformation.
  • Camera model and inverse perspective transformation.
  • Camera calibration.

Module 5: Artificial Cognitive Systems

  • Paradigms of cognitive science.
  • Learning and development.
  • Memory and prospection.
  • Knowledge and representation.
  • Social cognition.

Module 6: Cognitive Architectures

  • Role and requirements.
  • Desirable characteristics.
  • Core cognitive abilities.
  • Example cognitive architectures: Soar, ACT-R, Clarion, ICARUS, BBD, ISAC.
  • The Common Model of Cognition.

Module 7: The CRAM Cognitive Architecture

  • Overview of the CRAM (Cognitive Robot Abstract Machine) cognitive architecture.
  • Programming paradigms.
  • Essentials of Common Lisp.
  • Emacs.
  • The CRAM plan language.
  • Description of object pose with quaternions in ROS.
  • CRAM and the ROS Turtlesim turtlebot simulator.
  • Creating a CRAM package.
  • Controlling a turtle.
  • Implementing simple plans to move a turtle.
  • Using Prolog for reasoning.
  • Creating motion designators for the TurtleSim.
  • Creating process modules.
  • Automatically choosing a process module for a motion.
  • Using location designators with TurtleSim.
  • Writing high-level plans for TurtleSim.
  • Implementing failure handling for TurtleSim.
  • Fetch-and-place CRAM plan for the PR2 robot.
  • Bullet Real-time Physics Simulation.
  • Error handling.
  • Recovery behaviors.


Lecture Notes

Module 1: Overview of Cognitive Robotics
Lecture 1. Component disciplines; the nature of cognition; definition of cognitive robotics.
Lecture 2. Operation of a cognitive robot; reasons for studying cognitive robotics.
Lecture 3. Industrial requirements; resources.
Lecture 4. Installation of software development environments for exercises.

Module 2: Mobile Robots
Lecture 1. Introduction to ROS (Robot Operating System); the Turtlesim turtlebot simulator.
Lecture 2. Writing ROS software in C++.
Lecture 3. Types of mobile robots; the challenge of robot navigation; relative inertial position estimation.
Lecture 4. Relative odometry position estimation; kinematics of a two-wheel differential drive robot; absolute position estimation.
Lecture 5. Closed-loop control and PID control; the go-to-position and go-to-pose problems; divide-and-conquer and MIMO controllers.
Lecture 6. Finding a shortest path in a map; breadth-first search algorithm; other search approaches.

Module 3: Robot Manipulators
Lecture 1. Robot programming; description of object pose with homogenous transformations.
Lecture 2. Robot programming by frame-based task specification; specification of position and orientation.
Lecture 3. Pick-and-place example; implementation for a Lynxmotion AL5D robot arm using Frame classes in C++.
Lecture 4. Kinematics; Denavit-Hartenberg representation; kinematics and inverse kinematics of the LynxMotion AL5D arm.

Module 4: Robot Vision
Lecture 1. Computer vision; optics and sensors; image acquisition; image representation; image processing.
Lecture 2. Introduction to OpenCV.
Lecture 3. Segmentation; region-based approaches; feature-based thresholding; graph cuts.
Lecture 4. Segmentation; boundary-based approaches; edge detection.
Lecture 5. Image analysis; feature extraction.
Lecture 6. K-nearest neighbour, minimum distance, linear, maximum likelihood and Bayes classifiers.
Lecture 7. Perspective transformation; camera model; camera calibration.
Lecture 8. Inverse perspective transformation; stereo vision; epipolar geometry.

Module 5: Artificial Cognitive Systems
Lecture 1. The paradigms of cognitive science.
Lecture 2. Learning and development.
Lecture 3. Memory and prospection; internal simulation, the symbol grounding problem.
Lecture 4. Interaction, social cognition; theory of mind, instrumental helping, collaboration.

Module 6: Cognitive Architectures
Lecture 1. Role and requirements; desirable characteristics; core cognitive abilities.
Lecture 2. Example cognitive architectures; the common model of cognition.

Module 7: The CRAM Cognitive Architecture
Lecture 1. Overview: design principles; structure.
Lecture 2. Overview: operation.
Lecture 3. Common Lisp - REPL, lists, structures, equality, conditionals, CONS, CAR, CDR, dotted and assoc-lists.
Lecture 4. Common Lisp - functions, I/O, recursion, iteration, lambda and mapping functions, CLOS, inference.
Lecture 5. Emacs.
Lecture 6. CRAM plan language, fluents, concurrency, reasoning, exception handling.
Lecture 7. CRAM plan language, designators, process modules.
Lecture 8. Specifying pose in ROS, quaternions.
Lecture 9. Turtlesim with CRAM; creating a CRAM package; controlling a turtle.
Lecture 10. Turtlesim with CRAM; implementing plans to move a turtle.
Lecture 11. Using Prolog for reasoning.
Lecture 12. Creating motion designators for the TurtleSim.
Lecture 13. Creating process modules.
Lecture 14. Automatically choosing a process module for a motion.
Lecture 15. Using location designators with TurtleSim.
Lecture 16. Writing high-level plans for TurtleSim.
Lecture 17. Implementing failure handling for TurtleSim.
Lecture 18. Fetch-and-place CRAM plan for the PR2 robot; Bullet real-time physics simulation.
Lecture 19. Fetch-and-place CRAM plan for the PR2 robot; error handling and recovery looking in different places.
Lecture 20. Fetch-and-place CRAM plan for the PR2 robot; error handling and recovery using different arms.

If you are an instructor and would like a copy of the complete set of PowerPoint slides, please contact me by email at davidvernon.eu.


Course Textbook

Future project.


Recommended Reading

Cangelosi, A. and Asada, M. (Eds.), Cognitive Robotics, MIT Press, in press.

Corke, P. (2016). Robotics, Vision and Control, 2nd Edition, Springer.

O'Kane, J. M. (2018). A Gentle Introduction to ROS.

Paul, R. (1981). Robot Manipulators: Mathematics, Programming, and Control. MIT Press.

Sandini, G., Sciutti, A., and Vernon, D. (2021) "Cognitive Robotics" in Encyclopedia of Robotics, Ang, M., Khatib, O., Siciliano, B. (Eds.), Springer, Berlin, Heidelberg.

Szeliski, R. (2010). Computer Vision: Algorithms and Applications, Springer.

Vernon, D. (1991). Machine Vision: Automated Visual Inspection and Robot Vision, Prentice-Hall.

Vernon, D. Artificial Cognitive Systems, MIT Press, 2014.

Vernon, D. and Vincze, M. "Industrial Priorities for Cognitive Robotics", Proceedings of EUCognition 2016, Cognitive Robot Architectures, European Society for Cognitive Systems, Vienna, 8-9 December, 2016, R. Chrisley. V. C. Müller, Y. Sandamirskaya. M. Vincze (eds.), CEUR-WS Vol-1855, ISSN 1613-0073, pp. 6-9.

Vernon, D. (2020). "Cognitive Architectures", in Cognitive Robotics, Cangelosi, A. and Asada, M. (Eds.), MIT Press.


Software Development Environment

Lecture 4 in Module 1 has detailed instructions for installing the software required for the various exercises in the course.


Resources

Additional material can be found on the Resources page of the IEEE Technical Committee for Cognitive Robotics website.


Acknowledgements

Support for the preparation of this course was provided by the IEEE Robotics and Automation Society under the program Creation of Educational Material in Robotics and Automation (CEMRA) 2020.

The material on mobile robots is benefitted greatly from a course developed by Alessandro Saffiotti, Örebro University, Sweden, on Artificial Intelligence Techniques for Mobile Robots.

The material on CRAM (Cognitive Robot Abstract Machine) was derived from tutorials on the CRAM website. I am indebted to Michael Beetz and Gayane Kazhoyan for the time and effort they invested explaining CRAM and teaching me how to write CRAM Plan Language programs.

All images and diagrams are either original or have their source credited. Apologies in advance for any unintended omissions. Technical drawings were produced in LaTeX using TikZ and the 3D Plot package.


www.vernon.eu