Cognitive Robotics

David Vernon
Institute for Artificial Intelligence
University of Bremen
Germany

Beta version (Final version January 2021)
Outstanding tasks: add a simulator for the LynxMotion AL5D arm, add material on using LynxMotion AL5D arm and a Pepper robot with CRAM



A PR2 robot sets a table during a demonstration of cognitively-enabled robot manipulation using CRAM.
Image courtesy of the EASE interdisciplinary research center at the University of Bremen, Germany.

 


Course Description  |  Learning Objectives  |  Content  |  Lecture Notes  |  Course Textbook  |  Recommended Reading |  Software |  Resources  |  Acknowledgements

Course Description

This course does not assume you have already studied robotics and it covers both the essentials of classical robotics and the core topics in cognitive robotics.

The focus of cognitive robotics is on flexible context-sensitive goal-directed action. A cognitive robot anticipates the need to act and the outcome of the action. The action itself is guided by prospection. A cognitive robot can also adapt to changing circumstances, adjusting existing action policies and creating new action policies when required.

After a general overview of the field, the course begins with the key elements of mobile robots, robot manipulators, and robot vision, using ROS (Robot Operating System) and OpenCV. It then progresses to the main topics in artificial cognitive systems, including the different paradigms of cognitive science and cognitive architectures. These components form the foundation for the remainder of the course, involving a detailed study of the CRAM (Cognitive Robot Abstract Machine) cognitive architecture, building on ROS, and exploiting functional programming to reason about and execute under-determined tasks in everyday activities.

The course covers both theory and practice, using robot simulators as well as low-cost robots and cameras for practical examples and exercises.

Support for the preparation of this course was provided by grant from the IEEE Robotics and Automation Society under the program Creation of Educational Material in Robotics and Automation (CEMRA) 2020.

"Cognitive Robotics is the field that combines insights and methods from robotics, AI and cognitive and biological sciences to design an integrated cognitive system combining the sensorimotor behavior and higher-level functions and social capabilities of an intelligent robot"

A. Cangelosi and M. Asada, Cognitive Robotics, MIT Press, in press.


Learning Objectives

After completing this course, students should be able to:

  1. Apply their knowledge of machine vision and robot kinematics to create computer programs that control mobile robots and robot arms, enabling the robots to recognize and manipulate objects and navigate their environments.

  2. Explain how a robot can be designed to exhibit cognitive goal-directed behaviour through the integration of computer models of vision, reasoning, learning, prospection, and social interaction.

  3. Use computer programs that realize limited instances of these faculties.


Course Content

Goals of the course Learning objectives Resources Overview of Cognitive Robotics (Lectures 1 - 3)

  • The interdisciplinary nature of cognitive robotics.
  • Definition of cognitive robotics.
  • What is cognition?
  • Why is cognition useful in robotics?
  • How do cognitive robots work?
  • Goals of the course.
  • Learning outcomes.
  • Industrial requirements for cognitive robots.
  • Installation of software development environments for exercises and assignments.

Mobile Robots (Lectures 4 - 9)

  • Introduction to ROS (Robot Operating System).
  • Using the ROS Turtlebot simulator.
  • Writing ROS software.
  • Types of mobile robots.
  • The challenge of robot navigation
  • Relative inertial position estimation.
  • Kinematics of a two-wheel differential drive robot.
  • Relative odometry-based position estimation.
  • Absolute position estimation.
  • Closed-loop feedback control.
  • PID control.
  • Go-to-position and go-to-pose problems.
  • Divide and conquer controller.
  • MIMO controller.
  • Path tracking through waypoints
  • Finding a shortest path in a map.
  • Breadth-First Search (BFS).
  • Other search strategies.

Robot Manipulators (Lectures 10 - 13)

  • Robot programming.
  • Description of object pose with homogenous transformations.
  • Robot programming by frame-based task specification.
  • Convention for embedding T6 and end-effector frames.
  • Specifying orientation: roll, pitch, yaw; Euler angles.
  • Pick-and-place example of task-level robot programming.
  • Implementation for a Lynxmotion AL5D robot arm using Frame classes in C++.
  • Forward and inverse kinematics of a manipulator.
  • Denavit-Hartenberg reporesentation.
  • Kinematic specification of the LynxMotion AL5D arm.
  • Inverse kinematics of the LynxMotion AL5D arm.

Robot Vision (Lectures 14 - 20)

  • Computer vision.
  • Optics and sensors.
  • Image acquisition and image representation.
  • Image processing.
  • Introduction to OpenCV.
  • Segmentation.
  • Region-based approaches: feature-based thresholding, connected component analysis, graph cuts.
  • Boundary-based approaches: edge detection, boundary representations.
  • Image Analysis. Inspection, location, identification.
  • Feature extraction.
  • Classification: k-nearest neighbour, minimum distance, linear, maximum likelihood and Bayes classifiers.
  • Homogeneous coordinates and transformations.
  • Perspective transformation.
  • Camera model and inverse perspective transformation.
  • Camera calibration.

Artificial Cognitive Systems (Lectures 21 - 24)

  • Paradigms of cognitive science.
  • Learning and development.
  • Memory and prospection.
  • Knowledge and representation.
  • Social cognition.

Cognitive Architectures (Lectures 25 - 26)

  • Specification and operation.
  • Desirable characteristics.
  • Core cognitive abilities.
  • Example cognitive architectures: Soar, ACT-R, Clarion, ICARUS, BBD, ISAC.
  • The Common Model of Cognition.

The CRAM Cognitive Architecture (Lectures 27 - 45)

  • Overview of the CRAM (Cognitive Robot Abstract Machine) cognitive architecture.
  • Programming paradigms.
  • Essentials of Common Lisp.
  • Emacs.
  • The CRAM plan language.
  • Description of object pose with quaternions in ROS.
  • CRAM and the ROS Turtlesim turtlebot simulator.
  • Creating a CRAM package.
  • Controlling a turtle.
  • Implementing simple plans to move a turtle.
  • Using Prolog for reasoning.
  • Creating motion designators for the TurtleSim.
  • Creating process modules.
  • Automatically choosing a process module for a motion.
  • Using location designators with TurtleSim.
  • Writing high-level plans for TurtleSim.
  • Implementing failure handling for TurtleSim.
  • Fetch-and-place CRAM plan for the PR2 robot.
  • Bullet Real-time Physics Simulation.
  • Error handling.
  • Recovery behaviors.


Lecture Notes

If you are an instructor and would like a copy of the complete set of PowerPoint slides, please contact me by email at davidvernon.eu.

Lecture 1. Overview of cognitive robotics: component disciplines; defining cognition; the need for cognitive robots; how cognitive robots operate.
Lecture 2. Overview of cognitive robotics: industrial requirements.
Lecture 3. Overview of cognitive robotics: installation of software development environments for exercises.
Lecture 4. Mobile robots: introduction to ROS (Robot Operating System); the Turtlesim turtlebot simulator.
Lecture 5. Mobile robots: writing ROS software in C++.
Lecture 6. Mobile robots: types of mobile robots; the challenge of robot navigation; relative inertial position estimation.
Lecture 7. Mobile robots: relative odometry position estimation; kinematics of a two-wheel differential drive robot; absolute position estimation.
Lecture 8. Mobile robots: closed-loop control and PID control; the go-to-position and go-to-pose problems; divide-and-conquer and MIMO controllers.
Lecture 9. Mobile robots: finding a shortest path in a map; breadth-first search algorithm; other search approaches.
Lecture 10. Robot manipulators: robot programming; description of object pose with homogenous transformations.
Lecture 11. Robot manipulators: task specification; embedding T6 and end-effector frames; specifying orientation; roll, pitch, yaw; Euler angles.
Lecture 12. Robot manipulators: pick-and-place example; implementation for a Lynxmotion AL5D robot arm using Frame classes in C++.
Lecture 13. Robot manipulators: kinematics; Denavit-Hartenberg representation; kinematics and inverse kinematics of the LynxMotion AL5D arm.
Lecture 14. Robot vision: computer vision; optics and sensors; image acquisition; image representation; image processing; introduction to OpenCV.
Lecture 15. Robot vision: segmentation; region-based approaches; feature-based thresholding; graph cuts.
Lecture 16. Robot vision: segmentation; boundary-based approaches; edge detection.
Lecture 17. Robot vision: image analysis; feature extraction.
Lecture 18. Robot vision: k-nearest neighbour, minimum distance, linear, maximum likelihood and Bayes classifiers.
Lecture 19. Robot vision: perspective transformation; camera model; camera calibration.
Lecture 20. Robot vision: inverse perspective transformation; stereo vision; epipolar geometry.
Lecture 21. Artificial cognitive systems: the paradigms of cognitive science.
Lecture 22. Artificial cognitive systems: learning and development.
Lecture 23. Artificial cognitive systems: memory; prospection; internal simulation, the symbol grounding problem.
Lecture 24. Artificial cognitive systems: interaction, social cognition; theory of mind, instrumental helping, collaboration.
Lecture 25. Cognitive architectures: specification and operation; desirable characteristics; core cognitive abilities.
Lecture 26. Cognitive architectures: Example cognitive architectures: Soar, ACT-R, Clarion, ICARUS, BBD, ISAC; The Common Model of Cognition.
Lecture 27. The CRAM cognitive architecture: overview of CRAM (Cognitive Robot Abstract Machine).
Lecture 28. The CRAM cognitive architecture: Common Lisp - REPL, lists, structures, equality, conditionals, CONS, CAR, CDR, dotted and assoc-lists.
Lecture 29. The CRAM cognitive architecture: Common Lisp - functions, I/O, recursion, iteration, lambda and mapping functions, CLOS, inference.
Lecture 30. The CRAM cognitive architecture: Emacs.
Lecture 31. The CRAM cognitive architecture: CRAM plan language, fluents, concurrency, reasoning, exception handling.
Lecture 32. The CRAM cognitive architecture: CRAM plan language, designators, process modules.
Lecture 33. The CRAM cognitive architecture: Specifying pose in ROS, quaternions.
Lecture 34. The CRAM cognitive architecture: Turtlesim with CRAM; creating a CRAM package; controlling a turtle.
Lecture 35. The CRAM cognitive architecture: Turtlesim with CRAM; implementing plans to move a turtle.
Lecture 36. The CRAM cognitive architecture: using Prolog for reasoning.
Lecture 37. The CRAM cognitive architecture: creating motion designators for the TurtleSim.
Lecture 38. The CRAM cognitive architecture: creating process modules.
Lecture 39. The CRAM cognitive architecture: automatically choosing a process module for a motion.
Lecture 40. The CRAM cognitive architecture: using location designators with TurtleSim.
Lecture 41. The CRAM cognitive architecture: writing high-level plans for TurtleSim.
Lecture 42. The CRAM cognitive architecture: implementing failure handling for TurtleSim.
Lecture 43. The CRAM cognitive architecture: fetch-and-place CRAM plan for the PR2 robot; Bullet real-time physics simulation.
Lecture 44. The CRAM cognitive architecture: fetch-and-place CRAM plan for the PR2 robot; error handling and recovery looking in different places.
Lecture 45. The CRAM cognitive architecture: fetch-and-place CRAM plan for the PR2 robot; error handling and recovery using different arms.


Course Textbook

Future project.


Recommended Reading

Beetz, M., Mösenlechner, L., and Tenorth, M. (2010). "CRAM - A Cognitive Robot Abstract Machine for Everyday Manipulation in Human Environments", IEEE/RSJ International Conference on Intelligent Robots and Systems, 1012-1017.

Cangelosi, A. and Asada, M. (Eds.), Cognitive Robotics, MIT Press, in press.

Corke, P. (2016). Robotics, Vision and Control, 2nd Edition, Springer.

Paul, R. (1981). Robot Manipulators: Mathematics, Programming, and Control. MIT Press.

Szeliski, R. (2010). Computer Vision: Algorithms and Applications, Springer.

Vernon, D. (1991). Machine Vision: Automated Visual Inspection and Robot Vision, Prentice-Hall.

Vernon, D. Artificial Cognitive Systems, MIT Press, 2014.

Vernon, D. and Vincze, M. "Industrial Priorities for Cognitive Robotics", Proceedings of EUCognition 2016, Cognitive Robot Architectures, European Society for Cognitive Systems, Vienna, 8-9 December, 2016, R. Chrisley. V. C. Müller, Y. Sandamirskaya. M. Vincze (eds.), CEUR-WS Vol-1855, ISSN 1613-0073, pp. 6-9.


Software Development Environment

Lecture 3 has detailed instructions for installing the software required for the various exercises in the course.


Resources

Additional material can be found on the Resources page of the IEEE Technical Committee for Cognitive Robotics website.


Acknowledgements

The material on mobile robots is benefitted greatly from a course developed by Alessandro Saffiotti, Örebro University, Sweden, on Artificial Intelligence Techniques for Mobile Robots.

The material on CRAM (Cognitive Robot Abstract Machine) was derived from tutorials on the CRAM website. I am indebted to Michael Beetz and Gayane Kazhoyan for the time and effort they invested explaining CRAM and teaching me how to write CRAM Plan Language programs.

All images and diagrams are either original or have their source credited. Apologies in advance for any unintended omissions. Technical drawings were produced in LaTeX using TikZ and the 3D Plot package.


www.vernon.eu