Motivation
The integration of advanced AI with robotics represents one of the most promising frontiers in autonomous systems research. At the Lincoln Centre for Autonomous Systems (L-CAS), we’re exploring how multi-modal foundation models can revolutionise human-robot interaction. This project focuses on developing innovative approaches to enable natural, intuitive communication between humans and robots using cutting-edge AI technology. By combining the reasoning capabilities of Large Language Models with multi-modal sensor data (LiDAR, cameras, IMUs) and state representations (discrete positions, global positions, obstacles, maps) from robotic platforms, we aim to create systems that understand and communicate about their environment in human-like ways. This research will make robotic systems more accessible to untrained users through conversation-based interfaces, bridging the gap between sophisticated autonomous systems and everyday human interaction patterns. Your contribution will help shape the future of intuitive human-robot collaboration in various domains.
Required Skills
- Strong programming skills in Python
- Experience with ROS2 or willingness to learn
- Keen interest in robotics and multi-modal foundation (or LLM) models and agentic behaviour (e.g. Model Context Protocol)
- Understanding of different sensor modalities used in robotics
- Enthusiasm for learning new technologies
- Excellent communication abilities
- Problem-solving mindset and creative thinking
Skills to Be Gained
This project offers an exceptional opportunity to develop expertise at the intersection of artificial intelligence and robotics. Students will gain hands-on experience implementing Retrieval Augmented Generation (RAG) techniques to provide real-time contextual awareness to robotic systems using ROS2. You’ll develop advanced skills in integrating multi-modal foundation models with diverse robot sensor data (LiDAR, cameras, IMUs) and state representations, creating intuitive interfaces between humans and machines. Working with both real robots and recorded data streams (ROS bags) will provide flexibility in testing and evaluating your implementation. The experience gained with state-of-the-art foundation models and robotics programming frameworks will be valuable for cutting-edge research and industry applications. The project will enhance your understanding of both natural language processing and robotics system integration, preparing you for careers in AI research, robotics development, human-computer interaction, and related fields.
This is a project suitable as a final year project for any Lincoln students studying Computer Science or Robotics, or as an internship position in robotics. If you are interested to work on this as an intern fill out our Expression of Interest Form, choosing Professor Marc Hanheide as the researcher to supervise the project. If you are a Lincoln student wishing to pursue this project as part of your studies, please refer to your respective project module’s procedure on project selection and allocation.