Highlights
- 6 interconnected ROS2 packages for multi-modal robot control
- Hand gesture recognition with MediaPipe translating to velocity commands
- Voice-activated commands via OpenAI Whisper with silence detection
- Web-based camera streaming for wireless video input
- Modular architecture with custom ROS2 message types
- Reinforcement learning for autonomous user-following (in progress)
About This Project
A ROS2-based system that integrates computer vision, motion control, speech recognition, and text-to-speech capabilities. The system consists of six interconnected ROS2 packages that enable gesture and voice-based interaction with the iRobot Create 3 robot.
The hand gesture recognition node uses MediaPipe and OpenCV to translate hand positions into velocity commands, allowing intuitive directional control without physical input devices. A web-based camera streaming node enables any browser-capable device to serve as a video source for the ROS2 network.
Speech recognition is powered by OpenAI Whisper integrated into a ROS2 node for voice-activated commands, with automatic silence detection to filter out background noise. The modular architecture uses custom message types allowing vision, audio, and motion nodes to operate independently while communicating through ROS2 topics and services.