SignTalkis

Abstract

SignTalkis an assistive system designed to enhance communication between the deaf and hearing community using sign language technology. The system includes three main features, each addressing a unique interaction challenge and helping to break down communication barriers in public and private settings, such as hospitals, airports, and schools. Text/speech to sign language (letter-by-letter): Through a mobile app developed using the React framework, the user can input text or voice. The input is then sent to a 3D-printed robotic hand, controlled by servo motors, which moves individual sign language letters one by one, helping the hearing-impaired communicate. Sign language to text/speech recognition: A camera captures the sign gestures produced by the person, and using MediaPipe and a trained CNN model (via PyTorch), the system recognizes the hand gestures and classifies them by letter. The recognized letters are then converted into text that is displayed on an LCD screen and spoken aloud via a loudspeaker, helping deaf people communicate with their peers. Rock, Paper, Scissors: A fun and interactive application, the system allows users to play the classic game of rock, paper, scissors. Using computer vision and gesture detection on a Raspberry Pi 4, the user plays against the system, which randomly selects a move and announces the winner with visual and audio feedback. A robotic hand then displays the system's actual movement. The hardware used includes an Arduino Mega 2560, an ESP32, and a Raspberry Pi 4 Model B (4GB RAM). The ESP32 manages the wireless connection between the application and the main device (Arduino Mega), while the Raspberry Pi processes video inputs and runs the AI models. The system integrates mechanical motion, AI-based recognition, and real-time interaction into a single, compact, and scalable solution aimed at making communication more inclusive and accessible.