Intelligent Multi-Mode Robotic Platform Autonomous Pick-and-Place • AI Tic-Tac-Toe • Mobile Control • Xbox Control

Abstract

Robotics and computer vision are increasingly being applied to create intelligent systems that bridge the gap between digital perception and physical interaction. Toy sorting and object manipulation, tasks that traditionally require direct human involvement, present a compelling opportunity to demonstrate how automation can be made interactive, engaging, and accessible. This project is significant because it explores that intersection — combining real-time object detection, autonomous robotic manipulation, and interactive gameplay into a single low-cost platform, demonstrating that intelligent robotic systems are not limited to industrial settings but can be designed for interactive and educational purposes as well. The most important aspects of this project are the vision-based perception system, the autonomous decision-making pipeline, the mechanical manipulation capability, and the multi- modal control architecture. Together, these aspects form a complete end-to-end system that takes a visual input, processes it intelligently, and produces a meaningful physical action — which is the fundamental challenge at the heart of modern robotics. The main objectives of this project are: to develop a real-time object detection system capable of identifying and classifying toy objects using a custom-trained YOLOv8 model; to enable a mobile robot to autonomously navigate toward detected objects, pick them up using a servo- driven gripper arm, and deliver them to their correct destination boxes; to implement an interactive Tic-Tac-Toe mode in which the robot plays against a human opponent by reading the board state and physically placing game pieces; and to support manual control through both a Flutter mobile application and an Xbox controller connected via Bluetooth. The system was developed using a stand-mounted OAK-D depth camera positioned at a near- vertical angle above the play area to capture an overhead view of the workspace. A YOLOv8 object detection model was trained from scratch on a custom-labeled dataset covering seven classes, built and annotated using Roboflow and trained via Google Colab. The detection pipeline runs on a connected laptop and transmits movement commands wirelessly to the robot via an HC-05 Bluetooth module. The robot itself is built on a Mecanum-wheel platform controlled by an Arduino Mega, with a 3D-printed servo-driven arm handling object manipulation. The Flutter mobile application and Xbox controller provide additional manual control interfaces, both communicating through the same Bluetooth channel. Similar robotic sorting systems have been explored in both academic and commercial contexts. Projects such as robotic arms guided by overhead cameras for bin sorting and warehouse automation share conceptual similarities with this work. Consumer products such as robotic vacuum cleaners also demonstrate environment-aware autonomous navigation. However, the specific combination of overhead vision-guided toy sorting, interactive Tic-Tac-Toe gameplay, vi multi-modal manual control, and a fully custom-trained detection model in a single low-cost prototype represents a novel and distinct contribution that distinguishes this project from existing work.