Gesture and Voice-Control Based Virtual Mouse Using Machine Learning
Abstract
Human-Computer Interaction (HCI) has experienced significant advancements through the integration of machine learning, computer vision, and speech recognition technologies. Traditional input devices such as keyboards and mice present challenges for individuals with physical disabilities and in environments where contactless interaction is required.
This paper proposes a Gesture and Voice-Control Based Virtual Mouse system that enables users to perform computer operations through hand gestures and voice commands without using physical peripherals. The system employs MediaPipe Hands for hand landmark extraction, a machine learning-based gesture recognition model for gesture classification, and Google Speech Recognition for voice command processing.
The proposed solution supports cursor movement, clicking, scrolling, dragging, and application control functions. Experimental results indicate a gesture recognition accuracy of 94.7% and voice recognition accuracy of 88.3% while maintaining real-time performance at 31 FPS.
The developed system provides an accessible, low-cost, and efficient alternative for contactless human-computer interaction.
Citation
Tandwe Patrick & Sesu Ovid. (2026)
Gesture and Voice-Control Based Virtual Mouse Using Machine Learning
International Journal of Current Science Research (IJCSR)
e-ISSN: 2454-5422
12(6): 2026: 8–13
License
© 2026 The Author(s). Published by Dr. BGR Publications .
The authors retain copyright of this article.
This is an open access article distributed under the terms of the Creative Commons Attribution 4.0 International License (CC BY 4.0) , which permits unrestricted use, distribution, and reproduction in any medium, provided the original author(s) and source are credited.
References
- S. Mitra and T. Acharya, “Gesture Recognition: A Survey,” IEEE Transactions on Systems, Man, and Cybernetics, vol. 37, no. 3, pp. 311–324, 2007.
- S. S. Rautaray and A. Agrawal, “Vision-Based Hand Gesture Recognition for Human Computer Interaction,” Artificial Intelligence Review, vol. 43, no. 1, pp. 1–54, 2015.
- F. Zhang et al., “MediaPipe Hands: On-Device Real-Time Hand Tracking,” Google Research, 2020.
- P. Molchanov et al., “Hand Gesture Recognition with 3D Convolutional Neural Networks,” Proceedings of CVPR, pp. 1–9, 2016.
- J. Liu et al., “Hand Gesture Recognition Using LSTM Networks,” Pattern Recognition Letters, vol. 125, pp. 367–373, 2019.
- O. Oyedele et al., “Contactless Mouse Using CNN and OpenCV,” International Journal of Computer Applications, vol. 184, no. 22, pp. 12–18, 2022.
- M. Khan and R. Jain, “Voice and Gesture Multimodal Human Computer Interaction System,” IEEE Access, vol. 9, pp. 112345–112356, 2021.
- M. J. Cheok, Z. Omar, and M. H. Jaward, “A Review of Hand Gesture and Sign Language Recognition Technologies,” Neural Computing and Applications, vol. 31, pp. 131–153, 2019.
- Google Research, “MediaPipe Hands Documentation,” 2020.
- TensorFlow Team, “TensorFlow Machine Learning Framework Documentation,” 2024.
