Intelligent Control and Automation, 2015, 6, 147-157 Published Online May 2015 in SciRes. http://www.scirp.org/journal/ica http://dx.doi.org/10.4236/ica.2015.62015 How to cite this paper: Al-Shabi, M. (2015) Simulation and Implementation of Real-Time Vision-Based Control System for 2-DoF Robotic Arm Using PID with Hardware-in-the-Loop. Intelligent Control and Automation, 6, 147-157. http://dx.doi.org/10.4236/ica.2015.62015 Simulation and Implementation of Real-Time Vision-Based Control System for 2-DoF Robotic Arm Using PID with Hardware-in-the-Loop Mohammad Al-Shabi Department of Mechatronics Engineering, Philadelphia University, Philadelphia, USA Email: mshabi@philadelphia.edu.jo Received 1 March 2015; accepted 23 May 2015; published 26 May 2015 Copyright © 2015 by author and Scientific Research Publishing Inc. This work is licensed under the Creative Commons Attribution International License (CC BY). http://creativecommons.org/licenses/by/4.0/ Abstract Microsoft Kinect sensor has shown the research community that it's more than just an interactive gaming device, due to its multi-functional abilities and high reliability. In this work, online HIL (Hardware-in-the-Loop) experimental data are used to apply human motion imitation to a 2-degree of freedom Lego Mind storm NXT robotic arm. A model simulation of the dc motor used in this ex- periment is also present in this paper. The acquired input data from the Kinect sensor are processed in a closed loop PID controller with feedback from motors encoders. The applied algo- rithms solve the overlapping input problem, conducting a simultaneous control of both shoulder and elbow joints, and solving the overlapping input problem as well. The work in this paper is presented as a prototype to assure the applicability of the algorithms, for further development. Keywords HIL, NXT, Kinect, PID, Real-Time, Image Processing, Imitation, 2-DoF 1. Introduction Three-dimensional data recognition and processing are recently a vital research area for many modern applica- tions in Mechatronic and the field of engineering in general. Kinect, which is a motion detection and recognition smart sensor developed by the engineers at Microsoft, allows human/computer interaction needless to any phys- ical controllers. The sensor contains two cameras for image acquisition, an RGB camera that captures colors and