IEEE/ASME TRANSACTIONS ON MECHATRONICS, VOL. 9, NO. 4, DECEMBER 2004 661 Cooperative Teleoperation of a Multirobot System With Force Reflection via Internet Wang-tai Lo, Yunhui Liu, Senior Member, IEEE, Imad H. Elhajj, Member, IEEE, Ning Xi, Member, IEEE, Yuechao Wang, and Toshio Fukuda, Fellow, IEEE Abstract—With the rapid development of information tech- nology, the Internet has evolved from a simple data-sharing media to an amazing information world where people can enjoy various services. Recently, the use of the Internet has been expanded to the field of automation, i.e., using the Internet as a tool to control equipment located at remote sites. This paper presents a system that enables multiple operators at different sites to cooperatively control multiple robots with real-time force reflecting via the Internet. To overcome instability and reliability problems caused by random time delay of the Internet communication, we adopt an event as the reference for the controller design. To improve real-time efficiency and reduce the complexity of the controller, a distributed approach is proposed for the control of remote robots, so that the time delay in one control loop does not affect performance of the others. A vision-based method is developed to monitor and render interactions between the robots. The usefulness and effectiveness of the developed method and system have been verified by teleoperation experiments on a two-robots cooperative system among Hong Kong, mainland China, and the U.S. Index Terms—Cooperative control, distributed systems, force feedback, internet robots, teleoperation. I. INTRODUCTION W ITH THE rapid development of information technology, the Internet has been growing considerably in the past few years. It has evolved from a simple data-sharing media to an amazing information world where people can enjoy various ser- vices. And, its potential is still expanding. The Internet is widely spread and accessible all over the world. It has a high-transmis- Manuscript received December 27, 2002; revised April 7, 2003, July 21, 2003. This work was supported in part by the Hong Kong RGC and National Science Foundations of China under Grants N_CUHK404/01 and CUHK4173/00E, in part by the National Science Foundations of China under Project 60334010, and in part by the Chinese High-Tech Program (863) under Projects AA422250 and AA135220. W. Lo is with the Department of Automation and Computer-Aided Engi- neering, The Chinese University of Hong Kong, Shatin, N.T., Hong Kong. Y. Liu is with the Department of Automation and Computer-Aided Engineering, The Chinese University of Hong Kong, Shatin, N.T., Hong Kong, and is also with the Joint Center for Intelligent Sensing and Sys- tems, National University of Defense Technology, Changsha, China (e-mail: yhliu@acae.cuhk.edu.hk). I. H. Elhajj is with the Department of Computer Science and Engineering, Oakland University, Rochester, MI 48309 USA. N. Xi is with the Department of Electrical and Computer Engineering, Michigan State University, East Lansing, MI 48824 USA, and is also with the Shenyang Institute of Automation, Chinese Academy of Sciences, Shenyang 110015, China. Y. Wang is with the Shenyang Institue of Automation, Chinese Academy of Science, Shenyang 110015, China. T. Fukuda is with the Center for Cooperative Research in Advanced Science and Technology, Nagoya University, Nagoya 464-8603, Japan. Digital Object Identifier 10.1109/TMECH.2004.839040 sion rate while the cost is relatively low. With these character- istics, the Internet is one of the most convenient data-transmis- sion medias for teleoperation tasks, and therefore, the use of the Internet for teleoperating robots and mechatronic systems from remote sites has become recently one of the hottest topics in robotics and automation [1]–[6]. Existing telerobotic systems can be classified as two cat- egories: teleprogramming systems or telecontrol systems. In teleprogramming systems, operators rely on the visual informa- tion only to instruct robots to execute a motion that has already been programmed offline, and the method is basically an open-loop control. In telecontrol systems, operators have direct access to real-time control of the robot at the remote site based on real-time feedback of supermedia information, including images, force, and other information for haptic redering. One of the biggest challenges in the telecontrol method using the Internet is how to solve various problems caused by time delay of the Internet communication. Due to varying network load and changing routing path, the transmission time of data packets over the Internet from one point to another is random and, therefore, cannot be predicted accurately. The existence of the random time delay in the control loop makes the synchro- nization of tasks and actions of different units difficult. As a result, the stability of the system is significantly affected and its operation becomes unreliable. A few approaches [7]–[13] have been proposed to solve the critical issues caused by the time delay. One of the effective methods is the use of a nontime-based reference for controller design, developed in our earlier work [14], [15]. The systems mentioned above mainly concern teleoperation of a single robot by a single operator. Cooperative telecontrol of multiple robots by multi-operators can accomplish compli- cated and sophisticated tasks that cannot be performed by a single robot or operator. Teleoperation of multirobots by multi- operators via the Internet is being extensively studied in recent years. Goldberg et al. [16] have set up a collaborative teleop- erated system, which enables several users, through an Internet browser, to play the well-known Ouija board game together. El- hajj et al. [17] have developed the first multisite Internet-based teleoperation system with real-time force reflecting. This system allows operators from Hong Kong and Japan to interactively and cooperatively control a mobile manipulator located in the U.S. Chong et al. [18] have built a telemanipulation test bed in which one local operator and one remote operator control the robot with a local online graphics simulator to cope with the time delay. Kheddar et al. [19] developed a multirobot teleop- eration system between Japan and France using an intermediate 1083-4435/04$20.00 © 2004 IEEE