IEEE TRANSACTIONS ON INDUSTRY APPLICATIONS, VOL. 39, NO. 1, JANUARY/FEBRUARY 2003 21 Stereo Vision in LHD Automation Mark Whitehorn, Student Member, IEEE, Tyrone Vincent, Member, IEEE, Christian H. Debrunner, Member, IEEE, and John Steele, Member, IEEE Abstract—This paper details work in applying stereo vision for the enhancement of safety and productivity in the operation of a load–haul–dump (LHD) vehicle in underground mining. The pri- mary goal of this portion of the research is to provide three–di- mensional (3-D) models of the LHD’s environment. Availability of these models facilitates performance of automated or teleoperated loading tasks and enhances safety through identification and loca- tion of humans in the path of the vehicle. Generation of an accurate 3-D model of the immediate surroundings of the LHD is accom- plished through processing of stereo visual imagery. Stereo video is acquired using a pair of digital cameras mounted above the cab of the LHD. The video data are processed into a dense depth map plus confidence information. These depths and the stereo rig cal- ibration data are then used to construct a 3-D surface model. We demonstrate useful models obtained under both well-illuminated and low-light conditions. Index Terms—Machine vision, mining, modeling, stereo. I. INTRODUCTION B Y ITS NATURE, mining involves heavy equipment and large forces. This type of environment is not conducive to the safety and health of those who do the work unless they are removed from the point of direct application of these forces. This project is focused on loading automation for load–haul–dump vehicles (LHDs). While several automation demonstration projects are under development, production LHDs are all manually operated at the time of this writing, with the exception of Inco mines in Sudbury, ON, Canada, and LKAB’s 1 Kiruna Mine in Sweden where the tramming and dumping tasks have been automated and the loading operation is done via remote control (see Section IV for more detail on the current state of the art). The application of advanced technology to LHDs has thus far been limited to the haul and dump tasks. Loading has proven the most difficult to automate due to the fact that it requires perception of the shape of the muckpile in order to plan the efficient and safe removal of each scoop. Navigation in the vicinity of the muckpile during the loading operation is also complicated by the dynamic nature of the environment. A three–dimensional (3-D) model of the muckpile provides the necessary information for automation of the loading operation, and better quality information for remote operators. Paper PID 02–43, presented at the 2001 Industry Applications Society Annual Meeting, Chicago, IL, September 30–October 5, and approved for publication in the IEEE TRANSACTIONS ON INDUSTRY APPLICATIONS by the Mining Industry Committee of the IEEE Industry Applications Society. Manuscript submitted for review October 15, 2001 and released for publication October 22, 2002. The authors are with the Colorado School of Mines, Golden, CO 80401-1843 USA (e-mail: mcwhite@mines.edu). Digital Object Identifier 10.1109/TIA.2002.807245 1 Luossavaara-Kiirunavaara Aktiebolag (LKAB) is a mining company owned by the Swedish State. TABLE I FATAL ACCIDENTS UNDERGROUND INVOLVING POWER HAULAGE Our current effort is just one example of the application of advanced sensing to move a miner further from harm’s way and allow her/him to operate and manage mining equipment from a healthier, less stressful environment. For example, we believe stereo vision sensing and its fusion with other sensory data will allow us to move the coal miner away from the long-wall shear to a location that is both safer and healthier while simultane- ously improving control over the long-wall operation. II. PROBLEM STATEMENT The goal of this project is to improve the health and safety of underground miners. The approach taken is to move the opera- tors of LHDs to remote locations, away from the vehicle where they can telemanage the operation of the LHD. This has the fol- lowing benefits: • reducing risk of accident; • reducing exposure to hydrocarbon particulates; • reducing exposure to repetitive shock loading. This project is focused on developing the stereo vision and 3-D modeling techniques required to build models of the under- ground to enable automation of the LHD loading operation. A 3-D model of the muckpile (and surrounding area) is necessary for planning the location of each scoop, monitoring the slope of the face and obstacle avoidance. A broader goal is to apply this technology to other mining situations and environments, and to understand the requirements of implementing this technology in a number of mining venues. III. HEALTH AND SAFETY MOTIVATION Operation of power haulage equipment is a major source of fatal injuries in the mining industry. Table I shows the number of fatal accidents for each of the years 1995–2000 that were asso- ciated with underground power haulage. Fourteen miners have been killed since 1995. These numbers include only metal/non- metal underground operations. If we include the numbers for underground coal involving power haulage, we would see a sig- nificant increase. The systems being developed for this project will be applicable to all of these situations. If the equipment can be automated such that the miner is moved to a remote location, both his/her safety and health will be improved. In addition, many current implementations of remote control do not pro- vide sufficient information for the operator to make decisions 0093-9994/03$17.00 © 2003 IEEE