Computer Methods and Programs in Biomedicine 177 (2019) 253–268 Contents lists available at ScienceDirect Computer Methods and Programs in Biomedicine journal homepage: www.elsevier.com/locate/cmpb A novel mixed reality in breast and constructive jaw surgical tele-presence Haritha Sallepalli Venkata a , Abeer Alsadoon a,* , P.W.C. Prasad a , Omar Hisham Alsadoon b , Sami Haddad c,d , Anand Deva e , Jeremy Hsu e a School of Computing and Mathematics, Charles Sturt University, Sydney, Australia b Department of Islamic Sciences, Al Iraqia University, Baghdad, Iraq c Department of Oral and Maxillofacial Services, Greater Western Sydney Area Health Services, Sydney, Australia d Department of Oral and Maxillofacial Services, Central Coast Area Health, Australia e Faculty of Medicine and Health Sciences, Macquarie University, Sydney, Australia a r t i c l e i n f o Article history: Received 18 January 2019 Revised 25 April 2019 Accepted 28 May 2019 Keywords: Augmented reality Virtual reality Mixed reality Surgical telepresence Visualization a b s t r a c t Background and aim: Surgical telepresence has been implemented using Mixed reality (MR) but, MR is theory based and only used for investigating research. The Aim of this paper is to propose and implement a new solution by merging augmented video (generating in local site) and virtual expertise surgeon hand (remote site). This system is to improve the visualization of surgical area, overlay accuracy in the merged video without having any discoloured patterns on hand, smudging artefacts on surgeon hand boundary and occluded areas of surgical area. Methodology: The Proposed system consists of an Enhanced Multi-Layer Mean Value Cloning (EMLMV) algorithm that improves the overlay accuracy, visualization accuracy and the processing time. This pro- posed algorithm includes trimap and alpha matting as a pre-processing stage of merging process, which helps to remove the smudging and discoloured artefacts surrounded by remote surgeon hand. Results: Results showing that the proposed system improved the accuracy by reducing the overlay error of merging image from 1.3 mm (Millimeter) to 0.9 mm. Furthermore, it improves the visibility of surgeon hand in the final merged image from 98.4% (visibility of pixels) to 99.1% (visibility of pixels). Similarly, the processing time in our proposed solution is reduced, which is computed as 10 s to produce 50 frames, whilst, the state of art solution computes 11 s for the same number of frames. Conclusion: The proposed system focuses on the merging of augmented reality video (local site), and the virtual reality video (remote site) with the accurate visualization. we consider discoloured areas, smudg- ing artefacts and occlusion as the main aspects to improve the accuracy of merged video in terms of overlay error and visualization error. So, the proposed system would produce the merged video with the removal of artefacts around the expert surgeon hand. © 2019 Elsevier B.V. All rights reserved. 1. Introduction The surgical field requires outstanding technical skills and spon- taneous decision making depending on the patient’s situation. However, Expertise or speciality surgical care is not available glob- ally. In addition, Surgeries for hard and soft tissues are taking place rapidly. Some complex surgical situations require a real-time as- sistance from another expertise surgeon either virtually from his remote site or physically by operating in local site (surgery site). However, surgeon attentedence at local site is time consuming and not efficient [1]. Several digital technologies are developed to solve * Corresponding author. E-mail address: aalsadoon@studygroup.com (A. Alsadoon). these issues which includes video-assisted systems, tele monitor- ing systems, telemedicine systems, and telepresence. These tech- nologies are created as an initiative to support remote collabora- tion. In recent decades, tele surgical platforms has evolved, but has not been amended globally due to cost barriers and lack of flexi- bility [2]. Currently, the telepresence concept is introduced to replace the traditional methods to support remote collaboration [3]. Shenai et al. [3] described telepresence as two different fields located at different sites, these can visualize the common field which is digi- tally merged and shared to each site. One of the latest technology is Mixed reality, which can satisfy the telepresence essentials. It is introduced to contribute the remote collaboration that combines https://doi.org/10.1016/j.cmpb.2019.05.025 0169-2607/© 2019 Elsevier B.V. All rights reserved.