Vol.:(0123456789) 1 3 Arabian Journal for Science and Engineering https://doi.org/10.1007/s13369-020-04811-0 RESEARCH ARTICLE-COMPUTER ENGINEERING AND COMPUTER SCIENCE ML‑DCNNet: Multi‑level Deep Convolutional Neural Network for Facial Expression Recognition and Intensity Estimation Muhammad Aamir 1  · Tariq Ali 2  · Ahmad Shaf 1  · Muhammad Irfan 2  · Muhammad Qaiser Saleem 3 Received: 14 April 2020 / Accepted: 17 July 2020 © King Fahd University of Petroleum & Minerals 2020 Abstract The human face has a great accumulation and a diversity of facial expressions. It explores the feelings of a person and can be used to judge the emotional intents of the person to a certain level. By using facial detection and recognition systems, varieties of applications are working in computer vision, surveillance system, security, authentication, or verifcation of a person and home automation system based on digital image processing with the help of the Internet of Things. The state of the art in these applications is to detect expressions with their intensity level. It is an attention-grabbing problem due to the complex nature of facial features, which is associated with emotions. For that purpose, it is essential to develop an innovative deep learning model to detect and estimate the facial expression intensity level. To do this, a multi-level deep convolutional neural network is proposed to recognize facial expression and their intensity level. At the frst level, Expression-Net classi- fes face expressions, and at the second level, Intensity-Net estimates the intensity of the facial expression. Evaluation of the proposed model for facial expression recognition and intensity estimation is carried out by using the extended Cohn–Kanade and Japanese Female Facial Expression datasets. The proposed method shows an outstanding performance in terms of accu- racy of 98.8% and 97.7% for both the datasets as compared to state-of-the-art techniques. Keywords CNN · Facial expressions recognition · Facial expression intensity estimation ML-DCNNet · Deep learning · Computer vision 1 Introduction At present, the communication which occurs only in one direction is considered as an active one. However, the com- puter may listen to human dialogues through the usage of exclusive audio and speech recognition equipment. Letting computers to recognize humanoid emotions through visuali- zation will connect the gap between computer and human communication, which will make the computer very active, smart, and user-friendly. The facial expressions related to humans are a reliable source of nonverbal communication. To analyze and recognize expressions of the face using com- puters so that they will understand whether users feel some type of emotions like happy, bored, etc., or not. Many appli- cations are working based on digital image processing with the help of the Internet of Things (IoT) like home automa- tion system, monitoring of diferent industrial parameters, detection and positioning of diferent objects, and security systems [1]. It would be phenomenal experimentation and a real-world development in the computer-vision system * Tariq Ali tariqhsp@gmail.com Muhammad Aamir muhammadaamir@cuisahiwal.edu.pk Ahmad Shaf ahmadshaf@cuisahiwal.edu.pk Muhammad Irfan irfan16.uetian@gmail.com Muhammad Qaiser Saleem muhammad.qaiser.saleem@gmail.com 1 Computer Science Department, COMSATS University Islamabad, Sahiwal Campus, Sahiwal, Pakistan 2 College of Engineering, Electrical Engineering Department, Najran University, Najran 61441, Kingdom of Saudi Arabia 3 College of Computer Science and Information Technology, Al Baha University, Al Baha, Kingdom of Saudi Arabia