Multimedia Tools and Applications https://doi.org/10.1007/s11042-019-08547-4 Artist-based painting classification using Markov random fields with convolution neural network Kai-Lung Hua 1 · Trang-Thi Ho 1 · Kevin-Alfianto Jangtjik 1 · Yu-Jen Chen 2 · Mei-Chen Yeh 3 Received: 14 February 2019 / Revised: 22 August 2019 / Accepted: 27 November 2019 / © Springer Science+Business Media, LLC, part of Springer Nature 2020 Abstract Determining the authorship of a painting image is a challenging task because paintings of an artist may not have a unique style and various artists may have similar painting styles. In this paper, we present a new approach to categorize digital painting images based on artist. We construct a multi-scale pyramid from a painting image to consider both globally and locally the information contained in one image. For each layer, we train a Convolu- tional Neural Network (CNN) model to determine the class label. To build the relationship within local image patches, we employ Markov random fields (MRFs) by optimizing the Gibbs energy function defined by (1) the data term measuring the compatibility of labeling with given data, and (2) the smoothness term penalizing assignments that label neighboring patches differently. A new fusion scheme is proposed to aggregate patch-level classification results. The proposed CNN-MRF method is validated using two challenging painting image datasets. Experimental results show that the proposed method is effective and achieves state-of-the-art performance. Keywords Image classification · Multi-scale pyramid · Markov random fields · Convolutional neural network 1 Introduction In recent years, deep learning techniques have been implemented in various multimedia applications, including image recognition [3, 14, 18, 44–46] and multimodal data analysis [4, 6, 37, 38, 41]. Many deep learning based systems have been built to enhance the image  Mei-Chen Yeh myeh@csie.ntnu.edu.tw 1 Department of Computer Science and Information Engineering, National Taiwan University of Science and Technology, Taipei, Taiwan 2 MacKay Memorial Hospital, Taipei, Taiwan 3 Department of Computer Science and Information Engineering, National Taiwan Normal University, Taipei, Taiwan