International Journal of Electrical and Computer Engineering (IJECE) Vol. 13, No. 2, April 2023, pp. 2142∼2155 ISSN: 2088-8708, DOI: 10.11591/ijece.v13i2.pp2142-2155 ❒ 2142 Self-admitted technical debt classification using natural language processing word embeddings Ahmed F. Sabbah 1 , Abualsoud A. Hanani 2 1 Software Engineering Department, Faculty of Engineering and Technology, Birzeit University, Birzeit, Palestine 2 Electrical and Computer Engineering Department, Faculty of Engineering and Technology, Birzeit University, Birzeit, Palestine Article Info Article history: Received Sep 3, 2021 Revised Sep 30, 2022 Accepted Oct 24, 2022 Keywords: Bidirectional encoder representations from transformers Convolutional neural network FastText Self-admitted technical dept Software engineering Technical dept Word2Vec ABSTRACT Recent studies show that it is possible to detect technical dept automatically from source code comments intentionally created by developers, a phenomenon known as self-admitted technical debt. This study proposes a system by which a comment or commit is classified as one of five dept types, namely, require- ment, design, defect, test, and documentation. In addition to the traditional term frequency-inverse document frequency (TF-IDF), several word embed- dings methods produced by different pre-trained language models were used for feature extraction, such as Word2Vec, GolVe, bidirectional encoder represen- tations from transformers (BERT), and FastText. The generated features were used to train a set of classifiers including naive Bayes (NB), random forest (RF), support vector machines (SVM), and two configurations of convolutional neu- ral network (CNN). Two datasets were used to train and test the proposed sys- tems. Our collected dataset (A-dataset) includes a total of 1,513 comments and commits manually labeled. Additionally, a dataset, consisting of 4,071 labeled comments, used in previous studies (M-dataset) was also used in this study. The RF classifier achieved an accuracy of 0.822 with A-dataset and 0.820 with the M-dataset. CNN with A-dataset achieved an accuracy of 0.838 using BERT fea- tures. With M-dataset, the CNN achieves an accuracy of 0.809 and 0.812 with BERT and Word2Vec, respectively. This is an open access article under the CC BY-SA license. Corresponding Author: Abualsoud A. Hanani Department of Electrical and Computer Engineering, Birzeit University Birzeit, Palestine Email: ahanani@birzeit.edu 1. INTRODUCTION Technical debt (TD) is a metaphor, coined by Cunningham [1]. It reflects the additional cost that is implied by rework caused by a sub-optimal solution instead of using the better approach in the software development life cycle. The concept of TD is derived from financial debt, as the interest results from the late payment. TD has an interest rate, and the cost increases if the developer does not pay the debt early, by refactoring the code at a suitable time, to avoid interest in the future. Technical debt is extremely correlated with immature software and issues in software development, such as requirement debt, which measures the difference between the requirement specification and the actual software implementation [2]. Design debt refers to the violation of the good design principle, where code debt includes poor maintenance and readability, and it needs to be refactored. The documentation debt expresses the lack of information that describes the code, and the testing debt describes the shortage of accepted testing [1]. Journal homepage: http://ijece.iaescore.com