Computer Science & Engineering: An International Journal (CSEIJ), Vol 15, No 3, June 2025 DOI:10.5121/cseij.2025.15301 1 CAN WE TRUST MACHINES? A CRITICAL LOOK AT SOME MACHINE TRANSLATION EVALUATION METRICS Muhammad Zayyanu Zaki 1 and Nazir Ibrahim Abbas 2 1 Department of French, Usmanu Danfodiyo University, Sokoto, Nigeria 2 Department of Nigerian Languages, Usmanu Danfodiyo University, Sokoto, Nigeria ABSTRACT The growing interconnection of the globalised world necessitates seamless cross-lingual communication, making Machine Translation (MT) a crucial tool for bridging communication gaps. In this research, the authors have critically evaluated two prominent Machine Translation Evaluation (MTE) metrics: BLEU and METEOR, examining their strengths, weaknesses, and limitations in assessing translation quality, focusing on Hausa-French and Hausa-English translation of some selected proverbs. The authors compared automated metric scores with human judgments of machine-translated text from Google Translate (GT) software. The analysis explores how well BLEU and METEOR capture the nuances of meaning, particularly with culturally bounded expressions of Hausa proverbs, which often have meaning and philosophy. By analysing the performance of the translator’s datasets, they aim to provide a comprehensive overview of the utility of these metrics in Machine Translation (MT) system development research. The authors examined the relationship between automated metrics and human evaluations, identifying where these metrics may be lacking. Their work contributes to a deeper understanding of the challenges of Machine Translation Evaluation (MTE) and suggests potential future directions for creating more robust and reliable evaluation methods. The authors have explored the reasons behind human evaluation of MT quality examining its relationship with automated metrics and its importance in enhancing MT systems. They have analysed the performance of GT as a prominent MT system in translating Hausa proverbs, highlighting the challenges in capturing the language ’s cultural and contextual nuances. KEYWORDS Machine Translation, Quality Metrics, Evaluation, BLEU, METEOR 1. INTRODUCTION The increasing interconnectedness of the globalised world has created an unprecedented demand for seamless cross-lingual communication. As Sajo et al. (71) emphasise, translation is vital for effective communication across diverse cultures. This surge in cross-lingual interaction has propelled Machine Translation (MT) to the forefront of technological innovation. MT plays a crucial role in bridging communication gaps from facilitating international business transactions and fostering cross-cultural understanding to providing access to information and knowledge across linguistic barriers. Its applications are diverse, ranging from instant translation of web pages and social media content to powering multilingual customer support systems and enabling real-time interpretation in international conferences. Consequently, the development and refinement of robust and accurate MT systems have become paramount, driving significant research and investment in the field. This is further evidenced by the development of AI-powered