Heat Transfer—Asian Res. 2019;1–13. wileyonlinelibrary.com/journal/htj © 2019 Wiley Periodicals, Inc. | 1 Received: 31 July 2019 | Revised: 25 October 2019 | Accepted: 28 November 2019 DOI: 10.1002/htj.21648 ORIGINAL ARTICLE Stability analysis between two concentric rotating cylinders with heat and mass transfer Mukesh K. Awasthi | Shivam Agarwal Department of Mathematics, Babasaheb Bhimrao Ambedkar University, Lucknow, India Correspondence Mukesh K. Awasthi, Department of Mathematics, Babasaheb Bhimrao Ambedkar University, Lucknow 226025, India. Email: mukeshiitr.kumar@gmail.com Funding information University Grants Commission, Grant/Award Number: No.F.30-442/2018 Abstract The stability of the liquid/vapor interface between two concentric revolving cylinders is examined. The transfer of heat/mass is allowed at the interface. Both the cylinders rotate with different angular velocities. The fluids inside the annular region are taken as incompres- sible and viscous. The theory of viscous potential flow analysis is used to add the viscous effects. The normal mode technique is used to calculate the growth of perturbations. If we rotate the inner cylinder, it is seen that asymmetric disturbances have a destabilizing character at the interface but the rotation of the outer cylinder has a stabilizing effect. We found that an asymmetric disturbance destabilizes the interface if the inner cylinder rotates. It is found that the arrangement gets destabilized on rotating of the inner cylinder but rotation of the outer cylinder induces stability, and the most stable case is found when the inner cylinder is stationary and the outer cylinder is rotating. KEYWORDS concentric cylinders, heat transfer, interfacial stability, mass transfer, rotation 1 | INTRODUCTION The study of the phenomenon of heat/mass transport in multilayer flow has made a lot of progress in recent years. This phenomenon is widely used in boiling heat transport in chemical and geophysical engineering. The mathematical framework of this phenomenon was made by Hsieh 1 in 1972, in which he developed the equations of heat/mass transfer through potential flow. To know both the linear and nonlinear stability, Hsieh 2 developed a mathematical