Int. J. Computational Science and Engineering, Vol. 20, No. 4, 2019 501 Copyright © 2019 Inderscience Enterprises Ltd. CUDA GPU libraries and novel sparse matrix-vector multiplication – implementation and performance enhancement in unstructured finite element computations Richard Haney The MITRE Corporation, 7515 Colshire Drive, McLean, Virginia, 22102-7539, USA Email: rhaney@mitre.org Ram Mohan* North Carolina A&T State University, 2907 East Gate City Blvd., Greensboro, North Carolina, 27401, USA Email: rvmohan@ncat.edu *Corresponding author Abstract: The efficient solution to systems of linear and nonlinear equations arising from sparse matrix operations is a ubiquitous challenge for computing applications that can be exacerbated by the employment of heterogeneous architectures such as CPU-GPU computing systems. This paper presents our implementation of a novel sparse matrix-vector multiplication (a significant compute load operation in the iterative solution via pre-conditioned conjugate gradient based methods) employing LightSpMV with compressed sparse row (CSR) format, and the resulting performance characteristics using an unstructured finite element-based computational simulation. Computational performance analysed indicates that LightSpMV can provide an asset to boost performance for these computational modelling applications. This work also investigates potential improvements in the LightSpMV algorithm using CUDA 35 intrinsic, which results in an additional performance boost by 1%. While this may not be significant, it supports the idea that LightSpMV can potentially be used for other full-solution finite element-based computational implementations. Keywords: general purpose GPU computing; GPGPU; sparse matrix-vector; finite element method; FEM; Compute Unified Device Architecture; CUDA; performance analysis. Reference to this paper should be made as follows: Haney, R. and Mohan, R. (2019) ‘CUDA GPU libraries and novel sparse matrix-vector multiplication – implementation and performance enhancement in unstructured finite element computations’, Int. J. Computational Science and Engineering, Vol. 20, No. 4, pp.501–507. Biographical notes: Richard Haney received his BS in Computer Science and PhD in Computational Science and Engineering with a focus on GPGPU and High Performance Computing in 2013. Currently, he is a Senior Computer Scientist at The MITRE Corporation J84B Simulation Engineering division. His research interests include distributed high performance computing and GPGPU. Ram Mohan is currently a Professor of Nanoengineering with Department of Nanoengineering in Joint School of Nanoscience and Nanoengineering at North Carolina A&T State University. He leads the computational nanoengineering focus area with research interests in physics-based modelling and enabling scalable, high performance computational modelling developments, performance, and their applications. 1 Introduction As heterogeneous architectures become the standard paradigm for high performance computing (HPC), interest in efficient sparse matrix-vector multiplication using graphics processing units (GPU) has increased. The sparse matrix-vector multiplication is the largest cost of many physics-based modelling solutions that utilise finite element (FE) or finite difference (FD) discretisation. These methods result in a need to solve a system of linear equations using iterative methods such as conjugate gradient (CG) or other