Math.Comput.Sci. https://doi.org/10.1007/s11786-018-0375-z Mathematics in Computer Science Computer Algebra Challenges in Nanotechnology: Accurate Modeling of Nanoscale Electro-optic Devices Using Finite Elements Method Avi Karsenty · Yaakov Mandelbaum Received: 19 September 2017 / Revised: 11 May 2018 / Accepted: 6 June 2018 © Springer Nature Switzerland AG 2018 Abstract The simulation of silicon-based light-emitting and photodetectors nanodevices using computer algebra became a challenge. These devices couple the hyperbolic equations of electromagnetic radiation, the parabolic equations of heat conduction, the elliptic equations describing electric potential, and the eigenvalue equations of quantum mechanics—with the nonlinear drift–diffusion equations of the semiconductor physics. These complex equations must be solved by using generally mixed Dirichlet–Neumann boundary conditions in three-dimensional geometries. Comsol Multiphysics modeling software is employed integrated with MATLAB–SIMULINK and Zemax. The physical equations are discretized on a mesh using the Galerkin finite element method (FEM) and to a lesser extent the method of finite volumes. The equations can be implemented in a variety of forms such as directly as a partial differential equation, or as a variational integral, the so-called weak form. Boundary conditions may also be imposed directly or using variational constraint and reaction forces. Both choices have implication for convergence and physicality of the solution. The mesh is assembled from triangular or quadrilateral elements in two-dimensions, and hexahedral or prismatic elements in three dimensions, using a variety of algorithms. Solution is achieved using direct or iterative linear solvers and nonlinear solvers. The former are based on conjugate gradients, the latter generally on Newton–Raphson iterations. The general framework of FEM discretization, meshing and solver algorithms will be presented together with techniques for dealing with challenges such as multiple time scales, shocks and nonconvergence; these include load ramping, segregated iterations, and adaptive meshing. Keywords Finite element method (FEM) · Finite volumes (FVM) · Partial differential equation (PDE) · Computer aided design (CAD) · Nanoscale body devices (NSB) · Simulations · Nanotechnology Mathematics Subject Classification 81V80 · 76M10 · 65M60 A. Karsenty (B ) · Y. Mandelbaum Department of Applied Physics/Electro-Optics Engineering, Advanced Lab for Electro-Optics Simulations (ALEO), Lev Academic Center, Jerusalem 9116001, Israel e-mail: karsenty@jct.ac.il https://www.aleo.solutions/ Y. Mandelbaum e-mail: yaaqovm@gmail.com