To appear as a part of an upcoming textbook on dimensionality reduction and manifold learning. Locally Linear Embedding and its Variants: Tutorial and Survey Benyamin Ghojogh BGHOJOGH@UWATERLOO. CA Department of Electrical and Computer Engineering, Machine Learning Laboratory, University of Waterloo, Waterloo, ON, Canada Ali Ghodsi ALI . GHODSI @UWATERLOO. CA Department of Statistics and Actuarial Science & David R. Cheriton School of Computer Science, Data Analytics Laboratory, University of Waterloo, Waterloo, ON, Canada Fakhri Karray KARRAY@UWATERLOO. CA Department of Electrical and Computer Engineering, Centre for Pattern Analysis and Machine Intelligence, University of Waterloo, Waterloo, ON, Canada Mark Crowley MCROWLEY@UWATERLOO. CA Department of Electrical and Computer Engineering, Machine Learning Laboratory, University of Waterloo, Waterloo, ON, Canada Abstract This is a tutorial and survey paper for Locally Linear Embedding (LLE) and its variants. The idea of LLE is fitting the local structure of man- ifold in the embedding space. In this paper, we first cover LLE, kernel LLE, inverse LLE, and feature fusion with LLE. Then, we cover out- of-sample embedding using linear reconstruc- tion, eigenfunctions, and kernel mapping. Incre- mental LLE is explained for embedding stream- ing data. Landmark LLE methods using the Nystrom approximation and locally linear land- marks are explained for big data embedding. We introduce the methods for parameter selec- tion of number of neighbors using residual vari- ance, Procrustes statistics, preservation neigh- borhood error, and local neighborhood selection. Afterwards, Supervised LLE (SLLE), enhanced SLLE, SLLE projection, probabilistic SLLE, su- pervised guided LLE (using Hilbert-Schmidt in- dependence criterion), and semi-supervised LLE are explained for supervised and semi-supervised embedding. Robust LLE methods using least squares problem and penalty functions are also introduced for embedding in the presence of out- liers and noise. Then, we introduce fusion of LLE with other manifold learning methods in- cluding Isomap (i.e., ISOLLE), principal compo- nent analysis, Fisher discriminant analysis, dis- criminant LLE, and Isotop. Finally, we explain weighted LLE in which the distances, recon- struction weights, or the embeddings are adjusted for better embedding; we cover weighted LLE for deformed distributed data, weighted LLE us- ing probability of occurrence, SLLE by adjusting weights, modified LLE, and iterative LLE. 1. Introduction Locally Linear Embedding (LLE) (Roweis & Saul, 2000; Chen & Liu, 2011) is a nonlinear spectral dimensionality reduction method (Saul et al., 2006) which can be used for manifold embedding and feature extraction (Ghojogh et al., 2019e). LLE tries to preserve the local structure of data in the embedding space. In other words, the close points in the high-dimensional input space should also be close to each other in the low-dimensional embedding space. By this lo- cal fitting, hopefully the far points in the input space also fall far away from each other in the embedding space. This idea of fitting locally and thinking globally is the main idea of LLE (Saul & Roweis, 2002; 2003; Yotov et al., 2005; Wu et al., 2018). In another perspective, the idea of local fitting by LLE is similar to idea of piece-wise spline regres- sion (Marsh & Cormier, 2001). LLE unfolds the nonlinear manifold by locally unfolding of manifold piece by piece and it hopes that these local unfoldings result in a suitable total manifold unfolding (see Fig. 1). In general, we can say that most of the unsupervised manifold learning meth- arXiv:2011.10925v1 [stat.ML] 22 Nov 2020