SIViP (2014) 8:1613–1624
DOI 10.1007/s11760-012-0401-6
ORIGINAL PAPER
Sparsity aware consistent and high precision variable selection
T. Yousefi Rezaii · M. A. Tinati · S. Beheshti
Received: 19 April 2012 / Revised: 8 November 2012 / Accepted: 11 November 2012 / Published online: 9 December 2012
© Springer-Verlag London 2012
Abstract Variable selection is fundamental while dealing
with sparse signals that contain only a few number of
nonzero elements. This is the case in many signal process-
ing areas extending from high-dimensional statistical mod-
eling to sparse signal estimation. This paper explores a new
and efficient approach to model a system with underlying
sparse parameters. The idea is to get the noisy observations
and estimate the minimum number of underlying parameters
with acceptable estimation accuracy. The main challenge is
due to the non-convex optimization problem to be solved.
The reconstruction stage deals with some suitable objective
function in order to estimate the original sparse signal by per-
forming variable selection procedure. This paper introduces a
suitable objective function in order to simultaneously recover
the true support of the underlying sparse signal while still
achieving an acceptable estimation error. It is shown that the
proposed method performs the best variable selection com-
pared to the other algorithms, while approaching the lowest
least mean squared error in almost all the cases.
Keywords Sparse signal reconstruction · Lasso · Variable
selection · Estimation
T. Y. Rezaii (B ) · M. A. Tinati
Faculty of Electrical and Computer Engineering, University
of Tabriz, 29 Bahman Ave., 51666-15813 Tabriz, Iran
e-mail: yousefi@tabrizu.ac.ir
T. Y. Rezaii · S. Beheshti
Department of Electrical and Computer Engineering,
Ryerson University, Toronto, ON M5B 2K3, Canada
1 Introduction
The fact that many of the signals in nature are sparse or could
be modeled via limited degrees of freedom is the motiva-
tion to do variable selection in statistical modeling applica-
tions like high-dimensional and compressed sensing signal
processing [1, 2, 24, 26].
The main challenge in variable selection is the non-convex
optimization problem needed to be solved in order to recon-
struct the underlying sparse parameter vector.
Suppose the sparse vector θ ∈ R
d
, ‖
θ
‖
0
= K ≪ d is to
be estimated via the following linear regression model ( ‖.‖
0
indicates the number of nonzero elements, and although
it is not a norm, it is considered as L
0
norm in some
literature),
y
i
= x
T
i
θ + v
i
, i = 1, 2,..., n, (1)
where y
i
, x
i
∈ R
d
and v
i
are the observation, regres-
sor and observation noise at time index i , respectively.
v
i
is assumed to be additive white Gaussian noise with
a mean of 0 and variance of σ
2
. In the matrix form,
the linear regression model (1) would be of the following
form:
y = Xθ + v, (2)
where y = [ y
1
,..., y
n
]
T
, X =
x
T
1
,..., x
T
n
T
and v =
[v
1
,...,v
n
]
T
are the observation vector, regression matrix
and noise vector, respectively.
There are two main concerns in sparsity aware parame-
ter vector reconstruction. First, the estimator has to correctly
select the active variables in the model which is called vari-
able selection in the literature. Second, it has to achieve
high prediction accuracy based on the known support of
the signal. The variable selection procedure is necessary for
123