# 11.1: Introduction to Variable Selection
+ factorbased models: classification, clustering, regression
+ why limit number of factors?
1. avoid overfitting: especially problematic when # of factors
is => # of data p
...
# 11.1: Introduction to Variable Selection
+ factorbased models: classification, clustering, regression
+ why limit number of factors?
1. avoid overfitting: especially problematic when # of factors
is => # of data points; cauases model to fit to random effects
2. simplicity:
+ also implies less data collection is required
+ reduces chances of identifying insignificant factors
+ easier to interpret and communicate results (i.e. for
prescriptive analysis)
+ sometimes factors can be illegal to use (e.g. race,
gender, religion, etc.), so must be careful not to use
corresponding highly correlated factors; also, it can be
difficult to prove that an overly complex model avoids using
these factors
+ check: simpler models with fewer factors avoids (1) overfitting
and (2) difficulty of interpretation
(not necessarily low prediction quality nor bias in the most
important factors)
# 11.2: Models for Variable Selection
+ forward selection, backward elimination, and stepwise
regression
+ "greedy": takes action without considering future options
+ criteria other than pvalue can be used to determine factor
importance (e.g. R^2, AIC, BIC)
+ lasso:
+ adds constraint to standard regression optimization
objective of minimizing sum of scquared errors by adding a
"budget" tau to use for sum of coefficients (i.e. adds constraint
to minimize size of coefficients, which is similar to SVM)
+ data must be scaled beforehand
+ some coefficients may be calculated to be 0
+ how to choose tau? try different values and
+ consider number of variables
+ consider quality of model
+ ridge regression:
+ similar to lasso, but constrains on a combination of the
absolute value of the coefficients and their squares
+ same issues regarding scaling and choosing parameters (in
this case, tau and lambda)
+ ridge regression:
[Show More]