ISYE 6501 Comprehensive Midterm 2 Notes:
Week 8 Variable Selection:
- Important to limit the number of factors in the model for 2 reasons:
o Overfitting – When the number of factors is close to or larger than the
...
ISYE 6501 Comprehensive Midterm 2 Notes:
Week 8 Variable Selection:
- Important to limit the number of factors in the model for 2 reasons:
o Overfitting – When the number of factors is close to or larger than the number of data points the model might fit too closely to random effects
o Simplicity – on aggregate simple models are better than complex ones. Using less factors means that less data is required and the is a smaller chance of including insignificant factors. Interpretability is also crucial. Some factors are even illegal to use such as race and gender in addition to factors that are also predictive of these attributes.
- Forward Selection: A method of variable selection method where we start with a model containing no factors. At each step individual step, we find the best new factor to add to the model via iteration. When there is no longer another factor that meets quality thresholds, or we reach a max number of factors then we stop iterating and arrive at the final model.
- Backward Elimination: This process is the opposite of forward selection as we start with a full model where at each step, we remove insignificant variables until we arrive at a satisfying model.
- Stepwise Regression: Combination of both forward selection and backward elimination. There are two types backwards which starts with a full model or forward which starts with the null model. Then implements a hybrid approach of the two adding and selecting variables iteratively to return a satisfying model.
- Each of the stepwise approaches are known as greedy algorithms as each decision is made at each step with only enough consideration for the immediate result of the step and not the global state or future steps. At each step takes the one thing that looks like the immediate best decision. Future options are not considered.
- Lasso Approach: A more modern optimized approach to variable selection using global optimization. Add a constraint to the standard regression equation which sets a budget on the sum of the models’ coefficients. This constraint in effect limits the size of coefficients thus making our model a lot more of this coefficient size budget to the most important coefficients / variables. All non-important variables will be allotted zero in the coefficient budget which thus leaves them out of the new selection. Since we are implementing a global coefficient budget it is important that we use scaled data as the budget needs to treat the scale of variables the same otherwise magnitude of variables would impact the models budget allotment.
o Min ∑ni=1 (yi – (a0 + a1x1i + a2x2i + … + aixji))2
o S.t. ∑ji=1 |ai| ≤ T
- The lasso approach requires the tuning parameter T of the model to decide the size and quality of variables.
- Elastic regression: takes the general same approach as lasso regression however, instead of just constraining just the absolute value of the coefficients, we constrain a combination of the absolute values of the coefficients and their squares. This is the hybrid of ridge and lasso regression which brings with it the advantages of both as well as the bias disadvantages of both.
o Min ∑ni=1 (yi – (a0 + a1x1i + a2x2i + … + aixji))2
[Show More]
