ISYE 6501 Midterm 2 Notes Top Notch

Week 8 Variable Selection: - Important to limit the number of factors in the model for 2 reasons: o Overfitting – When the number of factors is close to or larger than the number of data points th e model might fit too c ...

Week 8 Variable Selection: - Important to limit the number of factors in the model for 2 reasons: o Overfitting – When the number of factors is close to or larger than the number of data points th e model might fit too closely to random effects o Simplicity – on aggregate simple models are better than complex ones. Using less factors means that less data is required and the is a smaller chance of including insignificant factors. Interpretability is also crucial. Some factors are even illegal to use such as race and gender in addition to factors that are also predictive of these attributes. - Forward Selection: A method of variable selection method where we start with a model containing no factors. At each step individual step, we find the best new factor to add to the model via iteration. When there is no longer another factor that meets quality thresholds, or we reach a max number of factors then we stop iterating and arrive at the final model. - Backward Elimination: This process is the opposite of forward selection as we start with a full model where at each step, we remove insignificant variables until we arrive at a satisfying model. - Stepwise Regression: Combination of both forward selection and backward elimination. There are two types backwards which starts with a full model or forward which starts with the null model. Then implements a hybrid approach of the two adding and selecting variables iteratively to return a satisfying model. - Each of the stepwise approaches are known as greedy algorithms as each decision is made at each step with only enough consideration for the immediate result of the step and not the global state or future steps. At each step takes the one thing that looks like the immediate best decision. Future options are not considered.