Question 10.1
Using the same crime data set uscrime.txt as in Questions 8.2 and 9.1, find the best model you
can using
(a) a regression tree model, and
(b) a random forest model.
In R, you can use the tree package o
...
Question 10.1
Using the same crime data set uscrime.txt as in Questions 8.2 and 9.1, find the best model you
can using
(a) a regression tree model, and
(b) a random forest model.
In R, you can use the tree package or the rpart package, and the randomForest package. For
each model, describe one or two qualitative takeaways you get from analyzing the results (i.e., don’t
just stop when you have a good model, but interpret it too).
regression tree model
As by now we know that the dataset contains only 47 points, for the regression tree model it might be
hard to produce many splits or it might end up overfitting and we won’t be able to say for sure that the
model would work as effectively with a large dataset. For this classification tree, I did not split the data
in training and validation, rather used all the datapoints to create the model. The initial model used
"Po1" "Pop" "LF" "NW" , the Residual mean deviance was 47390. This tree had 7 terminal nodes and
looked as below –
In the next step I pruned this tree with 6 , 4, 4,3 and 2 leaf nodes to look at the residual mean deviances,
which kept increasing as I dropped a node. It might seem like leaf nodes = 7 is the best fit model, but
because of a very small sample set this is overfitted. To solve this issue I chose to apply cross validation.
cv.tree is shows a cross-validated version of the model. Instead of computing the deviance on the full
training data, it uses cross-validated values for each of the 6 successive prunings. We can compare theISYE 6501 Week 7 HW
deviance in the outputs of just using prune.tree with the cross validated deviance and see that the crossvalidated values are rather higher at every step. Just using prune.tree tests on the training data and so
under-reports the deviance. The cv values are more realistic.
My random cross validation revealed that even for leafnode = 6 the RMSE is very close to that of 7. So I
chose to prune the tree with 6 leaf nodes and then calculated the R2 of both unpruned and pruned
models which happened to be very close to each other, withing .72 - .7 range.
If the cross validation sampling were done differently, we could get minimum RMSE for some # of leaf
nodes, and similarly the regression tree model with “limited” training data may become overfitted.
Takeaway –
The model shows that po1 is the first variable on which the first split happens and possibly LF is least
important one as in the prunes tree this gets dropped first. It also shows that NW is probably more
important the Pop as in the same brunch, pruning removed Pop. But kept NW.
random forest model
For deciding the NodeSize and mtry of the random forest model I created a loop for node size 2 to 15
and mtry values between 1 to 10 and charted their R square values to find the optimal numbers and
found that mtry=3 and NOdeSize = 3 gave the highrest R sqr = 0.4551208
I applied these values to create the model and
Looked at the importance of the variables in the model.ISYE 6501 Week 7 HW
Takeaway –
The random forest used more number of variables as compared to the regression tree, but did not
produce better R sqr values. Possibly it’s because we don’t have enough sample of data for using this
method and most of the trees were very similar to each other. From the charts we can see that it seems
like increased the number of variables used in ‘sampling and split’ is actually decreasing the accuracy of
this model.
Question 10.2
Describe a situation or problem from your job, everyday life, current events, etc., for which a logistic
regression model would be appropriate. List some (up to 5) predictors that you might use.
While sending out targeted emails with offers, our marketing team at a leading automotive company
would do a logistic regression modelling to determine the types of email flyers offers certain groups of
customers would enact to. The predictors that could be used are – Customer age group, Types od Car
they own, age of car, frequency of services availed at dealership, past offer redemption types etc. Based
on these customer segmentations and created and the emails are formatted accordingly through sales
force. Once the recipients click through them and we get back the sales and service data from
dealerships, they constitute back to the model for further adjustments.
Question 10.3
1. Using the GermanCredit data set germancredit.txt from
http://archive.ics.uci.edu/ml/machine-learning-databases/statlog/german / (description at
http://archive.ics.uci.edu/ml/datasets/Statlog+%28German+Credit+Data%29 ), use logistic
regression to find a good predictive model for whether credit applicants are good credit risks
or not. Show your model (factors used and their coefficients), the software output, and the
quality of fit. You can use the glm function in R. To get a logistic regression (logit) model on
data where the response is either zero or one, use family=binomial(link=”logit”)
in your glm function call.
2. Because the model gives a result between 0 and 1, it requires setting a threshold probability
to separate between “good” and “bad” answers. In this data set, they estimate that
incorrectly identifying a bad customer as good, is 5 times worse than incorrectly classifying a
good customer as bad. Determine a good threshold probability based on your model.
After loading the data in a table, the first thing I did is convert the output column values from 1
to 0 and 2 to 1 as the response is expected to be 0 or 1 as in yes or no.
Then I split the data in 70% training and 30% validation set and created the model using the
training set.ISYE 6501 Week 7 HW
Looking at the logistic regression model coefficients, I picked the significant predictors with p
value < = .01. As there were multiple categorical values under one column, but only few of the
combinations were significant, by looking as the coefficients, I also manually removed the nonsignificant coefficients
Then created another model using the significant ones from the first model.
This model showed I could still remove some non-significant coefficients and by applying the
same logic I came up with a model of variables V1A14, V4A41, V8, V2, V4A43, V6A65. In the
next step I applied this model on my validation data after transforming it the way I did it for the
training set.
I created the confusion matrix by rounding up the predictions, i.e. using a threshold of .5.
Observed
Predicted 0 1
0 190 64
1 22 24
Although this model correctly classified a big chunk of good borrowers, but it misclassifies a lot of
bad borrowers too. Now that cost of not being able to identify bad borrowers is 5 times more than
not identifying good borrowers, which brings us to the idea of increasing that threshold for our
classification. Hence, I tested with threshold = .7
Observed
Predicted 0 1
0 209 78
1 3 10
Tested with threshold = .9
Observed
Predicted 0 1
0 212 87
1 0 1
I calculated the specificity and accuracy of the model with thresholds of .5, .7 and .9 and found
although the accuracy with the threshold .7 did not increase much from threshold .5, for .9 it has
gone down. Whereas the specificity has gone to .98 from .89 to 1, I would prefer to stick to .7 as .9
might be very stringent and beyond this dataset, might not be a good choice with no a great model
accuracy
[Show More]
