ISIYE 6501 Step 2: Midterm Quiz 1 - GT Students (Launch Proctortrack first before taking the Midterm Quiz 1)

Course Midter… Step 2: … Step 2: ISIYE 6501 Step 2: Midterm Quiz 1 - GT Students (Launch Proctortrack first before taking the Midterm Quiz 1) 95 Minute Time Limit Instructions Work alone. Do not collaborate with ...

Course Midter… Step 2: … Step 2: ISIYE 6501 Step 2: Midterm Quiz 1 - GT Students (Launch Proctortrack first before taking the Midterm Quiz 1) 95 Minute Time Limit Instructions Work alone. Do not collaborate with or copy from anyone else. You may use any of the following resources: One sheet (both sides) of handwritten (not photocopied or scanned) notes If any question seems ambiguous, use the most reasonable interpretation (i.e. don't be like Calvin): Good Luck! Question 0 -- Practice with Drag & Drop 0 points possible (ungraded) Keyboard Help Some of the quiz questions are Drag-and-Drop. You'll need to drag one or more answers to a location. Some answers might not be used at all, and some answers will be used once. To get full credit you might need to drag more than one answer to some locations, just one answer to other locations, and some locations might not have any correct answers. Please do this quick practice question. The question will give you feedback to make sure you've done it correctly, but the real quiz questions will not. You have used 6 of 10 attempts. Reset Show Answer FEEDBACK Correctly placed 3 items. Good work! You have completed this drag and drop problem. Note that: (1) There are two places you could've put (x=2,y=3); either one would be correct. (2) One location (x+y=2) had nothing dragged to it. Another location had two answers dragged to it. (3) One choice (x=1,y=7) was not dragged anywhere, since it wasn't correct for anything. Question 1 9/13 points (graded) Keyboard Help Drag each model or method to a category of question it is commonly used for. For models/methods that have more than one correct category, choose any one correct category; for models/methods that have no correct category listed, do not drag them. You have used 1 of 1 attempts. Reset Show Answer FEEDBACK Correctly placed 8 items. Misplaced 1 item. Did not place 3 required items. Good work! You have completed this drag and drop problem. Final attempt was used, highest score is 9.0 Question 2 2.19/3.0 points (graded) Select all of the following models that are designed for use with time series data: k-nearest-neighbor Principal component analysis ARIMA k-means CUSUM Logistic regression GARCH Exponential smoothing Random forest Linear regression Support vector machine You have used 1 of 1 attempt Answers are displayed within the problem Information for Questions 3a, 3b, 3c Figures A and B show the training data for a soft classification problem, using two predictors (x1 and x2 ) to separate between black and white points. The dashed lines are the classifiers found using SVM. Figure A uses a linear kernel, and Figure B uses a nonlinear kernel that required fitting 16 parameter values. Figure A Figure B Question 3a 2.4/3.0 points (graded) 3a. Select all of the following statements that are true. Figure A's classifier is based only on the value of x2 . Figure A has fewer classification errors in the training data. Figure A's classifier has a wider margin in the training data. Figure A's classifier incorrectly classifies exactly 4 white points in the training data. Figure A shows that the black point (7.2,1.4) is an outlier. You have used 1 of 1 attempt Answers are displayed within the problem Question 3b 2.25/3.0 points (graded) 3b. Select all of the following statements that are true. Figure B's classifier has a narrower margin in the training data. Figure B's classifier is more likely to be over-fit. Figure B's classifier incorrectly classifies exactly 5 white points in the training data. Figure B shows that the black point (7.2,1.4) should be white. You have used 1 of 1 attempt Answers are displayed within the problem Question 3c 1.5/3.0 points (graded) 3c. Select all of the following statements that are true. A new point at (3,3) would be classified as white by Figure A's classifier. A new point at (3,3) would be classified as white by Figure B's classifier. A new point at (3,3) would be classified as white by a k-nearest-neighbor algorithm for 5 ≤ k ≤ 10. In Figure A, if the training data had 1000 more white points to the right of the classifier, a 1000-nearest-neighbor algorithm would classify a new point at (3,3) as white. You have used 1 of 1 attempt Answers are displayed within the problem Question 3d 3.0/3.0 points (graded) In the soft classification SVM model where we select coefficients a0 ... am to minimize n m m ∑ max{0, 1 − (∑ ai xij + a0 ) yj} + C ∑ a2 j−1 i=1 i=1 3d. Select each of the following cases when we would want to decrease the value of C. We want a larger margin even if it induces more classification errors in the training set. We are willing to accept a smaller margin in order to reduce classification errors in the training set. Neither. You have used 1 of 1 attempt Answers are displayed within the problem Question 3e 0.99/3.0 points (graded) 3e. In the hard classification SVM model, it might be desirable to not put the classifier in a location that has equal margin on both sides... (select all correct answers): ...because moving the classifier will usually result in fewer classification errors in the validation data. ...because moving the classifier will usually result in fewer classification errors in the test data. ...when the costs of misclassifying the two types of points are significantly different. You have used 1 of 1 attempt Answers are displayed within the problem Information for Questions 4a, 4b, 4c Seven different regression models have been fitted, using different sets of variables. The figure below shows the resulting adjusted R-squared value for various models, as measured by cross-validation. Question 4a 0.0/3.0 points (graded) Which of the models would you expect to perform worst on a test data set? Model 6, because it has a slightly lower Adjusted R2 than Model 5 and uses one more predictor. Model 2, because it's the simplest of those with a high Adjusted R2 . Model 5, because it has the highest Adjusted R2 . Model 1, because it has much lower Adjusted R2 . You have used 1 of 1 attempt Answers are displayed within the problem Question 4b 1.5/3.0 points (graded) Under which of the following conditions would Model 3 be the most appropriate to use (select all correct answers)? Data collection for x6 is too expensive for it to be used in the model. Government regulations require using x2 for this sort of model. It is important to find the simplest good model that includes x3 . The value of x3 is not known in time for use in the model. You have used 1 of 1 attempt Answers are displayed within the problem Additional Information for Question 4c The table below shows the Akaike Information Criterion (AIC), Corrected AIC, and Bayesian Information Criterion (BIC) for each of the models. Model AIC Corrected AIC BIC 1 -5.58 -5.32 2.07 2 -5.67 -5.15 3.89 3 -6.51 -5.62 4.96 4 -4.77 -3.41 8.61 5 -2.80 -0.85 12.49 6 -1.31 1.35 15.90 7 0.19 3.71 19.31 Question 4c 0.75/3.0 points (graded) Based on the table above and the figure shown for Question 4a, select all of the following statements that are correct. Adjusted R2 (see figure above 4a) and BIC (see table above 4c) give qualitatively opposite evaluations of Model 7. Among Models 2 and 4, AIC suggests that Model 2 is e(−5.67−(−4.77))/2 = 63.8% as likely as Model 4 to be better. Among Models 2 and 4, AIC suggests that Model 4 is e(−5.67−(−4.77))/2 = 63.8% as likely as Model 2 to be better. BIC suggests that Model 7 is very likely to be better than Model 5. You have used 1 of 1 attempt Answers are displayed within the problem Information for all parts of Question 5 Atlanta’s main library has collected the following day-by-day data over the past six years (more than 2000 data points): x1 = Number of books borrowed from the library on that day x2 = Day of the week x3 = Temperature x4 = Amount of rainfall x5 = Whether the library was closed that day x6 = Whether public schools were open that day Question 5a 2.0/2.0 points (graded) Select all data that are categorical (including binary data): Number of books borrowed from the library on that day Day of the week Temperature Amount of rainfall Whether the library was closed that day Whether public schools were open that day You have used 1 of 1 attempt Answers are displayed within the problem Questions 5b and 5c 0.0/4.0 points (graded) The library believes that if it was hotter yesterday, fewer books will be borrowed today (and if it was cooler yesterday, more books will be borrowed today), so they add a new predictor: x7 = temperature the day before b. If the library is correct that on average, if it was hotter yesterday, fewer books will be borrowed today (and if it was cooler yesterday, more books will be borrowed today), what sign (positive or negative) would you expect the new predictor's coefficient a7 to have? Positive, because the response (books borrowed today) is a positive number Negative, because higher values of x7 decrease the response (books borrowed today) Positive, because higher values of x7 increase the response (books borrowed today) c. Does x7 make the model autoregressive? No, because the model does not use previous response data to predict the day t response. Yes, because the model uses day t − 1 data to predict day t circulation. Yes, because the model uses both day t − 1 and day t temperature data as predictors. You have used 1 of 1 attempt Answers are displayed within the problem Information for Question 5d The library believes that as the temperature gets either too cold or too hot, more people come indoors to the library to borrow books. They have fit the data to a quadratic function (see the figure below). Question 5d 0.0/4.0 points (graded) How would you incorporate the new information above into the library's regression model? Add a (temperature)2 variable to the model. Replace the temperature variable with a (temperature)2 variable in the model. Change the model to estimate the square root of the books borrowed, as a function of temperature, day of the week, inches of rainfall, whether the day is a holiday, and whether schools were open. You have used 1 of 1 attempt Answers are displayed within the problem Question 5e-i,ii 6.0/6.0 points (graded) The library has built a triple exponential smoothing (Holt-Winters) model of the number of books borrowed each day, using a multiplicative annual cycle of seasonality. i. Every Wednesday, local schools bring children to visit the library and check out books. So, the number of books borrowed on those days is much higher than an average day. The model only has an annual seasonal cycle length, not a weekly one. Is the model likely to over-predict or under-predict books borrowed on Wednesdays? Over-predict Under-predict Neither ii. Is the model likely to over-predict or under-predict books borrowed on Thursdays, when it is open for a full business day? Over-predict Under-predict Neither You have used 1 of 1 attempt Answers are displayed within the problem Question 5e-iii 3.0/3.0 points (graded) iii. Aside from seasonal and trend effects, the library believes that the random variation in books borrowed each day is small. Should they expect the best value of α (the baseline smoothing constant) to be: α < 0 0 < α < 1 2 1 < α < 1 2 α > 1 You have used 1 of 1 attempt Answers are displayed within the problem Information for Questions 5f, 5g, 5h The library would like to compare the regression and exponential smoothing models to determine which is a better predictor, using the mean absolute error |(books borrowed) – (model’s estimate)|/n as a measure of prediction quality. Question 5f 0.0/4.0 points (graded) Select the best of the following four options for splitting the data: 70% for training, 15% for validation, 15% for test 15% for training, 70% for validation, 15% for test 15% for training, 15% for validation, 70% for test 55% for training, 15% for cross-validation, 15% for validation, 15% for test You have used 1 of 1 attempt Answers are displayed within the problem Question 5g 4/4 points (graded) Keyboard Help Match each data set with its purpose. Drag the purpose next to the appropriate data set. You have used 1 of 1 attempts. Reset Show Answer FEEDBACK Correctly placed 3 items. Good work! You have completed this drag and drop problem. Final attempt was used, highest score is 4.0 Question 5h 3.0/4.0 points (graded) The person who built these models discovered that although the regression model performed much better on the training set, the two models performed about the same on the validation set: Mean absolute error (training set) Mean absolute error (validation set) Regression model 117 152 Exponential smoothing model 148 153 Select all of the reasonable suggestions below: To choose between the models, we should see which one does better on the training set. The regression model is clearly better, because it does better on the training set and about the same on the validation set. The exponential smoothing model is probably fit too much to random patterns (i.e., it is overfit), because it performs much worse than the regression model on the training set. If there had been 20 models, the one that performed best on the validation set would probably not perform as well on the test set as it did on the validation set. You have used 1 of 1 attempt Answers are displayed within the problem Question 5i 2.01/3.0 points (graded) Fewer books are borrowed on Fridays than any other day. The library would like to determine whether there has been a change in the Friday effect on borrowing, over the past forty years (for this part only, assume there are forty years of data available). Select all of the approaches that might reasonably be correct. Use CUSUM on the number of additional books borrowed on the average Friday compared to the average other day over the past forty years. Use exponential smoothing (with L = 7) to find the seasonal mulitplier values Ct for each Friday, and use CUSUM on those values. Build a regression model for each of the forty years, and use CUSUM on the coefficients of the Friday variable. You have used 1 of 1 attempt Answers are displayed within the problem Information for Questions 6a, 6b A logistic regression model was built to model the probability that a retailer’s inventory of a popular product will run out before the next delivery from the manufacturer, based on a number of factors (amount of current inventory, past demand, promotions, etc.). If the logistic regression’s output is greater than a threshold value p, the retailer pays an additional amount D for a quick delivery to avoid running out. There are three confusion matrices below, for three different threshold values of p: Question 6a 0.0/3.0 points (graded) Let D be the cost of paying for a quick delivery (if the model's output is above p). Let C be the cost of running out of inventory. Select all of the statements that are correct: When p=0.7, the total cost is (53D + 47C + 8D). When p=0.7, the total cost is (53D + 47D + 8C). The total cost when p=0.7 must be higher than when p=0.5 and p=0.3. You have used 1 of 1 attempt Answers are displayed within the problem Question 6b 3.0/3.0 points (graded) The retailer’s primary goal right now is to build its market share, so it estimates the cost C of running out to be 20 times worse than the cost D of paying for an early delivery (i.e., C = 20D). Which threshold value of p would you suggest? p = 0.3 p = 0.5 p = 0.7 You have used 1 of 1 attempt Answers are displayed within the problem Question 7 8/8 points (graded) Keyboard Help The figures below each show a data set that will be used in k-means clustering algorithms (where distance between values is important). Each data set has two attributes. For each data set, drag to it the data preparations that are needed for k-means to work well on the data set. You have used 1 of 1 attempts. Reset Show Answer FEEDBACK Correctly placed 4 items. Good work! You have completed this drag and drop problem. Final attempt was used, highest score is 8.0 Information for Questions 8a, 8b A regression tree approach was used to describe the effect of 7 different covariates (x1 through x7) on monthly sales. The tree is shown below. In each model, only the significant covariates are shown. Question 8a 2.0/5.0 points (graded) Select all of the following statements that are true according to this regression tree: The effect of x7 is the same regardless of the values of x1 and x2 . x1 is only relevant when its value is small (less than 3.5). x4 is irrelevant when predicting monthly sales. The regression tree can predict monthly sales better for values of x1 greater than or equal to 3.5 than for values of x1 less than 3.5. The effect of x3 on sales is greatest when x1 is small and x2 is large (x1 < 3.5 and x2 > 2.0). You have used 1 of 1 attempt Answers are displayed within the problem Question 8b 2.01/3.0 points (graded) A random forest model was built for the same purpose, using the same 7 covariates. Which of the following statements are true? The random forest model uses many trees, but returns a single tree solution that can be analyzed. The random forest model uses a single tree solution. The random forest model can report the relative importance of each variable. You have used 1 of 1 attempt Answers are displayed within the problem Information for Question 8c A data scientist has run principal component analysis on the 7 covariates, with the following results: Component Eigenvalue 1 2.20 2 0.12 3 0.10 4 0.09 5 0.08 6 0.06 7 0.05 Question 8c 2.0/4.0 points (graded) Select all of the following statements that are correct: It is likely that the first principal component has much more predictive power than principal components 2-7. It is likely that the first original covariate has much more predictive power than covariates 2-7. It is likely that the last original covariate has much less predictive power than covariates 1-6. The first principal component cannot contain information from all 7 original covariates. You have used 1 of 1 attempt Answers are displayed within the problem