
R version 3.0.1 (2013-05-16) -- "Good Sport"
Copyright (C) 2013 The R Foundation for Statistical Computing
Platform: x86_64-apple-darwin10.8.0 (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

> ################################################################################
> ### R code from Applied Predictive Modeling (2013) by Kuhn and Johnson.
> ### Copyright 2013 Kuhn and Johnson
> ### Web Page: http://www.appliedpredictivemodeling.com
> ### Contact: Max Kuhn (mxkuhn@gmail.com) 
> ###
> ### Chapter 14 Classification Trees and Rule Based Models
> ###
> ### Required packages: AppliedPredictiveModeling, C50, caret, doMC (optional),
> ###                    gbm, lattice, partykit, pROC, randomForest, reshape2,
> ###                    rpart, RWeka
> ###
> ### Data used: The grant application data. See the file 'CreateGrantData.R'
> ###
> ### Notes: 
> ### 1) This code is provided without warranty.
> ###
> ### 2) This code should help the user reproduce the results in the
> ### text. There will be differences between this code and what is is
> ### the computing section. For example, the computing sections show
> ### how the source functions work (e.g. randomForest() or plsr()),
> ### which were not directly used when creating the book. Also, there may be 
> ### syntax differences that occur over time as packages evolve. These files 
> ### will reflect those changes.
> ###
> ### 3) In some cases, the calculations in the book were run in 
> ### parallel. The sub-processes may reset the random number seed.
> ### Your results may slightly vary.
> ###
> ################################################################################
> 
> ### NOTE: Many of the models here are computationally expensive. If
> ### this script is run as-is, the memory requirements will accumulate
> ### until it exceeds 32gb. 
> 
> ################################################################################
> ### Section 14.1 Basic Classification Trees
> 
> library(caret)
Loading required package: lattice
Loading required package: ggplot2
> 
> load("grantData.RData")
> 
> ctrl <- trainControl(method = "LGOCV",
+                      summaryFunction = twoClassSummary,
+                      classProbs = TRUE,
+                      index = list(TrainSet = pre2008),
+                      savePredictions = TRUE)
> 
> set.seed(476)
> rpartFit <- train(x = training[,fullSet], 
+                   y = training$Class,
+                   method = "rpart",
+                   tuneLength = 30,
+                   metric = "ROC",
+                   trControl = ctrl)
Loading required package: rpart
Loading required package: pROC
Loading required package: plyr
Type 'citation("pROC")' for a citation.

Attaching package: 'pROC'

The following object is masked from 'package:stats':

    cov, smooth, var

> rpartFit
CART 

8190 samples
1070 predictors
   2 classes: 'successful', 'unsuccessful' 

No pre-processing
Resampling: Repeated Train/Test Splits Estimated (1 reps, 0.75%) 

Summary of sample sizes: 6633 

Resampling results across tuning parameters:

  cp        ROC    Sens   Spec 
  0.000351  0.895  0.779  0.837
  0.000394  0.895  0.779  0.837
  0.000526  0.896  0.804  0.841
  0.000657  0.897  0.823  0.83 
  0.000789  0.897  0.793  0.839
  0.000877  0.897  0.877  0.818
  0.000894  0.897  0.877  0.818
  0.00092   0.897  0.877  0.818
  0.00105   0.898  0.881  0.806
  0.00131   0.906  0.882  0.816
  0.00145   0.91   0.844  0.848
  0.00158   0.911  0.847  0.846
  0.0021    0.912  0.811  0.862
  0.00224   0.912  0.811  0.862
  0.00237   0.912  0.811  0.862
  0.00272   0.912  0.811  0.862
  0.00276   0.912  0.811  0.862
  0.0028    0.912  0.8    0.865
  0.00289   0.912  0.8    0.865
  0.00394   0.883  0.886  0.811
  0.00421   0.875  0.858  0.81 
  0.0046    0.875  0.858  0.81 
  0.00526   0.874  0.858  0.81 
  0.00736   0.884  0.837  0.813
  0.0113    0.884  0.837  0.813
  0.021     0.871  0.947  0.727
  0.0227    0.871  0.947  0.727
  0.0465    0.85   0.944  0.735
  0.0715    0.852  0.944  0.738
  0.387     0.815  0.991  0.638

ROC was used to select the optimal model using  the largest value.
The final value used for the model was cp = 0.00289. 
> 
> library(partykit)
Loading required package: grid
> plot(as.party(rpartFit$finalModel))
> 
> rpart2008 <- merge(rpartFit$pred,  rpartFit$bestTune)
> rpartCM <- confusionMatrix(rpartFit, norm = "none")
> rpartCM
Repeated Train/Test Splits Estimated (1 reps, 0.75%) Confusion Matrix 

(entries are un-normalized counts)
 
Loading required package: class
Confusion Matrix and Statistics

              Reference
Prediction     successful unsuccessful
  successful          456          133
  unsuccessful        114          854
                                          
               Accuracy : 0.8414          
                 95% CI : (0.8223, 0.8592)
    No Information Rate : 0.6339          
    P-Value [Acc > NIR] : <2e-16          
                                          
                  Kappa : 0.6606          
 Mcnemar's Test P-Value : 0.2521          
                                          
            Sensitivity : 0.8000          
            Specificity : 0.8652          
         Pos Pred Value : 0.7742          
         Neg Pred Value : 0.8822          
             Prevalence : 0.3661          
         Detection Rate : 0.2929          
   Detection Prevalence : 0.3783          
      Balanced Accuracy : 0.8326          
                                          
       'Positive' Class : successful      
                                          

> rpartRoc <- roc(response = rpartFit$pred$obs,
+                 predictor = rpartFit$pred$successful,
+                 levels = rev(levels(rpartFit$pred$obs)))
> 
> set.seed(476)
> rpartFactorFit <- train(x = training[,factorPredictors], 
+                         y = training$Class,
+                         method = "rpart",
+                         tuneLength = 30,
+                         metric = "ROC",
+                         trControl = ctrl)
> rpartFactorFit 
CART 

8190 samples
1488 predictors
   2 classes: 'successful', 'unsuccessful' 

No pre-processing
Resampling: Repeated Train/Test Splits Estimated (1 reps, 0.75%) 

Summary of sample sizes: 6633 

Resampling results across tuning parameters:

  cp        ROC    Sens   Spec 
  0.000175  0.901  0.735  0.87 
  0.00021   0.901  0.735  0.87 
  0.000263  0.901  0.735  0.87 
  0.000368  0.891  0.761  0.864
  0.000376  0.891  0.761  0.864
  0.000394  0.891  0.761  0.864
  0.000526  0.891  0.775  0.865
  0.000657  0.895  0.795  0.866
  0.000789  0.899  0.821  0.864
  0.000877  0.899  0.821  0.864
  0.00092   0.899  0.821  0.864
  0.00105   0.897  0.825  0.856
  0.00118   0.898  0.825  0.853
  0.00131   0.894  0.837  0.847
  0.00145   0.894  0.837  0.847
  0.00184   0.902  0.825  0.855
  0.00237   0.902  0.825  0.858
  0.0025    0.903  0.821  0.866
  0.00263   0.903  0.821  0.866
  0.00289   0.91   0.812  0.872
  0.00394   0.892  0.847  0.831
  0.00539   0.892  0.847  0.831
  0.0071    0.892  0.847  0.831
  0.00763   0.901  0.847  0.831
  0.0116    0.899  0.828  0.834
  0.0146    0.899  0.828  0.834
  0.0318    0.9    0.823  0.841
  0.0652    0.867  0.865  0.779
  0.153     0.817  0.988  0.645
  0.393     0.817  0.988  0.645

ROC was used to select the optimal model using  the largest value.
The final value used for the model was cp = 0.00289. 
> plot(as.party(rpartFactorFit$finalModel))
> 
> rpartFactor2008 <- merge(rpartFactorFit$pred,  rpartFactorFit$bestTune)
> rpartFactorCM <- confusionMatrix(rpartFactorFit, norm = "none")
> rpartFactorCM
Repeated Train/Test Splits Estimated (1 reps, 0.75%) Confusion Matrix 

(entries are un-normalized counts)
 
Confusion Matrix and Statistics

              Reference
Prediction     successful unsuccessful
  successful          463          126
  unsuccessful        107          861
                                          
               Accuracy : 0.8504          
                 95% CI : (0.8317, 0.8677)
    No Information Rate : 0.6339          
    P-Value [Acc > NIR] : <2e-16          
                                          
                  Kappa : 0.6798          
 Mcnemar's Test P-Value : 0.2383          
                                          
            Sensitivity : 0.8123          
            Specificity : 0.8723          
         Pos Pred Value : 0.7861          
         Neg Pred Value : 0.8895          
             Prevalence : 0.3661          
         Detection Rate : 0.2974          
   Detection Prevalence : 0.3783          
      Balanced Accuracy : 0.8423          
                                          
       'Positive' Class : successful      
                                          

> 
> rpartFactorRoc <- roc(response = rpartFactorFit$pred$obs,
+                       predictor = rpartFactorFit$pred$successful,
+                       levels = rev(levels(rpartFactorFit$pred$obs)))
> 
> plot(rpartRoc, type = "s", print.thres = c(.5),
+      print.thres.pch = 3,
+      print.thres.pattern = "",
+      print.thres.cex = 1.2,
+      col = "red", legacy.axes = TRUE,
+      print.thres.col = "red")

Call:
roc.default(response = rpartFit$pred$obs, predictor = rpartFit$pred$successful,     levels = rev(levels(rpartFit$pred$obs)))

Data: rpartFit$pred$successful in 29610 controls (rpartFit$pred$obs unsuccessful) < 17100 cases (rpartFit$pred$obs successful).
Area under the curve: 0.8915
> plot(rpartFactorRoc,
+      type = "s",
+      add = TRUE,
+      print.thres = c(.5),
+      print.thres.pch = 16, legacy.axes = TRUE,
+      print.thres.pattern = "",
+      print.thres.cex = 1.2)

Call:
roc.default(response = rpartFactorFit$pred$obs, predictor = rpartFactorFit$pred$successful,     levels = rev(levels(rpartFactorFit$pred$obs)))

Data: rpartFactorFit$pred$successful in 29610 controls (rpartFactorFit$pred$obs unsuccessful) < 17100 cases (rpartFactorFit$pred$obs successful).
Area under the curve: 0.8856
> legend(.75, .2,
+        c("Grouped Categories", "Independent Categories"),
+        lwd = c(1, 1),
+        col = c("black", "red"),
+        pch = c(16, 3))
> 
> set.seed(476)
> j48FactorFit <- train(x = training[,factorPredictors], 
+                       y = training$Class,
+                       method = "J48",
+                       metric = "ROC",
+                       trControl = ctrl)
Loading required package: RWeka
> j48FactorFit
C4.5-like Trees 

8190 samples
1488 predictors
   2 classes: 'successful', 'unsuccessful' 

No pre-processing
Resampling: Repeated Train/Test Splits Estimated (1 reps, 0.75%) 

Summary of sample sizes: 6633 

Resampling results

  ROC    Sens   Spec 
  0.835  0.839  0.817

Tuning parameter 'C' was held constant at a value of 0.25
 
> 
> j48Factor2008 <- merge(j48FactorFit$pred,  j48FactorFit$bestTune)
> j48FactorCM <- confusionMatrix(j48FactorFit, norm = "none")
> j48FactorCM
Repeated Train/Test Splits Estimated (1 reps, 0.75%) Confusion Matrix 

(entries are un-normalized counts)
 
Confusion Matrix and Statistics

              Reference
Prediction     successful unsuccessful
  successful          478          181
  unsuccessful         92          806
                                          
               Accuracy : 0.8247          
                 95% CI : (0.8048, 0.8432)
    No Information Rate : 0.6339          
    P-Value [Acc > NIR] : < 2.2e-16       
                                          
                  Kappa : 0.6343          
 Mcnemar's Test P-Value : 1.004e-07       
                                          
            Sensitivity : 0.8386          
            Specificity : 0.8166          
         Pos Pred Value : 0.7253          
         Neg Pred Value : 0.8976          
             Prevalence : 0.3661          
         Detection Rate : 0.3070          
   Detection Prevalence : 0.4232          
      Balanced Accuracy : 0.8276          
                                          
       'Positive' Class : successful      
                                          

> 
> j48FactorRoc <- roc(response = j48FactorFit$pred$obs,
+                     predictor = j48FactorFit$pred$successful,
+                     levels = rev(levels(j48FactorFit$pred$obs)))
> 
> set.seed(476)
> j48Fit <- train(x = training[,fullSet], 
+                 y = training$Class,
+                 method = "J48",
+                 metric = "ROC",
+                 trControl = ctrl)
> 
> j482008 <- merge(j48Fit$pred,  j48Fit$bestTune)
> j48CM <- confusionMatrix(j48Fit, norm = "none")
> j48CM
Repeated Train/Test Splits Estimated (1 reps, 0.75%) Confusion Matrix 

(entries are un-normalized counts)
 
Confusion Matrix and Statistics

              Reference
Prediction     successful unsuccessful
  successful          438          160
  unsuccessful        132          827
                                          
               Accuracy : 0.8125          
                 95% CI : (0.7922, 0.8316)
    No Information Rate : 0.6339          
    P-Value [Acc > NIR] : <2e-16          
                                          
                  Kappa : 0.6001          
 Mcnemar's Test P-Value : 0.1141          
                                          
            Sensitivity : 0.7684          
            Specificity : 0.8379          
         Pos Pred Value : 0.7324          
         Neg Pred Value : 0.8624          
             Prevalence : 0.3661          
         Detection Rate : 0.2813          
   Detection Prevalence : 0.3841          
      Balanced Accuracy : 0.8032          
                                          
       'Positive' Class : successful      
                                          

> 
> j48Roc <- roc(response = j48Fit$pred$obs,
+               predictor = j48Fit$pred$successful,
+               levels = rev(levels(j48Fit$pred$obs)))
> 
> 
> plot(j48FactorRoc, type = "s", print.thres = c(.5), 
+      print.thres.pch = 16, print.thres.pattern = "", 
+      print.thres.cex = 1.2, legacy.axes = TRUE)

Call:
roc.default(response = j48FactorFit$pred$obs, predictor = j48FactorFit$pred$successful,     levels = rev(levels(j48FactorFit$pred$obs)))

Data: j48FactorFit$pred$successful in 987 controls (j48FactorFit$pred$obs unsuccessful) < 570 cases (j48FactorFit$pred$obs successful).
Area under the curve: 0.8353
> plot(j48Roc, type = "s", print.thres = c(.5), 
+      print.thres.pch = 3, print.thres.pattern = "", 
+      print.thres.cex = 1.2, legacy.axes = TRUE,
+      add = TRUE, col = "red", print.thres.col = "red")

Call:
roc.default(response = j48Fit$pred$obs, predictor = j48Fit$pred$successful,     levels = rev(levels(j48Fit$pred$obs)))

Data: j48Fit$pred$successful in 987 controls (j48Fit$pred$obs unsuccessful) < 570 cases (j48Fit$pred$obs successful).
Area under the curve: 0.842
> legend(.75, .2,
+        c("Grouped Categories", "Independent Categories"),
+        lwd = c(1, 1),
+        col = c("black", "red"),
+        pch = c(16, 3))
> 
> plot(rpartFactorRoc, type = "s", add = TRUE, 
+      col = rgb(.2, .2, .2, .2), legacy.axes = TRUE)

Call:
roc.default(response = rpartFactorFit$pred$obs, predictor = rpartFactorFit$pred$successful,     levels = rev(levels(rpartFactorFit$pred$obs)))

Data: rpartFactorFit$pred$successful in 29610 controls (rpartFactorFit$pred$obs unsuccessful) < 17100 cases (rpartFactorFit$pred$obs successful).
Area under the curve: 0.8856
> 
> ################################################################################
> ### Section 14.2 Rule-Based Models
> 
> set.seed(476)
> partFit <- train(x = training[,fullSet], 
+                  y = training$Class,
+                  method = "PART",
+                  metric = "ROC",
+                  trControl = ctrl)
> partFit
Rule-Based Classifier 

8190 samples
1070 predictors
   2 classes: 'successful', 'unsuccessful' 

No pre-processing
Resampling: Repeated Train/Test Splits Estimated (1 reps, 0.75%) 

Summary of sample sizes: 6633 

Resampling results

  ROC    Sens   Spec 
  0.809  0.779  0.802

Tuning parameter 'threshold' was held constant at a value of 0.25

Tuning parameter 'pruned' was held constant at a value of yes
 
> 
> part2008 <- merge(partFit$pred,  partFit$bestTune)
> partCM <- confusionMatrix(partFit, norm = "none")
> partCM
Repeated Train/Test Splits Estimated (1 reps, 0.75%) Confusion Matrix 

(entries are un-normalized counts)
 
Confusion Matrix and Statistics

              Reference
Prediction     successful unsuccessful
  successful          444          195
  unsuccessful        126          792
                                          
               Accuracy : 0.7938          
                 95% CI : (0.7729, 0.8137)
    No Information Rate : 0.6339          
    P-Value [Acc > NIR] : < 2.2e-16       
                                          
                  Kappa : 0.5669          
 Mcnemar's Test P-Value : 0.0001474       
                                          
            Sensitivity : 0.7789          
            Specificity : 0.8024          
         Pos Pred Value : 0.6948          
         Neg Pred Value : 0.8627          
             Prevalence : 0.3661          
         Detection Rate : 0.2852          
   Detection Prevalence : 0.4104          
      Balanced Accuracy : 0.7907          
                                          
       'Positive' Class : successful      
                                          

> 
> partRoc <- roc(response = partFit$pred$obs,
+                predictor = partFit$pred$successful,
+                levels = rev(levels(partFit$pred$obs)))
> partRoc

Call:
roc.default(response = partFit$pred$obs, predictor = partFit$pred$successful,     levels = rev(levels(partFit$pred$obs)))

Data: partFit$pred$successful in 987 controls (partFit$pred$obs unsuccessful) < 570 cases (partFit$pred$obs successful).
Area under the curve: 0.809
> 
> set.seed(476)
> partFactorFit <- train(training[,factorPredictors], training$Class,
+                        method = "PART",
+                        metric = "ROC",
+                        trControl = ctrl)
> partFactorFit
Rule-Based Classifier 

8190 samples
1488 predictors
   2 classes: 'successful', 'unsuccessful' 

No pre-processing
Resampling: Repeated Train/Test Splits Estimated (1 reps, 0.75%) 

Summary of sample sizes: 6633 

Resampling results

  ROC    Sens   Spec 
  0.835  0.807  0.766

Tuning parameter 'threshold' was held constant at a value of 0.25

Tuning parameter 'pruned' was held constant at a value of yes
 
> 
> partFactor2008 <- merge(partFactorFit$pred,  partFactorFit$bestTune)
> partFactorCM <- confusionMatrix(partFactorFit, norm = "none")
> partFactorCM
Repeated Train/Test Splits Estimated (1 reps, 0.75%) Confusion Matrix 

(entries are un-normalized counts)
 
Confusion Matrix and Statistics

              Reference
Prediction     successful unsuccessful
  successful          460          231
  unsuccessful        110          756
                                          
               Accuracy : 0.781           
                 95% CI : (0.7596, 0.8013)
    No Information Rate : 0.6339          
    P-Value [Acc > NIR] : < 2.2e-16       
                                          
                  Kappa : 0.5484          
 Mcnemar's Test P-Value : 8.12e-11        
                                          
            Sensitivity : 0.8070          
            Specificity : 0.7660          
         Pos Pred Value : 0.6657          
         Neg Pred Value : 0.8730          
             Prevalence : 0.3661          
         Detection Rate : 0.2954          
   Detection Prevalence : 0.4438          
      Balanced Accuracy : 0.7865          
                                          
       'Positive' Class : successful      
                                          

> 
> partFactorRoc <- roc(response = partFactorFit$pred$obs,
+                      predictor = partFactorFit$pred$successful,
+                      levels = rev(levels(partFactorFit$pred$obs)))
> partFactorRoc

Call:
roc.default(response = partFactorFit$pred$obs, predictor = partFactorFit$pred$successful,     levels = rev(levels(partFactorFit$pred$obs)))

Data: partFactorFit$pred$successful in 987 controls (partFactorFit$pred$obs unsuccessful) < 570 cases (partFactorFit$pred$obs successful).
Area under the curve: 0.8347
> 
> ################################################################################
> ### Section 14.3 Bagged Trees
> 
> set.seed(476)
> treebagFit <- train(x = training[,fullSet], 
+                     y = training$Class,
+                     method = "treebag",
+                     nbagg = 50,
+                     metric = "ROC",
+                     trControl = ctrl)
Loading required package: ipred
Loading required package: MASS
Loading required package: survival
Loading required package: splines

Attaching package: 'survival'

The following object is masked from 'package:caret':

    cluster

Loading required package: nnet
Loading required package: prodlim
KernSmooth 2.23 loaded
Copyright M. P. Wand 1997-2009
> treebagFit
Bagged CART 

8190 samples
1070 predictors
   2 classes: 'successful', 'unsuccessful' 

No pre-processing
Resampling: Repeated Train/Test Splits Estimated (1 reps, 0.75%) 

Summary of sample sizes: 6633 

Resampling results

  ROC    Sens  Spec 
  0.921  0.83  0.857

 
> 
> treebag2008 <- merge(treebagFit$pred,  treebagFit$bestTune)
> treebagCM <- confusionMatrix(treebagFit, norm = "none")
> treebagCM
Repeated Train/Test Splits Estimated (1 reps, 0.75%) Confusion Matrix 

(entries are un-normalized counts)
 
Confusion Matrix and Statistics

              Reference
Prediction     successful unsuccessful
  successful          473          141
  unsuccessful         97          846
                                          
               Accuracy : 0.8471          
                 95% CI : (0.8283, 0.8647)
    No Information Rate : 0.6339          
    P-Value [Acc > NIR] : < 2.2e-16       
                                          
                  Kappa : 0.6759          
 Mcnemar's Test P-Value : 0.005315        
                                          
            Sensitivity : 0.8298          
            Specificity : 0.8571          
         Pos Pred Value : 0.7704          
         Neg Pred Value : 0.8971          
             Prevalence : 0.3661          
         Detection Rate : 0.3038          
   Detection Prevalence : 0.3943          
      Balanced Accuracy : 0.8435          
                                          
       'Positive' Class : successful      
                                          

> 
> treebagRoc <- roc(response = treebagFit$pred$obs,
+                   predictor = treebagFit$pred$successful,
+                   levels = rev(levels(treebagFit$pred$obs)))
> set.seed(476)
> treebagFactorFit <- train(x = training[,factorPredictors], 
+                           y = training$Class,
+                           method = "treebag",
+                           nbagg = 50,
+                           metric = "ROC",
+                           trControl = ctrl)
> treebagFactorFit
Bagged CART 

8190 samples
1488 predictors
   2 classes: 'successful', 'unsuccessful' 

No pre-processing
Resampling: Repeated Train/Test Splits Estimated (1 reps, 0.75%) 

Summary of sample sizes: 6633 

Resampling results

  ROC    Sens   Spec 
  0.917  0.835  0.861

 
> 
> treebagFactor2008 <- merge(treebagFactorFit$pred,  treebagFactorFit$bestTune)
> treebagFactorCM <- confusionMatrix(treebagFactorFit, norm = "none")
> treebagFactorCM
Repeated Train/Test Splits Estimated (1 reps, 0.75%) Confusion Matrix 

(entries are un-normalized counts)
 
Confusion Matrix and Statistics

              Reference
Prediction     successful unsuccessful
  successful          476          137
  unsuccessful         94          850
                                         
               Accuracy : 0.8516         
                 95% CI : (0.833, 0.8689)
    No Information Rate : 0.6339         
    P-Value [Acc > NIR] : < 2e-16        
                                         
                  Kappa : 0.6854         
 Mcnemar's Test P-Value : 0.00572        
                                         
            Sensitivity : 0.8351         
            Specificity : 0.8612         
         Pos Pred Value : 0.7765         
         Neg Pred Value : 0.9004         
             Prevalence : 0.3661         
         Detection Rate : 0.3057         
   Detection Prevalence : 0.3937         
      Balanced Accuracy : 0.8481         
                                         
       'Positive' Class : successful     
                                         

> treebagFactorRoc <- roc(response = treebagFactorFit$pred$obs,
+                         predictor = treebagFactorFit$pred$successful,
+                         levels = rev(levels(treebagFactorFit$pred$obs)))
> 
> 
> plot(rpartRoc, type = "s", col = rgb(.2, .2, .2, .2), legacy.axes = TRUE)

Call:
roc.default(response = rpartFit$pred$obs, predictor = rpartFit$pred$successful,     levels = rev(levels(rpartFit$pred$obs)))

Data: rpartFit$pred$successful in 29610 controls (rpartFit$pred$obs unsuccessful) < 17100 cases (rpartFit$pred$obs successful).
Area under the curve: 0.8915
> plot(j48FactorRoc, type = "s", add = TRUE, col = rgb(.2, .2, .2, .2), 
+      legacy.axes = TRUE)

Call:
roc.default(response = j48FactorFit$pred$obs, predictor = j48FactorFit$pred$successful,     levels = rev(levels(j48FactorFit$pred$obs)))

Data: j48FactorFit$pred$successful in 987 controls (j48FactorFit$pred$obs unsuccessful) < 570 cases (j48FactorFit$pred$obs successful).
Area under the curve: 0.8353
> plot(treebagRoc, type = "s", add = TRUE, print.thres = c(.5), 
+      print.thres.pch = 3, legacy.axes = TRUE, print.thres.pattern = "", 
+      print.thres.cex = 1.2,
+      col = "red", print.thres.col = "red")

Call:
roc.default(response = treebagFit$pred$obs, predictor = treebagFit$pred$successful,     levels = rev(levels(treebagFit$pred$obs)))

Data: treebagFit$pred$successful in 987 controls (treebagFit$pred$obs unsuccessful) < 570 cases (treebagFit$pred$obs successful).
Area under the curve: 0.9205
> plot(treebagFactorRoc, type = "s", add = TRUE, print.thres = c(.5), 
+      print.thres.pch = 16, print.thres.pattern = "", legacy.axes = TRUE, 
+      print.thres.cex = 1.2)

Call:
roc.default(response = treebagFactorFit$pred$obs, predictor = treebagFactorFit$pred$successful,     levels = rev(levels(treebagFactorFit$pred$obs)))

Data: treebagFactorFit$pred$successful in 987 controls (treebagFactorFit$pred$obs unsuccessful) < 570 cases (treebagFactorFit$pred$obs successful).
Area under the curve: 0.9173
> legend(.75, .2,
+        c("Grouped Categories", "Independent Categories"),
+        lwd = c(1, 1),
+        col = c("black", "red"),
+        pch = c(16, 3))
> 
> ################################################################################
> ### Section 14.4 Random Forests
> 
> ### For the book, this model was run with only 500 trees (by
> ### accident). More than 1000 trees usually required to get consistent
> ### results.
> 
> mtryValues <- c(5, 10, 20, 32, 50, 100, 250, 500, 1000)
> set.seed(476)
> rfFit <- train(x = training[,fullSet], 
+                y = training$Class,
+                method = "rf",
+                ntree = 500,
+                tuneGrid = data.frame(mtry = mtryValues),
+                importance = TRUE,
+                metric = "ROC",
+                trControl = ctrl)
Loading required package: randomForest
randomForest 4.6-7
Type rfNews() to see new features/changes/bug fixes.
> rfFit
Random Forest 

8190 samples
1070 predictors
   2 classes: 'successful', 'unsuccessful' 

No pre-processing
Resampling: Repeated Train/Test Splits Estimated (1 reps, 0.75%) 

Summary of sample sizes: 6633 

Resampling results across tuning parameters:

  mtry  ROC    Sens   Spec 
  5     0.876  0.805  0.769
  10    0.901  0.828  0.812
  20    0.924  0.861  0.827
  32    0.931  0.879  0.835
  50    0.936  0.877  0.835
  100   0.939  0.867  0.846
  250   0.937  0.856  0.858
  500   0.93   0.844  0.862
  1000  0.923  0.837  0.853

ROC was used to select the optimal model using  the largest value.
The final value used for the model was mtry = 100. 
> 
> rf2008 <- merge(rfFit$pred,  rfFit$bestTune)
> rfCM <- confusionMatrix(rfFit, norm = "none")
> rfCM
Repeated Train/Test Splits Estimated (1 reps, 0.75%) Confusion Matrix 

(entries are un-normalized counts)
 
Confusion Matrix and Statistics

              Reference
Prediction     successful unsuccessful
  successful          494          152
  unsuccessful         76          835
                                         
               Accuracy : 0.8536         
                 95% CI : (0.835, 0.8708)
    No Information Rate : 0.6339         
    P-Value [Acc > NIR] : < 2e-16        
                                         
                  Kappa : 0.6931         
 Mcnemar's Test P-Value : 6.8e-07        
                                         
            Sensitivity : 0.8667         
            Specificity : 0.8460         
         Pos Pred Value : 0.7647         
         Neg Pred Value : 0.9166         
             Prevalence : 0.3661         
         Detection Rate : 0.3173         
   Detection Prevalence : 0.4149         
      Balanced Accuracy : 0.8563         
                                         
       'Positive' Class : successful     
                                         

> 
> rfRoc <- roc(response = rfFit$pred$obs,
+              predictor = rfFit$pred$successful,
+              levels = rev(levels(rfFit$pred$obs)))
> 
> gc()
             used    (Mb) gc trigger    (Mb)   max used    (Mb)
Ncells    8050579   430.0   13156139   702.7   13156139   702.7
Vcells 4127289672 31488.8 6062765953 46255.3 5498501682 41950.3
> 
> ## The randomForest package cannot handle factors with more than 32
> ## levels, so we make a new set of predictors where the sponsor code
> ## factor is entered as dummy variables instead of a single factor. 
> 
> sponsorVars <- grep("Sponsor", names(training), value = TRUE)
> sponsorVars <- sponsorVars[sponsorVars != "SponsorCode"]
> 
> rfPredictors <- factorPredictors
> rfPredictors <- rfPredictors[rfPredictors != "SponsorCode"]
> rfPredictors <- c(rfPredictors, sponsorVars)
> 
> set.seed(476)
> rfFactorFit <- train(x = training[,rfPredictors], 
+                      y = training$Class,
+                      method = "rf",
+                      ntree = 1500,
+                      tuneGrid = data.frame(mtry = mtryValues),
+                      importance = TRUE,
+                      metric = "ROC",
+                      trControl = ctrl)
> rfFactorFit
Random Forest 

8190 samples
1733 predictors
   2 classes: 'successful', 'unsuccessful' 

No pre-processing
Resampling: Repeated Train/Test Splits Estimated (1 reps, 0.75%) 

Summary of sample sizes: 6633 

Resampling results across tuning parameters:

  mtry  ROC    Sens   Spec 
  5     0.808  0.619  0.817
  10    0.855  0.726  0.815
  20    0.891  0.754  0.84 
  32    0.911  0.774  0.855
  50    0.921  0.802  0.865
  100   0.93   0.823  0.87 
  250   0.937  0.842  0.871
  500   0.936  0.847  0.876
  1000  0.931  0.837  0.872

ROC was used to select the optimal model using  the largest value.
The final value used for the model was mtry = 250. 
> 
> rfFactor2008 <- merge(rfFactorFit$pred,  rfFactorFit$bestTune)
> rfFactorCM <- confusionMatrix(rfFactorFit, norm = "none")
> rfFactorCM
Repeated Train/Test Splits Estimated (1 reps, 0.75%) Confusion Matrix 

(entries are un-normalized counts)
 
Confusion Matrix and Statistics

              Reference
Prediction     successful unsuccessful
  successful          480          127
  unsuccessful         90          860
                                          
               Accuracy : 0.8606          
                 95% CI : (0.8424, 0.8775)
    No Information Rate : 0.6339          
    P-Value [Acc > NIR] : < 2e-16         
                                          
                  Kappa : 0.7038          
 Mcnemar's Test P-Value : 0.01453         
                                          
            Sensitivity : 0.8421          
            Specificity : 0.8713          
         Pos Pred Value : 0.7908          
         Neg Pred Value : 0.9053          
             Prevalence : 0.3661          
         Detection Rate : 0.3083          
   Detection Prevalence : 0.3899          
      Balanced Accuracy : 0.8567          
                                          
       'Positive' Class : successful      
                                          

> 
> rfFactorRoc <- roc(response = rfFactorFit$pred$obs,
+                    predictor = rfFactorFit$pred$successful,
+                    levels = rev(levels(rfFactorFit$pred$obs)))
> 
> plot(treebagRoc, type = "s", col = rgb(.2, .2, .2, .2), legacy.axes = TRUE)

Call:
roc.default(response = treebagFit$pred$obs, predictor = treebagFit$pred$successful,     levels = rev(levels(treebagFit$pred$obs)))

Data: treebagFit$pred$successful in 987 controls (treebagFit$pred$obs unsuccessful) < 570 cases (treebagFit$pred$obs successful).
Area under the curve: 0.9205
> plot(rpartRoc, type = "s", add = TRUE, col = rgb(.2, .2, .2, .2), legacy.axes = TRUE)

Call:
roc.default(response = rpartFit$pred$obs, predictor = rpartFit$pred$successful,     levels = rev(levels(rpartFit$pred$obs)))

Data: rpartFit$pred$successful in 29610 controls (rpartFit$pred$obs unsuccessful) < 17100 cases (rpartFit$pred$obs successful).
Area under the curve: 0.8915
> plot(j48FactorRoc, type = "s", add = TRUE, col = rgb(.2, .2, .2, .2), 
+      legacy.axes = TRUE)

Call:
roc.default(response = j48FactorFit$pred$obs, predictor = j48FactorFit$pred$successful,     levels = rev(levels(j48FactorFit$pred$obs)))

Data: j48FactorFit$pred$successful in 987 controls (j48FactorFit$pred$obs unsuccessful) < 570 cases (j48FactorFit$pred$obs successful).
Area under the curve: 0.8353
> plot(rfRoc, type = "s", add = TRUE, print.thres = c(.5), 
+      print.thres.pch = 3, legacy.axes = TRUE, print.thres.pattern = "", 
+      print.thres.cex = 1.2,
+      col = "red", print.thres.col = "red")

Call:
roc.default(response = rfFit$pred$obs, predictor = rfFit$pred$successful,     levels = rev(levels(rfFit$pred$obs)))

Data: rfFit$pred$successful in 8883 controls (rfFit$pred$obs unsuccessful) < 5130 cases (rfFit$pred$obs successful).
Area under the curve: 0.9179
> plot(rfFactorRoc, type = "s", add = TRUE, print.thres = c(.5), 
+      print.thres.pch = 16, print.thres.pattern = "", legacy.axes = TRUE, 
+      print.thres.cex = 1.2)

Call:
roc.default(response = rfFactorFit$pred$obs, predictor = rfFactorFit$pred$successful,     levels = rev(levels(rfFactorFit$pred$obs)))

Data: rfFactorFit$pred$successful in 8883 controls (rfFactorFit$pred$obs unsuccessful) < 5130 cases (rfFactorFit$pred$obs successful).
Area under the curve: 0.9049
> legend(.75, .2,
+        c("Grouped Categories", "Independent Categories"),
+        lwd = c(1, 1),
+        col = c("black", "red"),
+        pch = c(16, 3))
> 
> 
> ################################################################################
> ### Section 14.5 Boosting
> 
> gbmGrid <- expand.grid(interaction.depth = c(1, 3, 5, 7, 9),
+                        n.trees = (1:20)*100,
+                        shrinkage = c(.01, .1))
> 
> set.seed(476)
> gbmFit <- train(x = training[,fullSet], 
+                 y = training$Class,
+                 method = "gbm",
+                 tuneGrid = gbmGrid,
+                 metric = "ROC",
+                 verbose = FALSE,
+                 trControl = ctrl)
Loading required package: gbm
Loading required package: parallel
Loaded gbm 2.1
> gbmFit
Stochastic Gradient Boosting 

8190 samples
1070 predictors
   2 classes: 'successful', 'unsuccessful' 

No pre-processing
Resampling: Repeated Train/Test Splits Estimated (1 reps, 0.75%) 

Summary of sample sizes: 6633 

Resampling results across tuning parameters:

  shrinkage  interaction.depth  n.trees  ROC    Sens   Spec 
  0.01       1                  100      0.879  0.947  0.73 
  0.01       1                  200      0.887  0.947  0.73 
  0.01       1                  300      0.888  0.951  0.73 
  0.01       1                  400      0.911  0.951  0.73 
  0.01       1                  500      0.906  0.951  0.73 
  0.01       1                  600      0.908  0.904  0.8  
  0.01       1                  700      0.907  0.905  0.799
  0.01       1                  800      0.91   0.905  0.798
  0.01       1                  900      0.911  0.904  0.799
  0.01       1                  1000     0.914  0.904  0.799
  0.01       1                  1100     0.914  0.9    0.807
  0.01       1                  1200     0.915  0.898  0.814
  0.01       1                  1300     0.916  0.893  0.816
  0.01       1                  1400     0.917  0.889  0.821
  0.01       1                  1500     0.918  0.875  0.822
  0.01       1                  1600     0.919  0.877  0.826
  0.01       1                  1700     0.919  0.865  0.831
  0.01       1                  1800     0.919  0.839  0.842
  0.01       1                  1900     0.92   0.858  0.841
  0.01       1                  2000     0.92   0.835  0.847
  0.01       3                  100      0.913  0.947  0.729
  0.01       3                  200      0.918  0.889  0.81 
  0.01       3                  300      0.919  0.889  0.809
  0.01       3                  400      0.922  0.889  0.809
  0.01       3                  500      0.924  0.889  0.818
  0.01       3                  600      0.926  0.882  0.829
  0.01       3                  700      0.926  0.868  0.84 
  0.01       3                  800      0.928  0.875  0.845
  0.01       3                  900      0.928  0.868  0.847
  0.01       3                  1000     0.929  0.867  0.851
  0.01       3                  1100     0.93   0.865  0.854
  0.01       3                  1200     0.931  0.863  0.86 
  0.01       3                  1300     0.931  0.86   0.863
  0.01       3                  1400     0.932  0.863  0.863
  0.01       3                  1500     0.932  0.86   0.865
  0.01       3                  1600     0.932  0.856  0.869
  0.01       3                  1700     0.932  0.853  0.865
  0.01       3                  1800     0.933  0.853  0.864
  0.01       3                  1900     0.933  0.851  0.865
  0.01       3                  2000     0.933  0.851  0.867
  0.01       5                  100      0.914  0.947  0.726
  0.01       5                  200      0.917  0.895  0.802
  0.01       5                  300      0.925  0.904  0.804
  0.01       5                  400      0.928  0.895  0.83 
  0.01       5                  500      0.93   0.872  0.839
  0.01       5                  600      0.932  0.87   0.845
  0.01       5                  700      0.932  0.87   0.851
  0.01       5                  800      0.934  0.867  0.855
  0.01       5                  900      0.934  0.865  0.854
  0.01       5                  1000     0.935  0.86   0.861
  0.01       5                  1100     0.935  0.861  0.86 
  0.01       5                  1200     0.935  0.861  0.861
  0.01       5                  1300     0.935  0.86   0.865
  0.01       5                  1400     0.935  0.854  0.865
  0.01       5                  1500     0.935  0.856  0.868
  0.01       5                  1600     0.935  0.854  0.868
  0.01       5                  1700     0.935  0.849  0.872
  0.01       5                  1800     0.935  0.844  0.873
  0.01       5                  1900     0.934  0.846  0.873
  0.01       5                  2000     0.935  0.837  0.875
  0.01       7                  100      0.913  0.893  0.798
  0.01       7                  200      0.92   0.911  0.802
  0.01       7                  300      0.926  0.898  0.828
  0.01       7                  400      0.931  0.87   0.842
  0.01       7                  500      0.932  0.867  0.849
  0.01       7                  600      0.933  0.865  0.854
  0.01       7                  700      0.934  0.863  0.858
  0.01       7                  800      0.934  0.858  0.861
  0.01       7                  900      0.935  0.853  0.863
  0.01       7                  1000     0.935  0.849  0.865
  0.01       7                  1100     0.935  0.847  0.864
  0.01       7                  1200     0.935  0.84   0.867
  0.01       7                  1300     0.935  0.839  0.872
  0.01       7                  1400     0.935  0.837  0.875
  0.01       7                  1500     0.935  0.83   0.874
  0.01       7                  1600     0.935  0.83   0.875
  0.01       7                  1700     0.935  0.832  0.878
  0.01       7                  1800     0.935  0.826  0.878
  0.01       7                  1900     0.935  0.819  0.876
  0.01       7                  2000     0.935  0.825  0.876
  0.01       9                  100      0.919  0.895  0.796
  0.01       9                  200      0.927  0.902  0.818
  0.01       9                  300      0.93   0.872  0.844
  0.01       9                  400      0.933  0.863  0.854
  0.01       9                  500      0.935  0.86   0.859
  0.01       9                  600      0.935  0.863  0.861
  0.01       9                  700      0.936  0.858  0.865
  0.01       9                  800      0.936  0.851  0.866
  0.01       9                  900      0.936  0.846  0.87 
  0.01       9                  1000     0.936  0.849  0.869
  0.01       9                  1100     0.936  0.846  0.87 
  0.01       9                  1200     0.936  0.846  0.873
  0.01       9                  1300     0.936  0.842  0.875
  0.01       9                  1400     0.936  0.842  0.876
  0.01       9                  1500     0.936  0.837  0.878
  0.01       9                  1600     0.935  0.84   0.879
  0.01       9                  1700     0.935  0.835  0.877
  0.01       9                  1800     0.935  0.837  0.879
  0.01       9                  1900     0.935  0.832  0.878
  0.01       9                  2000     0.935  0.823  0.877
  0.1        1                  100      0.914  0.889  0.813
  0.1        1                  200      0.92   0.805  0.864
  0.1        1                  300      0.921  0.828  0.859
  0.1        1                  400      0.923  0.821  0.86 
  0.1        1                  500      0.922  0.816  0.865
  0.1        1                  600      0.923  0.809  0.869
  0.1        1                  700      0.922  0.819  0.87 
  0.1        1                  800      0.922  0.818  0.869
  0.1        1                  900      0.922  0.819  0.871
  0.1        1                  1000     0.921  0.823  0.869
  0.1        1                  1100     0.92   0.816  0.868
  0.1        1                  1200     0.918  0.814  0.869
  0.1        1                  1300     0.917  0.816  0.867
  0.1        1                  1400     0.918  0.811  0.866
  0.1        1                  1500     0.916  0.807  0.868
  0.1        1                  1600     0.915  0.807  0.867
  0.1        1                  1700     0.916  0.804  0.871
  0.1        1                  1800     0.914  0.807  0.869
  0.1        1                  1900     0.913  0.802  0.866
  0.1        1                  2000     0.913  0.802  0.865
  0.1        3                  100      0.925  0.856  0.847
  0.1        3                  200      0.932  0.839  0.871
  0.1        3                  300      0.933  0.835  0.874
  0.1        3                  400      0.932  0.83   0.877
  0.1        3                  500      0.93   0.821  0.88 
  0.1        3                  600      0.928  0.826  0.868
  0.1        3                  700      0.927  0.809  0.875
  0.1        3                  800      0.925  0.814  0.877
  0.1        3                  900      0.924  0.802  0.879
  0.1        3                  1000     0.923  0.804  0.878
  0.1        3                  1100     0.923  0.804  0.876
  0.1        3                  1200     0.923  0.8    0.873
  0.1        3                  1300     0.921  0.796  0.876
  0.1        3                  1400     0.922  0.793  0.877
  0.1        3                  1500     0.921  0.793  0.878
  0.1        3                  1600     0.921  0.791  0.877
  0.1        3                  1700     0.922  0.784  0.878
  0.1        3                  1800     0.92   0.775  0.883
  0.1        3                  1900     0.921  0.784  0.881
  0.1        3                  2000     0.918  0.786  0.881
  0.1        5                  100      0.934  0.86   0.868
  0.1        5                  200      0.935  0.846  0.87 
  0.1        5                  300      0.933  0.833  0.872
  0.1        5                  400      0.932  0.828  0.875
  0.1        5                  500      0.931  0.816  0.875
  0.1        5                  600      0.93   0.832  0.877
  0.1        5                  700      0.929  0.818  0.879
  0.1        5                  800      0.926  0.8    0.882
  0.1        5                  900      0.927  0.802  0.883
  0.1        5                  1000     0.926  0.796  0.878
  0.1        5                  1100     0.926  0.807  0.881
  0.1        5                  1200     0.925  0.807  0.875
  0.1        5                  1300     0.925  0.805  0.877
  0.1        5                  1400     0.924  0.796  0.875
  0.1        5                  1500     0.924  0.809  0.877
  0.1        5                  1600     0.924  0.807  0.878
  0.1        5                  1700     0.923  0.811  0.878
  0.1        5                  1800     0.923  0.811  0.878
  0.1        5                  1900     0.921  0.809  0.876
  0.1        5                  2000     0.922  0.809  0.871
  0.1        7                  100      0.934  0.84   0.875
  0.1        7                  200      0.931  0.809  0.875
  0.1        7                  300      0.93   0.796  0.879
  0.1        7                  400      0.928  0.793  0.877
  0.1        7                  500      0.926  0.804  0.873
  0.1        7                  600      0.924  0.784  0.872
  0.1        7                  700      0.922  0.782  0.877
  0.1        7                  800      0.923  0.789  0.873
  0.1        7                  900      0.924  0.796  0.873
  0.1        7                  1000     0.924  0.793  0.875
  0.1        7                  1100     0.924  0.793  0.872
  0.1        7                  1200     0.923  0.791  0.876
  0.1        7                  1300     0.925  0.782  0.877
  0.1        7                  1400     0.923  0.775  0.878
  0.1        7                  1500     0.923  0.767  0.877
  0.1        7                  1600     0.923  0.767  0.877
  0.1        7                  1700     0.922  0.772  0.878
  0.1        7                  1800     0.922  0.779  0.879
  0.1        7                  1900     0.922  0.768  0.878
  0.1        7                  2000     0.921  0.77   0.878
  0.1        9                  100      0.933  0.828  0.871
  0.1        9                  200      0.931  0.814  0.889
  0.1        9                  300      0.929  0.796  0.887
  0.1        9                  400      0.928  0.793  0.881
  0.1        9                  500      0.926  0.789  0.884
  0.1        9                  600      0.927  0.779  0.883
  0.1        9                  700      0.928  0.791  0.883
  0.1        9                  800      0.928  0.791  0.884
  0.1        9                  900      0.926  0.777  0.881
  0.1        9                  1000     0.925  0.772  0.886
  0.1        9                  1100     0.925  0.777  0.887
  0.1        9                  1200     0.925  0.772  0.887
  0.1        9                  1300     0.925  0.763  0.883
  0.1        9                  1400     0.924  0.772  0.883
  0.1        9                  1500     0.922  0.763  0.88 
  0.1        9                  1600     0.922  0.761  0.884
  0.1        9                  1700     0.922  0.76   0.883
  0.1        9                  1800     0.922  0.758  0.882
  0.1        9                  1900     0.923  0.76   0.884
  0.1        9                  2000     0.923  0.765  0.886

ROC was used to select the optimal model using  the largest value.
The final values used for the model were n.trees = 1300, interaction.depth =
 9 and shrinkage = 0.01. 
> 
> gbmFit$pred <- merge(gbmFit$pred,  gbmFit$bestTune)
> gbmCM <- confusionMatrix(gbmFit, norm = "none")
> gbmCM
Repeated Train/Test Splits Estimated (1 reps, 0.75%) Confusion Matrix 

(entries are un-normalized counts)
 
Confusion Matrix and Statistics

              Reference
Prediction     successful unsuccessful
  successful          480          123
  unsuccessful         90          864
                                          
               Accuracy : 0.8632          
                 95% CI : (0.8451, 0.8799)
    No Information Rate : 0.6339          
    P-Value [Acc > NIR] : < 2e-16         
                                          
                  Kappa : 0.7088          
 Mcnemar's Test P-Value : 0.02834         
                                          
            Sensitivity : 0.8421          
            Specificity : 0.8754          
         Pos Pred Value : 0.7960          
         Neg Pred Value : 0.9057          
             Prevalence : 0.3661          
         Detection Rate : 0.3083          
   Detection Prevalence : 0.3873          
      Balanced Accuracy : 0.8587          
                                          
       'Positive' Class : successful      
                                          

> 
> gbmRoc <- roc(response = gbmFit$pred$obs,
+               predictor = gbmFit$pred$successful,
+               levels = rev(levels(gbmFit$pred$obs)))
> 
> set.seed(476)
> gbmFactorFit <- train(x = training[,factorPredictors], 
+                       y = training$Class,
+                       method = "gbm",
+                       tuneGrid = gbmGrid,
+                       verbose = FALSE,
+                       metric = "ROC",
+                       trControl = ctrl)
> gbmFactorFit
Stochastic Gradient Boosting 

8190 samples
1488 predictors
   2 classes: 'successful', 'unsuccessful' 

No pre-processing
Resampling: Repeated Train/Test Splits Estimated (1 reps, 0.75%) 

Summary of sample sizes: 6633 

Resampling results across tuning parameters:

  shrinkage  interaction.depth  n.trees  ROC    Sens   Spec 
  0.01       1                  100      0.881  0.658  0.797
  0.01       1                  200      0.886  0.872  0.821
  0.01       1                  300      0.887  0.882  0.824
  0.01       1                  400      0.888  0.886  0.8  
  0.01       1                  500      0.886  0.886  0.8  
  0.01       1                  600      0.883  0.888  0.799
  0.01       1                  700      0.883  0.888  0.799
  0.01       1                  800      0.881  0.888  0.799
  0.01       1                  900      0.883  0.884  0.8  
  0.01       1                  1000     0.884  0.884  0.8  
  0.01       1                  1100     0.885  0.884  0.801
  0.01       1                  1200     0.883  0.882  0.802
  0.01       1                  1300     0.88   0.882  0.8  
  0.01       1                  1400     0.877  0.882  0.801
  0.01       1                  1500     0.873  0.884  0.8  
  0.01       1                  1600     0.87   0.882  0.8  
  0.01       1                  1700     0.869  0.881  0.802
  0.01       1                  1800     0.867  0.884  0.804
  0.01       1                  1900     0.866  0.884  0.803
  0.01       1                  2000     0.864  0.884  0.803
  0.01       3                  100      0.907  0.884  0.792
  0.01       3                  200      0.909  0.886  0.793
  0.01       3                  300      0.905  0.886  0.795
  0.01       3                  400      0.902  0.884  0.799
  0.01       3                  500      0.894  0.884  0.796
  0.01       3                  600      0.888  0.884  0.797
  0.01       3                  700      0.881  0.884  0.8  
  0.01       3                  800      0.878  0.886  0.803
  0.01       3                  900      0.874  0.888  0.804
  0.01       3                  1000     0.873  0.886  0.802
  0.01       3                  1100     0.872  0.886  0.805
  0.01       3                  1200     0.872  0.884  0.806
  0.01       3                  1300     0.872  0.881  0.807
  0.01       3                  1400     0.872  0.882  0.806
  0.01       3                  1500     0.872  0.881  0.807
  0.01       3                  1600     0.872  0.882  0.809
  0.01       3                  1700     0.872  0.881  0.81 
  0.01       3                  1800     0.872  0.888  0.81 
  0.01       3                  1900     0.872  0.884  0.807
  0.01       3                  2000     0.873  0.881  0.81 
  0.01       5                  100      0.909  0.86   0.805
  0.01       5                  200      0.906  0.875  0.792
  0.01       5                  300      0.899  0.879  0.799
  0.01       5                  400      0.894  0.882  0.798
  0.01       5                  500      0.886  0.882  0.798
  0.01       5                  600      0.881  0.882  0.801
  0.01       5                  700      0.878  0.879  0.802
  0.01       5                  800      0.877  0.879  0.803
  0.01       5                  900      0.876  0.877  0.803
  0.01       5                  1000     0.876  0.879  0.806
  0.01       5                  1100     0.876  0.879  0.806
  0.01       5                  1200     0.876  0.881  0.809
  0.01       5                  1300     0.876  0.879  0.806
  0.01       5                  1400     0.876  0.882  0.806
  0.01       5                  1500     0.876  0.884  0.809
  0.01       5                  1600     0.876  0.881  0.806
  0.01       5                  1700     0.876  0.882  0.806
  0.01       5                  1800     0.876  0.882  0.809
  0.01       5                  1900     0.876  0.879  0.805
  0.01       5                  2000     0.876  0.882  0.804
  0.01       7                  100      0.917  0.882  0.78 
  0.01       7                  200      0.904  0.879  0.797
  0.01       7                  300      0.896  0.881  0.797
  0.01       7                  400      0.886  0.875  0.804
  0.01       7                  500      0.88   0.877  0.804
  0.01       7                  600      0.878  0.875  0.803
  0.01       7                  700      0.876  0.877  0.806
  0.01       7                  800      0.876  0.877  0.807
  0.01       7                  900      0.876  0.879  0.813
  0.01       7                  1000     0.876  0.879  0.811
  0.01       7                  1100     0.875  0.875  0.81 
  0.01       7                  1200     0.875  0.875  0.811
  0.01       7                  1300     0.875  0.874  0.811
  0.01       7                  1400     0.875  0.875  0.811
  0.01       7                  1500     0.875  0.875  0.811
  0.01       7                  1600     0.875  0.874  0.811
  0.01       7                  1700     0.875  0.875  0.807
  0.01       7                  1800     0.875  0.875  0.806
  0.01       7                  1900     0.875  0.875  0.807
  0.01       7                  2000     0.875  0.877  0.811
  0.01       9                  100      0.913  0.882  0.789
  0.01       9                  200      0.904  0.881  0.789
  0.01       9                  300      0.893  0.879  0.795
  0.01       9                  400      0.883  0.881  0.804
  0.01       9                  500      0.879  0.881  0.806
  0.01       9                  600      0.877  0.879  0.806
  0.01       9                  700      0.876  0.881  0.811
  0.01       9                  800      0.876  0.881  0.811
  0.01       9                  900      0.875  0.881  0.811
  0.01       9                  1000     0.875  0.875  0.814
  0.01       9                  1100     0.875  0.874  0.81 
  0.01       9                  1200     0.875  0.874  0.81 
  0.01       9                  1300     0.875  0.874  0.81 
  0.01       9                  1400     0.875  0.872  0.81 
  0.01       9                  1500     0.875  0.874  0.81 
  0.01       9                  1600     0.874  0.874  0.81 
  0.01       9                  1700     0.874  0.875  0.811
  0.01       9                  1800     0.874  0.875  0.81 
  0.01       9                  1900     0.874  0.879  0.809
  0.01       9                  2000     0.874  0.879  0.809
  0.1        1                  100      0.882  0.891  0.8  
  0.1        1                  200      0.865  0.888  0.801
  0.1        1                  300      0.857  0.891  0.798
  0.1        1                  400      0.858  0.882  0.802
  0.1        1                  500      0.858  0.884  0.801
  0.1        1                  600      0.859  0.888  0.801
  0.1        1                  700      0.858  0.884  0.804
  0.1        1                  800      0.857  0.886  0.799
  0.1        1                  900      0.857  0.884  0.797
  0.1        1                  1000     0.856  0.886  0.8  
  0.1        1                  1100     0.857  0.886  0.801
  0.1        1                  1200     0.856  0.889  0.801
  0.1        1                  1300     0.856  0.891  0.804
  0.1        1                  1400     0.855  0.886  0.801
  0.1        1                  1500     0.855  0.882  0.804
  0.1        1                  1600     0.855  0.884  0.807
  0.1        1                  1700     0.856  0.888  0.801
  0.1        1                  1800     0.855  0.882  0.811
  0.1        1                  1900     0.855  0.881  0.807
  0.1        1                  2000     0.855  0.888  0.811
  0.1        3                  100      0.875  0.886  0.799
  0.1        3                  200      0.873  0.882  0.813
  0.1        3                  300      0.872  0.891  0.81 
  0.1        3                  400      0.872  0.889  0.809
  0.1        3                  500      0.871  0.888  0.812
  0.1        3                  600      0.87   0.893  0.812
  0.1        3                  700      0.87   0.888  0.811
  0.1        3                  800      0.87   0.889  0.81 
  0.1        3                  900      0.869  0.881  0.813
  0.1        3                  1000     0.869  0.879  0.815
  0.1        3                  1100     0.869  0.879  0.814
  0.1        3                  1200     0.868  0.884  0.811
  0.1        3                  1300     0.868  0.872  0.812
  0.1        3                  1400     0.867  0.877  0.807
  0.1        3                  1500     0.865  0.874  0.811
  0.1        3                  1600     0.865  0.881  0.81 
  0.1        3                  1700     0.864  0.877  0.812
  0.1        3                  1800     0.865  0.879  0.812
  0.1        3                  1900     0.865  0.879  0.815
  0.1        3                  2000     0.864  0.87   0.817
  0.1        5                  100      0.873  0.879  0.807
  0.1        5                  200      0.872  0.891  0.8  
  0.1        5                  300      0.871  0.875  0.814
  0.1        5                  400      0.87   0.882  0.806
  0.1        5                  500      0.868  0.879  0.806
  0.1        5                  600      0.869  0.87   0.807
  0.1        5                  700      0.868  0.875  0.809
  0.1        5                  800      0.866  0.881  0.811
  0.1        5                  900      0.865  0.879  0.805
  0.1        5                  1000     0.865  0.879  0.806
  0.1        5                  1100     0.864  0.868  0.81 
  0.1        5                  1200     0.863  0.877  0.807
  0.1        5                  1300     0.863  0.879  0.806
  0.1        5                  1400     0.863  0.875  0.805
  0.1        5                  1500     0.862  0.879  0.802
  0.1        5                  1600     0.862  0.872  0.806
  0.1        5                  1700     0.862  0.879  0.809
  0.1        5                  1800     0.862  0.877  0.807
  0.1        5                  1900     0.862  0.875  0.809
  0.1        5                  2000     0.861  0.879  0.803
  0.1        7                  100      0.876  0.893  0.809
  0.1        7                  200      0.873  0.879  0.804
  0.1        7                  300      0.87   0.882  0.799
  0.1        7                  400      0.868  0.882  0.798
  0.1        7                  500      0.864  0.879  0.8  
  0.1        7                  600      0.863  0.879  0.804
  0.1        7                  700      0.863  0.87   0.802
  0.1        7                  800      0.863  0.872  0.802
  0.1        7                  900      0.863  0.874  0.801
  0.1        7                  1000     0.862  0.868  0.8  
  0.1        7                  1100     0.861  0.863  0.794
  0.1        7                  1200     0.862  0.861  0.793
  0.1        7                  1300     0.861  0.863  0.796
  0.1        7                  1400     0.86   0.861  0.797
  0.1        7                  1500     0.86   0.867  0.796
  0.1        7                  1600     0.859  0.861  0.799
  0.1        7                  1700     0.859  0.87   0.797
  0.1        7                  1800     0.86   0.863  0.801
  0.1        7                  1900     0.86   0.868  0.799
  0.1        7                  2000     0.859  0.858  0.796
  0.1        9                  100      0.872  0.874  0.811
  0.1        9                  200      0.868  0.872  0.801
  0.1        9                  300      0.866  0.872  0.806
  0.1        9                  400      0.865  0.868  0.8  
  0.1        9                  500      0.863  0.872  0.801
  0.1        9                  600      0.861  0.879  0.803
  0.1        9                  700      0.861  0.874  0.8  
  0.1        9                  800      0.861  0.87   0.801
  0.1        9                  900      0.861  0.874  0.796
  0.1        9                  1000     0.86   0.868  0.795
  0.1        9                  1100     0.86   0.874  0.798
  0.1        9                  1200     0.859  0.868  0.797
  0.1        9                  1300     0.859  0.868  0.796
  0.1        9                  1400     0.859  0.87   0.797
  0.1        9                  1500     0.86   0.874  0.796
  0.1        9                  1600     0.859  0.868  0.796
  0.1        9                  1700     0.858  0.874  0.796
  0.1        9                  1800     0.86   0.874  0.799
  0.1        9                  1900     0.859  0.877  0.796
  0.1        9                  2000     0.859  0.879  0.795

ROC was used to select the optimal model using  the largest value.
The final values used for the model were n.trees = 100, interaction.depth =
 7 and shrinkage = 0.01. 
> 
> gbmFactorFit$pred <- merge(gbmFactorFit$pred,  gbmFactorFit$bestTune)
> gbmFactorCM <- confusionMatrix(gbmFactorFit, norm = "none")
> gbmFactorCM
Repeated Train/Test Splits Estimated (1 reps, 0.75%) Confusion Matrix 

(entries are un-normalized counts)
 
Confusion Matrix and Statistics

              Reference
Prediction     successful unsuccessful
  successful          503          217
  unsuccessful         67          770
                                          
               Accuracy : 0.8176          
                 95% CI : (0.7975, 0.8365)
    No Information Rate : 0.6339          
    P-Value [Acc > NIR] : < 2.2e-16       
                                          
                  Kappa : 0.6277          
 Mcnemar's Test P-Value : < 2.2e-16       
                                          
            Sensitivity : 0.8825          
            Specificity : 0.7801          
         Pos Pred Value : 0.6986          
         Neg Pred Value : 0.9200          
             Prevalence : 0.3661          
         Detection Rate : 0.3231          
   Detection Prevalence : 0.4624          
      Balanced Accuracy : 0.8313          
                                          
       'Positive' Class : successful      
                                          

> 
> gbmFactorRoc <- roc(response = gbmFactorFit$pred$obs,
+                     predictor = gbmFactorFit$pred$successful,
+                     levels = rev(levels(gbmFactorFit$pred$obs)))
> 
> gbmROCRange <- extendrange(cbind(gbmFactorFit$results$ROC,gbmFit$results$ROC))
> 
> plot(gbmFactorFit, ylim = gbmROCRange, 
+      auto.key = list(columns = 4, lines = TRUE))
> 
> 
> plot(gbmFit, ylim = gbmROCRange, 
+      auto.key = list(columns = 4, lines = TRUE))
> 
> 
> plot(treebagRoc, type = "s", col = rgb(.2, .2, .2, .2), legacy.axes = TRUE)

Call:
roc.default(response = treebagFit$pred$obs, predictor = treebagFit$pred$successful,     levels = rev(levels(treebagFit$pred$obs)))

Data: treebagFit$pred$successful in 987 controls (treebagFit$pred$obs unsuccessful) < 570 cases (treebagFit$pred$obs successful).
Area under the curve: 0.9205
> plot(rpartRoc, type = "s", add = TRUE, col = rgb(.2, .2, .2, .2), legacy.axes = TRUE)

Call:
roc.default(response = rpartFit$pred$obs, predictor = rpartFit$pred$successful,     levels = rev(levels(rpartFit$pred$obs)))

Data: rpartFit$pred$successful in 29610 controls (rpartFit$pred$obs unsuccessful) < 17100 cases (rpartFit$pred$obs successful).
Area under the curve: 0.8915
> plot(j48FactorRoc, type = "s", add = TRUE, col = rgb(.2, .2, .2, .2), legacy.axes = TRUE)

Call:
roc.default(response = j48FactorFit$pred$obs, predictor = j48FactorFit$pred$successful,     levels = rev(levels(j48FactorFit$pred$obs)))

Data: j48FactorFit$pred$successful in 987 controls (j48FactorFit$pred$obs unsuccessful) < 570 cases (j48FactorFit$pred$obs successful).
Area under the curve: 0.8353
> plot(rfFactorRoc, type = "s", add = TRUE, col = rgb(.2, .2, .2, .2), legacy.axes = TRUE)

Call:
roc.default(response = rfFactorFit$pred$obs, predictor = rfFactorFit$pred$successful,     levels = rev(levels(rfFactorFit$pred$obs)))

Data: rfFactorFit$pred$successful in 8883 controls (rfFactorFit$pred$obs unsuccessful) < 5130 cases (rfFactorFit$pred$obs successful).
Area under the curve: 0.9049
> plot(gbmRoc, type = "s", print.thres = c(.5), print.thres.pch = 3, 
+      print.thres.pattern = "", print.thres.cex = 1.2,
+      add = TRUE, col = "red", print.thres.col = "red", legacy.axes = TRUE)

Call:
roc.default(response = gbmFit$pred$obs, predictor = gbmFit$pred$successful,     levels = rev(levels(gbmFit$pred$obs)))

Data: gbmFit$pred$successful in 987 controls (gbmFit$pred$obs unsuccessful) < 570 cases (gbmFit$pred$obs successful).
Area under the curve: 0.9361
> plot(gbmFactorRoc, type = "s", print.thres = c(.5), print.thres.pch = 16, 
+      legacy.axes = TRUE, print.thres.pattern = "", print.thres.cex = 1.2,
+      add = TRUE)

Call:
roc.default(response = gbmFactorFit$pred$obs, predictor = gbmFactorFit$pred$successful,     levels = rev(levels(gbmFactorFit$pred$obs)))

Data: gbmFactorFit$pred$successful in 987 controls (gbmFactorFit$pred$obs unsuccessful) < 570 cases (gbmFactorFit$pred$obs successful).
Area under the curve: 0.9168
> legend(.75, .2,
+        c("Grouped Categories", "Independent Categories"),
+        lwd = c(1, 1),
+        col = c("black", "red"),
+        pch = c(16, 3))
> 
> ################################################################################
> ### Section 14.5 C5.0
> 
> c50Grid <- expand.grid(trials = c(1:9, (1:10)*10),
+                        model = c("tree", "rules"),
+                        winnow = c(TRUE, FALSE))
> set.seed(476)
> c50FactorFit <- train(training[,factorPredictors], training$Class,
+                       method = "C5.0",
+                       tuneGrid = c50Grid,
+                       verbose = FALSE,
+                       metric = "ROC",
+                       trControl = ctrl)
Loading required package: C50
> c50FactorFit
C5.0 

8190 samples
1488 predictors
   2 classes: 'successful', 'unsuccessful' 

No pre-processing
Resampling: Repeated Train/Test Splits Estimated (1 reps, 0.75%) 

Summary of sample sizes: 6633 

Resampling results across tuning parameters:

  model  winnow  trials  ROC    Sens   Spec 
  rules  FALSE   1       0.877  0.886  0.796
  rules  FALSE   2       0.886  0.621  0.919
  rules  FALSE   3       0.9    0.782  0.844
  rules  FALSE   4       0.905  0.816  0.858
  rules  FALSE   5       0.907  0.802  0.846
  rules  FALSE   6       0.917  0.832  0.841
  rules  FALSE   7       0.922  0.796  0.873
  rules  FALSE   8       0.924  0.847  0.866
  rules  FALSE   9       0.923  0.832  0.867
  rules  FALSE   10      0.92   0.818  0.87 
  rules  FALSE   20      0.934  0.823  0.888
  rules  FALSE   30      0.937  0.844  0.875
  rules  FALSE   40      0.938  0.844  0.88 
  rules  FALSE   50      0.939  0.835  0.88 
  rules  FALSE   60      0.94   0.842  0.882
  rules  FALSE   70      0.939  0.839  0.884
  rules  FALSE   80      0.941  0.847  0.886
  rules  FALSE   90      0.941  0.842  0.884
  rules  FALSE   100     0.942  0.849  0.888
  rules  TRUE    1       0.859  0.886  0.81 
  rules  TRUE    2       0.892  0.784  0.851
  rules  TRUE    3       0.895  0.796  0.85 
  rules  TRUE    4       0.914  0.811  0.862
  rules  TRUE    5       0.919  0.828  0.865
  rules  TRUE    6       0.923  0.795  0.875
  rules  TRUE    7       0.927  0.856  0.854
  rules  TRUE    8       0.93   0.818  0.876
  rules  TRUE    9       0.931  0.846  0.867
  rules  TRUE    10      0.932  0.854  0.869
  rules  TRUE    20      0.932  0.854  0.869
  rules  TRUE    30      0.933  0.849  0.869
  rules  TRUE    40      0.935  0.856  0.871
  rules  TRUE    50      0.936  0.856  0.87 
  rules  TRUE    60      0.936  0.856  0.868
  rules  TRUE    70      0.936  0.868  0.867
  rules  TRUE    80      0.937  0.858  0.873
  rules  TRUE    90      0.937  0.867  0.869
  rules  TRUE    100     0.937  0.87   0.874
  tree   FALSE   1       0.906  0.874  0.832
  tree   FALSE   2       0.903  0.886  0.838
  tree   FALSE   3       0.908  0.809  0.853
  tree   FALSE   4       0.908  0.84   0.859
  tree   FALSE   5       0.909  0.818  0.835
  tree   FALSE   6       0.908  0.835  0.844
  tree   FALSE   7       0.909  0.825  0.835
  tree   FALSE   8       0.913  0.842  0.844
  tree   FALSE   9       0.921  0.847  0.839
  tree   FALSE   10      0.921  0.847  0.838
  tree   FALSE   20      0.929  0.853  0.855
  tree   FALSE   30      0.933  0.858  0.868
  tree   FALSE   40      0.934  0.853  0.875
  tree   FALSE   50      0.934  0.847  0.872
  tree   FALSE   60      0.935  0.86   0.872
  tree   FALSE   70      0.935  0.854  0.872
  tree   FALSE   80      0.935  0.856  0.867
  tree   FALSE   90      0.935  0.853  0.866
  tree   FALSE   100     0.936  0.847  0.867
  tree   TRUE    1       0.904  0.877  0.826
  tree   TRUE    2       0.895  0.874  0.85 
  tree   TRUE    3       0.91   0.856  0.835
  tree   TRUE    4       0.911  0.826  0.83 
  tree   TRUE    5       0.912  0.816  0.848
  tree   TRUE    6       0.918  0.856  0.852
  tree   TRUE    7       0.919  0.833  0.856
  tree   TRUE    8       0.92   0.837  0.854
  tree   TRUE    9       0.921  0.83   0.854
  tree   TRUE    10      0.923  0.833  0.846
  tree   TRUE    20      0.929  0.856  0.863
  tree   TRUE    30      0.932  0.867  0.86 
  tree   TRUE    40      0.933  0.865  0.867
  tree   TRUE    50      0.934  0.868  0.873
  tree   TRUE    60      0.935  0.865  0.869
  tree   TRUE    70      0.934  0.877  0.854
  tree   TRUE    80      0.935  0.865  0.86 
  tree   TRUE    90      0.934  0.861  0.869
  tree   TRUE    100     0.935  0.872  0.866

ROC was used to select the optimal model using  the largest value.
The final values used for the model were trials = 100, model = rules and
 winnow = FALSE. 
> 
> c50FactorFit$pred <- merge(c50FactorFit$pred,  c50FactorFit$bestTune)
> c50FactorCM <- confusionMatrix(c50FactorFit, norm = "none")
> c50FactorCM
Repeated Train/Test Splits Estimated (1 reps, 0.75%) Confusion Matrix 

(entries are un-normalized counts)
 
Confusion Matrix and Statistics

              Reference
Prediction     successful unsuccessful
  successful          484          111
  unsuccessful         86          876
                                          
               Accuracy : 0.8735          
                 95% CI : (0.8559, 0.8896)
    No Information Rate : 0.6339          
    P-Value [Acc > NIR] : < 2e-16         
                                          
                  Kappa : 0.7299          
 Mcnemar's Test P-Value : 0.08728         
                                          
            Sensitivity : 0.8491          
            Specificity : 0.8875          
         Pos Pred Value : 0.8134          
         Neg Pred Value : 0.9106          
             Prevalence : 0.3661          
         Detection Rate : 0.3109          
   Detection Prevalence : 0.3821          
      Balanced Accuracy : 0.8683          
                                          
       'Positive' Class : successful      
                                          

> 
> c50FactorRoc <- roc(response = c50FactorFit$pred$obs,
+                     predictor = c50FactorFit$pred$successful,
+                     levels = rev(levels(c50FactorFit$pred$obs)))
> 
> set.seed(476)
> c50Fit <- train(training[,fullSet], training$Class,
+                 method = "C5.0",
+                 tuneGrid = c50Grid,
+                 metric = "ROC",
+                 verbose = FALSE,
+                 trControl = ctrl)
> c50Fit
C5.0 

8190 samples
1070 predictors
   2 classes: 'successful', 'unsuccessful' 

No pre-processing
Resampling: Repeated Train/Test Splits Estimated (1 reps, 0.75%) 

Summary of sample sizes: 6633 

Resampling results across tuning parameters:

  model  winnow  trials  ROC    Sens   Spec 
  rules  FALSE   1       0.893  0.768  0.87 
  rules  FALSE   2       0.877  0.872  0.831
  rules  FALSE   3       0.896  0.747  0.874
  rules  FALSE   4       0.901  0.823  0.858
  rules  FALSE   5       0.901  0.753  0.883
  rules  FALSE   6       0.914  0.851  0.855
  rules  FALSE   7       0.919  0.805  0.87 
  rules  FALSE   8       0.919  0.839  0.859
  rules  FALSE   9       0.924  0.833  0.872
  rules  FALSE   10      0.921  0.839  0.867
  rules  FALSE   20      0.928  0.846  0.866
  rules  FALSE   30      0.932  0.842  0.868
  rules  FALSE   40      0.934  0.84   0.872
  rules  FALSE   50      0.931  0.826  0.872
  rules  FALSE   60      0.933  0.842  0.872
  rules  FALSE   70      0.934  0.839  0.869
  rules  FALSE   80      0.935  0.84   0.873
  rules  FALSE   90      0.935  0.832  0.872
  rules  FALSE   100     0.935  0.844  0.871
  rules  TRUE    1       0.85   0.847  0.847
  rules  TRUE    2       0.882  0.868  0.829
  rules  TRUE    3       0.899  0.775  0.868
  rules  TRUE    4       0.91   0.854  0.834
  rules  TRUE    5       0.918  0.821  0.854
  rules  TRUE    6       0.915  0.839  0.839
  rules  TRUE    7       0.917  0.786  0.867
  rules  TRUE    8       0.921  0.842  0.853
  rules  TRUE    9       0.917  0.814  0.865
  rules  TRUE    10      0.919  0.825  0.862
  rules  TRUE    20      0.927  0.84   0.858
  rules  TRUE    30      0.923  0.809  0.869
  rules  TRUE    40      0.927  0.84   0.866
  rules  TRUE    50      0.927  0.844  0.862
  rules  TRUE    60      0.928  0.839  0.867
  rules  TRUE    70      0.928  0.837  0.866
  rules  TRUE    80      0.929  0.833  0.864
  rules  TRUE    90      0.93   0.823  0.873
  rules  TRUE    100     0.931  0.825  0.872
  tree   FALSE   1       0.9    0.753  0.878
  tree   FALSE   2       0.874  0.805  0.858
  tree   FALSE   3       0.908  0.758  0.872
  tree   FALSE   4       0.914  0.832  0.852
  tree   FALSE   5       0.921  0.814  0.857
  tree   FALSE   6       0.916  0.826  0.851
  tree   FALSE   7       0.921  0.805  0.869
  tree   FALSE   8       0.923  0.835  0.852
  tree   FALSE   9       0.924  0.809  0.866
  tree   FALSE   10      0.924  0.825  0.864
  tree   FALSE   20      0.932  0.823  0.873
  tree   FALSE   30      0.932  0.819  0.88 
  tree   FALSE   40      0.932  0.828  0.881
  tree   FALSE   50      0.932  0.83   0.878
  tree   FALSE   60      0.933  0.842  0.874
  tree   FALSE   70      0.934  0.842  0.87 
  tree   FALSE   80      0.934  0.835  0.868
  tree   FALSE   90      0.934  0.837  0.872
  tree   FALSE   100     0.935  0.842  0.875
  tree   TRUE    1       0.905  0.837  0.854
  tree   TRUE    2       0.877  0.782  0.851
  tree   TRUE    3       0.896  0.753  0.864
  tree   TRUE    4       0.902  0.774  0.862
  tree   TRUE    5       0.908  0.791  0.852
  tree   TRUE    6       0.908  0.805  0.856
  tree   TRUE    7       0.914  0.798  0.868
  tree   TRUE    8       0.915  0.795  0.865
  tree   TRUE    9       0.916  0.782  0.867
  tree   TRUE    10      0.919  0.809  0.864
  tree   TRUE    20      0.919  0.807  0.874
  tree   TRUE    30      0.926  0.804  0.873
  tree   TRUE    40      0.927  0.809  0.877
  tree   TRUE    50      0.928  0.814  0.873
  tree   TRUE    60      0.926  0.809  0.872
  tree   TRUE    70      0.928  0.812  0.871
  tree   TRUE    80      0.929  0.816  0.869
  tree   TRUE    90      0.929  0.816  0.872
  tree   TRUE    100     0.929  0.818  0.869

ROC was used to select the optimal model using  the largest value.
The final values used for the model were trials = 90, model = rules and
 winnow = FALSE. 
> 
> c50Fit$pred <- merge(c50Fit$pred,  c50Fit$bestTune)
> c50CM <- confusionMatrix(c50Fit, norm = "none")
> c50CM
Repeated Train/Test Splits Estimated (1 reps, 0.75%) Confusion Matrix 

(entries are un-normalized counts)
 
Confusion Matrix and Statistics

              Reference
Prediction     successful unsuccessful
  successful          474          126
  unsuccessful         96          861
                                          
               Accuracy : 0.8574          
                 95% CI : (0.8391, 0.8744)
    No Information Rate : 0.6339          
    P-Value [Acc > NIR] : < 2e-16         
                                          
                  Kappa : 0.6962          
 Mcnemar's Test P-Value : 0.05161         
                                          
            Sensitivity : 0.8316          
            Specificity : 0.8723          
         Pos Pred Value : 0.7900          
         Neg Pred Value : 0.8997          
             Prevalence : 0.3661          
         Detection Rate : 0.3044          
   Detection Prevalence : 0.3854          
      Balanced Accuracy : 0.8520          
                                          
       'Positive' Class : successful      
                                          

> 
> c50Roc <- roc(response = c50Fit$pred$obs,
+               predictor = c50Fit$pred$successful,
+               levels = rev(levels(c50Fit$pred$obs)))
> 
> update(plot(c50FactorFit), ylab = "ROC AUC (2008 Hold-Out Data)")
> 
> 
> plot(treebagRoc, type = "s", col = rgb(.2, .2, .2, .2), legacy.axes = TRUE)

Call:
roc.default(response = treebagFit$pred$obs, predictor = treebagFit$pred$successful,     levels = rev(levels(treebagFit$pred$obs)))

Data: treebagFit$pred$successful in 987 controls (treebagFit$pred$obs unsuccessful) < 570 cases (treebagFit$pred$obs successful).
Area under the curve: 0.9205
> plot(rpartRoc, type = "s", add = TRUE, col = rgb(.2, .2, .2, .2), legacy.axes = TRUE)

Call:
roc.default(response = rpartFit$pred$obs, predictor = rpartFit$pred$successful,     levels = rev(levels(rpartFit$pred$obs)))

Data: rpartFit$pred$successful in 29610 controls (rpartFit$pred$obs unsuccessful) < 17100 cases (rpartFit$pred$obs successful).
Area under the curve: 0.8915
> plot(j48FactorRoc, type = "s", add = TRUE, col = rgb(.2, .2, .2, .2), legacy.axes = TRUE)

Call:
roc.default(response = j48FactorFit$pred$obs, predictor = j48FactorFit$pred$successful,     levels = rev(levels(j48FactorFit$pred$obs)))

Data: j48FactorFit$pred$successful in 987 controls (j48FactorFit$pred$obs unsuccessful) < 570 cases (j48FactorFit$pred$obs successful).
Area under the curve: 0.8353
> plot(rfFactorRoc, type = "s", add = TRUE, col = rgb(.2, .2, .2, .2), legacy.axes = TRUE)

Call:
roc.default(response = rfFactorFit$pred$obs, predictor = rfFactorFit$pred$successful,     levels = rev(levels(rfFactorFit$pred$obs)))

Data: rfFactorFit$pred$successful in 8883 controls (rfFactorFit$pred$obs unsuccessful) < 5130 cases (rfFactorFit$pred$obs successful).
Area under the curve: 0.9049
> plot(gbmRoc, type = "s",  col = rgb(.2, .2, .2, .2), add = TRUE, legacy.axes = TRUE)

Call:
roc.default(response = gbmFit$pred$obs, predictor = gbmFit$pred$successful,     levels = rev(levels(gbmFit$pred$obs)))

Data: gbmFit$pred$successful in 987 controls (gbmFit$pred$obs unsuccessful) < 570 cases (gbmFit$pred$obs successful).
Area under the curve: 0.9361
> plot(c50Roc, type = "s", print.thres = c(.5), print.thres.pch = 3, 
+      print.thres.pattern = "", print.thres.cex = 1.2,
+      add = TRUE, col = "red", print.thres.col = "red", legacy.axes = TRUE)

Call:
roc.default(response = c50Fit$pred$obs, predictor = c50Fit$pred$successful,     levels = rev(levels(c50Fit$pred$obs)))

Data: c50Fit$pred$successful in 987 controls (c50Fit$pred$obs unsuccessful) < 570 cases (c50Fit$pred$obs successful).
Area under the curve: 0.9352
> plot(c50FactorRoc, type = "s", print.thres = c(.5), print.thres.pch = 16, 
+      print.thres.pattern = "", print.thres.cex = 1.2,
+      add = TRUE, legacy.axes = TRUE)

Call:
roc.default(response = c50FactorFit$pred$obs, predictor = c50FactorFit$pred$successful,     levels = rev(levels(c50FactorFit$pred$obs)))

Data: c50FactorFit$pred$successful in 987 controls (c50FactorFit$pred$obs unsuccessful) < 570 cases (c50FactorFit$pred$obs successful).
Area under the curve: 0.942
> legend(.75, .2,
+        c("Grouped Categories", "Independent Categories"),
+        lwd = c(1, 1),
+        col = c("black", "red"),
+        pch = c(16, 3))
> 
> ################################################################################
> ### Section 14.7 Comparing Two Encodings of Categorical Predictors
> 
> ## Pull the hold-out results from each model and merge
> 
> rp1 <- caret:::getTrainPerf(rpartFit)
> names(rp1) <- gsub("Train", "Independent", names(rp1))
> rp2 <- caret:::getTrainPerf(rpartFactorFit)
> rp2$Label <- "CART"
> names(rp2) <- gsub("Train", "Grouped", names(rp2))
> rp <- cbind(rp1, rp2)
> 
> j481 <- caret:::getTrainPerf(j48Fit)
> names(j481) <- gsub("Train", "Independent", names(j481))
> j482 <- caret:::getTrainPerf(j48FactorFit)
> j482$Label <- "J48"
> names(j482) <- gsub("Train", "Grouped", names(j482))
> j48 <- cbind(j481, j482)
> 
> part1 <- caret:::getTrainPerf(partFit)
> names(part1) <- gsub("Train", "Independent", names(part1))
> part2 <- caret:::getTrainPerf(partFactorFit)
> part2$Label <- "PART"
> names(part2) <- gsub("Train", "Grouped", names(part2))
> part <- cbind(part1, part2)
> 
> tb1 <- caret:::getTrainPerf(treebagFit)
> names(tb1) <- gsub("Train", "Independent", names(tb1))
> tb2 <- caret:::getTrainPerf(treebagFactorFit)
> tb2$Label <- "Bagged Tree"
> names(tb2) <- gsub("Train", "Grouped", names(tb2))
> tb <- cbind(tb1, tb2)
> 
> rf1 <- caret:::getTrainPerf(rfFit)
> names(rf1) <- gsub("Train", "Independent", names(rf1))
> rf2 <- caret:::getTrainPerf(rfFactorFit)
> rf2$Label <- "Random Forest"
> names(rf2) <- gsub("Train", "Grouped", names(rf2))
> rf <- cbind(rf1, rf2)
> 
> gbm1 <- caret:::getTrainPerf(gbmFit)
> names(gbm1) <- gsub("Train", "Independent", names(gbm1))
> gbm2 <- caret:::getTrainPerf(gbmFactorFit)
> gbm2$Label <- "Boosted Tree"
> names(gbm2) <- gsub("Train", "Grouped", names(gbm2))
> bst <- cbind(gbm1, gbm2)
> 
> 
> c501 <- caret:::getTrainPerf(c50Fit)
> names(c501) <- gsub("Train", "Independent", names(c501))
> c502 <- caret:::getTrainPerf(c50FactorFit)
> c502$Label <- "C5.0"
> names(c502) <- gsub("Train", "Grouped", names(c502))
> c5 <- cbind(c501, c502)
> 
> 
> trainPerf <- rbind(rp, j48, part, tb, rf, bst, c5)
> 
> library(lattice)
> library(reshape2)
> trainPerf <- melt(trainPerf)
Using method, method, Label as id variables
> trainPerf$metric <- "ROC"
> trainPerf$metric[grepl("Sens", trainPerf$variable)] <- "Sensitivity"
> trainPerf$metric[grepl("Spec", trainPerf$variable)] <- "Specificity"
> trainPerf$model <- "Grouped"
> trainPerf$model[grepl("Independent", trainPerf$variable)] <- "Independent"
> 
> trainPerf <- melt(trainPerf)
Using method, method.1, Label, variable, metric, model as id variables
> trainPerf$metric <- "ROC"
> trainPerf$metric[grepl("Sens", trainPerf$variable)] <- "Sensitivity"
> trainPerf$metric[grepl("Spec", trainPerf$variable)] <- "Specificity"
> trainPerf$model <- "Independent"
> trainPerf$model[grepl("Grouped", trainPerf$variable)] <- "Grouped"
> trainPerf$Label <- factor(trainPerf$Label,
+                           levels = rev(c("CART", "Cond. Trees", "J48", "Ripper",
+                                          "PART", "Bagged Tree", "Random Forest", 
+                                          "Boosted Tree", "C5.0")))
> 
> dotplot(Label ~ value|metric,
+         data = trainPerf,
+         groups = model,
+         horizontal = TRUE,
+         auto.key = list(columns = 2),
+         between = list(x = 1),
+         xlab = "")
> 
> 
> sessionInfo()
R version 3.0.1 (2013-05-16)
Platform: x86_64-apple-darwin10.8.0 (64-bit)

locale:
[1] C

attached base packages:
 [1] parallel  splines   grid      stats     graphics  grDevices utils    
 [8] datasets  methods   base     

other attached packages:
 [1] reshape2_1.2.2     C50_0.1.0-15       gbm_2.1            randomForest_4.6-7
 [5] ipred_0.9-1        prodlim_1.3.7      nnet_7.3-6         survival_2.37-4   
 [9] MASS_7.3-26        RWeka_0.4-17       e1071_1.6-1        class_7.3-7       
[13] partykit_0.1-5     pROC_1.5.4         plyr_1.8           rpart_4.1-1       
[17] caret_6.0-22       ggplot2_0.9.3.1    lattice_0.20-15   

loaded via a namespace (and not attached):
 [1] KernSmooth_2.23-10 RColorBrewer_1.0-5 RWekajars_3.7.9-1  car_2.0-17        
 [5] codetools_0.2-8    colorspace_1.2-2   compiler_3.0.1     dichromat_2.0-0   
 [9] digest_0.6.3       foreach_1.4.0      gtable_0.1.2       iterators_1.0.6   
[13] labeling_0.1       munsell_0.4        proto_0.3-10       rJava_0.9-4       
[17] scales_0.2.3       stringr_0.6.2     
> 
> q("no")
> proc.time()
      user     system    elapsed 
208496.296    776.829 209791.456 
