
R version 3.0.1 (2013-05-16) -- "Good Sport"
Copyright (C) 2013 The R Foundation for Statistical Computing
Platform: x86_64-apple-darwin10.8.0 (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

> ################################################################################
> ### R code from Applied Predictive Modeling (2013) by Kuhn and Johnson.
> ### Copyright 2013 Kuhn and Johnson
> ### Web Page: http://www.appliedpredictivemodeling.com
> ### Contact: Max Kuhn (mxkuhn@gmail.com) 
> ###
> ### Chapter 13 Non-Linear Classification Models
> ###
> ### Required packages: AppliedPredictiveModeling, caret, doMC (optional) 
> ###                    kernlab, klaR, lattice, latticeExtra, MASS, mda, nnet,
> ###                    pROC
> ###
> ### Data used: The grant application data. See the file 'CreateGrantData.R'
> ###
> ### Notes: 
> ### 1) This code is provided without warranty.
> ###
> ### 2) This code should help the user reproduce the results in the
> ### text. There will be differences between this code and what is is
> ### the computing section. For example, the computing sections show
> ### how the source functions work (e.g. randomForest() or plsr()),
> ### which were not directly used when creating the book. Also, there may be 
> ### syntax differences that occur over time as packages evolve. These files 
> ### will reflect those changes.
> ###
> ### 3) In some cases, the calculations in the book were run in 
> ### parallel. The sub-processes may reset the random number seed.
> ### Your results may slightly vary.
> ###
> ################################################################################
> 
> ################################################################################
> ### Section 13.1 Nonlinear Discriminant Analysis
> 
> 
> load("grantData.RData")
> 
> library(caret)
Loading required package: lattice
Loading required package: ggplot2
> 
> ### Optional: parallel processing can be used via the 'do' packages,
> ### such as doMC, doMPI etc. We used doMC (not on Windows) to speed
> ### up the computations.
>  
> ### WARNING: Be aware of how much memory is needed to parallel
> ### process. It can very quickly overwhelm the available hardware. We
> ### estimate the memory usage (VSIZE = total memory size) to be 
> ### 2700M/core.
> 
> library(doMC)
Loading required package: foreach
Loading required package: iterators
Loading required package: parallel
> registerDoMC(12)
> 
> ## This control object will be used across multiple models so that the
> ## data splitting is consistent
> 
> ctrl <- trainControl(method = "LGOCV",
+                      summaryFunction = twoClassSummary,
+                      classProbs = TRUE,
+                      index = list(TrainSet = pre2008),
+                      savePredictions = TRUE)
> 
> set.seed(476)
> mdaFit <- train(x = training[,reducedSet], 
+                 y = training$Class,
+                 method = "mda",
+                 metric = "ROC",
+                 tries = 40,
+                 tuneGrid = expand.grid(subclasses = 1:8),
+                 trControl = ctrl)
Loading required package: mda
Loading required package: class
Loading required package: pROC
Loading required package: plyr
Type 'citation("pROC")' for a citation.

Attaching package: ‘pROC’

The following object is masked from ‘package:stats’:

    cov, smooth, var

> mdaFit
Mixture Discriminant Analysis 

8190 samples
 252 predictors
   2 classes: 'successful', 'unsuccessful' 

No pre-processing
Resampling: Repeated Train/Test Splits Estimated (1 reps, 0.75%) 

Summary of sample sizes: 6633 

Resampling results across tuning parameters:

  subclasses  ROC    Sens   Spec 
  1           0.887  0.811  0.822
  2           0.865  0.789  0.813
  3           0.831  0.835  0.726
  4           0.852  0.732  0.82 
  5           0.842  0.733  0.797
  6           0.822  0.733  0.782
  7           0.836  0.823  0.734
  8           0.791  0.649  0.851

ROC was used to select the optimal model using  the largest value.
The final value used for the model was subclasses = 1. 
> 
> mdaFit$results <- mdaFit$results[!is.na(mdaFit$results$ROC),]                
> mdaFit$pred <- merge(mdaFit$pred,  mdaFit$bestTune)
> mdaCM <- confusionMatrix(mdaFit, norm = "none")
> mdaCM
Repeated Train/Test Splits Estimated (1 reps, 0.75%) Confusion Matrix 

(entries are un-normalized counts)
 
Confusion Matrix and Statistics

              Reference
Prediction     successful unsuccessful
  successful          462          176
  unsuccessful        108          811
                                          
               Accuracy : 0.8176          
                 95% CI : (0.7975, 0.8365)
    No Information Rate : 0.6339          
    P-Value [Acc > NIR] : < 2.2e-16       
                                          
                  Kappa : 0.6167          
 Mcnemar's Test P-Value : 7.017e-05       
                                          
            Sensitivity : 0.8105          
            Specificity : 0.8217          
         Pos Pred Value : 0.7241          
         Neg Pred Value : 0.8825          
             Prevalence : 0.3661          
         Detection Rate : 0.2967          
   Detection Prevalence : 0.4098          
      Balanced Accuracy : 0.8161          
                                          
       'Positive' Class : successful      
                                          

> 
> mdaRoc <- roc(response = mdaFit$pred$obs,
+               predictor = mdaFit$pred$successful,
+               levels = rev(levels(mdaFit$pred$obs)))
> mdaRoc

Call:
roc.default(response = mdaFit$pred$obs, predictor = mdaFit$pred$successful,     levels = rev(levels(mdaFit$pred$obs)))

Data: mdaFit$pred$successful in 987 controls (mdaFit$pred$obs unsuccessful) < 570 cases (mdaFit$pred$obs successful).
Area under the curve: 0.8874
> 
> update(plot(mdaFit,
+             ylab = "ROC AUC (2008 Hold-Out Data)"))
> 
> ################################################################################
> ### Section 13.2 Neural Networks
> 
> nnetGrid <- expand.grid(size = 1:10, decay = c(0, .1, 1, 2))
> maxSize <- max(nnetGrid$size)
> 
> 
> ## Four different models are evaluate based on the data pre-processing and 
> ## whethera single or multiple models are used
> 
> set.seed(476)
> nnetFit <- train(x = training[,reducedSet], 
+                  y = training$Class,
+                  method = "nnet",
+                  metric = "ROC",
+                  preProc = c("center", "scale"),
+                  tuneGrid = nnetGrid,
+                  trace = FALSE,
+                  maxit = 2000,
+                  MaxNWts = 1*(maxSize * (length(reducedSet) + 1) + maxSize + 1),
+                  trControl = ctrl)
Loading required package: nnet
> nnetFit
Neural Network 

8190 samples
 252 predictors
   2 classes: 'successful', 'unsuccessful' 

Pre-processing: centered, scaled 
Resampling: Repeated Train/Test Splits Estimated (1 reps, 0.75%) 

Summary of sample sizes: 6633 

Resampling results across tuning parameters:

  size  decay  ROC    Sens   Spec 
  1     0      0.778  0.765  0.791
  1     0.1    0.845  0.793  0.794
  1     1      0.844  0.795  0.79 
  1     2      0.853  0.782  0.811
  2     0      0.804  0.811  0.753
  2     0.1    0.846  0.807  0.806
  2     1      0.86   0.73   0.841
  2     2      0.864  0.758  0.834
  3     0      0.841  0.805  0.757
  3     0.1    0.822  0.786  0.728
  3     1      0.857  0.73   0.833
  3     2      0.859  0.747  0.81 
  4     0      0.828  0.795  0.754
  4     0.1    0.854  0.74   0.814
  4     1      0.869  0.802  0.796
  4     2      0.864  0.779  0.785
  5     0      0.819  0.767  0.719
  5     0.1    0.843  0.786  0.787
  5     1      0.845  0.716  0.817
  5     2      0.851  0.728  0.829
  6     0      0.844  0.728  0.806
  6     0.1    0.8    0.693  0.775
  6     1      0.848  0.782  0.778
  6     2      0.869  0.777  0.82 
  7     0      0.833  0.807  0.757
  7     0.1    0.806  0.728  0.768
  7     1      0.831  0.746  0.777
  7     2      0.863  0.758  0.822
  8     0      0.833  0.761  0.784
  8     0.1    0.847  0.751  0.78 
  8     1      0.857  0.753  0.803
  8     2      0.866  0.77   0.814
  9     0      0.848  0.784  0.789
  9     0.1    0.836  0.719  0.798
  9     1      0.843  0.753  0.781
  9     2      0.854  0.746  0.803
  10    0      0.806  0.707  0.779
  10    0.1    0.82   0.726  0.76 
  10    1      0.846  0.73   0.807
  10    2      0.863  0.749  0.817

ROC was used to select the optimal model using  the largest value.
The final values used for the model were size = 4 and decay = 1. 
> 
> set.seed(476)
> nnetFit2 <- train(x = training[,reducedSet], 
+                   y = training$Class,
+                   method = "nnet",
+                   metric = "ROC",
+                   preProc = c("center", "scale", "spatialSign"),
+                   tuneGrid = nnetGrid,
+                   trace = FALSE,
+                   maxit = 2000,
+                   MaxNWts = 1*(maxSize * (length(reducedSet) + 1) + maxSize + 1),
+                   trControl = ctrl)
> nnetFit2
Neural Network 

8190 samples
 252 predictors
   2 classes: 'successful', 'unsuccessful' 

Pre-processing: centered, scaled, spatial sign transformation 
Resampling: Repeated Train/Test Splits Estimated (1 reps, 0.75%) 

Summary of sample sizes: 6633 

Resampling results across tuning parameters:

  size  decay  ROC    Sens   Spec 
  1     0      0.782  0.782  0.78 
  1     0.1    0.863  0.784  0.809
  1     1      0.874  0.807  0.805
  1     2      0.88   0.804  0.807
  2     0      0.776  0.804  0.711
  2     0.1    0.892  0.767  0.861
  2     1      0.897  0.804  0.839
  2     2      0.881  0.805  0.811
  3     0      0.841  0.653  0.876
  3     0.1    0.887  0.737  0.851
  3     1      0.898  0.805  0.851
  3     2      0.884  0.805  0.812
  4     0      0.786  0.756  0.715
  4     0.1    0.871  0.716  0.829
  4     1      0.899  0.793  0.84 
  4     2      0.883  0.804  0.812
  5     0      0.862  0.867  0.705
  5     0.1    0.858  0.718  0.836
  5     1      0.902  0.788  0.857
  5     2      0.883  0.804  0.812
  6     0      0.808  0.691  0.796
  6     0.1    0.859  0.712  0.844
  6     1      0.896  0.795  0.842
  6     2      0.883  0.804  0.812
  7     0      0.807  0.732  0.782
  7     0.1    0.843  0.693  0.829
  7     1      0.902  0.789  0.857
  7     2      0.883  0.804  0.813
  8     0      0.73   0.661  0.795
  8     0.1    0.858  0.681  0.834
  8     1      0.903  0.791  0.853
  8     2      0.883  0.804  0.813
  9     0      0.857  0.779  0.804
  9     0.1    0.87   0.739  0.833
  9     1      0.902  0.788  0.857
  9     2      0.883  0.804  0.813
  10    0      0.788  0.684  0.823
  10    0.1    0.876  0.721  0.845
  10    1      0.897  0.796  0.842
  10    2      0.883  0.804  0.813

ROC was used to select the optimal model using  the largest value.
The final values used for the model were size = 8 and decay = 1. 
> 
> nnetGrid$bag <- FALSE
> 
> set.seed(476)
> nnetFit3 <- train(x = training[,reducedSet], 
+                   y = training$Class,
+                   method = "avNNet",
+                   metric = "ROC",
+                   preProc = c("center", "scale"),
+                   tuneGrid = nnetGrid,
+                   repeats = 10,
+                   trace = FALSE,
+                   maxit = 2000,
+                   MaxNWts = 10*(maxSize * (length(reducedSet) + 1) + maxSize + 1),
+                   allowParallel = FALSE, ## this will cause to many workers to be launched.
+                   trControl = ctrl)
> nnetFit3
Model Averaged Neural Network 

8190 samples
 252 predictors
   2 classes: 'successful', 'unsuccessful' 

Pre-processing: centered, scaled 
Resampling: Repeated Train/Test Splits Estimated (1 reps, 0.75%) 

Summary of sample sizes: 6633 

Resampling results across tuning parameters:

  size  decay  ROC    Sens   Spec 
  1     0      0.884  0.867  0.762
  1     0.1    0.868  0.779  0.812
  1     1      0.847  0.774  0.812
  1     2      0.849  0.777  0.81 
  2     0      0.892  0.825  0.791
  2     0.1    0.886  0.784  0.854
  2     1      0.895  0.788  0.844
  2     2      0.895  0.796  0.845
  3     0      0.887  0.826  0.793
  3     0.1    0.882  0.8    0.825
  3     1      0.89   0.795  0.842
  3     2      0.899  0.798  0.838
  4     0      0.883  0.821  0.805
  4     0.1    0.887  0.8    0.821
  4     1      0.899  0.781  0.853
  4     2      0.902  0.798  0.86 
  5     0      0.886  0.83   0.79 
  5     0.1    0.874  0.788  0.824
  5     1      0.901  0.8    0.844
  5     2      0.9    0.8    0.851
  6     0      0.885  0.819  0.807
  6     0.1    0.882  0.789  0.827
  6     1      0.893  0.786  0.854
  6     2      0.9    0.8    0.849
  7     0      0.881  0.832  0.761
  7     0.1    0.883  0.791  0.821
  7     1      0.898  0.811  0.834
  7     2      0.899  0.807  0.859
  8     0      0.889  0.818  0.793
  8     0.1    0.88   0.786  0.83 
  8     1      0.891  0.8    0.823
  8     2      0.901  0.791  0.845
  9     0      0.887  0.8    0.806
  9     0.1    0.889  0.786  0.817
  9     1      0.894  0.791  0.848
  9     2      0.9    0.802  0.836
  10    0      0.883  0.811  0.805
  10    0.1    0.881  0.784  0.825
  10    1      0.898  0.793  0.844
  10    2      0.896  0.802  0.839

Tuning parameter 'bag' was held constant at a value of FALSE
ROC was used to select the optimal model using  the largest value.
The final values used for the model were size = 4, decay = 2 and bag = FALSE. 
> 
> set.seed(476)
> nnetFit4 <- train(x = training[,reducedSet], 
+                   y = training$Class,
+                   method = "avNNet",
+                   metric = "ROC",
+                   preProc = c("center", "scale", "spatialSign"),
+                   tuneGrid = nnetGrid,
+                   trace = FALSE,
+                   maxit = 2000,
+                   repeats = 10,
+                   MaxNWts = 10*(maxSize * (length(reducedSet) + 1) + maxSize + 1),
+                   allowParallel = FALSE, 
+                   trControl = ctrl)
> nnetFit4
Model Averaged Neural Network 

8190 samples
 252 predictors
   2 classes: 'successful', 'unsuccessful' 

Pre-processing: centered, scaled, spatial sign transformation 
Resampling: Repeated Train/Test Splits Estimated (1 reps, 0.75%) 

Summary of sample sizes: 6633 

Resampling results across tuning parameters:

  size  decay  ROC    Sens   Spec 
  1     0      0.867  0.784  0.8  
  1     0.1    0.857  0.782  0.81 
  1     1      0.874  0.804  0.807
  1     2      0.882  0.795  0.802
  2     0      0.882  0.782  0.845
  2     0.1    0.897  0.754  0.872
  2     1      0.875  0.798  0.811
  2     2      0.881  0.795  0.804
  3     0      0.89   0.796  0.833
  3     0.1    0.907  0.788  0.864
  3     1      0.876  0.795  0.81 
  3     2      0.881  0.795  0.804
  4     0      0.889  0.795  0.838
  4     0.1    0.911  0.782  0.867
  4     1      0.874  0.798  0.809
  4     2      0.881  0.795  0.805
  5     0      0.893  0.786  0.861
  5     0.1    0.909  0.779  0.87 
  5     1      0.875  0.796  0.809
  5     2      0.881  0.795  0.805
  6     0      0.893  0.786  0.848
  6     0.1    0.904  0.754  0.865
  6     1      0.876  0.793  0.81 
  6     2      0.881  0.795  0.805
  7     0      0.89   0.782  0.849
  7     0.1    0.905  0.76   0.866
  7     1      0.881  0.796  0.817
  7     2      0.881  0.795  0.805
  8     0      0.898  0.795  0.856
  8     0.1    0.904  0.756  0.865
  8     1      0.878  0.795  0.813
  8     2      0.881  0.795  0.805
  9     0      0.893  0.782  0.857
  9     0.1    0.902  0.761  0.869
  9     1      0.878  0.795  0.813
  9     2      0.881  0.795  0.805
  10    0      0.895  0.786  0.865
  10    0.1    0.901  0.76   0.858
  10    1      0.878  0.795  0.814
  10    2      0.881  0.795  0.806

Tuning parameter 'bag' was held constant at a value of FALSE
ROC was used to select the optimal model using  the largest value.
The final values used for the model were size = 4, decay = 0.1 and bag = FALSE. 
> 
> nnetFit4$pred <- merge(nnetFit4$pred,  nnetFit4$bestTune)
> nnetCM <- confusionMatrix(nnetFit4, norm = "none")
> nnetCM
Repeated Train/Test Splits Estimated (1 reps, 0.75%) Confusion Matrix 

(entries are un-normalized counts)
 
Confusion Matrix and Statistics

              Reference
Prediction     successful unsuccessful
  successful          446          131
  unsuccessful        124          856
                                          
               Accuracy : 0.8362          
                 95% CI : (0.8169, 0.8543)
    No Information Rate : 0.6339          
    P-Value [Acc > NIR] : <2e-16          
                                          
                  Kappa : 0.648           
 Mcnemar's Test P-Value : 0.7071          
                                          
            Sensitivity : 0.7825          
            Specificity : 0.8673          
         Pos Pred Value : 0.7730          
         Neg Pred Value : 0.8735          
             Prevalence : 0.3661          
         Detection Rate : 0.2864          
   Detection Prevalence : 0.3706          
      Balanced Accuracy : 0.8249          
                                          
       'Positive' Class : successful      
                                          

> 
> nnetRoc <- roc(response = nnetFit4$pred$obs,
+                predictor = nnetFit4$pred$successful,
+                levels = rev(levels(nnetFit4$pred$obs)))
> 
> 
> nnet1 <- nnetFit$results
> nnet1$Transform <- "No Transformation"
> nnet1$Model <- "Single Model"
> 
> nnet2 <- nnetFit2$results
> nnet2$Transform <- "Spatial Sign"
> nnet2$Model <- "Single Model"
> 
> nnet3 <- nnetFit3$results
> nnet3$Transform <- "No Transformation"
> nnet3$Model <- "Model Averaging"
> nnet3$bag <- NULL
> 
> nnet4 <- nnetFit4$results
> nnet4$Transform <- "Spatial Sign"
> nnet4$Model <- "Model Averaging"
> nnet4$bag <- NULL
> 
> nnetResults <- rbind(nnet1, nnet2, nnet3, nnet4)
> nnetResults$Model <- factor(as.character(nnetResults$Model),
+                             levels = c("Single Model", "Model Averaging"))
> library(latticeExtra)
Loading required package: RColorBrewer

Attaching package: ‘latticeExtra’

The following object is masked from ‘package:ggplot2’:

    layer

> useOuterStrips(
+   xyplot(ROC ~ size|Model*Transform,
+          data = nnetResults,
+          groups = decay,
+          as.table = TRUE,
+          type = c("p", "l", "g"),
+          lty = 1,
+          ylab = "ROC AUC (2008 Hold-Out Data)",
+          xlab = "Number of Hidden Units",
+          auto.key = list(columns = 4, 
+                          title = "Weight Decay", 
+                          cex.title = 1)))
> 
> plot(nnetRoc, type = "s", legacy.axes = TRUE)

Call:
roc.default(response = nnetFit4$pred$obs, predictor = nnetFit4$pred$successful,     levels = rev(levels(nnetFit4$pred$obs)))

Data: nnetFit4$pred$successful in 987 controls (nnetFit4$pred$obs unsuccessful) < 570 cases (nnetFit4$pred$obs successful).
Area under the curve: 0.9111
> 
> ################################################################################
> ### Section 13.3 Flexible Discriminant Analysis
> 
> set.seed(476)
> fdaFit <- train(x = training[,reducedSet], 
+                 y = training$Class,
+                 method = "fda",
+                 metric = "ROC",
+                 tuneGrid = expand.grid(degree = 1, nprune = 2:25),
+                 trControl = ctrl)
Loading required package: earth
Loading required package: leaps
Loading required package: plotmo
Loading required package: plotrix
> fdaFit
Flexible Discriminant Analysis 

8190 samples
 252 predictors
   2 classes: 'successful', 'unsuccessful' 

No pre-processing
Resampling: Repeated Train/Test Splits Estimated (1 reps, 0.75%) 

Summary of sample sizes: 6633 

Resampling results across tuning parameters:

  nprune  ROC    Sens   Spec 
  2       0.815  0.991  0.638
  3       0.809  0.995  0.567
  4       0.86   0.947  0.73 
  5       0.869  0.963  0.728
  6       0.877  0.968  0.727
  7       0.893  0.823  0.806
  8       0.903  0.779  0.851
  9       0.909  0.83   0.841
  10      0.915  0.816  0.853
  11      0.919  0.825  0.859
  12      0.92   0.816  0.865
  13      0.918  0.809  0.865
  14      0.92   0.807  0.865
  15      0.92   0.819  0.861
  16      0.921  0.826  0.858
  17      0.921  0.818  0.863
  18      0.922  0.821  0.86 
  19      0.924  0.825  0.864
  20      0.922  0.825  0.858
  21      0.919  0.816  0.869
  22      0.919  0.811  0.872
  23      0.918  0.811  0.867
  24      0.918  0.809  0.868
  25      0.918  0.809  0.865

Tuning parameter 'degree' was held constant at a value of 1
ROC was used to select the optimal model using  the largest value.
The final values used for the model were degree = 1 and nprune = 19. 
> 
> fdaFit$pred <- merge(fdaFit$pred,  fdaFit$bestTune)
> fdaCM <- confusionMatrix(fdaFit, norm = "none")
> fdaCM
Repeated Train/Test Splits Estimated (1 reps, 0.75%) Confusion Matrix 

(entries are un-normalized counts)
 
Confusion Matrix and Statistics

              Reference
Prediction     successful unsuccessful
  successful          470          134
  unsuccessful        100          853
                                         
               Accuracy : 0.8497         
                 95% CI : (0.831, 0.8671)
    No Information Rate : 0.6339         
    P-Value [Acc > NIR] : < 2e-16        
                                         
                  Kappa : 0.6802         
 Mcnemar's Test P-Value : 0.03098        
                                         
            Sensitivity : 0.8246         
            Specificity : 0.8642         
         Pos Pred Value : 0.7781         
         Neg Pred Value : 0.8951         
             Prevalence : 0.3661         
         Detection Rate : 0.3019         
   Detection Prevalence : 0.3879         
      Balanced Accuracy : 0.8444         
                                         
       'Positive' Class : successful     
                                         

> 
> fdaRoc <- roc(response = fdaFit$pred$obs,
+               predictor = fdaFit$pred$successful,
+               levels = rev(levels(fdaFit$pred$obs)))
> 
> update(plot(fdaFit), ylab = "ROC AUC (2008 Hold-Out Data)")
> 
> plot(nnetRoc, type = "s", col = rgb(.2, .2, .2, .2), legacy.axes = TRUE)

Call:
roc.default(response = nnetFit4$pred$obs, predictor = nnetFit4$pred$successful,     levels = rev(levels(nnetFit4$pred$obs)))

Data: nnetFit4$pred$successful in 987 controls (nnetFit4$pred$obs unsuccessful) < 570 cases (nnetFit4$pred$obs successful).
Area under the curve: 0.9111
> plot(fdaRoc, type = "s", add = TRUE, legacy.axes = TRUE)

Call:
roc.default(response = fdaFit$pred$obs, predictor = fdaFit$pred$successful,     levels = rev(levels(fdaFit$pred$obs)))

Data: fdaFit$pred$successful in 987 controls (fdaFit$pred$obs unsuccessful) < 570 cases (fdaFit$pred$obs successful).
Area under the curve: 0.924
> 
> 
> ################################################################################
> ### Section 13.4 Support Vector Machines
> 
> library(kernlab)
> 
> set.seed(201)
> sigmaRangeFull <- sigest(as.matrix(training[,fullSet]))
> svmRGridFull <- expand.grid(sigma =  as.vector(sigmaRangeFull)[1],
+                             C = 2^(-3:4))
> set.seed(476)
> svmRFitFull <- train(x = training[,fullSet], 
+                      y = training$Class,
+                      method = "svmRadial",
+                      metric = "ROC",
+                      preProc = c("center", "scale"),
+                      tuneGrid = svmRGridFull,
+                      trControl = ctrl)
> svmRFitFull
Support Vector Machines with Radial Basis Function Kernel 

8190 samples
1070 predictors
   2 classes: 'successful', 'unsuccessful' 

Pre-processing: centered, scaled 
Resampling: Repeated Train/Test Splits Estimated (1 reps, 0.75%) 

Summary of sample sizes: 6633 

Resampling results across tuning parameters:

  C      ROC    Sens   Spec 
  0.125  0.781  0.916  0.521
  0.25   0.851  0.861  0.694
  0.5    0.866  0.84   0.755
  1      0.873  0.83   0.774
  2      0.875  0.821  0.791
  4      0.875  0.811  0.803
  8      0.87   0.798  0.799
  16     0.866  0.798  0.81 

Tuning parameter 'sigma' was held constant at a value of 0.0002385724
ROC was used to select the optimal model using  the largest value.
The final values used for the model were sigma = 0.000239 and C = 2. 
> 
> set.seed(202)
> sigmaRangeReduced <- sigest(as.matrix(training[,reducedSet]))
> svmRGridReduced <- expand.grid(sigma = sigmaRangeReduced[1],
+                                C = 2^(seq(-4, 4)))
> set.seed(476)
> svmRFitReduced <- train(x = training[,reducedSet], 
+                         y = training$Class,
+                         method = "svmRadial",
+                         metric = "ROC",
+                         preProc = c("center", "scale"),
+                         tuneGrid = svmRGridReduced,
+                         trControl = ctrl)
> svmRFitReduced
Support Vector Machines with Radial Basis Function Kernel 

8190 samples
 252 predictors
   2 classes: 'successful', 'unsuccessful' 

Pre-processing: centered, scaled 
Resampling: Repeated Train/Test Splits Estimated (1 reps, 0.75%) 

Summary of sample sizes: 6633 

Resampling results across tuning parameters:

  C       ROC    Sens   Spec 
  0.0625  0.866  0.916  0.691
  0.125   0.88   0.86   0.758
  0.25    0.89   0.849  0.781
  0.5     0.894  0.83   0.8  
  1       0.895  0.811  0.815
  2       0.891  0.805  0.83 
  4       0.887  0.805  0.822
  8       0.885  0.798  0.821
  16      0.882  0.8    0.82 

Tuning parameter 'sigma' was held constant at a value of 0.001166986
ROC was used to select the optimal model using  the largest value.
The final values used for the model were sigma = 0.00117 and C = 1. 
> 
> svmPGrid <-  expand.grid(degree = 1:2,
+                          scale = c(0.01, .005),
+                          C = 2^(seq(-6, -2, length = 10)))
> 
> set.seed(476)
> svmPFitFull <- train(x = training[,fullSet], 
+                      y = training$Class,
+                      method = "svmPoly",
+                      metric = "ROC",
+                      preProc = c("center", "scale"),
+                      tuneGrid = svmPGrid,
+                      trControl = ctrl)
> svmPFitFull
Support Vector Machines with Polynomial Kernel 

8190 samples
1070 predictors
   2 classes: 'successful', 'unsuccessful' 

Pre-processing: centered, scaled 
Resampling: Repeated Train/Test Splits Estimated (1 reps, 0.75%) 

Summary of sample sizes: 6633 

Resampling results across tuning parameters:

  degree  scale  C       ROC    Sens   Spec 
  1       0.005  0.0156  0.856  0.886  0.706
  1       0.005  0.0213  0.861  0.87   0.733
  1       0.005  0.0289  0.863  0.868  0.758
  1       0.005  0.0394  0.867  0.863  0.768
  1       0.005  0.0536  0.87   0.863  0.777
  1       0.005  0.0729  0.872  0.856  0.782
  1       0.005  0.0992  0.872  0.84   0.789
  1       0.005  0.135   0.873  0.825  0.798
  1       0.005  0.184   0.872  0.816  0.798
  1       0.005  0.25    0.872  0.814  0.803
  1       0.01   0.0156  0.864  0.868  0.758
  1       0.01   0.0213  0.868  0.865  0.768
  1       0.01   0.0289  0.87   0.861  0.78 
  1       0.01   0.0394  0.872  0.849  0.784
  1       0.01   0.0536  0.873  0.84   0.79 
  1       0.01   0.0729  0.873  0.825  0.798
  1       0.01   0.0992  0.872  0.814  0.801
  1       0.01   0.135   0.871  0.812  0.802
  1       0.01   0.184   0.868  0.812  0.795
  1       0.01   0.25    0.862  0.791  0.795
  2       0.005  0.0156  0.838  0.812  0.752
  2       0.005  0.0213  0.845  0.816  0.766
  2       0.005  0.0289  0.852  0.819  0.776
  2       0.005  0.0394  0.856  0.819  0.78 
  2       0.005  0.0536  0.86   0.814  0.784
  2       0.005  0.0729  0.865  0.825  0.782
  2       0.005  0.0992  0.866  0.823  0.787
  2       0.005  0.135   0.865  0.816  0.788
  2       0.005  0.184   0.862  0.802  0.789
  2       0.005  0.25    0.86   0.807  0.784
  2       0.01   0.0156  0.845  0.816  0.765
  2       0.01   0.0213  0.851  0.811  0.774
  2       0.01   0.0289  0.856  0.811  0.778
  2       0.01   0.0394  0.857  0.812  0.78 
  2       0.01   0.0536  0.855  0.809  0.779
  2       0.01   0.0729  0.854  0.796  0.786
  2       0.01   0.0992  0.854  0.789  0.783
  2       0.01   0.135   0.852  0.788  0.78 
  2       0.01   0.184   0.851  0.782  0.778
  2       0.01   0.25    0.85   0.784  0.78 

ROC was used to select the optimal model using  the largest value.
The final values used for the model were degree = 1, scale = 0.01 and C
 = 0.0729. 
> 
> svmPGrid2 <-  expand.grid(degree = 1:2,
+                           scale = c(0.01, .005),
+                           C = 2^(seq(-6, -2, length = 10)))
> set.seed(476)
> svmPFitReduced <- train(x = training[,reducedSet], 
+                         y = training$Class,
+                         method = "svmPoly",
+                         metric = "ROC",
+                         preProc = c("center", "scale"),
+                         tuneGrid = svmPGrid2,
+                         fit = FALSE,
+                         trControl = ctrl)
line search fails -2.047663 -0.1283902 1.181205e-05 2.17076e-06 -2.621876e-08 -4.051435e-09 -3.184921e-13Warning messages:
1: In nominalTrainWorkflow(x = x, y = y, wts = weights, info = trainInfo,  :
  There were missing values in resampled performance measures.
2: In train.default(x = training[, reducedSet], y = training$Class,  :
  missing values found in aggregated results
> svmPFitReduced
Support Vector Machines with Polynomial Kernel 

8190 samples
 252 predictors
   2 classes: 'successful', 'unsuccessful' 

Pre-processing: centered, scaled 
Resampling: Repeated Train/Test Splits Estimated (1 reps, 0.75%) 

Summary of sample sizes: 6633 

Resampling results across tuning parameters:

  degree  scale  C       ROC    Sens   Spec 
  1       0.005  0.0156  0.867  0.94   0.653
  1       0.005  0.0213  0.875  0.926  0.707
  1       0.005  0.0289  0.881  0.912  0.738
  1       0.005  0.0394  0.887  0.909  0.743
  1       0.005  0.0536  0.892  0.904  0.762
  1       0.005  0.0729  0.895  0.895  0.772
  1       0.005  0.0992  0.897  0.863  0.781
  1       0.005  0.135   0.896  0.854  0.797
  1       0.005  0.184   0.896  0.849  0.804
  1       0.005  0.25    0.895  0.844  0.811
  1       0.01   0.0156  0.883  0.916  0.74 
  1       0.01   0.0213  0.888  0.911  0.749
  1       0.01   0.0289  0.893  0.898  0.762
  1       0.01   0.0394  0.896  0.886  0.775
  1       0.01   0.0536  0.896  0.863  0.785
  1       0.01   0.0729  0.897  0.853  0.8  
  1       0.01   0.0992  0.896  0.847  0.81 
  1       0.01   0.135   0.894  0.844  0.81 
  1       0.01   0.184   0.891  0.837  0.818
  1       0.01   0.25    0.888  0.816  0.825
  2       0.005  0.0156  0.88   0.902  0.746
  2       0.005  0.0213  0.886  0.896  0.759
  2       0.005  0.0289  0.89   0.879  0.774
  2       0.005  0.0394  0.894  0.877  0.777
  2       0.005  0.0536  0.896  0.854  0.794
  2       0.005  0.0729  0.898  0.842  0.805
  2       0.005  0.0992  0.898  0.83   0.815
  2       0.005  0.135   0.896  0.828  0.828
  2       0.005  0.184   0.896  0.819  0.828
  2       0.005  0.25    0.893  0.818  0.828
  2       0.01   0.0156  0.891  0.863  0.781
  2       0.01   0.0213  0.894  0.856  0.788
  2       0.01   0.0289  0.896  0.832  0.803
  2       0.01   0.0394  0.897  0.826  0.81 
  2       0.01   0.0536  0.896  0.833  0.818
  2       0.01   0.0729  0.893  0.819  0.821
  2       0.01   0.0992  NaN    NaN    NaN  
  2       0.01   0.135   0.887  0.802  0.83 
  2       0.01   0.184   0.883  0.804  0.832
  2       0.01   0.25    0.88   0.8    0.831

ROC was used to select the optimal model using  the largest value.
The final values used for the model were degree = 2, scale = 0.005 and C
 = 0.0729. 
> 
> svmPFitReduced$pred <- merge(svmPFitReduced$pred,  svmPFitReduced$bestTune)
> svmPCM <- confusionMatrix(svmPFitReduced, norm = "none")
> svmPRoc <- roc(response = svmPFitReduced$pred$obs,
+                predictor = svmPFitReduced$pred$successful,
+                levels = rev(levels(svmPFitReduced$pred$obs)))
> 
> 
> svmRadialResults <- rbind(svmRFitReduced$results,
+                           svmRFitFull$results)
> svmRadialResults$Set <- c(rep("Reduced Set", nrow(svmRFitReduced$result)),
+                           rep("Full Set", nrow(svmRFitFull$result)))
> svmRadialResults$Sigma <- paste("sigma = ", 
+                                 format(svmRadialResults$sigma, 
+                                        scientific = FALSE, digits= 5))
> svmRadialResults <- svmRadialResults[!is.na(svmRadialResults$ROC),]
> xyplot(ROC ~ C|Set, data = svmRadialResults,
+        groups = Sigma, type = c("g", "o"),
+        xlab = "Cost",
+        ylab = "ROC (2008 Hold-Out Data)",
+        auto.key = list(columns = 2),
+        scales = list(x = list(log = 2)))
> 
> svmPolyResults <- rbind(svmPFitReduced$results,
+                         svmPFitFull$results)
> svmPolyResults$Set <- c(rep("Reduced Set", nrow(svmPFitReduced$result)),
+                         rep("Full Set", nrow(svmPFitFull$result)))
> svmPolyResults <- svmPolyResults[!is.na(svmPolyResults$ROC),]
> svmPolyResults$scale <- paste("scale = ", 
+                               format(svmPolyResults$scale, 
+                                      scientific = FALSE))
> svmPolyResults$Degree <- "Linear"
> svmPolyResults$Degree[svmPolyResults$degree == 2] <- "Quadratic"
> useOuterStrips(xyplot(ROC ~ C|Degree*Set, data = svmPolyResults,
+                       groups = scale, type = c("g", "o"),
+                       xlab = "Cost",
+                       ylab = "ROC (2008 Hold-Out Data)",
+                       auto.key = list(columns = 2),
+                       scales = list(x = list(log = 2))))
> 
> plot(nnetRoc, type = "s", col = rgb(.2, .2, .2, .2), legacy.axes = TRUE)

Call:
roc.default(response = nnetFit4$pred$obs, predictor = nnetFit4$pred$successful,     levels = rev(levels(nnetFit4$pred$obs)))

Data: nnetFit4$pred$successful in 987 controls (nnetFit4$pred$obs unsuccessful) < 570 cases (nnetFit4$pred$obs successful).
Area under the curve: 0.9111
> plot(fdaRoc, type = "s", add = TRUE, col = rgb(.2, .2, .2, .2), legacy.axes = TRUE)

Call:
roc.default(response = fdaFit$pred$obs, predictor = fdaFit$pred$successful,     levels = rev(levels(fdaFit$pred$obs)))

Data: fdaFit$pred$successful in 987 controls (fdaFit$pred$obs unsuccessful) < 570 cases (fdaFit$pred$obs successful).
Area under the curve: 0.924
> plot(svmPRoc, type = "s", add = TRUE, legacy.axes = TRUE)

Call:
roc.default(response = svmPFitReduced$pred$obs, predictor = svmPFitReduced$pred$successful,     levels = rev(levels(svmPFitReduced$pred$obs)))

Data: svmPFitReduced$pred$successful in 987 controls (svmPFitReduced$pred$obs unsuccessful) < 570 cases (svmPFitReduced$pred$obs successful).
Area under the curve: 0.8982
> 
> ################################################################################
> ### Section 13.5 K-Nearest Neighbors
> 
> 
> set.seed(476)
> knnFit <- train(x = training[,reducedSet], 
+                 y = training$Class,
+                 method = "knn",
+                 metric = "ROC",
+                 preProc = c("center", "scale"),
+                 tuneGrid = data.frame(k = c(4*(0:5)+1,20*(1:5)+1,50*(2:9)+1)),
+                 trControl = ctrl)
> knnFit
k-Nearest Neighbors 

8190 samples
 252 predictors
   2 classes: 'successful', 'unsuccessful' 

Pre-processing: centered, scaled 
Resampling: Repeated Train/Test Splits Estimated (1 reps, 0.75%) 

Summary of sample sizes: 6633 

Resampling results across tuning parameters:

  k    ROC    Sens   Spec   ROC SD  Sens SD  Spec SD
  1    0.622  0.547  0.694  NA      NA       NA     
  5    0.7    0.553  0.708  NA      NA       NA     
  9    0.706  0.542  0.746  NA      NA       NA     
  13   0.709  0.558  0.743  NA      NA       NA     
  17   0.711  0.565  0.737  NA      NA       NA     
  21   0.724  0.542  0.739  0       0        0.00143
  41   0.734  0.575  0.757  NA      NA       NA     
  61   0.75   0.556  0.785  NA      NA       NA     
  81   0.762  0.535  0.811  NA      NA       NA     
  101  0.766  0.52   0.825  0       0.00124  0      
  151  0.773  0.454  0.866  NA      NA       NA     
  201  0.779  0.395  0.891  NA      NA       NA     
  251  0.781  0.351  0.897  NA      NA       NA     
  301  0.787  0.333  0.907  NA      NA       NA     
  351  0.792  0.312  0.906  NA      NA       NA     
  401  0.797  0.337  0.905  NA      NA       NA     
  451  0.807  0.353  0.908  NA      NA       NA     

ROC was used to select the optimal model using  the largest value.
The final value used for the model was k = 451. 
> 
> knnFit$pred <- merge(knnFit$pred,  knnFit$bestTune)
> knnCM <- confusionMatrix(knnFit, norm = "none")
> knnCM
Repeated Train/Test Splits Estimated (1 reps, 0.75%) Confusion Matrix 

(entries are un-normalized counts)
 
Confusion Matrix and Statistics

              Reference
Prediction     successful unsuccessful
  successful          201           91
  unsuccessful        369          896
                                          
               Accuracy : 0.7046          
                 95% CI : (0.6812, 0.7271)
    No Information Rate : 0.6339          
    P-Value [Acc > NIR] : 2.461e-09       
                                          
                  Kappa : 0.2903          
 Mcnemar's Test P-Value : < 2.2e-16       
                                          
            Sensitivity : 0.3526          
            Specificity : 0.9078          
         Pos Pred Value : 0.6884          
         Neg Pred Value : 0.7083          
             Prevalence : 0.3661          
         Detection Rate : 0.1291          
   Detection Prevalence : 0.1875          
      Balanced Accuracy : 0.6302          
                                          
       'Positive' Class : successful      
                                          

> knnRoc <- roc(response = knnFit$pred$obs,
+               predictor = knnFit$pred$successful,
+               levels = rev(levels(knnFit$pred$obs)))
> 
> update(plot(knnFit, ylab = "ROC (2008 Hold-Out Data)"))
> 
> plot(fdaRoc, type = "s", col = rgb(.2, .2, .2, .2), legacy.axes = TRUE)

Call:
roc.default(response = fdaFit$pred$obs, predictor = fdaFit$pred$successful,     levels = rev(levels(fdaFit$pred$obs)))

Data: fdaFit$pred$successful in 987 controls (fdaFit$pred$obs unsuccessful) < 570 cases (fdaFit$pred$obs successful).
Area under the curve: 0.924
> plot(nnetRoc, type = "s", add = TRUE, col = rgb(.2, .2, .2, .2), legacy.axes = TRUE)

Call:
roc.default(response = nnetFit4$pred$obs, predictor = nnetFit4$pred$successful,     levels = rev(levels(nnetFit4$pred$obs)))

Data: nnetFit4$pred$successful in 987 controls (nnetFit4$pred$obs unsuccessful) < 570 cases (nnetFit4$pred$obs successful).
Area under the curve: 0.9111
> plot(svmPRoc, type = "s", add = TRUE, col = rgb(.2, .2, .2, .2), legacy.axes = TRUE)

Call:
roc.default(response = svmPFitReduced$pred$obs, predictor = svmPFitReduced$pred$successful,     levels = rev(levels(svmPFitReduced$pred$obs)))

Data: svmPFitReduced$pred$successful in 987 controls (svmPFitReduced$pred$obs unsuccessful) < 570 cases (svmPFitReduced$pred$obs successful).
Area under the curve: 0.8982
> plot(knnRoc, type = "s", add = TRUE, legacy.axes = TRUE)

Call:
roc.default(response = knnFit$pred$obs, predictor = knnFit$pred$successful,     levels = rev(levels(knnFit$pred$obs)))

Data: knnFit$pred$successful in 987 controls (knnFit$pred$obs unsuccessful) < 570 cases (knnFit$pred$obs successful).
Area under the curve: 0.8068
> 
> ################################################################################
> ### Section 13.6 Naive Bayes
> 
> ## Create factor versions of some of the predictors so that they are treated
> ## as categories and not dummy variables
> 
> factors <- c("SponsorCode", "ContractValueBand", "Month", "Weekday")
> nbPredictors <- factorPredictors[factorPredictors %in% reducedSet]
> nbPredictors <- c(nbPredictors, factors)
> nbPredictors <- nbPredictors[nbPredictors != "SponsorUnk"]
> 
> nbTraining <- training[, c("Class", nbPredictors)]
> nbTesting <- testing[, c("Class", nbPredictors)]
> 
> for(i in nbPredictors)
+ {
+   if(length(unique(training[,i])) <= 15)
+   {
+     nbTraining[, i] <- factor(nbTraining[,i], levels = paste(sort(unique(training[,i]))))
+     nbTesting[, i] <- factor(nbTesting[,i], levels = paste(sort(unique(training[,i]))))
+   }
+ }
> 
> set.seed(476)
> nBayesFit <- train(x = nbTraining[,nbPredictors],
+                    y = nbTraining$Class,
+                    method = "nb",
+                    metric = "ROC",
+                    tuneGrid = data.frame(usekernel = c(TRUE, FALSE), fL = 2),
+                    trControl = ctrl)
Loading required package: klaR
Loading required package: MASS
> nBayesFit
Naive Bayes 

8190 samples
 205 predictors
   2 classes: 'successful', 'unsuccessful' 

No pre-processing
Resampling: Repeated Train/Test Splits Estimated (1 reps, 0.75%) 

Summary of sample sizes: 6633 

Resampling results across tuning parameters:

  usekernel  ROC    Sens   Spec 
  FALSE      0.782  0.588  0.796
  TRUE       0.814  0.644  0.824

Tuning parameter 'fL' was held constant at a value of 2
ROC was used to select the optimal model using  the largest value.
The final values used for the model were fL = 2 and usekernel = TRUE. 
> 
> nBayesFit$pred <- merge(nBayesFit$pred,  nBayesFit$bestTune)
> nBayesCM <- confusionMatrix(nBayesFit, norm = "none")
> nBayesCM
Repeated Train/Test Splits Estimated (1 reps, 0.75%) Confusion Matrix 

(entries are un-normalized counts)
 
Confusion Matrix and Statistics

              Reference
Prediction     successful unsuccessful
  successful          367          174
  unsuccessful        203          813
                                         
               Accuracy : 0.7579         
                 95% CI : (0.7358, 0.779)
    No Information Rate : 0.6339         
    P-Value [Acc > NIR] : <2e-16         
                                         
                  Kappa : 0.4726         
 Mcnemar's Test P-Value : 0.1493         
                                         
            Sensitivity : 0.6439         
            Specificity : 0.8237         
         Pos Pred Value : 0.6784         
         Neg Pred Value : 0.8002         
             Prevalence : 0.3661         
         Detection Rate : 0.2357         
   Detection Prevalence : 0.3475         
      Balanced Accuracy : 0.7338         
                                         
       'Positive' Class : successful     
                                         

> nBayesRoc <- roc(response = nBayesFit$pred$obs,
+                  predictor = nBayesFit$pred$successful,
+                  levels = rev(levels(nBayesFit$pred$obs)))
> nBayesRoc

Call:
roc.default(response = nBayesFit$pred$obs, predictor = nBayesFit$pred$successful,     levels = rev(levels(nBayesFit$pred$obs)))

Data: nBayesFit$pred$successful in 987 controls (nBayesFit$pred$obs unsuccessful) < 570 cases (nBayesFit$pred$obs successful).
Area under the curve: 0.8137
> 
> 
> sessionInfo()
R version 3.0.1 (2013-05-16)
Platform: x86_64-apple-darwin10.8.0 (64-bit)

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] parallel  stats     graphics  grDevices utils     datasets  methods  
[8] base     

other attached packages:
 [1] klaR_0.6-7          MASS_7.3-26         kernlab_0.9-16     
 [4] earth_3.2-3         plotrix_3.4-6       plotmo_1.3-2       
 [7] leaps_2.9           latticeExtra_0.6-24 RColorBrewer_1.0-5 
[10] nnet_7.3-6          e1071_1.6-1         pROC_1.5.4         
[13] plyr_1.8            mda_0.4-2           class_7.3-7        
[16] doMC_1.3.0          iterators_1.0.6     foreach_1.4.0      
[19] caret_6.0-22        ggplot2_0.9.3.1     lattice_0.20-15    

loaded via a namespace (and not attached):
 [1] car_2.0-16       codetools_0.2-8  colorspace_1.2-1 compiler_3.0.1  
 [5] dichromat_2.0-0  digest_0.6.3     grid_3.0.1       gtable_0.1.2    
 [9] labeling_0.1     munsell_0.4      proto_0.3-10     reshape2_1.2.2  
[13] scales_0.2.3     stringr_0.6.2   
> 
> q("no")
> proc.time()
     user    system   elapsed 
313451.24   2270.67  52861.72 
