
R version 3.0.1 (2013-05-16) -- "Good Sport"
Copyright (C) 2013 The R Foundation for Statistical Computing
Platform: x86_64-apple-darwin10.8.0 (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

> ################################################################################
> ### R code from Applied Predictive Modeling (2013) by Kuhn and Johnson.
> ### Copyright 2013 Kuhn and Johnson
> ### Web Page: http://www.appliedpredictivemodeling.com
> ### Contact: Max Kuhn (mxkuhn@gmail.com) 
> ###
> ### Chapter 12 Discriminant Analysis and Other Linear Classification Models
> ###
> ### Required packages: AppliedPredictiveModeling, caret, doMC (optional),  
> ###                    glmnet, lattice, MASS, pamr, pls, pROC, sparseLDA
> ###
> ### Data used: The grant application data. See the file 'CreateGrantData.R'
> ###
> ### Notes: 
> ### 1) This code is provided without warranty.
> ###
> ### 2) This code should help the user reproduce the results in the
> ### text. There will be differences between this code and what is is
> ### the computing section. For example, the computing sections show
> ### how the source functions work (e.g. randomForest() or plsr()),
> ### which were not directly used when creating the book. Also, there may be 
> ### syntax differences that occur over time as packages evolve. These files 
> ### will reflect those changes.
> ###
> ### 3) In some cases, the calculations in the book were run in 
> ### parallel. The sub-processes may reset the random number seed.
> ### Your results may slightly vary.
> ###
> ################################################################################
> 
> ################################################################################
> ### Section 12.1 Case Study: Predicting Successful Grant Applications
> 
> load("grantData.RData")
> 
> library(caret)
Loading required package: lattice
Loading required package: ggplot2
> library(doMC)
Loading required package: foreach
Loading required package: iterators
Loading required package: parallel
> registerDoMC(12)
> library(plyr)
> library(reshape2)
> 
> ## Look at two different ways to split and resample the data. A support vector
> ## machine is used to illustrate the differences. The full set of predictors
> ## is used. 
> 
> pre2008Data <- training[pre2008,]
> year2008Data <- rbind(training[-pre2008,], testing)
> 
> set.seed(552)
> test2008 <- createDataPartition(year2008Data$Class, p = .25)[[1]]
> 
> allData <- rbind(pre2008Data, year2008Data[-test2008,])
> holdout2008 <- year2008Data[test2008,]
> 
> ## Use a common tuning grid for both approaches. 
> svmrGrid <- expand.grid(sigma = c(.00007, .00009, .0001, .0002),
+                         C = 2^(-3:8))
> 
> ## Evaluate the model using overall 10-fold cross-validation
> ctrl0 <- trainControl(method = "cv",
+                       summaryFunction = twoClassSummary,
+                       classProbs = TRUE)
> set.seed(477)
> svmFit0 <- train(pre2008Data[,fullSet], pre2008Data$Class,
+                  method = "svmRadial",
+                  tuneGrid = svmrGrid,
+                  preProc = c("center", "scale"),
+                  metric = "ROC",
+                  trControl = ctrl0)
Loading required package: kernlab
Loading required package: pROC
Type 'citation("pROC")' for a citation.

Attaching package: 'pROC'

The following object is masked from 'package:stats':

    cov, smooth, var

> svmFit0
Support Vector Machines with Radial Basis Function Kernel 

6633 samples
1070 predictors
   2 classes: 'successful', 'unsuccessful' 

Pre-processing: centered, scaled 
Resampling: Cross-Validated (10 fold) 

Summary of sample sizes: 5970, 5970, 5969, 5970, 5970, 5969, ... 

Resampling results across tuning parameters:

  sigma  C      ROC    Sens   Spec   ROC SD  Sens SD  Spec SD
  7e-05  0.125  0.806  0.88   0.562  0.0231  0.023    0.0168 
  7e-05  0.25   0.81   0.876  0.574  0.022   0.0254   0.0157 
  7e-05  0.5    0.836  0.837  0.677  0.018   0.029    0.0194 
  7e-05  1      0.853  0.803  0.757  0.0173  0.0308   0.0288 
  7e-05  2      0.863  0.805  0.78   0.0177  0.0275   0.0318 
  7e-05  4      0.869  0.8    0.789  0.0168  0.0279   0.0285 
  7e-05  8      0.874  0.798  0.798  0.0189  0.0313   0.0279 
  7e-05  16     0.876  0.796  0.797  0.0193  0.03     0.0235 
  7e-05  32     0.877  0.793  0.801  0.0184  0.0242   0.0287 
  7e-05  64     0.877  0.793  0.81   0.0178  0.034    0.0182 
  7e-05  128    0.876  0.793  0.812  0.0163  0.0233   0.0164 
  7e-05  256    0.873  0.794  0.812  0.0165  0.0239   0.0162 
  9e-05  0.125  0.8    0.876  0.551  0.0249  0.0209   0.023  
  9e-05  0.25   0.811  0.87   0.581  0.0219  0.0236   0.0186 
  9e-05  0.5    0.842  0.816  0.715  0.018   0.031    0.0258 
  9e-05  1      0.856  0.8    0.769  0.0176  0.0314   0.0306 
  9e-05  2      0.866  0.801  0.785  0.0173  0.0277   0.0315 
  9e-05  4      0.871  0.8    0.792  0.0172  0.0271   0.0269 
  9e-05  8      0.875  0.796  0.796  0.0188  0.0295   0.0259 
  9e-05  16     0.877  0.795  0.8    0.0186  0.0258   0.0246 
  9e-05  32     0.878  0.793  0.804  0.0179  0.0291   0.025  
  9e-05  64     0.877  0.794  0.813  0.0169  0.0297   0.0187 
  9e-05  128    0.876  0.795  0.813  0.0156  0.0228   0.0153 
  9e-05  256    0.874  0.788  0.814  0.0164  0.0205   0.017  
  1e-04  0.125  0.797  0.878  0.546  0.0257  0.0241   0.016  
  1e-04  0.25   0.814  0.863  0.596  0.0212  0.0319   0.0189 
  1e-04  0.5    0.845  0.81   0.728  0.018   0.0296   0.0247 
  1e-04  1      0.857  0.799  0.771  0.0179  0.0321   0.0298 
  1e-04  2      0.867  0.804  0.785  0.0173  0.0285   0.0312 
  1e-04  4      0.872  0.801  0.794  0.0174  0.0279   0.0266 
  1e-04  8      0.875  0.792  0.797  0.0187  0.0304   0.0242 
  1e-04  16     0.878  0.794  0.799  0.0184  0.0249   0.025  
  1e-04  32     0.878  0.795  0.806  0.0179  0.0335   0.0222 
  1e-04  64     0.878  0.796  0.812  0.0163  0.0245   0.0168 
  1e-04  128    0.876  0.796  0.811  0.0159  0.0215   0.0143 
  1e-04  256    0.874  0.788  0.816  0.0165  0.0209   0.0127 
  2e-04  0.125  0.786  0.861  0.542  0.0282  0.0356   0.0198 
  2e-04  0.25   0.836  0.81   0.701  0.0192  0.0382   0.0232 
  2e-04  0.5    0.853  0.792  0.765  0.0177  0.0342   0.0308 
  2e-04  1      0.864  0.8    0.782  0.0177  0.028    0.036  
  2e-04  2      0.87   0.796  0.789  0.0174  0.0258   0.0277 
  2e-04  4      0.875  0.795  0.793  0.0182  0.0295   0.026  
  2e-04  8      0.878  0.793  0.801  0.0176  0.0293   0.0196 
  2e-04  16     0.879  0.796  0.809  0.0167  0.033    0.0203 
  2e-04  32     0.88   0.795  0.811  0.0153  0.0227   0.0169 
  2e-04  64     0.879  0.792  0.813  0.0155  0.0194   0.0171 
  2e-04  128    0.877  0.786  0.816  0.0162  0.0235   0.0128 
  2e-04  256    0.877  0.789  0.822  0.0156  0.0241   0.0159 

ROC was used to select the optimal model using  the largest value.
The final values used for the model were sigma = 2e-04 and C = 32. 
> 
> ### Now fit the single 2008 test set
> ctrl00 <- trainControl(method = "LGOCV",
+                        summaryFunction = twoClassSummary,
+                        classProbs = TRUE,
+                        index = list(TestSet = 1:nrow(pre2008Data)))
> 
> 
> set.seed(476)
> svmFit00 <- train(allData[,fullSet], allData$Class,
+                   method = "svmRadial",
+                   tuneGrid = svmrGrid,
+                   preProc = c("center", "scale"),
+                   metric = "ROC",
+                   trControl = ctrl00)
> svmFit00
Support Vector Machines with Radial Basis Function Kernel 

8189 samples
1070 predictors
   2 classes: 'successful', 'unsuccessful' 

Pre-processing: centered, scaled 
Resampling: Repeated Train/Test Splits Estimated (1 reps, 0.75%) 

Summary of sample sizes: 6633 

Resampling results across tuning parameters:

  sigma  C      ROC    Sens   Spec 
  7e-05  0.125  0.806  0.968  0.494
  7e-05  0.25   0.814  0.965  0.512
  7e-05  0.5    0.855  0.921  0.651
  7e-05  1      0.873  0.882  0.753
  7e-05  2      0.882  0.873  0.783
  7e-05  4      0.886  0.856  0.802
  7e-05  8      0.887  0.835  0.813
  7e-05  16     0.883  0.812  0.813
  7e-05  32     0.875  0.786  0.814
  7e-05  64     0.872  0.794  0.816
  7e-05  128    0.872  0.791  0.807
  7e-05  256    0.869  0.793  0.811
  9e-05  0.125  0.798  0.97   0.478
  9e-05  0.25   0.819  0.96   0.536
  9e-05  0.5    0.864  0.902  0.688
  9e-05  1      0.876  0.868  0.765
  9e-05  2      0.885  0.863  0.785
  9e-05  4      0.888  0.84   0.807
  9e-05  8      0.887  0.822  0.806
  9e-05  16     0.88   0.801  0.816
  9e-05  32     0.874  0.791  0.821
  9e-05  64     0.873  0.8    0.811
  9e-05  128    0.872  0.791  0.812
  9e-05  256    0.865  0.775  0.803
  1e-04  0.125  0.795  0.961  0.476
  1e-04  0.25   0.825  0.946  0.563
  1e-04  0.5    0.867  0.895  0.709
  1e-04  1      0.877  0.87   0.765
  1e-04  2      0.885  0.858  0.786
  1e-04  4      0.888  0.835  0.804
  1e-04  8      0.887  0.819  0.809
  1e-04  16     0.88   0.791  0.818
  1e-04  32     0.875  0.798  0.814
  1e-04  64     0.874  0.796  0.809
  1e-04  128    0.871  0.794  0.807
  1e-04  256    0.863  0.78   0.79 
  2e-04  0.125  0.791  0.942  0.504
  2e-04  0.25   0.86   0.888  0.684
  2e-04  0.5    0.875  0.865  0.752
  2e-04  1      0.884  0.849  0.783
  2e-04  2      0.886  0.833  0.798
  2e-04  4      0.888  0.821  0.803
  2e-04  8      0.883  0.805  0.814
  2e-04  16     0.88   0.803  0.817
  2e-04  32     0.877  0.796  0.816
  2e-04  64     0.868  0.78   0.805
  2e-04  128    0.862  0.779  0.791
  2e-04  256    0.857  0.78   0.779

ROC was used to select the optimal model using  the largest value.
The final values used for the model were sigma = 2e-04 and C = 4. 
> 
> ## Combine the two sets of results and plot
> 
> grid0 <- subset(svmFit0$results,  sigma == svmFit0$bestTune$sigma)
> grid0$Model <- "10-Fold Cross-Validation"
> 
> grid00 <- subset(svmFit00$results,  sigma == svmFit00$bestTune$sigma)
> grid00$Model <- "Single 2008 Test Set"
> 
> plotData <- rbind(grid00, grid0)
> 
> plotData <- plotData[!is.na(plotData$ROC),]
> xyplot(ROC ~ C, data = plotData,
+        groups = Model,
+        type = c("g", "o"),
+        scales = list(x = list(log = 2)),
+        auto.key = list(columns = 1))
> 
> ################################################################################
> ### Section 12.2 Logistic Regression
> 
> modelFit <- glm(Class ~ Day, data = training[pre2008,], family = binomial)
> dataGrid <- data.frame(Day = seq(0, 365, length = 500))
> dataGrid$Linear <- 1 - predict(modelFit, dataGrid, type = "response")
> linear2008 <- auc(roc(response = training[-pre2008, "Class"],
+                       predictor = 1 - predict(modelFit, 
+                                               training[-pre2008,], 
+                                               type = "response"),
+                       levels = rev(levels(training[-pre2008, "Class"]))))
> 
> 
> modelFit2 <- glm(Class ~ Day + I(Day^2), 
+                  data = training[pre2008,], 
+                  family = binomial)
> dataGrid$Quadratic <- 1 - predict(modelFit2, dataGrid, type = "response")
> quad2008 <- auc(roc(response = training[-pre2008, "Class"],
+                     predictor = 1 - predict(modelFit2, 
+                                             training[-pre2008,], 
+                                             type = "response"),
+                     levels = rev(levels(training[-pre2008, "Class"]))))
> 
> dataGrid <- melt(dataGrid, id.vars = "Day")
> 
> byDay <- training[pre2008, c("Day", "Class")]
> byDay$Binned <- cut(byDay$Day, seq(0, 360, by = 5))
> 
> observedProps <- ddply(byDay, .(Binned),
+                        function(x) c(n = nrow(x), mean = mean(x$Class == "successful")))
> observedProps$midpoint <- seq(2.5, 357.5, by = 5)
> 
> xyplot(value ~ Day|variable, data = dataGrid,
+        ylab = "Probability of A Successful Grant",
+        ylim = extendrange(0:1),
+        between = list(x = 1),
+        panel = function(...)
+        {
+          panel.xyplot(x = observedProps$midpoint, observedProps$mean,
+                       pch = 16., col = rgb(.2, .2, .2, .5))
+          panel.xyplot(..., type = "l", col = "black", lwd = 2)
+        })
> 
> ## For the reduced set of factors, fit the logistic regression model (linear and
> ## quadratic) and evaluate on the 
> training$Day2 <- training$Day^2
> testing$Day2 <- testing$Day^2
> fullSet <- c(fullSet, "Day2")
> reducedSet <- c(reducedSet, "Day2")
> 
> ## This control object will be used across multiple models so that the
> ## data splitting is consistent
> 
> ctrl <- trainControl(method = "LGOCV",
+                      summaryFunction = twoClassSummary,
+                      classProbs = TRUE,
+                      index = list(TrainSet = pre2008),
+                      savePredictions = TRUE)
> 
> set.seed(476)
> lrFit <- train(x = training[,reducedSet], 
+                y = training$Class,
+                method = "glm",
+                metric = "ROC",
+                trControl = ctrl)
> lrFit
Generalized Linear Model 

8190 samples
 253 predictors
   2 classes: 'successful', 'unsuccessful' 

No pre-processing
Resampling: Repeated Train/Test Splits Estimated (1 reps, 0.75%) 

Summary of sample sizes: 6633 

Resampling results

  ROC    Sens   Spec 
  0.872  0.804  0.822

 
> set.seed(476)
> lrFit2 <- train(x = training[,c(fullSet, "Day2")], 
+                 y = training$Class,
+                 method = "glm",
+                 metric = "ROC",
+                 trControl = ctrl)
Warning messages:
1: glm.fit: algorithm did not converge 
2: glm.fit: fitted probabilities numerically 0 or 1 occurred 
3: In predict.lm(object, newdata, se.fit, scale = 1, type = ifelse(type ==  :
  prediction from a rank-deficient fit may be misleading
4: In predict.lm(object, newdata, se.fit, scale = 1, type = ifelse(type ==  :
  prediction from a rank-deficient fit may be misleading
5: glm.fit: fitted probabilities numerically 0 or 1 occurred 
> lrFit2
Generalized Linear Model 

8190 samples
1072 predictors
   2 classes: 'successful', 'unsuccessful' 

No pre-processing
Resampling: Repeated Train/Test Splits Estimated (1 reps, 0.75%) 

Summary of sample sizes: 6633 

Resampling results

  ROC    Sens  Spec 
  0.782  0.77  0.761

 
> 
> lrFit$pred <- merge(lrFit$pred,  lrFit$bestTune)
> 
> ## Get the confusion matrices for the hold-out set
> lrCM <- confusionMatrix(lrFit, norm = "none")
> lrCM
Repeated Train/Test Splits Estimated (1 reps, 0.75%) Confusion Matrix 

(entries are un-normalized counts)
 
Loading required package: class
Confusion Matrix and Statistics

              Reference
Prediction     successful unsuccessful
  successful          458          176
  unsuccessful        112          811
                                         
               Accuracy : 0.815          
                 95% CI : (0.7948, 0.834)
    No Information Rate : 0.6339         
    P-Value [Acc > NIR] : < 2.2e-16      
                                         
                  Kappa : 0.6107         
 Mcnemar's Test P-Value : 0.0002054      
                                         
            Sensitivity : 0.8035         
            Specificity : 0.8217         
         Pos Pred Value : 0.7224         
         Neg Pred Value : 0.8787         
             Prevalence : 0.3661         
         Detection Rate : 0.2942         
   Detection Prevalence : 0.4072         
      Balanced Accuracy : 0.8126         
                                         
       'Positive' Class : successful     
                                         

> lrCM2 <- confusionMatrix(lrFit2, norm = "none")
> lrCM2
Repeated Train/Test Splits Estimated (1 reps, 0.75%) Confusion Matrix 

(entries are un-normalized counts)
 
Confusion Matrix and Statistics

              Reference
Prediction     successful unsuccessful
  successful          439          236
  unsuccessful        131          751
                                          
               Accuracy : 0.7643          
                 95% CI : (0.7424, 0.7852)
    No Information Rate : 0.6339          
    P-Value [Acc > NIR] : < 2.2e-16       
                                          
                  Kappa : 0.5112          
 Mcnemar's Test P-Value : 5.675e-08       
                                          
            Sensitivity : 0.7702          
            Specificity : 0.7609          
         Pos Pred Value : 0.6504          
         Neg Pred Value : 0.8515          
             Prevalence : 0.3661          
         Detection Rate : 0.2820          
   Detection Prevalence : 0.4335          
      Balanced Accuracy : 0.7655          
                                          
       'Positive' Class : successful      
                                          

> 
> ## Get the area under the ROC curve for the hold-out set
> lrRoc <- roc(response = lrFit$pred$obs,
+              predictor = lrFit$pred$successful,
+              levels = rev(levels(lrFit$pred$obs)))
> lrRoc2 <- roc(response = lrFit2$pred$obs,
+               predictor = lrFit2$pred$successful,
+               levels = rev(levels(lrFit2$pred$obs)))
> lrImp <- varImp(lrFit, scale = FALSE)
> 
> plot(lrRoc, legacy.axes = TRUE)

Call:
roc.default(response = lrFit$pred$obs, predictor = lrFit$pred$successful,     levels = rev(levels(lrFit$pred$obs)))

Data: lrFit$pred$successful in 987 controls (lrFit$pred$obs unsuccessful) < 570 cases (lrFit$pred$obs successful).
Area under the curve: 0.8715
> 
> ################################################################################
> ### Section 12.3 Linear Discriminant Analysis
> 
> ## Fit the model to the reduced set
> set.seed(476)
> ldaFit <- train(x = training[,reducedSet], 
+                 y = training$Class,
+                 method = "lda",
+                 preProc = c("center","scale"),
+                 metric = "ROC",
+                 trControl = ctrl)
Loading required package: MASS
> ldaFit
Linear Discriminant Analysis 

8190 samples
 253 predictors
   2 classes: 'successful', 'unsuccessful' 

Pre-processing: centered, scaled 
Resampling: Repeated Train/Test Splits Estimated (1 reps, 0.75%) 

Summary of sample sizes: 6633 

Resampling results

  ROC    Sens   Spec 
  0.889  0.804  0.823

 
> 
> ldaFit$pred <- merge(ldaFit$pred,  ldaFit$bestTune)
> ldaCM <- confusionMatrix(ldaFit, norm = "none")
> ldaCM
Repeated Train/Test Splits Estimated (1 reps, 0.75%) Confusion Matrix 

(entries are un-normalized counts)
 
Confusion Matrix and Statistics

              Reference
Prediction     successful unsuccessful
  successful          458          175
  unsuccessful        112          812
                                          
               Accuracy : 0.8157          
                 95% CI : (0.7955, 0.8346)
    No Information Rate : 0.6339          
    P-Value [Acc > NIR] : < 2.2e-16       
                                          
                  Kappa : 0.6119          
 Mcnemar's Test P-Value : 0.0002525       
                                          
            Sensitivity : 0.8035          
            Specificity : 0.8227          
         Pos Pred Value : 0.7235          
         Neg Pred Value : 0.8788          
             Prevalence : 0.3661          
         Detection Rate : 0.2942          
   Detection Prevalence : 0.4066          
      Balanced Accuracy : 0.8131          
                                          
       'Positive' Class : successful      
                                          

> ldaRoc <- roc(response = ldaFit$pred$obs,
+               predictor = ldaFit$pred$successful,
+               levels = rev(levels(ldaFit$pred$obs)))
> plot(lrRoc, type = "s", col = rgb(.2, .2, .2, .2), legacy.axes = TRUE)

Call:
roc.default(response = lrFit$pred$obs, predictor = lrFit$pred$successful,     levels = rev(levels(lrFit$pred$obs)))

Data: lrFit$pred$successful in 987 controls (lrFit$pred$obs unsuccessful) < 570 cases (lrFit$pred$obs successful).
Area under the curve: 0.8715
> plot(ldaRoc, add = TRUE, type = "s", legacy.axes = TRUE)

Call:
roc.default(response = ldaFit$pred$obs, predictor = ldaFit$pred$successful,     levels = rev(levels(ldaFit$pred$obs)))

Data: ldaFit$pred$successful in 987 controls (ldaFit$pred$obs unsuccessful) < 570 cases (ldaFit$pred$obs successful).
Area under the curve: 0.8892
> 
> ################################################################################
> ### Section 12.4 Partial Least Squares Discriminant Analysis
> 
> ## This model uses all of the predictors
> set.seed(476)
> plsFit <- train(x = training[,fullSet], 
+                 y = training$Class,
+                 method = "pls",
+                 tuneGrid = expand.grid(ncomp = 1:10),
+                 preProc = c("center","scale"),
+                 metric = "ROC",
+                 probMethod = "Bayes",
+                 trControl = ctrl)
Loading required package: pls

Attaching package: 'pls'

The following object is masked from 'package:caret':

    R2

The following object is masked from 'package:stats':

    loadings

> plsFit
Partial Least Squares 

8190 samples
1071 predictors
   2 classes: 'successful', 'unsuccessful' 

Pre-processing: centered, scaled 
Resampling: Repeated Train/Test Splits Estimated (1 reps, 0.75%) 

Summary of sample sizes: 6633 

Resampling results across tuning parameters:

  ncomp  ROC    Sens   Spec 
  1      0.821  0.863  0.667
  2      0.847  0.83   0.749
  3      0.863  0.851  0.749
  4      0.863  0.835  0.754
  5      0.864  0.839  0.77 
  6      0.87   0.837  0.77 
  7      0.865  0.816  0.776
  8      0.862  0.816  0.779
  9      0.864  0.825  0.778
  10     0.858  0.812  0.782

ROC was used to select the optimal model using  the largest value.
The final value used for the model was ncomp = 6. 
> 
> plsImpGrant <- varImp(plsFit, scale = FALSE)
> 
> bestPlsNcomp <- plsFit$results[best(plsFit$results, "ROC", maximize = TRUE), "ncomp"]
> bestPlsROC <- plsFit$results[best(plsFit$results, "ROC", maximize = TRUE), "ROC"]
> 
> ## Only keep the final tuning parameter data
> plsFit$pred <- merge(plsFit$pred,  plsFit$bestTune)
> 
> plsRoc <- roc(response = plsFit$pred$obs,
+               predictor = plsFit$pred$successful,
+               levels = rev(levels(plsFit$pred$obs)))
> 
> ### PLS confusion matrix information
> plsCM <- confusionMatrix(plsFit, norm = "none")
> plsCM
Repeated Train/Test Splits Estimated (1 reps, 0.75%) Confusion Matrix 

(entries are un-normalized counts)
 
Confusion Matrix and Statistics

              Reference
Prediction     successful unsuccessful
  successful          477          227
  unsuccessful         93          760
                                          
               Accuracy : 0.7945          
                 95% CI : (0.7735, 0.8143)
    No Information Rate : 0.6339          
    P-Value [Acc > NIR] : < 2.2e-16       
                                          
                  Kappa : 0.5781          
 Mcnemar's Test P-Value : 1.046e-13       
                                          
            Sensitivity : 0.8368          
            Specificity : 0.7700          
         Pos Pred Value : 0.6776          
         Neg Pred Value : 0.8910          
             Prevalence : 0.3661          
         Detection Rate : 0.3064          
   Detection Prevalence : 0.4522          
      Balanced Accuracy : 0.8034          
                                          
       'Positive' Class : successful      
                                          

> 
> ## Now fit a model that uses a smaller set of predictors chosen by unsupervised 
> ## filtering. 
> 
> set.seed(476)
> plsFit2 <- train(x = training[,reducedSet], 
+                  y = training$Class,
+                  method = "pls",
+                  tuneGrid = expand.grid(ncomp = 1:10),
+                  preProc = c("center","scale"),
+                  metric = "ROC",
+                  probMethod = "Bayes",
+                  trControl = ctrl)
> plsFit2
Partial Least Squares 

8190 samples
 253 predictors
   2 classes: 'successful', 'unsuccessful' 

Pre-processing: centered, scaled 
Resampling: Repeated Train/Test Splits Estimated (1 reps, 0.75%) 

Summary of sample sizes: 6633 

Resampling results across tuning parameters:

  ncomp  ROC    Sens   Spec 
  1      0.836  0.912  0.616
  2      0.868  0.858  0.752
  3      0.889  0.874  0.762
  4      0.895  0.86   0.777
  5      0.895  0.846  0.79 
  6      0.894  0.832  0.795
  7      0.89   0.823  0.806
  8      0.888  0.83   0.803
  9      0.887  0.83   0.803
  10     0.884  0.821  0.807

ROC was used to select the optimal model using  the largest value.
The final value used for the model was ncomp = 4. 
> 
> bestPlsNcomp2 <- plsFit2$results[best(plsFit2$results, "ROC", maximize = TRUE), "ncomp"]
> bestPlsROC2 <- plsFit2$results[best(plsFit2$results, "ROC", maximize = TRUE), "ROC"]
> 
> plsFit2$pred <- merge(plsFit2$pred,  plsFit2$bestTune)
> 
> plsRoc2 <- roc(response = plsFit2$pred$obs,
+                predictor = plsFit2$pred$successful,
+                levels = rev(levels(plsFit2$pred$obs)))
> plsCM2 <- confusionMatrix(plsFit2, norm = "none")
> plsCM2
Repeated Train/Test Splits Estimated (1 reps, 0.75%) Confusion Matrix 

(entries are un-normalized counts)
 
Confusion Matrix and Statistics

              Reference
Prediction     successful unsuccessful
  successful          490          220
  unsuccessful         80          767
                                          
               Accuracy : 0.8073          
                 95% CI : (0.7868, 0.8266)
    No Information Rate : 0.6339          
    P-Value [Acc > NIR] : < 2.2e-16       
                                          
                  Kappa : 0.6053          
 Mcnemar's Test P-Value : 1.014e-15       
                                          
            Sensitivity : 0.8596          
            Specificity : 0.7771          
         Pos Pred Value : 0.6901          
         Neg Pred Value : 0.9055          
             Prevalence : 0.3661          
         Detection Rate : 0.3147          
   Detection Prevalence : 0.4560          
      Balanced Accuracy : 0.8184          
                                          
       'Positive' Class : successful      
                                          

> 
> pls.ROC <- cbind(plsFit$results,Descriptors="Full Set")
> pls2.ROC <- cbind(plsFit2$results,Descriptors="Reduced Set")
> 
> plsCompareROC <- data.frame(rbind(pls.ROC,pls2.ROC))
> 
> xyplot(ROC ~ ncomp,
+        data = plsCompareROC,
+        xlab = "# Components",
+        ylab = "ROC (2008 Hold-Out Data)",
+        auto.key = list(columns = 2),
+        groups = Descriptors,
+        type = c("o", "g"))
> 
> ## Plot ROC curves and variable importance scores
> plot(ldaRoc, type = "s", col = rgb(.2, .2, .2, .2), legacy.axes = TRUE)

Call:
roc.default(response = ldaFit$pred$obs, predictor = ldaFit$pred$successful,     levels = rev(levels(ldaFit$pred$obs)))

Data: ldaFit$pred$successful in 987 controls (ldaFit$pred$obs unsuccessful) < 570 cases (ldaFit$pred$obs successful).
Area under the curve: 0.8892
> plot(lrRoc, type = "s", col = rgb(.2, .2, .2, .2), add = TRUE, legacy.axes = TRUE)

Call:
roc.default(response = lrFit$pred$obs, predictor = lrFit$pred$successful,     levels = rev(levels(lrFit$pred$obs)))

Data: lrFit$pred$successful in 987 controls (lrFit$pred$obs unsuccessful) < 570 cases (lrFit$pred$obs successful).
Area under the curve: 0.8715
> plot(plsRoc2, type = "s", add = TRUE, legacy.axes = TRUE)

Call:
roc.default(response = plsFit2$pred$obs, predictor = plsFit2$pred$successful,     levels = rev(levels(plsFit2$pred$obs)))

Data: plsFit2$pred$successful in 987 controls (plsFit2$pred$obs unsuccessful) < 570 cases (plsFit2$pred$obs successful).
Area under the curve: 0.895
> 
> plot(plsImpGrant, top=20, scales = list(y = list(cex = .95)))
> 
> ################################################################################
> ### Section 12.5 Penalized Models
> 
> ## The glmnet model
> glmnGrid <- expand.grid(alpha = c(0,  .1,  .2, .4, .6, .8, 1),
+                         lambda = seq(.01, .2, length = 40))
> set.seed(476)
> glmnFit <- train(x = training[,fullSet], 
+                  y = training$Class,
+                  method = "glmnet",
+                  tuneGrid = glmnGrid,
+                  preProc = c("center", "scale"),
+                  metric = "ROC",
+                  trControl = ctrl)
Loading required package: glmnet
Loading required package: Matrix
Loaded glmnet 1.9-3


Attaching package: 'glmnet'

The following object is masked from 'package:pROC':

    auc

> glmnFit
glmnet 

8190 samples
1071 predictors
   2 classes: 'successful', 'unsuccessful' 

Pre-processing: centered, scaled 
Resampling: Repeated Train/Test Splits Estimated (1 reps, 0.75%) 

Summary of sample sizes: 6633 

Resampling results across tuning parameters:

  alpha  lambda  ROC    Sens   Spec 
  0      0.01    0.856  0.8    0.791
  0      0.0149  0.856  0.8    0.791
  0      0.0197  0.856  0.8    0.791
  0      0.0246  0.858  0.804  0.796
  0      0.0295  0.86   0.804  0.801
  0      0.0344  0.861  0.802  0.8  
  0      0.0392  0.862  0.804  0.801
  0      0.0441  0.863  0.804  0.801
  0      0.049   0.863  0.798  0.8  
  0      0.0538  0.864  0.807  0.799
  0      0.0587  0.866  0.809  0.796
  0      0.0636  0.864  0.807  0.797
  0      0.0685  0.866  0.809  0.797
  0      0.0733  0.865  0.807  0.797
  0      0.0782  0.866  0.811  0.794
  0      0.0831  0.867  0.814  0.792
  0      0.0879  0.866  0.814  0.792
  0      0.0928  0.866  0.816  0.792
  0      0.0977  0.867  0.816  0.793
  0      0.103   0.867  0.819  0.791
  0      0.107   0.867  0.818  0.79 
  0      0.112   0.867  0.819  0.79 
  0      0.117   0.866  0.819  0.789
  0      0.122   0.866  0.816  0.791
  0      0.127   0.867  0.821  0.794
  0      0.132   0.866  0.819  0.791
  0      0.137   0.866  0.823  0.789
  0      0.142   0.867  0.823  0.789
  0      0.146   0.866  0.818  0.792
  0      0.151   0.865  0.818  0.792
  0      0.156   0.866  0.821  0.789
  0      0.161   0.866  0.819  0.79 
  0      0.166   0.866  0.825  0.788
  0      0.171   0.867  0.825  0.788
  0      0.176   0.866  0.821  0.792
  0      0.181   0.865  0.823  0.791
  0      0.185   0.866  0.826  0.788
  0      0.19    0.865  0.825  0.788
  0      0.195   0.866  0.821  0.791
  0      0.2     0.865  0.823  0.789
  0.1    0.01    0.866  0.809  0.797
  0.1    0.0149  0.874  0.823  0.798
  0.1    0.0197  0.881  0.826  0.797
  0.1    0.0246  0.886  0.828  0.803
  0.1    0.0295  0.89   0.835  0.803
  0.1    0.0344  0.892  0.839  0.81 
  0.1    0.0392  0.895  0.84   0.811
  0.1    0.0441  0.898  0.851  0.809
  0.1    0.049   0.9    0.851  0.809
  0.1    0.0538  0.9    0.853  0.814
  0.1    0.0587  0.902  0.858  0.805
  0.1    0.0636  0.903  0.86   0.809
  0.1    0.0685  0.904  0.868  0.803
  0.1    0.0733  0.906  0.874  0.799
  0.1    0.0782  0.906  0.87   0.802
  0.1    0.0831  0.906  0.872  0.801
  0.1    0.0879  0.907  0.877  0.8  
  0.1    0.0928  0.908  0.877  0.794
  0.1    0.0977  0.908  0.879  0.795
  0.1    0.103   0.907  0.877  0.795
  0.1    0.107   0.907  0.881  0.792
  0.1    0.112   0.907  0.881  0.797
  0.1    0.117   0.907  0.884  0.795
  0.1    0.122   0.908  0.886  0.791
  0.1    0.127   0.907  0.882  0.791
  0.1    0.132   0.909  0.884  0.79 
  0.1    0.137   0.908  0.886  0.789
  0.1    0.142   0.908  0.884  0.786
  0.1    0.146   0.909  0.886  0.784
  0.1    0.151   0.908  0.881  0.787
  0.1    0.156   0.908  0.881  0.785
  0.1    0.161   0.909  0.884  0.787
  0.1    0.166   0.908  0.882  0.787
  0.1    0.171   0.91   0.889  0.785
  0.1    0.176   0.91   0.889  0.784
  0.1    0.181   0.91   0.889  0.785
  0.1    0.185   0.909  0.886  0.788
  0.1    0.19    0.91   0.893  0.778
  0.1    0.195   0.909  0.889  0.784
  0.1    0.2     0.909  0.891  0.781
  0.2    0.01    0.878  0.83   0.8  
  0.2    0.0149  0.887  0.835  0.803
  0.2    0.0197  0.891  0.839  0.804
  0.2    0.0246  0.896  0.849  0.802
  0.2    0.0295  0.899  0.853  0.805
  0.2    0.0344  0.902  0.858  0.8  
  0.2    0.0392  0.902  0.86   0.798
  0.2    0.0441  0.903  0.874  0.794
  0.2    0.049   0.904  0.879  0.802
  0.2    0.0538  0.904  0.879  0.794
  0.2    0.0587  0.905  0.881  0.797
  0.2    0.0636  0.904  0.881  0.8  
  0.2    0.0685  0.905  0.888  0.793
  0.2    0.0733  0.907  0.888  0.793
  0.2    0.0782  0.905  0.886  0.792
  0.2    0.0831  0.906  0.884  0.793
  0.2    0.0879  0.907  0.886  0.788
  0.2    0.0928  0.905  0.882  0.789
  0.2    0.0977  0.906  0.881  0.791
  0.2    0.103   0.906  0.888  0.777
  0.2    0.107   0.907  0.889  0.778
  0.2    0.112   0.906  0.884  0.774
  0.2    0.117   0.905  0.882  0.777
  0.2    0.122   0.905  0.881  0.779
  0.2    0.127   0.905  0.879  0.778
  0.2    0.132   0.905  0.884  0.772
  0.2    0.137   0.905  0.884  0.77 
  0.2    0.142   0.904  0.877  0.779
  0.2    0.146   0.904  0.879  0.773
  0.2    0.151   0.905  0.884  0.77 
  0.2    0.156   0.904  0.879  0.778
  0.2    0.161   0.905  0.886  0.768
  0.2    0.166   0.905  0.898  0.761
  0.2    0.171   0.904  0.891  0.766
  0.2    0.176   0.904  0.884  0.775
  0.2    0.181   0.903  0.875  0.772
  0.2    0.185   0.905  0.898  0.759
  0.2    0.19    0.904  0.886  0.764
  0.2    0.195   0.903  0.879  0.772
  0.2    0.2     0.903  0.888  0.765
  0.4    0.01    0.887  0.84   0.798
  0.4    0.0149  0.893  0.853  0.796
  0.4    0.0197  0.896  0.858  0.795
  0.4    0.0246  0.897  0.863  0.796
  0.4    0.0295  0.897  0.87   0.793
  0.4    0.0344  0.897  0.875  0.786
  0.4    0.0392  0.897  0.868  0.799
  0.4    0.0441  0.898  0.875  0.793
  0.4    0.049   0.898  0.874  0.79 
  0.4    0.0538  0.898  0.874  0.794
  0.4    0.0587  0.897  0.874  0.78 
  0.4    0.0636  0.897  0.875  0.778
  0.4    0.0685  0.9    0.881  0.766
  0.4    0.0733  0.898  0.879  0.767
  0.4    0.0782  0.899  0.882  0.76 
  0.4    0.0831  0.9    0.879  0.765
  0.4    0.0879  0.899  0.877  0.765
  0.4    0.0928  0.902  0.888  0.758
  0.4    0.0977  0.902  0.888  0.756
  0.4    0.103   0.901  0.882  0.765
  0.4    0.107   0.902  0.886  0.769
  0.4    0.112   0.902  0.886  0.764
  0.4    0.117   0.904  0.9    0.757
  0.4    0.122   0.904  0.9    0.749
  0.4    0.127   0.903  0.902  0.748
  0.4    0.132   0.903  0.904  0.743
  0.4    0.137   0.901  0.893  0.747
  0.4    0.142   0.903  0.9    0.747
  0.4    0.146   0.9    0.914  0.719
  0.4    0.151   0.902  0.926  0.708
  0.4    0.156   0.901  0.944  0.699
  0.4    0.161   0.896  0.902  0.716
  0.4    0.166   0.897  0.912  0.716
  0.4    0.171   0.901  0.935  0.705
  0.4    0.176   0.898  0.954  0.688
  0.4    0.181   0.894  0.951  0.686
  0.4    0.185   0.891  0.919  0.7  
  0.4    0.19    0.877  0.891  0.693
  0.4    0.195   0.877  0.926  0.676
  0.4    0.2     0.881  0.94   0.674
  0.6    0.01    0.889  0.842  0.803
  0.6    0.0149  0.892  0.846  0.8  
  0.6    0.0197  0.892  0.863  0.793
  0.6    0.0246  0.893  0.868  0.787
  0.6    0.0295  0.893  0.865  0.785
  0.6    0.0344  0.894  0.875  0.78 
  0.6    0.0392  0.893  0.874  0.777
  0.6    0.0441  0.894  0.872  0.779
  0.6    0.049   0.893  0.865  0.775
  0.6    0.0538  0.897  0.879  0.767
  0.6    0.0587  0.897  0.875  0.765
  0.6    0.0636  0.9    0.879  0.76 
  0.6    0.0685  0.899  0.879  0.764
  0.6    0.0733  0.901  0.889  0.756
  0.6    0.0782  0.902  0.889  0.756
  0.6    0.0831  0.901  0.895  0.747
  0.6    0.0879  0.903  0.907  0.737
  0.6    0.0928  0.9    0.896  0.744
  0.6    0.0977  0.899  0.898  0.739
  0.6    0.103   0.902  0.918  0.721
  0.6    0.107   0.901  0.944  0.7  
  0.6    0.112   0.902  0.93   0.708
  0.6    0.117   0.89   0.909  0.702
  0.6    0.122   0.894  0.939  0.693
  0.6    0.127   0.888  0.968  0.669
  0.6    0.132   0.876  0.9    0.688
  0.6    0.137   0.876  0.916  0.679
  0.6    0.142   0.869  0.981  0.652
  0.6    0.146   0.87   0.977  0.656
  0.6    0.151   0.868  0.991  0.643
  0.6    0.156   0.868  0.984  0.648
  0.6    0.161   0.869  0.988  0.644
  0.6    0.166   0.872  0.991  0.638
  0.6    0.171   0.869  0.991  0.638
  0.6    0.176   0.869  0.991  0.638
  0.6    0.181   0.872  0.991  0.638
  0.6    0.185   0.823  0.991  0.638
  0.6    0.19    0.823  0.991  0.638
  0.6    0.195   0.823  0.991  0.638
  0.6    0.2     0.823  0.991  0.638
  0.8    0.01    0.89   0.842  0.801
  0.8    0.0149  0.89   0.858  0.795
  0.8    0.0197  0.888  0.861  0.784
  0.8    0.0246  0.89   0.867  0.777
  0.8    0.0295  0.89   0.87   0.773
  0.8    0.0344  0.892  0.867  0.775
  0.8    0.0392  0.892  0.87   0.766
  0.8    0.0441  0.895  0.86   0.768
  0.8    0.049   0.896  0.868  0.767
  0.8    0.0538  0.898  0.884  0.76 
  0.8    0.0587  0.899  0.882  0.76 
  0.8    0.0636  0.898  0.872  0.759
  0.8    0.0685  0.901  0.904  0.74 
  0.8    0.0733  0.902  0.918  0.723
  0.8    0.0782  0.897  0.898  0.72 
  0.8    0.0831  0.901  0.937  0.706
  0.8    0.0879  0.897  0.953  0.683
  0.8    0.0928  0.892  0.914  0.702
  0.8    0.0977  0.877  0.904  0.688
  0.8    0.103   0.881  0.954  0.671
  0.8    0.107   0.868  0.981  0.652
  0.8    0.112   0.868  0.974  0.656
  0.8    0.117   0.868  0.986  0.646
  0.8    0.122   0.868  0.982  0.648
  0.8    0.127   0.872  0.991  0.638
  0.8    0.132   0.868  0.991  0.638
  0.8    0.137   0.823  0.991  0.638
  0.8    0.142   0.823  0.991  0.638
  0.8    0.146   0.823  0.991  0.638
  0.8    0.151   0.823  0.991  0.638
  0.8    0.156   0.823  0.991  0.638
  0.8    0.161   0.823  0.991  0.638
  0.8    0.166   0.815  0.991  0.638
  0.8    0.171   0.815  0.991  0.638
  0.8    0.176   0.815  0.991  0.638
  0.8    0.181   0.815  0.991  0.638
  0.8    0.185   0.815  0.991  0.638
  0.8    0.19    0.815  0.991  0.638
  0.8    0.195   0.815  0.991  0.638
  0.8    0.2     0.815  0.991  0.638
  1      0.01    0.889  0.854  0.799
  1      0.0149  0.888  0.858  0.791
  1      0.0197  0.887  0.858  0.774
  1      0.0246  0.887  0.854  0.775
  1      0.0295  0.888  0.851  0.775
  1      0.0344  0.891  0.861  0.772
  1      0.0392  0.897  0.881  0.761
  1      0.0441  0.897  0.882  0.761
  1      0.049   0.898  0.889  0.751
  1      0.0538  0.9    0.891  0.752
  1      0.0587  0.899  0.904  0.737
  1      0.0636  0.901  0.926  0.708
  1      0.0685  0.897  0.949  0.688
  1      0.0733  0.898  0.94   0.693
  1      0.0782  0.891  0.965  0.671
  1      0.0831  0.868  0.949  0.666
  1      0.0879  0.868  0.944  0.667
  1      0.0928  0.867  0.991  0.643
  1      0.0977  0.867  0.991  0.643
  1      0.103   0.872  0.991  0.638
  1      0.107   0.867  0.991  0.638
  1      0.112   0.823  0.991  0.638
  1      0.117   0.823  0.991  0.638
  1      0.122   0.823  0.991  0.638
  1      0.127   0.823  0.991  0.638
  1      0.132   0.815  0.991  0.638
  1      0.137   0.815  0.991  0.638
  1      0.142   0.815  0.991  0.638
  1      0.146   0.815  0.991  0.638
  1      0.151   0.815  0.991  0.638
  1      0.156   0.815  0.991  0.638
  1      0.161   0.815  0.991  0.638
  1      0.166   0.815  0.991  0.638
  1      0.171   0.815  0.991  0.638
  1      0.176   0.815  0.991  0.638
  1      0.181   0.815  0.991  0.638
  1      0.185   0.815  0.991  0.638
  1      0.19    0.815  0.991  0.638
  1      0.195   0.815  0      1    
  1      0.2     0.815  0      1    

ROC was used to select the optimal model using  the largest value.
The final values used for the model were alpha = 0.1 and lambda = 0.176. 
> 
> glmnet2008 <- merge(glmnFit$pred,  glmnFit$bestTune)
> glmnetCM <- confusionMatrix(glmnFit, norm = "none")
> glmnetCM
Repeated Train/Test Splits Estimated (1 reps, 0.75%) Confusion Matrix 

(entries are un-normalized counts)
 
Confusion Matrix and Statistics

              Reference
Prediction     successful unsuccessful
  successful          507          213
  unsuccessful         63          774
                                          
               Accuracy : 0.8227          
                 95% CI : (0.8028, 0.8414)
    No Information Rate : 0.6339          
    P-Value [Acc > NIR] : < 2.2e-16       
                                          
                  Kappa : 0.6382          
 Mcnemar's Test P-Value : < 2.2e-16       
                                          
            Sensitivity : 0.8895          
            Specificity : 0.7842          
         Pos Pred Value : 0.7042          
         Neg Pred Value : 0.9247          
             Prevalence : 0.3661          
         Detection Rate : 0.3256          
   Detection Prevalence : 0.4624          
      Balanced Accuracy : 0.8368          
                                          
       'Positive' Class : successful      
                                          

> 
> glmnetRoc <- roc(response = glmnet2008$obs,
+                  predictor = glmnet2008$successful,
+                  levels = rev(levels(glmnet2008$obs)))
> 
> glmnFit0 <- glmnFit
> glmnFit0$results$lambda <- format(round(glmnFit0$results$lambda, 3))
> 
> glmnPlot <- plot(glmnFit0,
+                  plotType = "level",
+                  cuts = 15,
+                  scales = list(x = list(rot = 90, cex = .65)))
> 
> update(glmnPlot,
+        ylab = "Mixing Percentage\nRidge <---------> Lasso",
+        sub = "",
+        main = "Area Under the ROC Curve",
+        xlab = "Amount of Regularization")
> 
> plot(plsRoc2, type = "s", col = rgb(.2, .2, .2, .2), legacy.axes = TRUE)

Call:
roc.default(response = plsFit2$pred$obs, predictor = plsFit2$pred$successful,     levels = rev(levels(plsFit2$pred$obs)))

Data: plsFit2$pred$successful in 987 controls (plsFit2$pred$obs unsuccessful) < 570 cases (plsFit2$pred$obs successful).
Area under the curve: 0.895
> plot(ldaRoc, type = "s", add = TRUE, col = rgb(.2, .2, .2, .2), legacy.axes = TRUE)

Call:
roc.default(response = ldaFit$pred$obs, predictor = ldaFit$pred$successful,     levels = rev(levels(ldaFit$pred$obs)))

Data: ldaFit$pred$successful in 987 controls (ldaFit$pred$obs unsuccessful) < 570 cases (ldaFit$pred$obs successful).
Area under the curve: 0.8892
> plot(lrRoc, type = "s", col = rgb(.2, .2, .2, .2), add = TRUE, legacy.axes = TRUE)

Call:
roc.default(response = lrFit$pred$obs, predictor = lrFit$pred$successful,     levels = rev(levels(lrFit$pred$obs)))

Data: lrFit$pred$successful in 987 controls (lrFit$pred$obs unsuccessful) < 570 cases (lrFit$pred$obs successful).
Area under the curve: 0.8715
> plot(glmnetRoc, type = "s", add = TRUE, legacy.axes = TRUE)

Call:
roc.default(response = glmnet2008$obs, predictor = glmnet2008$successful,     levels = rev(levels(glmnet2008$obs)))

Data: glmnet2008$successful in 987 controls (glmnet2008$obs unsuccessful) < 570 cases (glmnet2008$obs successful).
Area under the curve: 0.91
> 
> ## Sparse logistic regression
> 
> set.seed(476)
> spLDAFit <- train(x = training[,fullSet], 
+                   y = training$Class,
+                   "sparseLDA",
+                   tuneGrid = expand.grid(lambda = c(.1),
+                                          NumVars = c(1:20, 50, 75, 100, 250, 500, 750, 1000)),
+                   preProc = c("center", "scale"),
+                   metric = "ROC",
+                   trControl = ctrl)
Loading required package: sparseLDA
Loading required package: lars
Loaded lars 1.2

Loading required package: elasticnet
Loading required package: mda
> spLDAFit
Sparse Linear Discriminant Analysis 

8190 samples
1071 predictors
   2 classes: 'successful', 'unsuccessful' 

Pre-processing: centered, scaled 
Resampling: Repeated Train/Test Splits Estimated (1 reps, 0.75%) 

Summary of sample sizes: 6633 

Resampling results across tuning parameters:

  NumVars  ROC    Sens   Spec 
  1        0.815  0.991  0.638
  2        0.823  0.991  0.638
  3        0.865  0.991  0.638
  4        0.868  0.96   0.663
  5        0.886  0.961  0.67 
  6        0.901  0.921  0.719
  7        0.899  0.891  0.751
  8        0.898  0.888  0.754
  9        0.897  0.886  0.751
  10       0.897  0.886  0.751
  11       0.897  0.886  0.751
  12       0.897  0.886  0.754
  13       0.897  0.886  0.755
  14       0.897  0.886  0.755
  15       0.897  0.886  0.755
  16       0.897  0.886  0.756
  17       0.897  0.884  0.764
  18       0.897  0.884  0.765
  19       0.897  0.882  0.766
  20       0.897  0.882  0.765
  50       0.899  0.877  0.78 
  75       0.9    0.877  0.785
  100      0.901  0.875  0.787
  250      0.9    0.856  0.797
  500      0.89   0.837  0.8  
  750      0.878  0.818  0.799
  1000     0.864  0.802  0.798

Tuning parameter 'lambda' was held constant at a value of 0.1
ROC was used to select the optimal model using  the largest value.
The final values used for the model were NumVars = 6 and lambda = 0.1. 
> 
> spLDA2008 <- merge(spLDAFit$pred,  spLDAFit$bestTune)
> spLDACM <- confusionMatrix(spLDAFit, norm = "none")
> spLDACM
Repeated Train/Test Splits Estimated (1 reps, 0.75%) Confusion Matrix 

(entries are un-normalized counts)
 
Confusion Matrix and Statistics

              Reference
Prediction     successful unsuccessful
  successful          525          277
  unsuccessful         45          710
                                          
               Accuracy : 0.7932          
                 95% CI : (0.7722, 0.8131)
    No Information Rate : 0.6339          
    P-Value [Acc > NIR] : < 2.2e-16       
                                          
                  Kappa : 0.5897          
 Mcnemar's Test P-Value : < 2.2e-16       
                                          
            Sensitivity : 0.9211          
            Specificity : 0.7194          
         Pos Pred Value : 0.6546          
         Neg Pred Value : 0.9404          
             Prevalence : 0.3661          
         Detection Rate : 0.3372          
   Detection Prevalence : 0.5151          
      Balanced Accuracy : 0.8202          
                                          
       'Positive' Class : successful      
                                          

> 
> spLDARoc <- roc(response = spLDA2008$obs,
+                 predictor = spLDA2008$successful,
+                 levels = rev(levels(spLDA2008$obs)))
> 
> update(plot(spLDAFit, scales = list(x = list(log = 10))),
+        ylab = "ROC AUC (2008 Hold-Out Data)")
> 
> plot(plsRoc2, type = "s", col = rgb(.2, .2, .2, .2), legacy.axes = TRUE)

Call:
roc.default(response = plsFit2$pred$obs, predictor = plsFit2$pred$successful,     levels = rev(levels(plsFit2$pred$obs)))

Data: plsFit2$pred$successful in 987 controls (plsFit2$pred$obs unsuccessful) < 570 cases (plsFit2$pred$obs successful).
Area under the curve: 0.895
> plot(glmnetRoc, type = "s", add = TRUE, col = rgb(.2, .2, .2, .2), legacy.axes = TRUE)

Call:
roc.default(response = glmnet2008$obs, predictor = glmnet2008$successful,     levels = rev(levels(glmnet2008$obs)))

Data: glmnet2008$successful in 987 controls (glmnet2008$obs unsuccessful) < 570 cases (glmnet2008$obs successful).
Area under the curve: 0.91
> plot(ldaRoc, type = "s", add = TRUE, col = rgb(.2, .2, .2, .2), legacy.axes = TRUE)

Call:
roc.default(response = ldaFit$pred$obs, predictor = ldaFit$pred$successful,     levels = rev(levels(ldaFit$pred$obs)))

Data: ldaFit$pred$successful in 987 controls (ldaFit$pred$obs unsuccessful) < 570 cases (ldaFit$pred$obs successful).
Area under the curve: 0.8892
> plot(lrRoc, type = "s", col = rgb(.2, .2, .2, .2), add = TRUE, legacy.axes = TRUE)

Call:
roc.default(response = lrFit$pred$obs, predictor = lrFit$pred$successful,     levels = rev(levels(lrFit$pred$obs)))

Data: lrFit$pred$successful in 987 controls (lrFit$pred$obs unsuccessful) < 570 cases (lrFit$pred$obs successful).
Area under the curve: 0.8715
> plot(spLDARoc, type = "s", add = TRUE, legacy.axes = TRUE)

Call:
roc.default(response = spLDA2008$obs, predictor = spLDA2008$successful,     levels = rev(levels(spLDA2008$obs)))

Data: spLDA2008$successful in 987 controls (spLDA2008$obs unsuccessful) < 570 cases (spLDA2008$obs successful).
Area under the curve: 0.9015
> 
> ################################################################################
> ### Section 12.6 Nearest Shrunken Centroids
> 
> set.seed(476)
> nscFit <- train(x = training[,fullSet], 
+                 y = training$Class,
+                 method = "pam",
+                 preProc = c("center", "scale"),
+                 tuneGrid = data.frame(threshold = seq(0, 25, length = 30)),
+                 metric = "ROC",
+                 trControl = ctrl)
Loading required package: pamr
Loading required package: cluster
Loading required package: survival
Loading required package: splines

Attaching package: 'survival'

The following object is masked from 'package:caret':

    cluster

11> nscFit
Nearest Shrunken Centroids 

8190 samples
1071 predictors
   2 classes: 'successful', 'unsuccessful' 

Pre-processing: centered, scaled 
Resampling: Repeated Train/Test Splits Estimated (1 reps, 0.75%) 

Summary of sample sizes: 6633 

Resampling results across tuning parameters:

  threshold  ROC    Sens   Spec 
  0          0.827  0.784  0.733
  0.862      0.864  0.842  0.73 
  1.72       0.871  0.865  0.736
  2.59       0.873  0.861  0.744
  3.45       0.873  0.849  0.752
  4.31       0.868  0.823  0.754
  5.17       0.866  0.821  0.753
  6.03       0.862  0.856  0.732
  6.9        0.852  0.844  0.721
  7.76       0.857  0.935  0.675
  8.62       0.872  0.991  0.638
  9.48       0.832  0.991  0.638
  10.3       0.823  0.991  0.638
  11.2       0.815  0.991  0.638
  12.1       0.815  0.991  0.638
  12.9       0.815  0.991  0.638
  13.8       0.815  0      1    
  14.7       0.815  0      1    
  15.5       0.815  0      1    
  16.4       0.815  0      1    
  17.2       0.815  0      1    
  18.1       0.5    0      1    
  19         0.5    0      1    
  19.8       0.5    0      1    
  20.7       0.5    0      1    
  21.6       0.5    0      1    
  22.4       0.5    0      1    
  23.3       0.5    0      1    
  24.1       0.5    0      1    
  25         0.5    0      1    

ROC was used to select the optimal model using  the largest value.
The final value used for the model was threshold = 2.59. 
> 
> nsc2008 <- merge(nscFit$pred,  nscFit$bestTune)
> nscCM <- confusionMatrix(nscFit, norm = "none")
> nscCM
Repeated Train/Test Splits Estimated (1 reps, 0.75%) Confusion Matrix 

(entries are un-normalized counts)
 
Confusion Matrix and Statistics

              Reference
Prediction     successful unsuccessful
  successful          491          253
  unsuccessful         79          734
                                          
               Accuracy : 0.7868          
                 95% CI : (0.7656, 0.8069)
    No Information Rate : 0.6339          
    P-Value [Acc > NIR] : < 2.2e-16       
                                          
                  Kappa : 0.5684          
 Mcnemar's Test P-Value : < 2.2e-16       
                                          
            Sensitivity : 0.8614          
            Specificity : 0.7437          
         Pos Pred Value : 0.6599          
         Neg Pred Value : 0.9028          
             Prevalence : 0.3661          
         Detection Rate : 0.3154          
   Detection Prevalence : 0.4778          
      Balanced Accuracy : 0.8025          
                                          
       'Positive' Class : successful      
                                          

> nscRoc <- roc(response = nsc2008$obs,
+               predictor = nsc2008$successful,
+               levels = rev(levels(nsc2008$obs)))
> update(plot(nscFit), ylab = "ROC AUC (2008 Hold-Out Data)")
> 
> 
> plot(plsRoc2, type = "s", col = rgb(.2, .2, .2, .2), legacy.axes = TRUE)

Call:
roc.default(response = plsFit2$pred$obs, predictor = plsFit2$pred$successful,     levels = rev(levels(plsFit2$pred$obs)))

Data: plsFit2$pred$successful in 987 controls (plsFit2$pred$obs unsuccessful) < 570 cases (plsFit2$pred$obs successful).
Area under the curve: 0.895
> plot(glmnetRoc, type = "s", add = TRUE, col = rgb(.2, .2, .2, .2), legacy.axes = TRUE)

Call:
roc.default(response = glmnet2008$obs, predictor = glmnet2008$successful,     levels = rev(levels(glmnet2008$obs)))

Data: glmnet2008$successful in 987 controls (glmnet2008$obs unsuccessful) < 570 cases (glmnet2008$obs successful).
Area under the curve: 0.91
> plot(ldaRoc, type = "s", add = TRUE, col = rgb(.2, .2, .2, .2), legacy.axes = TRUE)

Call:
roc.default(response = ldaFit$pred$obs, predictor = ldaFit$pred$successful,     levels = rev(levels(ldaFit$pred$obs)))

Data: ldaFit$pred$successful in 987 controls (ldaFit$pred$obs unsuccessful) < 570 cases (ldaFit$pred$obs successful).
Area under the curve: 0.8892
> plot(lrRoc, type = "s", col = rgb(.2, .2, .2, .2), add = TRUE, legacy.axes = TRUE)

Call:
roc.default(response = lrFit$pred$obs, predictor = lrFit$pred$successful,     levels = rev(levels(lrFit$pred$obs)))

Data: lrFit$pred$successful in 987 controls (lrFit$pred$obs unsuccessful) < 570 cases (lrFit$pred$obs successful).
Area under the curve: 0.8715
> plot(spLDARoc, type = "s", col = rgb(.2, .2, .2, .2), add = TRUE, legacy.axes = TRUE)

Call:
roc.default(response = spLDA2008$obs, predictor = spLDA2008$successful,     levels = rev(levels(spLDA2008$obs)))

Data: spLDA2008$successful in 987 controls (spLDA2008$obs unsuccessful) < 570 cases (spLDA2008$obs successful).
Area under the curve: 0.9015
> plot(nscRoc, type = "s", add = TRUE, legacy.axes = TRUE)

Call:
roc.default(response = nsc2008$obs, predictor = nsc2008$successful,     levels = rev(levels(nsc2008$obs)))

Data: nsc2008$successful in 987 controls (nsc2008$obs unsuccessful) < 570 cases (nsc2008$obs successful).
Area under the curve: 0.8733
> 
> sessionInfo()
R version 3.0.1 (2013-05-16)
Platform: x86_64-apple-darwin10.8.0 (64-bit)

locale:
[1] C

attached base packages:
[1] splines   parallel  stats     graphics  grDevices utils     datasets 
[8] methods   base     

other attached packages:
 [1] pamr_1.54       survival_2.37-4 cluster_1.14.4  sparseLDA_0.1-6
 [5] mda_0.4-2       elasticnet_1.1  lars_1.2        glmnet_1.9-3   
 [9] Matrix_1.0-12   klaR_0.6-8      pls_2.3-0       MASS_7.3-26    
[13] e1071_1.6-1     class_7.3-7     pROC_1.5.4      kernlab_0.9-18 
[17] reshape2_1.2.2  plyr_1.8        doMC_1.3.0      iterators_1.0.6
[21] foreach_1.4.0   caret_6.0-22    ggplot2_0.9.3.1 lattice_0.20-15

loaded via a namespace (and not attached):
 [1] RColorBrewer_1.0-5 car_2.0-17         codetools_0.2-8    colorspace_1.2-2  
 [5] compiler_3.0.1     dichromat_2.0-0    digest_0.6.3       grid_3.0.1        
 [9] gtable_0.1.2       labeling_0.1       munsell_0.4        proto_0.3-10      
[13] scales_0.2.3       stringr_0.6.2     
> 
> q("no")
> proc.time()
      user     system    elapsed 
376332.996   8337.928  35694.682 
