
R version 3.0.1 (2013-05-16) -- "Good Sport"
Copyright (C) 2013 The R Foundation for Statistical Computing
Platform: x86_64-apple-darwin10.8.0 (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

> ################################################################################
> ### R code from Applied Predictive Modeling (2013) by Kuhn and Johnson.
> ### Copyright 2013 Kuhn and Johnson
> ### Web Page: http://www.appliedpredictivemodeling.com
> ### Contact: Max Kuhn (mxkuhn@gmail.com)
> ###
> ### Chapter 7: Non-Linear Regression Models
> ###
> ### Required packages: AppliedPredictiveModeling, caret, doMC (optional), earth,
> ###                    kernlab, lattice, nnet
> ###
> ### Data used: The solubility from the AppliedPredictiveModeling package
> ###
> ### Notes: 
> ### 1) This code is provided without warranty.
> ###
> ### 2) This code should help the user reproduce the results in the
> ### text. There will be differences between this code and what is is
> ### the computing section. For example, the computing sections show
> ### how the source functions work (e.g. randomForest() or plsr()),
> ### which were not directly used when creating the book. Also, there may be 
> ### syntax differences that occur over time as packages evolve. These files 
> ### will reflect those changes.
> ###
> ### 3) In some cases, the calculations in the book were run in 
> ### parallel. The sub-processes may reset the random number seed.
> ### Your results may slightly vary.
> ###
> ################################################################################
> 
> ################################################################################
> ### Load the data
> 
> library(AppliedPredictiveModeling)
> data(solubility)
> 
> ### Create a control funciton that will be used across models. We
> ### create the fold assignments explictily instead of relying on the
> ### random number seed being set to identical values.
> 
> library(caret)
Loading required package: lattice
Loading required package: ggplot2
> set.seed(100)
> indx <- createFolds(solTrainY, returnTrain = TRUE)
> ctrl <- trainControl(method = "cv", index = indx)
> 
> ################################################################################
> ### Section 7.1 Neural Networks
> 
> ### Optional: parallel processing can be used via the 'do' packages,
> ### such as doMC, doMPI etc. We used doMC (not on Windows) to speed
> ### up the computations.
> 
> ### WARNING: Be aware of how much memory is needed to parallel
> ### process. It can very quickly overwhelm the availible hardware. We
> ### estimate the memory usuage (VSIZE = total memory size) to be 
> ### 2677M/core.
> 
> library(doMC)
Loading required package: foreach
Loading required package: iterators
Loading required package: parallel
> registerDoMC(10)
> 
> 
> library(caret)
> 
> nnetGrid <- expand.grid(decay = c(0, 0.01, .1), 
+                         size = c(1, 3, 5, 7, 9, 11, 13), 
+                         bag = FALSE)
> 
> set.seed(100)
> nnetTune <- train(x = solTrainXtrans, y = solTrainY,
+                   method = "avNNet",
+                   tuneGrid = nnetGrid,
+                   trControl = ctrl,
+                   preProc = c("center", "scale"),
+                   linout = TRUE,
+                   trace = FALSE,
+                   MaxNWts = 13 * (ncol(solTrainXtrans) + 1) + 13 + 1,
+                   maxit = 1000,
+                   allowParallel = FALSE)
Loading required package: nnet
> nnetTune
Model Averaged Neural Network 

951 samples
228 predictors

Pre-processing: centered, scaled 
Resampling: Cross-Validated (10 fold) 

Summary of sample sizes: 856, 857, 855, 856, 856, 855, ... 

Resampling results across tuning parameters:

  decay  size  RMSE   Rsquared  RMSE SD  Rsquared SD
  0      1     0.828  0.842     0.101    0.0326     
  0      3     0.813  0.846     0.0564   0.0257     
  0      5     0.757  0.864     0.0839   0.0274     
  0      7     0.821  0.846     0.0819   0.0293     
  0      9     0.89   0.819     0.0777   0.0259     
  0      11    0.835  0.839     0.0954   0.0382     
  0      13    0.82   0.844     0.0529   0.0274     
  0.01   1     0.74   0.871     0.069    0.0259     
  0.01   3     0.766  0.862     0.0696   0.0325     
  0.01   5     0.784  0.859     0.0507   0.0174     
  0.01   7     0.77   0.863     0.0715   0.0293     
  0.01   9     0.808  0.849     0.116    0.0426     
  0.01   11    0.733  0.876     0.052    0.0185     
  0.01   13    0.712  0.881     0.068    0.0304     
  0.1    1     0.752  0.867     0.0641   0.0241     
  0.1    3     0.781  0.857     0.0761   0.0272     
  0.1    5     0.742  0.873     0.0843   0.0254     
  0.1    7     0.694  0.887     0.0847   0.0239     
  0.1    9     0.682  0.89      0.0899   0.0242     
  0.1    11    0.676  0.892     0.0772   0.0247     
  0.1    13    0.674  0.892     0.0763   0.0241     

Tuning parameter 'bag' was held constant at a value of FALSE
RMSE was used to select the optimal model using  the smallest value.
The final values used for the model were size = 13, decay = 0.1 and bag = FALSE. 
> 
> plot(nnetTune)
> 
> testResults <- data.frame(obs = solTestY,
+                           NNet = predict(nnetTune, solTestXtrans))
> 
> ################################################################################
> ### Section 7.2 Multivariate Adaptive Regression Splines
> 
> set.seed(100)
> marsTune <- train(x = solTrainXtrans, y = solTrainY,
+                   method = "earth",
+                   tuneGrid = expand.grid(degree = 1, nprune = 2:38),
+                   trControl = ctrl)
Loading required package: earth
Loading required package: plotmo
Loading required package: plotrix
> marsTune
Multivariate Adaptive Regression Spline 

951 samples
228 predictors

No pre-processing
Resampling: Cross-Validated (10 fold) 

Summary of sample sizes: 856, 857, 855, 856, 856, 855, ... 

Resampling results across tuning parameters:

  nprune  RMSE   Rsquared  RMSE SD  Rsquared SD
  2       1.54   0.438     0.128    0.0802     
  3       1.12   0.7       0.0968   0.0647     
  4       1.06   0.73      0.0849   0.0594     
  5       1.02   0.75      0.102    0.0551     
  6       0.984  0.768     0.0733   0.042      
  7       0.919  0.796     0.0657   0.0432     
  8       0.862  0.821     0.0418   0.0237     
  9       0.855  0.823     0.0323   0.0263     
  10      0.854  0.823     0.0423   0.0329     
  11      0.841  0.829     0.042    0.0306     
  12      0.816  0.839     0.0348   0.0296     
  13      0.815  0.839     0.0466   0.0306     
  14      0.81   0.842     0.0407   0.0263     
  15      0.804  0.843     0.0518   0.0309     
  16      0.79   0.849     0.0363   0.0285     
  17      0.779  0.853     0.0382   0.0271     
  18      0.773  0.855     0.0465   0.0309     
  19      0.758  0.861     0.0502   0.0307     
  20      0.745  0.865     0.0539   0.0313     
  21      0.732  0.87      0.049    0.0295     
  22      0.721  0.874     0.0468   0.028      
  23      0.715  0.876     0.0558   0.0278     
  24      0.718  0.875     0.051    0.0274     
  25      0.706  0.879     0.0542   0.0276     
  26      0.702  0.881     0.0541   0.0268     
  27      0.699  0.881     0.0545   0.0267     
  28      0.698  0.882     0.0526   0.0269     
  29      0.698  0.882     0.056    0.0276     
  30      0.693  0.883     0.0545   0.0265     
  31      0.69   0.884     0.0511   0.0255     
  32      0.687  0.885     0.0515   0.0248     
  33      0.688  0.885     0.0539   0.026      
  34      0.685  0.886     0.0512   0.0248     
  35      0.684  0.887     0.0508   0.0247     
  36      0.685  0.886     0.0529   0.0254     
  37      0.685  0.886     0.0529   0.0258     
  38      0.683  0.887     0.0512   0.0251     

Tuning parameter 'degree' was held constant at a value of 1
RMSE was used to select the optimal model using  the smallest value.
The final values used for the model were nprune = 38 and degree = 1. 
> 
> plot(marsTune)
> 
> testResults$MARS <- predict(marsTune, solTestXtrans)
> 
> marsImp <- varImp(marsTune, scale = FALSE)
> plot(marsImp, top = 25)
> 
> ################################################################################
> ### Section 7.3 Support Vector Machines
> 
> ## In a recent update to caret, the method to estimate the
> ## sigma parameter was slightly changed. These results will
> ## slightly differ from the text for that reason.
> 
> set.seed(100)
> svmRTune <- train(x = solTrainXtrans, y = solTrainY,
+                   method = "svmRadial",
+                   preProc = c("center", "scale"),
+                   tuneLength = 14,
+                   trControl = ctrl)
Loading required package: kernlab
> svmRTune
Support Vector Machines with Radial Basis Function Kernel 

951 samples
228 predictors

Pre-processing: centered, scaled 
Resampling: Cross-Validated (10 fold) 

Summary of sample sizes: 856, 857, 855, 856, 856, 855, ... 

Resampling results across tuning parameters:

  C     RMSE   Rsquared  RMSE SD  Rsquared SD
  0.25  0.805  0.865     0.0542   0.0181     
  0.5   0.711  0.887     0.0502   0.0176     
  1     0.661  0.898     0.0555   0.0181     
  2     0.63   0.906     0.0618   0.0176     
  4     0.617  0.909     0.0644   0.0175     
  8     0.607  0.912     0.0674   0.0176     
  16    0.601  0.914     0.0665   0.0169     
  32    0.601  0.913     0.0662   0.0173     
  64    0.599  0.914     0.0667   0.0175     
  128   0.598  0.914     0.0654   0.0179     
  256   0.597  0.914     0.0642   0.0178     
  512   0.599  0.914     0.0631   0.0177     
  1020  0.599  0.914     0.0639   0.0179     
  2050  0.602  0.913     0.0634   0.0181     

Tuning parameter 'sigma' was held constant at a value of 0.002762998
RMSE was used to select the optimal model using  the smallest value.
The final values used for the model were sigma = 0.00276 and C = 256. 
> plot(svmRTune, scales = list(x = list(log = 2)))                 
> 
> svmGrid <- expand.grid(degree = 1:2, 
+                        scale = c(0.01, 0.005, 0.001), 
+                        C = 2^(-2:5))
> set.seed(100)
> svmPTune <- train(x = solTrainXtrans, y = solTrainY,
+                   method = "svmPoly",
+                   preProc = c("center", "scale"),
+                   tuneGrid = svmGrid,
+                   trControl = ctrl)
> 
> svmPTune
Support Vector Machines with Polynomial Kernel 

951 samples
228 predictors

Pre-processing: centered, scaled 
Resampling: Cross-Validated (10 fold) 

Summary of sample sizes: 856, 857, 855, 856, 856, 855, ... 

Resampling results across tuning parameters:

  degree  scale  C     RMSE   Rsquared  RMSE SD  Rsquared SD
  1       0.001  0.25  1.05   0.793     0.0949   0.0298     
  1       0.001  0.5   0.893  0.834     0.0755   0.0188     
  1       0.001  1     0.793  0.858     0.0568   0.0161     
  1       0.001  2     0.732  0.875     0.0432   0.015      
  1       0.001  4     0.701  0.883     0.0389   0.0155     
  1       0.001  8     0.686  0.888     0.0369   0.0152     
  1       0.001  16    0.693  0.885     0.0407   0.017      
  1       0.001  32    0.703  0.882     0.0439   0.0183     
  1       0.005  0.25  0.769  0.865     0.0537   0.0155     
  1       0.005  0.5   0.719  0.878     0.0407   0.0153     
  1       0.005  1     0.694  0.885     0.0366   0.0157     
  1       0.005  2     0.687  0.887     0.0384   0.0158     
  1       0.005  4     0.696  0.884     0.0407   0.0172     
  1       0.005  8     0.71   0.88      0.0467   0.019      
  1       0.005  16    0.724  0.875     0.0489   0.0189     
  1       0.005  32    0.735  0.871     0.057    0.0212     
  1       0.01   0.25  0.719  0.878     0.0407   0.0153     
  1       0.01   0.5   0.694  0.885     0.0366   0.0157     
  1       0.01   1     0.687  0.887     0.0384   0.0158     
  1       0.01   2     0.696  0.884     0.0407   0.0172     
  1       0.01   4     0.709  0.88      0.0466   0.019      
  1       0.01   8     0.723  0.875     0.049    0.0189     
  1       0.01   16    0.735  0.871     0.0568   0.0212     
  1       0.01   32    0.739  0.87      0.0564   0.0232     
  2       0.001  0.25  0.876  0.841     0.0747   0.0183     
  2       0.001  0.5   0.767  0.868     0.0549   0.0168     
  2       0.001  1     0.701  0.885     0.0461   0.0157     
  2       0.001  2     0.661  0.896     0.0454   0.0151     
  2       0.001  4     0.64   0.902     0.0537   0.016      
  2       0.001  8     0.626  0.906     0.0662   0.0183     
  2       0.001  16    0.633  0.904     0.0695   0.0196     
  2       0.001  32    0.636  0.903     0.0804   0.022      
  2       0.005  0.25  0.653  0.899     0.0602   0.0183     
  2       0.005  0.5   0.628  0.906     0.0668   0.019      
  2       0.005  1     0.617  0.909     0.0698   0.0196     
  2       0.005  2     0.615  0.909     0.0749   0.0203     
  2       0.005  4     0.614  0.91      0.0806   0.0211     
  2       0.005  8     0.616  0.909     0.078    0.0205     
  2       0.005  16    0.622  0.908     0.0775   0.0204     
  2       0.005  32    0.632  0.905     0.0782   0.0209     
  2       0.01   0.25  0.621  0.908     0.0703   0.02       
  2       0.01   0.5   0.618  0.908     0.0737   0.0208     
  2       0.01   1     0.611  0.91      0.079    0.0211     
  2       0.01   2     0.609  0.911     0.0766   0.0204     
  2       0.01   4     0.612  0.91      0.0756   0.0203     
  2       0.01   8     0.619  0.908     0.0737   0.0201     
  2       0.01   16    0.623  0.907     0.0742   0.0207     
  2       0.01   32    0.628  0.906     0.0728   0.0206     

RMSE was used to select the optimal model using  the smallest value.
The final values used for the model were degree = 2, scale = 0.01 and C = 2. 
> plot(svmPTune, 
+      scales = list(x = list(log = 2), 
+                    between = list(x = .5, y = 1)))                 
> 
> testResults$SVMr <- predict(svmRTune, solTestXtrans)
> testResults$SVMp <- predict(svmPTune, solTestXtrans)
> 
> ################################################################################
> ### Section 7.4 K-Nearest Neighbors
> 
> ### First we remove near-zero variance predictors
> knnDescr <- solTrainXtrans[, -nearZeroVar(solTrainXtrans)]
> 
> set.seed(100)
> knnTune <- train(x = knnDescr, y = solTrainY,
+                  method = "knn",
+                  preProc = c("center", "scale"),
+                  tuneGrid = data.frame(k = 1:20),
+                  trControl = ctrl)
>                  
> knnTune
k-Nearest Neighbors 

951 samples
225 predictors

Pre-processing: centered, scaled 
Resampling: Cross-Validated (10 fold) 

Summary of sample sizes: 856, 857, 855, 856, 856, 855, ... 

Resampling results across tuning parameters:

  k   RMSE  Rsquared  RMSE SD  Rsquared SD
  1   1.23  0.671     0.169    0.0864     
  2   1.1   0.723     0.144    0.0605     
  3   1.06  0.741     0.0969   0.0424     
  4   1.04  0.747     0.081    0.0364     
  5   1.06  0.738     0.0674   0.0445     
  6   1.05  0.739     0.0645   0.048      
  7   1.05  0.739     0.0625   0.0399     
  8   1.05  0.742     0.0651   0.043      
  9   1.05  0.738     0.0694   0.043      
  10  1.06  0.735     0.061    0.0402     
  11  1.06  0.733     0.0598   0.0425     
  12  1.08  0.727     0.0666   0.0456     
  13  1.08  0.723     0.0677   0.0445     
  14  1.09  0.72      0.0724   0.0451     
  15  1.1   0.715     0.0751   0.0473     
  16  1.11  0.711     0.0835   0.049      
  17  1.11  0.708     0.0883   0.0492     
  18  1.12  0.704     0.089    0.0504     
  19  1.13  0.7       0.0922   0.0511     
  20  1.14  0.697     0.0918   0.0503     

RMSE was used to select the optimal model using  the smallest value.
The final value used for the model was k = 4. 
> 
> plot(knnTune)
> 
> testResults$Knn <- predict(svmRTune, solTestXtrans[, names(knnDescr)])
Warning messages:
1: In predict.preProcess(preProc, newdata) :
  newdata does not contain some variables
2: In method$predict(modelFit = modelFit, newdata = newdata, submodels = param) :
  kernlab prediction calculations failed; returning NAs
> 
> ################################################################################
> ### Session Information
> 
> sessionInfo()
R version 3.0.1 (2013-05-16)
Platform: x86_64-apple-darwin10.8.0 (64-bit)

locale:
[1] C

attached base packages:
[1] parallel  stats     graphics  grDevices utils     datasets  methods  
[8] base     

other attached packages:
 [1] kernlab_0.9-18                  earth_3.2-6                    
 [3] plotrix_3.4-7                   plotmo_1.3-2                   
 [5] nnet_7.3-6                      doMC_1.3.0                     
 [7] iterators_1.0.6                 foreach_1.4.0                  
 [9] caret_6.0-22                    ggplot2_0.9.3.1                
[11] lattice_0.20-15                 AppliedPredictiveModeling_1.1-5

loaded via a namespace (and not attached):
 [1] CORElearn_0.9.41   MASS_7.3-26        RColorBrewer_1.0-5 car_2.0-17        
 [5] codetools_0.2-8    colorspace_1.2-2   compiler_3.0.1     dichromat_2.0-0   
 [9] digest_0.6.3       grid_3.0.1         gtable_0.1.2       labeling_0.1      
[13] munsell_0.4        plyr_1.8           proto_0.3-10       reshape2_1.2.2    
[17] scales_0.2.3       stringr_0.6.2      tools_3.0.1       
> 
> q("no")
> proc.time()
      user     system    elapsed 
112106.723    188.979  12272.168 
