
R version 3.0.1 (2013-05-16) -- "Good Sport"
Copyright (C) 2013 The R Foundation for Statistical Computing
Platform: x86_64-apple-darwin10.8.0 (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

> ################################################################################
> ### R code from Applied Predictive Modeling (2013) by Kuhn and Johnson.
> ### Copyright 2013 Kuhn and Johnson
> ### Web Page: http://www.appliedpredictivemodeling.com
> ### Contact: Max Kuhn (mxkuhn@gmail.com)
> ###
> ### Chapter 6: Linear Regression and Its Cousins
> ###
> ### Required packages: AppliedPredictiveModeling, lattice, corrplot, pls, 
> ###                    elasticnet,
> ###
> ### Data used: The solubility from the AppliedPredictiveModeling package
> ###
> ### Notes:
> ### 1) This code is provided without warranty.
> ###
> ### 2) This code should help the user reproduce the results in the
> ### text. There will be differences between this code and what is is
> ### the computing section. For example, the computing sections show
> ### how the source functions work (e.g. randomForest() or plsr()),
> ### which were not directly used when creating the book. Also, there may be 
> ### syntax differences that occur over time as packages evolve. These files 
> ### will reflect those changes.
> ###
> ### 3) In some cases, the calculations in the book were run in 
> ### parallel. The sub-processes may reset the random number seed.
> ### Your results may slightly vary.
> ###
> ################################################################################
> 
> ################################################################################
> ### Section 6.1 Case Study: Quantitative Structure- Activity
> ### Relationship Modeling
> 
> library(AppliedPredictiveModeling)
> data(solubility)
> 
> library(lattice)
> 
> ### Some initial plots of the data
> 
> xyplot(solTrainY ~ solTrainX$MolWeight, type = c("p", "g"),
+        ylab = "Solubility (log)",
+        main = "(a)",
+        xlab = "Molecular Weight")
> xyplot(solTrainY ~ solTrainX$NumRotBonds, type = c("p", "g"),
+        ylab = "Solubility (log)",
+        xlab = "Number of Rotatable Bonds")
> bwplot(solTrainY ~ ifelse(solTrainX[,100] == 1, 
+                           "structure present", 
+                           "structure absent"),
+        ylab = "Solubility (log)",
+        main = "(b)",
+        horizontal = FALSE)
> 
> ### Find the columns that are not fingerprints (i.e. the continuous
> ### predictors). grep will return a list of integers corresponding to
> ### column names that contain the pattern "FP".
> 
> notFingerprints <- grep("FP", names(solTrainXtrans))
> 
> library(caret)
Loading required package: ggplot2
> featurePlot(solTrainXtrans[, -notFingerprints],
+             solTrainY,
+             between = list(x = 1, y = 1),
+             type = c("g", "p", "smooth"),
+             labels = rep("", 2))
There were 50 or more warnings (use warnings() to see the first 50)
> 
> library(corrplot)
> 
> ### We used the full namespace to call this function because the pls
> ### package (also used in this chapter) has a function with the same
> ### name.
> 
> corrplot::corrplot(cor(solTrainXtrans[, -notFingerprints]), 
+                    order = "hclust", 
+                    tl.cex = .8)
> 
> ################################################################################
> ### Section 6.2 Linear Regression
> 
> ### Create a control function that will be used across models. We
> ### create the fold assignments explicitly instead of relying on the
> ### random number seed being set to identical values.
> 
> set.seed(100)
> indx <- createFolds(solTrainY, returnTrain = TRUE)
> ctrl <- trainControl(method = "cv", index = indx)
> 
> ### Linear regression model with all of the predictors. This will
> ### produce some warnings that a 'rank-deficient fit may be
> ### misleading'. This is related to the predictors being so highly
> ### correlated that some of the math has broken down.
> 
> set.seed(100)
> lmTune0 <- train(x = solTrainXtrans, y = solTrainY,
+                  method = "lm",
+                  trControl = ctrl)
Warning messages:
1: In predict.lm(modelFit, newdata) :
  prediction from a rank-deficient fit may be misleading
2: In predict.lm(modelFit, newdata) :
  prediction from a rank-deficient fit may be misleading
> 
> lmTune0                 
Linear Regression 

951 samples
228 predictors

No pre-processing
Resampling: Cross-Validated (10 fold) 

Summary of sample sizes: 856, 857, 855, 856, 856, 855, ... 

Resampling results

  RMSE   Rsquared  RMSE SD  Rsquared SD
  0.721  0.877     0.07     0.0247     

 
> 
> ### And another using a set of predictors reduced by unsupervised
> ### filtering. We apply a filter to reduce extreme between-predictor
> ### correlations. Note the lack of warnings.
> 
> tooHigh <- findCorrelation(cor(solTrainXtrans), .9)
> trainXfiltered <- solTrainXtrans[, -tooHigh]
> testXfiltered  <-  solTestXtrans[, -tooHigh]
> 
> set.seed(100)
> lmTune <- train(x = trainXfiltered, y = solTrainY,
+                 method = "lm",
+                 trControl = ctrl)
> 
> lmTune
Linear Regression 

951 samples
190 predictors

No pre-processing
Resampling: Cross-Validated (10 fold) 

Summary of sample sizes: 856, 857, 855, 856, 856, 855, ... 

Resampling results

  RMSE   Rsquared  RMSE SD  Rsquared SD
  0.711  0.879     0.0632   0.0243     

 
> 
> ### Save the test set results in a data frame                 
> testResults <- data.frame(obs = solTestY,
+                           Linear_Regression = predict(lmTune, testXfiltered))
> 
> 
> ################################################################################
> ### Section 6.3 Partial Least Squares
> 
> ## Run PLS and PCR on solubility data and compare results
> set.seed(100)
> plsTune <- train(x = solTrainXtrans, y = solTrainY,
+                  method = "pls",
+                  tuneGrid = expand.grid(ncomp = 1:20),
+                  trControl = ctrl)
Loading required package: pls

Attaching package: 'pls'

The following object is masked from 'package:corrplot':

    corrplot

The following object is masked from 'package:caret':

    R2

The following object is masked from 'package:stats':

    loadings

> plsTune
Partial Least Squares 

951 samples
228 predictors

No pre-processing
Resampling: Cross-Validated (10 fold) 

Summary of sample sizes: 856, 857, 855, 856, 856, 855, ... 

Resampling results across tuning parameters:

  ncomp  RMSE   Rsquared  RMSE SD  Rsquared SD
  1      1.75   0.263     0.084    0.065      
  2      1.27   0.613     0.0794   0.0535     
  3      1.04   0.743     0.0716   0.0276     
  4      0.837  0.832     0.0562   0.0257     
  5      0.746  0.866     0.0378   0.0193     
  6      0.711  0.878     0.0343   0.0228     
  7      0.692  0.884     0.0379   0.024      
  8      0.691  0.885     0.0328   0.0197     
  9      0.683  0.888     0.0291   0.0185     
  10     0.682  0.888     0.0305   0.0187     
  11     0.683  0.888     0.0291   0.0195     
  12     0.685  0.887     0.0373   0.0194     
  13     0.684  0.888     0.0397   0.0194     
  14     0.686  0.887     0.0398   0.0186     
  15     0.687  0.887     0.0422   0.0194     
  16     0.686  0.887     0.0436   0.0208     
  17     0.688  0.887     0.0463   0.0213     
  18     0.693  0.885     0.0481   0.0221     
  19     0.694  0.885     0.0486   0.0221     
  20     0.698  0.884     0.053    0.0225     

RMSE was used to select the optimal model using  the smallest value.
The final value used for the model was ncomp = 10. 
> 
> testResults$PLS <- predict(plsTune, solTestXtrans)
> 
> set.seed(100)
> pcrTune <- train(x = solTrainXtrans, y = solTrainY,
+                  method = "pcr",
+                  tuneGrid = expand.grid(ncomp = 1:35),
+                  trControl = ctrl)
> pcrTune                  
Principal Component Analysis 

951 samples
228 predictors

No pre-processing
Resampling: Cross-Validated (10 fold) 

Summary of sample sizes: 856, 857, 855, 856, 856, 855, ... 

Resampling results across tuning parameters:

  ncomp  RMSE   Rsquared  RMSE SD  Rsquared SD
  1      1.98   0.0659    0.11     0.0347     
  2      1.64   0.362     0.0983   0.0848     
  3      1.37   0.555     0.094    0.0453     
  4      1.37   0.552     0.0981   0.0476     
  5      1.34   0.571     0.105    0.0617     
  6      1.21   0.65      0.0879   0.0615     
  7      1.18   0.666     0.101    0.0605     
  8      1.15   0.688     0.0778   0.0408     
  9      1.05   0.737     0.082    0.037      
  10     1.01   0.757     0.0959   0.0417     
  11     0.972  0.774     0.0778   0.0284     
  12     0.969  0.776     0.0789   0.0291     
  13     0.953  0.783     0.0764   0.0272     
  14     0.94   0.789     0.0706   0.0244     
  15     0.942  0.788     0.0684   0.0241     
  16     0.87   0.818     0.0467   0.0251     
  17     0.87   0.818     0.0458   0.0249     
  18     0.872  0.817     0.0475   0.0258     
  19     0.87   0.818     0.0473   0.0266     
  20     0.868  0.819     0.0473   0.0268     
  21     0.81   0.842     0.0458   0.0245     
  22     0.812  0.841     0.0448   0.0243     
  23     0.809  0.842     0.0446   0.0251     
  24     0.81   0.842     0.0401   0.0233     
  25     0.81   0.842     0.039    0.0236     
  26     0.805  0.844     0.0368   0.0213     
  27     0.804  0.845     0.0338   0.0204     
  28     0.806  0.844     0.034    0.021      
  29     0.786  0.851     0.036    0.0189     
  30     0.782  0.853     0.0307   0.0206     
  31     0.78   0.854     0.0283   0.021      
  32     0.776  0.855     0.0357   0.0217     
  33     0.74   0.869     0.0306   0.0181     
  34     0.733  0.871     0.031    0.0212     
  35     0.731  0.871     0.0357   0.022      

RMSE was used to select the optimal model using  the smallest value.
The final value used for the model was ncomp = 35. 
> 
> plsResamples <- plsTune$results
> plsResamples$Model <- "PLS"
> pcrResamples <- pcrTune$results
> pcrResamples$Model <- "PCR"
> plsPlotData <- rbind(plsResamples, pcrResamples)
> 
> xyplot(RMSE ~ ncomp,
+        data = plsPlotData,
+        #aspect = 1,
+        xlab = "# Components",
+        ylab = "RMSE (Cross-Validation)",
+        auto.key = list(columns = 2),
+        groups = Model,
+        type = c("o", "g"))
> 
> plsImp <- varImp(plsTune, scale = FALSE)
> plot(plsImp, top = 25, scales = list(y = list(cex = .95)))
> 
> ################################################################################
> ### Section 6.4 Penalized Models
> 
> ## The text used the elasticnet to obtain a ridge regression model.
> ## There is now a simple ridge regression method.
> 
> ridgeGrid <- expand.grid(lambda = seq(0, .1, length = 15))
> 
> set.seed(100)
> ridgeTune <- train(x = solTrainXtrans, y = solTrainY,
+                    method = "ridge",
+                    tuneGrid = ridgeGrid,
+                    trControl = ctrl,
+                    preProc = c("center", "scale"))
Loading required package: elasticnet
Loading required package: lars
Loaded lars 1.2

> ridgeTune
Ridge Regression 

951 samples
228 predictors

Pre-processing: centered, scaled 
Resampling: Cross-Validated (10 fold) 

Summary of sample sizes: 856, 857, 855, 856, 856, 855, ... 

Resampling results across tuning parameters:

  lambda   RMSE   Rsquared  RMSE SD  Rsquared SD
  0        0.721  0.877     0.0699   0.0245     
  0.00714  0.705  0.882     0.045    0.0199     
  0.0143   0.696  0.885     0.0405   0.0187     
  0.0214   0.693  0.886     0.0378   0.018      
  0.0286   0.691  0.887     0.0359   0.0175     
  0.0357   0.69   0.887     0.0346   0.0171     
  0.0429   0.691  0.888     0.0336   0.0168     
  0.05     0.692  0.888     0.0329   0.0166     
  0.0571   0.693  0.887     0.0323   0.0164     
  0.0643   0.695  0.887     0.032    0.0162     
  0.0714   0.698  0.887     0.0319   0.016      
  0.0786   0.7    0.887     0.0318   0.0159     
  0.0857   0.703  0.886     0.0318   0.0158     
  0.0929   0.706  0.886     0.032    0.0157     
  0.1      0.709  0.885     0.0321   0.0156     

RMSE was used to select the optimal model using  the smallest value.
The final value used for the model was lambda = 0.0357. 
> 
> print(update(plot(ridgeTune), xlab = "Penalty"))
> 
> 
> enetGrid <- expand.grid(lambda = c(0, 0.01, .1), 
+                         fraction = seq(.05, 1, length = 20))
> set.seed(100)
> enetTune <- train(x = solTrainXtrans, y = solTrainY,
+                   method = "enet",
+                   tuneGrid = enetGrid,
+                   trControl = ctrl,
+                   preProc = c("center", "scale"))
> enetTune
Elasticnet 

951 samples
228 predictors

Pre-processing: centered, scaled 
Resampling: Cross-Validated (10 fold) 

Summary of sample sizes: 856, 857, 855, 856, 856, 855, ... 

Resampling results across tuning parameters:

  lambda  fraction  RMSE   Rsquared  RMSE SD  Rsquared SD
  0       0.05      0.871  0.834     0.0382   0.0274     
  0       0.1       0.688  0.886     0.043    0.0206     
  0       0.15      0.673  0.891     0.0394   0.0184     
  0       0.2       0.675  0.89      0.0381   0.0176     
  0       0.25      0.688  0.887     0.0438   0.0195     
  0       0.3       0.697  0.884     0.0481   0.0206     
  0       0.35      0.706  0.881     0.0519   0.0216     
  0       0.4       0.713  0.879     0.0535   0.0219     
  0       0.45      0.714  0.879     0.0534   0.0218     
  0       0.5       0.714  0.879     0.0546   0.0218     
  0       0.55      0.714  0.878     0.0558   0.0221     
  0       0.6       0.714  0.879     0.0574   0.0223     
  0       0.65      0.714  0.879     0.0594   0.0227     
  0       0.7       0.715  0.879     0.0612   0.023      
  0       0.75      0.715  0.878     0.0629   0.0234     
  0       0.8       0.716  0.878     0.0645   0.0237     
  0       0.85      0.717  0.878     0.0656   0.0238     
  0       0.9       0.718  0.878     0.0667   0.024      
  0       0.95      0.719  0.877     0.0683   0.0242     
  0       1         0.721  0.877     0.0699   0.0245     
  0.01    0.05      1.52   0.644     0.11     0.0788     
  0.01    0.1       1.13   0.767     0.075    0.0477     
  0.01    0.15      0.906  0.824     0.056    0.03       
  0.01    0.2       0.786  0.857     0.0493   0.0217     
  0.01    0.25      0.73   0.873     0.0417   0.0207     
  0.01    0.3       0.699  0.883     0.0426   0.0203     
  0.01    0.35      0.687  0.886     0.0421   0.0197     
  0.01    0.4       0.681  0.888     0.04     0.0185     
  0.01    0.45      0.678  0.89      0.0361   0.0172     
  0.01    0.5       0.676  0.89      0.0331   0.0162     
  0.01    0.55      0.674  0.891     0.0307   0.0157     
  0.01    0.6       0.675  0.891     0.0304   0.0158     
  0.01    0.65      0.677  0.89      0.0318   0.0164     
  0.01    0.7       0.68   0.89      0.0336   0.0171     
  0.01    0.75      0.683  0.889     0.0355   0.0176     
  0.01    0.8       0.686  0.888     0.0372   0.0179     
  0.01    0.85      0.69   0.887     0.0389   0.0182     
  0.01    0.9       0.693  0.886     0.0405   0.0186     
  0.01    0.95      0.697  0.884     0.0418   0.019      
  0.01    1         0.701  0.883     0.0428   0.0193     
  0.1     0.05      1.69   0.516     0.132    0.0888     
  0.1     0.1       1.41   0.695     0.107    0.0658     
  0.1     0.15      1.17   0.76      0.0865   0.0462     
  0.1     0.2       1.01   0.788     0.0659   0.0376     
  0.1     0.25      0.895  0.822     0.0583   0.0281     
  0.1     0.3       0.819  0.844     0.0517   0.0222     
  0.1     0.35      0.774  0.857     0.0472   0.0208     
  0.1     0.4       0.752  0.864     0.0418   0.0196     
  0.1     0.45      0.734  0.871     0.0381   0.0187     
  0.1     0.5       0.725  0.875     0.0354   0.0184     
  0.1     0.55      0.718  0.878     0.0329   0.0179     
  0.1     0.6       0.714  0.88      0.0318   0.0176     
  0.1     0.65      0.711  0.882     0.031    0.017      
  0.1     0.7       0.71   0.882     0.0297   0.0164     
  0.1     0.75      0.71   0.883     0.0295   0.016      
  0.1     0.8       0.71   0.884     0.03     0.0158     
  0.1     0.85      0.709  0.884     0.0306   0.0157     
  0.1     0.9       0.709  0.885     0.0311   0.0155     
  0.1     0.95      0.709  0.885     0.0317   0.0155     
  0.1     1         0.709  0.885     0.0321   0.0156     

RMSE was used to select the optimal model using  the smallest value.
The final values used for the model were fraction = 0.15 and lambda = 0. 
> 
> plot(enetTune)
> 
> testResults$Enet <- predict(enetTune, solTestXtrans)
> 
> ################################################################################
> ### Session Information
> 
> sessionInfo()
R version 3.0.1 (2013-05-16)
Platform: x86_64-apple-darwin10.8.0 (64-bit)

locale:
[1] C

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] elasticnet_1.1                  lars_1.2                       
[3] pls_2.3-0                       corrplot_0.71                  
[5] caret_6.0-22                    ggplot2_0.9.3.1                
[7] lattice_0.20-15                 AppliedPredictiveModeling_1.1-5

loaded via a namespace (and not attached):
 [1] CORElearn_0.9.41   MASS_7.3-26        RColorBrewer_1.0-5 car_2.0-17        
 [5] codetools_0.2-8    colorspace_1.2-2   compiler_3.0.1     dichromat_2.0-0   
 [9] digest_0.6.3       foreach_1.4.0      grid_3.0.1         gtable_0.1.2      
[13] iterators_1.0.6    labeling_0.1       munsell_0.4        plyr_1.8          
[17] proto_0.3-10       reshape2_1.2.2     scales_0.2.3       stringr_0.6.2     
[21] tools_3.0.1       
> 
> q("no")
> proc.time()
   user  system elapsed 
540.993  74.917 615.942 
