Agfdhyk

I'm fitting classification models for binary issues using MLR package in R. For each model, I perform a cross-validation with embedded feature selection using "selectFeatures" function and retrieve mean AUCs over test sets. I would like next to retrieve predictions on the test sets for each fold but this function does not seem to support that. I already tried to plug selected predictors into the "resample" function to get it. It works, but performance metrics are different which is not suitable for my analysis. I also tried to check in caret package if it is possible but I have not seen a solution at first glance. Any idea how to do it?

Here is my code with synthetic data and with my attempt with "resample" function (again: not suitable in this current version as performance metrics are different) .

# 1. Find a synthetic dataset for supervised learning (two classes)

###################################################################



install.packages("mlbench")

library(mlbench)

data(BreastCancer)



# generate 1000 rows, 21 quantitative candidate predictors and 1 target variable 

p<-mlbench.waveform(1000) 



# convert list into dataframe

dataset<-as.data.frame(p)



# drop thrid class to get 2 classes

dataset2  = subset(dataset, classes != 3)



# 2. Perform cross validation with embedded feature selection

#############################################################



library(BBmisc)

library(nnet)

library(mlr)



# Choice of algorithm i.e. neural network

mL <- makeLearner("classif.nnet", predict.type = "prob")



# Choice of sampling plan: 10 fold cross validation with stratification of target classes 

mRD = makeResampleDesc("CV", iters = 10,stratify = TRUE)



# Choice of feature selection strategy   

ctrl = makeFeatSelControlSequential(method = "sffs", maxit = NA,alpha = 0.001)



# Choice of feature selection technique (stepwize family) and p-value 

mFSCS = makeFeatSelControlSequential(method = "sffs", maxit = NA,alpha = 0.001)



# Choice of seed 

set.seed(12)



# Choice of data 

mCT <- makeClassifTask(data =dataset2, target = "classes")



# Perform the method

result = selectFeatures(mL,mCT, mRD, control = ctrl, measures = list(mlr::auc,mlr::acc,mlr::brier))



# Retrieve AUC and selected variables

analyzeFeatSelResult(result)

# Result: auc.test.mean=0.9614525 Variables selected: x.10, x.11, x.15, x.17, x.18    



# 3. Retrieve predictions on tests sets (to later perform Delong tests on AUCs derived from multiple sets of candidate variables)

#################################################################################################################################



# create new dataset with selected predictors

keep <- c("x.10","x.11","x.15","x.17","x.18","classes")

dataset3 <- dataset2[ , names(dataset2) %in% keep]



# Perform same tasks with  resample function instead of selectFeatures function to get predictions on tests set

mL <- makeLearner("classif.nnet", predict.type = "prob")   

ctrl = makeFeatSelControlSequential(method = "sffs", maxit = NA,alpha = 0.001)

mRD = makeResampleDesc("CV", iters = 10,stratify = TRUE)

set.seed(12)

mCT <- makeClassifTask(data =dataset3, target = "classes")

r1r = resample(mL, mCT, mRD, measures = list(mlr::auc,mlr::acc,mlr::brier))

# Result: auc.test.mean=0.9673023

ctrl is missing in your code.

For getting predictions of your resample object, just use getRRPredictions(r1r) or
r1r$measures.test.