Can't predict on randomForest when test set contains NA's in features

[florianfendt]

I don't know if this is a bug in some sort or if I'm overlooking something, but this baffled @ja-thomas and me a bit this morning.
Consider a simple case where you have a missing value somewhere in your test set like in this example:

lrn.rf = makeLearner("classif.randomForest")
mod = train(lrn.rf, iris.task)
test.df = getTaskData(iris.task)
test.df[1L, 1L] = NA

mlr then throws an error when you try to predict on this set, randomForest's predict method doesn't though:

# throws error: row names contain missing values
predict(mod, newdata = test.df)
# if I'm directly using the predict method from randomForest it works
predict(mod$learner.model, test.df)

I tried printing out .newdata in predictLearner.classif.randomForest to see if we do sth unwanted with the data.frame before sending it to the learner's predict method but row names / str etc. looks fine.
Any ideas?

[larskotthoff]

This sounds like a bug. Could you make a unit test that reproduces it please?

[berndbischl]

before you produce a unit test:
please really show the produced output in such cases

[berndbischl]

This sounds like a bug. Could you make a unit test that reproduces it please?

why are you asking him that? he already kinda posted that test....?

the problem is more: what happens here?

[berndbischl]

the real problem seems to be:
mlr claims that the rf does not handle missing values. so if you would put the NA into the task data,
neither training NOR prediction would work, and you would get a meaningful error message.

but here the prediction data frame is directly handled. and the underlying RF just creates an NA for the prediction. this also links to the issue
#1499

we should probably create the task description internally and sanity check it.

[QuayAu]

This problem does not only occur with the random forest. I ran every available learner on this problem and here is what i found out:

test.df = getTaskData(iris.task)
test.df[1L, 1L] = NA
learners = listLearners(obj = "classif", properties = "multiclass")$class
res = lapply(learners, function(x) {
  mod = train(makeLearner(x), iris.task)
  tryCatch(predict(mod, newdata = test.df), error = function(e) conditionMessage(e))
  }
)

So the problem here is, that only the test set contains missing values, and some learners support that, others don't. Here is a small summary of the results:

learner	predict
classif.bdk	setosa
classif.boosting	setosa
classif.C50	setosa
classif.cforest	setosa
classif.ctree	setosa
classif.cvglmnet	NA
classif.dbnDNN	NA
classif.earth	invalid subscript type 'list'
classif.evtree	setosa
classif.extraTrees	setosa
classif.featureless	versicolor
classif.fnn	no missing values are allowed
classif.gausspr	arguments imply differing number of rows: 150, 149
classif.gbm	setosa
classif.geoDA	NA
classif.glmnet	setosa
classif.h2o.deeplearning	setosa
classif.h2o.gbm	setosa
classif.h2o.randomForest	setosa
classif.IBk	setosa
classif.J48	setosa
classif.JRip	setosa
classif.kknn	arguments imply differing number of rows: 150, 149
classif.knn	no missing values are allowed
classif.ksvm	arguments imply differing number of rows: 150, 149
classif.lda	NA
classif.LiblineaRL1L2SVC	NA/NaN/Inf in foreign function call (arg 2)
classif.LiblineaRL1LogReg	NA/NaN/Inf in foreign function call (arg 2)
classif.LiblineaRL2L1SVC	NA/NaN/Inf in foreign function call (arg 2)
classif.LiblineaRL2LogReg	NA/NaN/Inf in foreign function call (arg 2)
classif.LiblineaRL2SVC	NA/NaN/Inf in foreign function call (arg 2)
classif.LiblineaRMultiClassSVC	NA/NaN/Inf in foreign function call (arg 2)
classif.linDA	NA
classif.lssvm	arguments imply differing number of rows: 150, 149
classif.lvq1	no missing values are allowed
classif.mda	arguments imply differing number of rows: 150, 149
classif.mlp	missing values in 'x'
classif.multinom	NA
classif.naiveBayes	setosa
classif.nnet	NA
classif.nnTrain	NA
classif.OneR	setosa
classif.PART	setosa
classif.qda	NA
classif.quaDA	NA
classif.randomForest	row names contain missing values
classif.randomForestSRC	setosa
classif.ranger	Missing data in columns: Sepal.Length.
classif.rda	virginica
classif.rFerns	NAs in predictors.
classif.rknn	no missing values are allowed
classif.rpart	setosa
classif.RRF	row names contain missing values
classif.rrlda	invalid subscript type 'list'
classif.saeDNN	NA
classif.sda	NA
classif.sparseLDA	NA
classif.svm	arguments imply differing number of rows: 150, 149
classif.xgboost	setosa
classif.xyf	setosa

[larskotthoff]

Well we already have the missing learner property for that, so we just need to check that in predict().