【数据分析】 处理缺失值的常规方法总结

发布者:湖北数据与分析中心发布时间:2016-05-27浏览次数:10

前言

  • 现实生活中的数据是纷繁杂乱的,收集来的数据有缺失和录入错误司空见惯,所以学习如果处理这些常见问题是每一个数据人必须掌握的技能,俗话说巧妇 难为无米之炊,不能很好的处理原始数据会给后来的建模带来麻烦,甚至引入不必要的偏差和错误,数据科学家都熟悉“垃圾进垃圾出”的说法,今天让我们来学习 成为合格数据人的必修课 —- 处理缺失值的常规办法。

数据介绍

  • 我们继续使用上一节的贷款数据集(数据内容请见上篇介绍)

  1. library(foreign)

  2. library(gmodels)

  3. library(mice)

  4. library(VIM)

  5. library(Hmisc)

  6. library(DMwR)

  7. loan.data = read.spss(Loan_ROC.sav, to.data.frame=TRUE)

  8. levels(loan.data$education)

  9. loan.data$education=factor(loan.data$education)

  10. levels(loan.data$education)

引入缺失值

  1. set.seed(1)

  2. data.copy = loan.data

  3. data.copy$debt_income[sample(1:nrow(data.copy), 10)] <- NA

  4. data.copy$education[sample(1:nrow(data.copy), 10)] <- NA

  5. levels(data.copy$education)

  6. data.copy$education=factor(data.copy$education)

  7. levels(data.copy$education)

MICE 包 —— 检查缺失值分布

  • 注意连续型数据和分类型数据的区别

  • 两类缺失值

    • MCAR: missing completely at random (desirable scenario; safe maximum threshold — 5% of total large dataset)

    • MNAR: missing not at random (wise and worthwhile to check data gathering process)

  • 检查行与列(自建公式)

  1. summary(data.copy)

  2. pMiss <- function(x){sum(is.na(x))/length(x)*100}

  3. apply(data.copy, 2, pMiss)

  4. apply(data.copy, 1, pMiss)

  5. md.pattern(data.copy)

  • r names(md.pattern(data.copy)[,1][1]) 个完整值

  • r names(md.pattern(data.copy)[,1][2]) 个观察值只缺失r names(md.pattern(data.copy)[1,][12])

  • r names(md.pattern(data.copy)[,1][3]) 个观察值只缺失r names(md.pattern(data.copy)[1,][13])

  • r names(md.pattern(data.copy)[,1][4]) 个观察值缺失前两个变量

VIM 包 + box plot 箱线图 —- 缺失值视觉化

  1. aggr_plot <- aggr(data.copy, col=c('blue','red'), numbers=TRUE, sortVars=TRUE, labels=names(data.copy), cex.axis=.7, gap=3, ylab=c(Missing data histogram,Missing value pattern))

  2. marginplot(data.copy[,c(5,2)], main='Missing value box plot\n Constrain: 2 variables')

  • debt_income 的红 - 缺失 education 的分布

  • debt_income 的蓝 - 其余不缺失 education 的分布

  • 希望看到两个变量分别的红和蓝条分布相似(注意数据类型的不同)

1. 处理方法 —- 删除行/列 (na.action=na.omit)

适用条件:


    1. 足够大数据集 (model doesn’t lose power)


    1. 不会引入偏差 (no disproportionate or non-representation of classes)


    1. 删列:不起重要预测作用的变量

2. 处理方法 —- 替换 (mean / median / mode)

适用条件:


    1. 不需要非常精确的估算


    1. 变量 variation is low,或者 low leverage over response variable

  1. impute(data.copy$debt_income, mean)  # replace with mean, indicated by star suffix

  2. impute(data.copy$debt_income, 20)  # replace with specific number

  3. # data.copy$debt_income[is.na(data.copy$debt_income)] <- mean(data.copy$debt_income, na.rm = T)  # impute manually

计算正确率 —- 替换 mean

  1. actuals <- loan.data$debt_income[is.na(data.copy$debt_income)]

  2. predicteds <- rep(mean(data.copy$debt_income, na.rm=T), length(actuals))

  3. regr.eval(actuals, predicteds)

3. 处理方法 —- 推测 (高大上的方法:kNN, rpart, and mice)

3.1. kNN

  1. # 推测方法:identify ‘k’ closest observations based on euclidean distance and computes the weighted average (weighted based on distance) of these ‘k’ obs.

  2. # 优点:can impute all missing values in all variables with one call to function. It takes whole dataframe as argument and don’t have to specify which variable to impute.

  3. # 注意:not to include response variable.

  1. knnOutput <- knnImputation(data.copy)

  2. anyNA(knnOutput)

  3. actuals <- loan.data$debt_income[is.na(data.copy$debt_income)]

  4. predicteds <- knnOutput[is.na(data.copy$debt_income), debt_income]

  5. regr.eval(actuals, predicteds)

  • The mean absolute percentage error (mape) has improved by ~ 65% compared to the imputation by mean.

3.2. rpart

  1. # 优点:比较于 kNN,rpart 和 mice 可以用于分类型数据;而且 rpart 仅需要一个 predictor variable to be non-NA

  2. # 注意:not to include response variable.

  3. # `method=class` for factor variable

  4. # `method=anova` for numeric variable

  1. library(rpart)

  2. class_mod <- rpart(education ~ ., data=data.copy[!is.na(data.copy$education), ], method=class, na.action=na.omit)

  3. anova_mod <- rpart(debt_income ~ ., data=data.copy[!is.na(data.copy$debt_income), ], method=anova, na.action=na.omit)

  4. education_pred <- predict(class_mod, data.copy[is.na(data.copy$education), ],type='class')

  5. debt_income_pred <- predict(anova_mod, data.copy[is.na(data.copy$debt_income), ])

  6. actuals <- loan.data$debt_income[is.na(data.copy$debt_income)] # debt_income accuracy

  7. predicteds <- debt_income_pred

  8. regr.eval(actuals, predicteds)


  9. actuals <- loan.data$education[is.na(data.copy$education)]

  10. mean(actuals != education_pred)  # misclass error for education accuracy

  • debt_income’s accuracy is slightly worse than kNN but still much better than imputation by mean

3.3. mice

  1. # 方法:2-step:mice() to build multiple models; complete() to generate one/several completed data (default is first).

  1. miceMod <- mice(data.copy[, !names(data.copy) %in% Loan], method=rf)  # based on random forests

  2. miceOutput <- complete(miceMod)  # generate completed data.

  3. anyNA(miceOutput)

  4. actuals <- loan.data$debt_income[is.na(data.copy$debt_income)]

  5. predicteds <- miceOutput[is.na(data.copy$debt_income), debt_income]

  6. regr.eval(actuals, predicteds)   # debt_income accuracy


  7. actuals <- loan.data$education[is.na(data.copy$education)]

  8. predicteds <- miceOutput[is.na(data.copy$education), education]

  9. mean(actuals != predicteds)  # misclass error education accuracy

  • The mean absolute percentage error (mape) has improved by ~ 62% compared to the imputation by mean.

  • Mis-classification error is the same as rpart’s 60%, which may be contributed to imbalanced classes and small sample.

  1. miceMod <- mice(data.copy[, !names(data.copy) %in% Loan], method=pmm)  # based on predictive mean matching

  2. miceOutput <- complete(miceMod)  # generate completed data.

  3. anyNA(miceOutput)

  4. actuals <- loan.data$debt_income[is.na(data.copy$debt_income)]

  5. predicteds <- miceOutput[is.na(data.copy$debt_income), debt_income]

  6. regr.eval(actuals, predicteds)   # debt_income accuracy


  7. actuals <- loan.data$education[is.na(data.copy$education)]

  8. predicteds <- miceOutput[is.na(data.copy$education), education]

  9. mean(actuals != predicteds)  # misclass error education accuracy

  • The mean absolute percentage error (mape) has improved by ~ 61% compared to the imputation by mean.

  • But mis-classification error is 50%, better than previous two methods.

4. 数据视觉化检测 —- 对比原始数据和推测值

4.1. scatterplot

  1. xyplot(miceMod,debt_income ~ age + year_emp + income, pch=18,cex=0.8)

  • Desirable scenario: magenta points (imputed) matches shape of blue  (observed). Matching shape 说明 imputed values are indeed “plausible  values”.

4.2. density plot

  1. densityplot(miceMod)

  • Magenta - density of imputed data for each imputed dataset

  • Blue - density of observed data

  • Desirable: similar distributions

4.3. stripplot plot

  1. stripplot(miceMod, pch = 20, cex = 1.2)

  • 点状图

5. 推测处理后的数据建模 —- pooling

  1. modelFit1 <- with(miceMod,lm(debt_income ~ age + year_emp + income))

  2. summary(pool(modelFit1))

  • fmi —- fraction of missing information

  • lambda —- proportion of total variance that is attributable to missing data

6. 备注(额外介绍)—- 处理 random seed initialization

  • mice function is initialed with a specific seed, so results are  dependent on initial choice. To reduce this effect, we impute a higher  number of dataset, by changing default m=5 parameter in mice() function

    1. modelFit2 <- with(tempData2,lm(debt_income ~ age + year_emp + income))

    2. summary(pool(modelFit2))

    1. tempData2 <- mice(data.copy[, !names(data.copy) %in% Loan], method='pmm',m=50, maxit=10, seed=0)