Planned functionality

[rhiever]

In the immediate future, datacleaner will:

• Encode all non-numerical variables as numerical variables
• Replace all NaNs with the median of the column or drop all NaN rows (configurable)

See this tweet chain for more ideas.

If anyone has more ideas, please add them here.

[jaumebp]

In my experience it is worth identifying ordinal variables (e.g. numerical grades) and handle then separately. In many cases these can be treated as continuous variables, but sometimes it is necessary to treat them as discrete ones. One example of this is missing value imputation. If treating them as continuous you may end up injecting fake values that then can mislead the downstream analysis.

Thanks for the project! I tested it on some of my biomedical datasets and compared the PCA before/after the cleaning. The only case where there were differences is a dataset with discrete variables (Exome sequencing) and specifically in the columns where some of the values were '0'. There was the following error message:
sys:1: DtypeWarning: Columns (6,19,131,225,404,416,515,651,833,945,975,986,1265,1327,1387,1494,1541,1558,1715,1737,1854,1875,1947,1980,2015,2024,2111,2132,2140,2165,2426,2652,2667,2668,2871,2943,2978,2997,3165,3335,3634,3807,3945,4010,4018,4177,4191,4196,4243,4245,4389,4463,4553,4772,4814,4841,4962) have mixed types. Specify dtype option on import or set low_memory=False.

[rhiever]

Indeed, which is why I'm trying to discover how to identify ordinal vs. continuous variables. I posted this question on StackOverflow to brainstorm.

[jaumebp]

In our software we went with a much simpler approach. Letting the user specify a list of attributes to be treated as ordinal. Of course, an automatic solution is far more elegant :)

[westurner]

"Convenience function: Detect if there are non-numerical features and encode them as numerical features" EpistasisLab/tpot#61

[westurner]

Do I have to do get_dummies() all by myself?
http://pandas.pydata.org/pandas-docs/stable/generated/pandas.get_dummies.html

... get_dummies() accepts a number of kwargs

[westurner]

Do I have to do get_dummies() all by myself?

I think it illogical to e.g. average Exterior1st in the Kaggle House Prices Dataset: the average of ImStucc and Wd Sdng seems nonsensical?

• https://www.kaggle.com/c/house-prices-advanced-regression-techniques/data
• https://www.kaggle.com/c/house-prices-advanced-regression-techniques/download/data_description.txt

[westurner]

CSVW as JSONLD may be a good way to specify a dataset header with the relevant metadata for such operations? pandas-dev/pandas#3402

[rhiever]

You should be able to use the sklearn OneHotEncoder to get the equivalent of the pandas get_dummies().

[westurner]

You should be able to use the sklearn OneHotEncoder to get the equivalent of the pandas get_dummies().

http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OneHotEncoder.html

Is there a way to specify that I only need certain columns to be expanded into multiple columns w/ OneHotEncoder?