Dev (#43)

* updated config

* added categorical_features to lgbm

* reverted to old config

* To english translate - notebook

* Replace 'talking-data' with 'avito' in logger

* Translate

* Delete notebook

* Text features

* Update pipelines.py

* Improve TextCleaner

* Improve TextCounter

* Fix

* added categorical encoding and missing and timestamp, not working yet

* updated config

* dropped image des

* added hashing trick for categorical encoding

* updated encoder configs

* Include suggested changes

* updated random search

* Update

* easthetic change with translation

* updated configs

* refactored text features

* fixed inference mode remnants

* added more groupings, moved item_seq to numericals, starter hyperparam search

* fixed pipelines eval mode defs

* running random search, feature extraction working

* fixed label encoding, and feature extraction for text, added train to lgb.train evals

* dropped user_id from categoricals, refactored _groupby_features

* week of year

* added log price

* added input missing, dropped feature dispatcher

* added tfidf, changed lightgbm to expect numpy, adjusted pipelines and configs

* params updated, small fixes

Author: jakubczakon

feature_cleaning.py

@@ -0,0 +1,23 @@
+import numpy as np
+
+from steps.base import BaseTransformer
+
+
+class InputMissing(BaseTransformer):
+    def __init__(self, text_columns,
+                 categorical_columns,
+                 numerical_columns,
+                 timestamp_columns):
+        self.text_columns = text_columns
+        self.categorical_columns = categorical_columns
+        self.numerical_columns = numerical_columns
+        self.timestamp_columns = timestamp_columns
+
+    def transform(self, X, **kwargs):
+        X_ = X.copy()
+        for col, input_value in [self.text_columns,
+                                 self.categorical_columns,
+                                 self.numerical_columns,
+                                 self.timestamp_columns]:
+            X_[col] = X_[col].fillna(input_value)
+        return {'clean_features': X_}