Machine learning (ML) models are very powerful and have been used to solve a wide range of problems in the industry. However, in reality, ML systems are much more than the models that encapsulate them. Building and deploying ML models can become a complex and time-consuming process. This is where AutoML comes in.
AutoML is a field of machine learning that automates the process of building production ML systems, which includes tasks such as data preprocessing, feature engineering, model selection, and hyperparameter tuning.
MLBox is a powerful Automated Machine Learning python library. It provides the following features:
More info on the library can be found on their wiki. Home — Welcome to MLBox’s official documentation — MLBox Documentation
It has done really well in Kaggle competitions like Two Sigma Connect: Rental Listing Inquiries | Kaggle (rank 85/2488) and Sberbank Russian Housing Market | Kaggle (rank 190/3274) demonstrating its efficacy
The library aims to abstract away all the tedious work of data munging, wrangling, modeling, etc. so that engineers can build solutions quickly and even people with less experience in ML can develop ML applications.
To explore MLBox’s capabilities, let’s build an ML pipeline to predict a movie’s popularity. I will be using the dataset from Kaggle’s Movie Popularity Prediction | Kaggle competition for this.
Installing MLBox is pretty easy, with a few extra steps if you’re on MacOS:
Because the library hasn’t been updated in over 4 years, it hasn’t really kept up with the evolving landscape of its dependencies. Hence, you are going to have to manually remove deprecated pieces of code or update them in order to get MLBox to work. I had to:
class Reader():
...
def clean(self, path, drop_duplicate=False):
df = pd.read_csv(path,
sep=self.sep,
header=self.header,
engine='c',
on_bad_lines='warn')
...
Once all that’s done, we can finally use MLBox like how it was meant to be used. However, it does not support complex data types in dataframes like lists, and dictionaries. Hence, we need to get rid of those columns in the dataset.
Read the dataset and remove columns containing complex data types
def read_df(path):
df = pd.read_csv(path, index_col=0)
columns_to_remove = ['genres', 'production_companies', 'production_countries', 'spoken_languages', 'keywords']
df = df.drop(columns_to_remove, axis=1)
return df
Finally, you’ll want to save this dataset to be read by MLBox
# Read
train_df = read_df('movies_train.csv')
test_df = read_df('movies_test.csv')
# Write
train_df.to_csv('train.csv', index=False)
test_df.to_csv('test.csv', index=False)
First, import MLBox
from mlbox.preprocessing import *
from mlbox.optimisation import *
from mlbox.prediction import *
Then let’s read the dataset using Reader which provides some handy abstractions such as train_test_split and clean. We use the train_test_split method directly which given a list of several paths and a target name, automatically creates and cleans train and test datasets. Keep in mind that the test dataset should not have a target column.
paths = ['train.csv', 'test.csv']
target_name = 'popularity'
data = Reader(sep=",").train_test_split(paths, target_name)
This gives some nice stats about the dataset
reading csv : train.csv ... cleaning data ...
CPU time: 16.03498101234436 seconds
reading csv : test.csv ... cleaning data ...
CPU time: 1.457719087600708 seconds
> Number of common features : 16
> gathering and crunching for train and test datasets ...
> reindexing for train and test datasets ...
> dropping training duplicates ...
> dropping constant variables on training set ...
> Number of categorical features: 11
> Number of numerical features: 5
> Number of training samples : 31801
> Number of test samples : 13629
> Top sparse features (% missing values on train set):
> homepage 82.8
> tagline 55.1
> overview 2.1
> poster_path 0.9
> runtime 0.6
> dtype: float64
> Task : regression
> count 31801.000000
> mean 2.927718
> std 6.258147
> min 0.000000
> 25% 0.382886
> 50% 1.123409
> 75% 3.669545
> max 547.488298
Name: popularity, dtype: float64
Then, we can also automatically analyze and remove columns with highest drift in the dataset using Drift_thresholder
data = Drift_thresholder().fit_transform(data)
Which tells you the variables with greatest drift (the ones it removed)
computing drifts ...
CPU time: 1.618678092956543 seconds
> Top 10 drifts
('crew', 0.9997069972417014)
('poster_path', 0.9989472555206005)
('overview', 0.9982590147930175)
('cast', 0.9970838494208243)
('original_title', 0.9490023691248313)
('title', 0.9117541246799499)
('tagline', 0.6895282076095619)
('release_date', 0.3518149143942706)
('homepage', 0.306738200768061)
('runtime', 0.010559629542386428) > Deleted variables : ['cast', 'crew', 'original_title', 'overview', 'poster_path', 'tagline', 'title']
> Drift coefficients dumped into directory : save
Columns with drift greater than 0.5 are said to be unstable. In our case, cast, crew, original_title, overview, poster_path, tagline and title have high drifts and are removed.
Now, we can quickly baseline the data using MLBox’s default configuration using the Optimiser class. If you want to use your own configuration, you can pass your params dictionary to the first argument. Read more here
opt = Optimiser()
opt.evaluate(None, data)
Which gives us the negative mean squared error of predicted movie popularity values using LightGBM.
No parameters set. Default configuration is tested
##################################################### testing hyper-parameters... #####################################################
>>> NA ENCODER :{'numerical_strategy': 'mean', 'categorical_strategy': '<NULL>'} >>> CA ENCODER :{'strategy': 'label_encoding'} >>> ESTIMATOR :{'strategy': 'LightGBM', 'boosting_type': 'gbdt', 'class_weight': None, 'colsample_bytree': 0.8, 'importance_type': 'split', 'learning_rate': 0.05, 'max_depth': -1, 'min_child_samples': 20, 'min_child_weight': 0.001, 'min_split_gain': 0.0, 'n_estimators': 500, 'n_jobs': None, 'num_leaves': 31, 'objective': None, 'random_state': None, 'reg_alpha': 0.0, 'reg_lambda': 0.0, 'subsample': 0.9, 'subsample_for_bin': 200000, 'subsample_freq': 0, 'nthread': -1, 'seed': 0} [LightGBM] [Info] Auto-choosing row-wise multi-threading, the overhead of testing was 0.001377 seconds.
You can set `force_row_wise=true` to remove the overhead.
And if memory is not enough, you can set `force_col_wise=true`.
[LightGBM] [Info] Total Bins 1641
[LightGBM] [Info] Number of data points in the train set: 15900, number of used features: 9
[LightGBM] [Info] Start training from score 2.962931
[LightGBM] [Info] Auto-choosing row-wise multi-threading, the overhead of testing was 0.000729 seconds.
You can set `force_row_wise=true` to remove the overhead.
And if memory is not enough, you can set `force_col_wise=true`.
[LightGBM] [Info] Total Bins 1635
[LightGBM] [Info] Number of data points in the train set: 15901, number of used features: 9
[LightGBM] [Info] Start training from score 2.892507
MEAN SCORE : neg_mean_squared_error = -88.25482101831459
VARIANCE : 42.215979239246586 (fold 1 = -130.47080025756117, fold 2 = -46.038841779068)
CPU time: 5.132155179977417 seconds
As you can see, it automatically encodes categorical columns, imputes missing values, fits and predicts a model with cross validation. In this case LightGBM
You can also automatically perform the best hyperparameter search. The hyperparameter syntax closely follows scikit-learns syntax for GridSearch or RandomSearch, presumably because it uses scikit-learn under the hood.
space = {
'ne__numerical_strategy' : {"space" : [0, 'mean']},
'ce__strategy' : {"space" : ["label_encoding", "random_projection", "entity_embedding"]},
'fs__strategy' : {"space" : ["variance", "rf_feature_importance"]},
'fs__threshold': {"search" : "choice", "space" : [0.1, 0.2, 0.3]},
'est__strategy' : {"space" : ["LightGBM"]},
'est__max_depth' : {"search" : "choice", "space" : [5,6]},
'est__subsample' : {"search" : "uniform", "space" : [0.6,0.9]}
}
best_params = opt.optimise(space, data, max_evals = 5) Which will give you the best hyperparameters in its output
Finally, it’s time to actually train and use the model to predict the popularity of a movie. This is can be done in just two lines with MLBox
p = Predictor()
predictions = p.fit_predict(best_params, data)
Barring, the bugs because of outdated code, it was quite easy to clean the data, train a fairly good model, and predict movie popularity. In a production system, one would definitely explore MLBox’s features more and perform extensive testing. Nevertheless, the library is quite good and does a decent enough job at automating machine learning.
MLBox has a lot of the advantages of AutoML, along with some distinct strengths:
While MLBox does offer all of these cool features, the repository is no longer maintained actively and has failed to keep up with updates to packages it depends on like scikit-learn, Pandas, and even Python. To get it working, you will have to look at the source code and remove or upgrade deprecated code yourself. Luckily, this is possible because it’s open source. Further, it is limited in its capabilities and seems to work well with standard or well formed datasets consisting of conventional data types. For e.g. you are not able to use bag-of-words of TfIdf vectorization.
In conclusion, it is a great library to get your feet wet with the world of AutoML, however, you’ll probably be able to find more up to date and robust AutoML frameworks online. MLBox can be a valuable tool in a production machine learning environment, but it is important to be aware of its strengths and limitations.