‘My thesis @ClearboxAI is a blogpost series that summarises the various graduate research projects conducted at Clearbox AI. These experimental works are conducted by Master students from Italian and European universities who collaborated with Clearbox AI to deep dive into advanced topics in Machine Learning to apply R&D results in practice.’
Artificial Intelligence (AI) methodologies are increasingly applied to a plethora of different and possibly critical decision making use cases. Their technical performance as well as their explainability and trustworthiness are crucial for them to meaningfully fit in human lives and become responsibly pervasive in society.
Despite the leap forward in performances that occurred during the last few years, most of the Machine Learning (ML) and Deep Learning (DL) algorithms still involve manual dataset-specific fine-tuning. This makes the models’ predictions tightly coupled with the input and resulting in the entire pipeline to be heavily problem-specific, requiring a high degree of domain expertise usually provided by ML experts. The rise of AutoML is trying to fill this gap by automating several steps in the process with the goal of providing optimised and accessible off-the-shelf optimized models that are agnostic of the input data.
my thesis is enclosed in the broader scope of the MLOps solution built by Clearbox AI which strives to explain black boxes models by using deep generative models and following the Trustworthy AI principles. In particular, the goal of this thesis was to build a robust and reliable optimization framework modular to the Clearbox AI Control Room and whose purpose would be to leverage cutting edge Automated Machine Learning (AutoML) techniques to tune generative models.
What is AutoML?
Automated Machine Learning (AutoML) refers to the set of processes and methods related to the automation of different steps across the ML pipeline on a constrained computational budget. In a nutshell, AutoML’s target is achieved through one or more of the following techniques:
- Automated Data Cleaning;
- Automated Feature Engineering (Auto FE);
- Hyperparameter Optimization (HPO);
- Neural Architecture Search (NAS).
In this thesis, AutoML was primarily applied to automate and optimise the hyperparameters of a generative model. Secondarily, this work explored automated feature engineering and neural architecture search methodologies.
Methodology and results
The purpose of this work was to develop a model and dataset agnostic framework complementary to the Clearbox AI Control Room, requiring production needs such as accuracy, latency and explainability. To achieve this objective, we tested different optimisation techniques on a set of different datasets and by using various model’s variants. In particular, the selected data sources had dissimilar statistical features as well as numerous instances and cardinalities.
Firstly, the input dataset triggers an automated data preparation pipeline built to optimally preprocess different sources and provide the models with the best possible representation of the data: this leverages multiple statistical indicators and historical examples to improve data quality.
This work explored different AutoML optimisation techniques. Starting with different families of HPO algorithms, we constantly monitored the trade-off between performances and speed of execution. Furthermore, given the inputs, the study involved NAS to find the top-performing architecture.
These experiments resulted in a comparative analysis which allowed us to evaluate the differences between the optimisation techniques principally in terms of accuracy and speed. Furthermore, we developed multiple quantitative and qualitative metrics to assess the robustness and precision of the results.
This experimental work, enclosed in the broader scope of the Clearbox AI Control Room, represents a stepping stone towards future studies aimed at enhancing the performances of Deep Learning models towards the implementation of AutoML methodologies.
More specifically, the scope of this framework can be extended to unstructured data such as images, texts and audios. Moreover, the work can be applied to further explore other branches of AutoML.