Google AI researchers have released a PyGlove library, a symbolic implementation of Automated Machine Learning (AutoML) that allows developers to experiment with search spaces, search algorithms, and search flows of an AutoML with only a few code lines. Now, developers can self-mutate Python classes and functions through brief Python annotations, making it much easier to write AutoML programs.
Developers previously had data and the outputs; they fed that into a machine learning algorithm, which automated the learning of rules governing input to output. Researchers later automated the selection and hyper-parameter tuning of those machine learning algorithms as well. One of the sub-classes of machine learning algorithms is neural networks, which are highly sensitive to architecture and hyper-parameters.
The possible combinations of architecture and hyper-parameter choices become humongous as researchers aim to build larger and larger neural models. They waste months in hand-crafting neural network architectures and selecting the right hyper-parameters. AutoML automated these aspects by formulating the problem as a search problem.
Also Read: What Is Liquid Machine Learning?
A search space is defined to represent all possible choices, and a search algorithm is used to find the best options. Neural Architecture Search (NAS) algorithms like ENAS and DARTS come under the purview of the AutoML. But the current implementations of NAS algorithms do not offer modularity to the components of NAS algorithms like the search space and search algorithm. Therefore, researchers had to face difficulties modifying the search space, search algorithm, or search flow alone.
The Google researchers introduced AutoML based on symbolic programming — a paradigm that allows self-mutating programs by manipulating its components — that makes components decoupled. This decoupling makes it easy for practitioners to change the search space and search algorithm with and without weight sharing, and add search capabilities to existing code and implement complex search flows.
On ImageNet and NAS-Bench-101 benchmarks, they showed that symbolic programming based PyGlove converts a static program into a search space, quickly iterates on the search spaces and search algorithms, and crafts complex search flows to achieve better results. PyGlove also allows easy plug-and-play of AutoML techniques in existing ML pipelines while also benefiting open-ended research.
