Hyper-Parameter Optimization

This module provides a hyperparameter optimization (HPO) framework for an arbitrary use of an arbitrary training program. This is the documentation for the hyper-parameter optimization (HPO) module.

Getting Started

To use this framework, a user must make some changes to their model training code that will be used to evaluate a set of hyperparameters, and write a small wrapper program to set up the hyperparameter optimization.

Steps to making changes to an existing model training program:

  • Add command line flags for hyperparameters to be optimized.

  • Add a unique string followed by a value for the figure of merit (FoM) to be minimized during training.

Steps to writing a wrapper program to set up HPO:

  • Create an Evaluator instance.

  • Create a Params instance.

  • Create an Optimizer instance, which takes the Evaluator as an argument.

  • Call the optimize method, which takes the Params as an argument.

The optimizer will search the hyperparameter space for the set of hyperparameters that yields the minimal figure of merit (FoM). To switch the behavior from miniminizing to maximinizing, one can reverse the sign on the FoM value being printed from their training program.

See the API documentation below for further details.

See examples/sin for a simple example and examples/mnist-distributed for an example of using the slurm-integration as well as distributed evaluation.

Frequently Asked Questions

How do I specify an allocation constraint for Slurm?

Allocation constraints can be specified in two different ways:

1. Set the alloc_args initializer argument of Evaluator, e.g. hpo.Evaluator(..., alloc_args='-C haswell')

2. Set SALLOC_CONSTRAINT, which changes the default value for -C flag. More info can be found in the Slurm documentation.

Contents: