HPO Analytics API

HPO visualization and hyperparameter recommendation

Code for recommending hyperparameters. For example usecases, please refer to example_hp_analysis.ipynb

class crayai.analytics.hpo.recommendation.HyperparameterRecommender(dataset: str = 'Point.log', path: str = '', hyperparameter_column: str = 'hyperparameters', parse: bool = True, metadata: str = 'metadata.json', lazy_loading: bool = False, name: str = '')

Class for recommending hyperparameter and show their interaction

calc_stat(self, df: pandas.core.frame.DataFrame, by_col: str, metric: str, op: str = 'max') → pandas.core.frame.DataFrame

Calculate min or max statistics on metric attribute for a given value of by_col attribute.

Parameters

df: pd.DataFrame

dataframe containing dataset

by_col: str

column of the dataframe for grouping by

metric: str

column to compute calculate stats on

op: str

flag used to determine type of statistics. minimum is used when flag is min

Returns

pd.DataFrame

a data frame containing aggregated data

check_data(self)

Check if the dataset is ready

static generate_scale_data(learning_rate_name: str, learning_rates: List[float]) → Dict[str, List[Any]]

Generate scale data such as num_nodes, runtime and add noise variation to learning rate

get_best_hyperparameter(self, by: str, metric: str, op: str = 'max')

Provide a table of hyperparameters with the maximum (or minimum) metric for each value of the attribute by

get_data(self) → pandas.core.frame.DataFrame

Return data as data frame

static infer_columns(df: pandas.core.frame.DataFrame, split_char: str = ' ', param_value_split_char: str = '=', hp_col: str = 'hyperparameters') → List[str]

Extract all the column names by looking at the entire data frame

load_dataset(self, filename: str = '')

Utility to load a dataset from file. Parsing of a hyperparameter field is not done when dataset is loaded. If parsing is required, then use load_log method

static load_experiments(files: List[str]) → pandas.core.frame.DataFrame

Load a list of HPO logs into memory and return as one dataframe

load_log(self, filename: str = '')

Utility to load hpo logs from file. Hyperparameter column (e.g., –lr=0.01 –weight_decay=1e-5) is parsed so that a separate column is created for each individual hyperparameter

static plot_experiments(experiments: List[_ForwardRef(‘HyperparameterRecommender’)])

Visualize HPO metrics over time for a list of HPO experiments

static plot_experiments_from_files(files: List[str])

Visualize HPO metrics over time for a list of HPO log files

plot_hyperparameter(self, x: str, y: str, color: str = '', xaxis_scale='linear', yaxis_scale='linear')

Provide scatter visualization of the hyperparameter interaction. Users must specify hyperparamters to plot along x-axis and y-axis and can choose to specify a third hyperparameter as color. Interactive visualization will allow users to change hyperparameters using dropdown menus. Switching across numeric and categorical hyperparamers using the dropdown menu is somewhat limited.

static preprocess_log_hp_entry(df: pandas.core.frame.DataFrame, split_char: str = ' ', param_value_split_char: str = '=', hp_col: str = 'hyperparameters') → pandas.core.frame.DataFrame

Parse hyperparameter entry from the hpo log file

preview(self, n: int = 5) → pandas.core.frame.DataFrame

Return first n records of the data

show_metadata(self)

Display metadata related to the HPO experiment. It requires a json file with metadata to exist in the experiment folder