Development Feature: HPO visualization and hyperparameter recommendation

Utility script visualizing and analyzing HPO results. Methods defined here often accepts dataframe that are the results of an HPO run.

crayai.development.analytics.hpo.plot_utilities._calc_perc_good_evaluations(experiment_df: pandas.core.frame.DataFrame, metric: str, metric_threshold: str) → float

Returns what percentage of the records with metric greater or equal to threshold. Dataframe contains the result of a HPO run

crayai.development.analytics.hpo.plot_utilities._derive_metric(experiment_df, metric, source_metric)

Calculate metric as (1-source_metric). Dataframe contains the result of a HPO run

crayai.development.analytics.hpo.plot_utilities._get_experiment_data(experiments: Dict[str, str])

Iterator method to load experiment data. Input is a dictionary with experiment name as key and log file name as value

crayai.development.analytics.hpo.plot_utilities._parallel_plot(df: ‘pandas.DataFrame’, color_column: ‘str’ = '', categorical_columns: ‘List[str]’ = [], drop_columns: ‘List[str]’ = [], title: ‘str’ = '', height: ‘int’ = 400) → ’plotly.figure’

Display parallel coordinate plot of the dataframe columns except the fields in drop_columns. Categorical columns are converted to numeric values. Dataframe contains the result of a HPO run

crayai.development.analytics.hpo.plot_utilities._plot_number_of_evaluations(experiments: Dict[str, str]) → ’matplotlib.axes._subplots.AxesSubplot’

Plot number of evaluations in each experiment. Input to the method include experiments which is a dictionary with key as experiment name and value as the log file name.

crayai.development.analytics.hpo.plot_utilities._plot_percentage_evaluations_passing_metric_threshold(experiments: Dict[str, str], derive_metric_flag: bool = True, metric: str = 'accuracy', source_metric: str = 'fom', metric_threshold: float = 0.8) → ’matplotlib.axes._subplots.AxesSubplot’

Plot percentange of evaluations with metric value greater or equal to metric_threshold. Input to the method include experiments which is a dictionary with key as experiment name and value as the log file name. This method also calculates metric as (1 - DataFrame[source_metric]) unless derive_metric_flag is set to False.