.. DO NOT EDIT.
.. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY.
.. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE:
.. "auto_examples/plot_xgboost_hyperparam_opt_comparison.py"
.. LINE NUMBERS ARE GIVEN BELOW.

.. only:: html

    .. note::
        :class: sphx-glr-download-link-note

        :ref:`Go to the end <sphx_glr_download_auto_examples_plot_xgboost_hyperparam_opt_comparison.py>`
        to download the full example code

.. rst-class:: sphx-glr-example-title

.. _sphx_glr_auto_examples_plot_xgboost_hyperparam_opt_comparison.py:


XGBoost - Genetic vs Grid Search vs Random Search vs Bayesian Optimization
==========================================================================
mloptimizer example optimization of iris dataset comparing hyperparameter tuning techniques:

1) Genetic optimization - mloptimizer
2) Grid Search - scikit-learn
3) Random Search - scikit-learn
4) Bayesian Optimization - hyperopt

.. GENERATED FROM PYTHON SOURCE LINES 13-16

Imports
-------
The necessary libraries for the example are imported.

.. GENERATED FROM PYTHON SOURCE LINES 16-40

.. code-block:: default


    import pandas as pd
    import numpy as np
    from time import time
    from functools import reduce
    import plotly

    from mloptimizer.domain.hyperspace import HyperparameterSpace, Hyperparam
    from mloptimizer.domain.evaluation import kfold_stratified_score
    from mloptimizer.application.reporting.plots import plotly_search_space
    from mloptimizer.interfaces import HyperparameterSpaceBuilder, GeneticSearch

    from sklearn.model_selection import GridSearchCV, RandomizedSearchCV, cross_val_score, \
        StratifiedKFold
    from sklearn.datasets import load_iris

    from xgboost import XGBClassifier

    from hyperopt import STATUS_OK, hp, tpe
    from hyperopt import Trials, fmin

    width = 1000
    height = 1000





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    /home/docs/checkouts/readthedocs.org/user_builds/mloptimizer/envs/alpha/lib/python3.10/site-packages/hyperopt/atpe.py:19: UserWarning:

    pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.





.. GENERATED FROM PYTHON SOURCE LINES 41-65

1) Dataset Description
----------------------
The Iris dataset is a classic dataset used for classification tasks.
It consists of 150 samples, each representing an iris flower.

Features:

- Sepal Length: Length of the sepal in centimeters.
- Sepal Width: Width of the sepal in centimeters.
- Petal Length: Length of the petal in centimeters.
- Petal Width: Width of the petal in centimeters.

Target:

- Species of the flower, which can be one of the following three classes:

  1. Setosa
  2. Versicolor
  3. Virginica

Characteristics:

- The dataset is balanced, containing 50 samples for each class.
- Features are continuous and have varying scales, which may require normalization or standardization for certain machine learning algorithms.

.. GENERATED FROM PYTHON SOURCE LINES 65-72

.. code-block:: default


    name = 'iris'
    X, y = load_iris(return_X_y=True)

    print(f"1) Description of the dataset")
    print(f"Dataset: {name}, X shape: {X.shape}, y shape: {y.shape}")





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    1) Description of the dataset
    Dataset: iris, X shape: (150, 4), y shape: (150,)




.. GENERATED FROM PYTHON SOURCE LINES 73-101

2) Genetic Search of XGBoost Algorithm
--------------------------------------------
Genetic search optimization is performed using the mloptimizer library to fine-tune the hyperparameters of the XGBoost
algorithm.

Hyperparameters to Optimize:

- `colsample_bytree`: Subsample ratio of columns when constructing each tree.
- `gamma`: Minimum loss reduction required to make a further partition on a leaf node of the tree.
- `learning_rate`: Step size shrinkage used in updates to prevent overfitting.
- `max_depth`: Maximum depth of a tree.
- `n_estimators`: Number of boosting rounds.
- `subsample`: Subsample ratio of the training instances.

Optimization Process:

- Population Size: 10
- Generations: 10
- Fitness Score: Balanced accuracy
- Evaluation Function: Stratified k-fold cross-validation with 5 folds

The genetic optimization explores the hyperparameter space defined by the evolvable hyperparameters.
It searches within the defined minimum and maximum values for each hyperparameter.

Advantages:

- Provides a more exhaustive search compared to grid search and random search.
- Can be more efficient in finding optimal hyperparameters for the XGBoost algorithm.

.. GENERATED FROM PYTHON SOURCE LINES 101-147

.. code-block:: default


    print(f"2) Genetic optimization of the algorithm XGBoost")

    fixed_hyperparams = {}
    evolvable_hyperparams = {
        'colsample_bytree': Hyperparam('colsample_bytree', 3, 10, 'float', 10),
        'gamma': Hyperparam('gamma', 0, 20, 'int'),
        'learning_rate': Hyperparam('learning_rate', 1, 100, 'float', 1000),
        'max_depth': Hyperparam('max_depth', 2, 20, 'int'),
        'n_estimators': Hyperparam('n_estimators', 100, 500, 'int'),
        'subsample': Hyperparam('subsample', 700, 1000, 'float', 1000)
    }
    hyperparameter_space = HyperparameterSpace(fixed_hyperparams, evolvable_hyperparams)

    population_size = 10
    generations = 10

    cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)
    opt = GeneticSearch(
        estimator_class=XGBClassifier,
        hyperparam_space=hyperparameter_space,
        **{"generations": generations, "population_size": population_size},
        #eval_function=kfold_stratified_score,
        cv=cv,
        scoring="balanced_accuracy", seed=0,
        use_parallel=False
    )


    t0_gen = time()
    clf = opt.fit(X,y)  # Aprox 100 elements
    t1_gen = time()
    print(f"Genetic optimization around {population_size * (generations + 1)} algorithm executions")
    execution_time_gen = round(t1_gen - t0_gen, 2)
    print(f"Time of the genetic optimization {execution_time_gen} s")
    population_df = opt.populations_
    print(f"Genetic optimization {population_df.shape[0]} algorithm executions")
    df = population_df[list(hyperparameter_space.evolvable_hyperparams.keys()) + ['fitness']]
    fig_gen = plotly_search_space(df).update_layout(
        autosize=True,
        width=width,  # Adjust width as needed
        height=height,  # Adjust height as needed
        margin=dict(l=20, r=20, t=50, b=20)  # Adjust margins as needed
    )
    plotly.io.show(fig_gen)




.. raw:: html
    :file: images/sphx_glr_plot_xgboost_hyperparam_opt_comparison_001.html


.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    2) Genetic optimization of the algorithm XGBoost

    Genetic execution:   0%|          | 0/11 [00:00<?, ?it/s, best fitness=?]
    Genetic execution:   9%|▉         | 1/11 [00:00<00:01,  6.42it/s, best fitness=0.94]
    Genetic execution:  18%|█▊        | 2/11 [00:01<00:08,  1.05it/s, best fitness=0.94]
    Genetic execution:  27%|██▋       | 3/11 [00:03<00:10,  1.30s/it, best fitness=0.94]
    Genetic execution:  27%|██▋       | 3/11 [00:04<00:10,  1.30s/it, best fitness=0.947]
    Genetic execution:  36%|███▋      | 4/11 [00:05<00:10,  1.45s/it, best fitness=0.947]
    Genetic execution:  45%|████▌     | 5/11 [00:06<00:08,  1.49s/it, best fitness=0.947]
    Genetic execution:  55%|█████▍    | 6/11 [00:08<00:07,  1.50s/it, best fitness=0.947]
    Genetic execution:  64%|██████▎   | 7/11 [00:10<00:06,  1.51s/it, best fitness=0.947]
    Genetic execution:  73%|███████▎  | 8/11 [00:11<00:04,  1.49s/it, best fitness=0.947]
    Genetic execution:  82%|████████▏ | 9/11 [00:13<00:03,  1.54s/it, best fitness=0.947]
    Genetic execution:  91%|█████████ | 10/11 [00:14<00:01,  1.56s/it, best fitness=0.947]
    Genetic execution: 100%|██████████| 11/11 [00:16<00:00,  1.63s/it, best fitness=0.947]
    Genetic execution: 100%|██████████| 11/11 [00:18<00:00,  1.66s/it, best fitness=0.947]
    Genetic optimization around 110 algorithm executions
    Time of the genetic optimization 18.76 s
    Genetic optimization 110 algorithm executions




.. GENERATED FROM PYTHON SOURCE LINES 148-172

3) Grid Search Optimization for XGBoost
---------------------------------------
Grid Search optimization is performed using the GridSearchCV class from the scikit-learn library.

Optimization Process:

- The hyperparameters to optimize are the same as those used in genetic optimization:

  - `colsample_bytree`
  - `gamma`
  - `learning_rate`
  - `max_depth`
  - `n_estimators`
  - `subsample`

- Search Strategy:

  - Grid Search performs an exhaustive search over a predefined hyperparameter space.
  - The search space and number of parameter combinations are fixed by the user.

Considerations:

- Grid Search is computationally expensive, especially for large hyperparameter spaces.
- It may not be feasible for extensive parameter tuning due to its exhaustive nature.

.. GENERATED FROM PYTHON SOURCE LINES 172-213

.. code-block:: default


    print(f"3) Grid Search of the algorithm Grid Search")


    xgb = XGBClassifier()
    parameters = {
        'colsample_bytree': (0.3, 0.5, 0.8),
        'gamma': (5, 20),
        'learning_rate': (0.001, 0.01, 0.1),
        'max_depth': (2, 10),
        'n_estimators': (300,),
        'subsample': (0.7, 0.8, 0.9)
    }
    gs_executions = reduce(lambda x, y: x * y, [len(parameters[k]) for k in parameters.keys()], 1)
    print(f"Grid Search optimization will run {gs_executions} algorithm executions")
    clf_gs = GridSearchCV(
        xgb,
        parameters,
        cv=5,
        scoring="balanced_accuracy",
    )

    t0_gs = time()
    clf_gs.fit(X, y)
    t1_gs = time()
    execution_time_gs = round(t1_gs - t0_gs, 2)

    print(f"Time of the grid search {execution_time_gs} s")

    synth_population_gs = pd.DataFrame(clf_gs.cv_results_['params'])
    synth_population_gs['fitness'] = clf_gs.cv_results_['mean_test_score']
    fig_gs = plotly_search_space(synth_population_gs).update_layout(
        autosize=True,
        width=width,  # Adjust width as needed
        height=height,  # Adjust height as needed
        margin=dict(l=20, r=20, t=50, b=20)  # Adjust margins as needed
    )
    plotly.io.show(fig_gs)

    print(f"Grid Search optimization has run {synth_population_gs.shape[0]} algorithm executions")




.. raw:: html
    :file: images/sphx_glr_plot_xgboost_hyperparam_opt_comparison_002.html


.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    3) Grid Search of the algorithm Grid Search
    Grid Search optimization will run 108 algorithm executions
    Time of the grid search 29.29 s
    Grid Search optimization has run 108 algorithm executions




.. GENERATED FROM PYTHON SOURCE LINES 214-238

4) Random Search Optimization for XGBoost
-----------------------------------------
Random Search optimization is performed using the RandomizedSearchCV class from the scikit-learn library.

Optimization Process:

- The hyperparameters to optimize are the same as those used in genetic optimization and grid search:

  - `colsample_bytree`
  - `gamma`
  - `learning_rate`
  - `max_depth`
  - `n_estimators`
  - `subsample`

- Search Strategy:

  - Random Search samples a specified number of random combinations from the predefined hyperparameter space.
  - The number of iterations (i.e., sampled parameter combinations) is fixed by the user.

Considerations:

- Random Search is less computationally expensive than Grid Search, making it more feasible for larger search spaces.
- It is not as exhaustive as genetic optimization but can be more efficient in exploring the hyperparameter space compared to Grid Search.

.. GENERATED FROM PYTHON SOURCE LINES 238-276

.. code-block:: default


    print(f"4) Random Search of the algorithm")
    distributions = {
        'colsample_bytree': np.linspace(0.3, 1, 10),
        'gamma': (0, 5, 20),
        'learning_rate': (0.001, 0.01, 0.1),
        'max_depth': (2, 5, 10, 20),
        'n_estimators': (100, 300, 500),
        'subsample': np.linspace(0.7, 0.9, 10)
    }
    rs_executions = reduce(lambda x, y: x * y, [len(distributions[k]) for k in distributions.keys()], 1)
    print(f"Random Search optimization could run {rs_executions} different algorithm executions")
    clf_rs = RandomizedSearchCV(
        xgb,
        distributions,
        cv=5,
        n_iter=110,
        random_state=0,
        scoring="balanced_accuracy"
    )

    t0_rs = time()
    search = clf_rs.fit(X, y)
    t1_rs = time()
    execution_time_rs = round(t1_rs - t0_rs, 2)
    print(f"Time of the grid search {execution_time_rs} s")

    synth_population_rs = pd.DataFrame(clf_rs.cv_results_['params'])
    synth_population_rs['fitness'] = clf_rs.cv_results_['mean_test_score']
    fig_rs = plotly_search_space(synth_population_rs).update_layout(
        autosize=True,
        width=width,  # Adjust width as needed
        height=height,  # Adjust height as needed
        margin=dict(l=20, r=20, t=50, b=20)  # Adjust margins as needed
    )
    print(f"Random Search optimization has run {synth_population_rs.shape[0]} algorithm executions")
    plotly.io.show(fig_rs)




.. raw:: html
    :file: images/sphx_glr_plot_xgboost_hyperparam_opt_comparison_003.html


.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    4) Random Search of the algorithm
    Random Search optimization could run 10800 different algorithm executions
    Time of the grid search 31.12 s
    Random Search optimization has run 110 algorithm executions




.. GENERATED FROM PYTHON SOURCE LINES 277-301

---------------------------------------
Bayesian Optimization is performed using the hyperopt library to optimize the hyperparameters of the XGBoost algorithm.

Optimization Process:

- The hyperparameters to optimize are the same as those used in genetic optimization, grid search, and random search:

  - `colsample_bytree`
  - `gamma`
  - `learning_rate`
  - `max_depth`
  - `n_estimators`
  - `subsample`

- Search Strategy:

  - Bayesian Optimization uses a probabilistic model to predict the performance of different hyperparameter combinations.
  - It iteratively evaluates the objective function (in this case, cross-validated balanced accuracy) to find the optimal hyperparameters.

Considerations:

- Bayesian Optimization is more efficient than random search and grid search for large search spaces.
- It can provide better results with fewer evaluations compared to random search and grid search.
- The number of evaluations is a parameter that can be adjusted to balance the trade-off between performance and computational cost.

.. GENERATED FROM PYTHON SOURCE LINES 301-351

.. code-block:: default


    print(f"5) Bayesian Optimization of the algorithm")
    bayesian_space = {
        'colsample_bytree': hp.uniform('colsample_by_tree', 0.3, 1.0),
        'gamma': hp.choice('gamma', np.arange(5, 50+1, dtype=int)),
        'learning_rate': hp.loguniform('learning_rate', np.log(0.01), np.log(1)),
        'max_depth': hp.choice('max_depth', np.arange(2, 10+1, dtype=int)),
        'n_estimators': hp.choice('n_estimators', np.arange(50, 500, dtype=int)),
        'subsample': hp.uniform('subsample', 0.6, 1.0)
    }

    def objective_function(params):
        clf = XGBClassifier(**params)
        score = cross_val_score(clf, X, y, cv=5, scoring="balanced_accuracy").mean()
        return {'loss': -score, 'status': STATUS_OK}

    trials = Trials()
    num_eval = 110

    print(f"Bayesian Search optimization will run {num_eval} algorithm executions")

    t0_bay = time()
    best_param = fmin(objective_function, bayesian_space, algo=tpe.suggest, max_evals=num_eval, trials=trials)
    t1_bay = time()
    execution_time_bay = round(t1_bay - t0_bay, 2)
    print(f"Time of the bayesian search {execution_time_bay} s")


    # Extract parameters and scores from trials
    trials_dict = trials.trials
    params_list = [{k: v[0] for k, v in trial['misc']['vals'].items()} for trial in trials_dict]
    scores_list = [-trial['result']['loss'] for trial in trials_dict]
    # Convert parameters to DataFrame
    synth_population_bay = pd.DataFrame(params_list)
    # Add scores to DataFrame
    synth_population_bay['fitness'] = scores_list
    # Rename columns to match the parameter names
    synth_population_bay.columns = [col.replace('vals_', '') for col in synth_population_bay.columns]

    fig_bay = plotly_search_space(synth_population_bay).update_layout(
        autosize=True,
        width=width,  # Adjust width as needed
        height=height,  # Adjust height as needed
        margin=dict(l=20, r=20, t=50, b=20)  # Adjust margins as needed
    )
    print(f"Bayesian Search optimization has run {num_eval} algorithm executions")
    plotly.io.show(fig_bay)






.. raw:: html
    :file: images/sphx_glr_plot_xgboost_hyperparam_opt_comparison_004.html


.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    5) Bayesian Optimization of the algorithm
    Bayesian Search optimization will run 110 algorithm executions

      0%|          | 0/110 [00:00<?, ?trial/s, best loss=?]
      1%|          | 1/110 [00:00<00:24,  4.49trial/s, best loss: -0.76]
      2%|▏         | 2/110 [00:00<00:31,  3.43trial/s, best loss: -0.9266666666666665]
      3%|▎         | 3/110 [00:00<00:30,  3.55trial/s, best loss: -0.9266666666666665]
      4%|▎         | 4/110 [00:01<00:35,  2.96trial/s, best loss: -0.9533333333333331]
      5%|▍         | 5/110 [00:01<00:36,  2.91trial/s, best loss: -0.9533333333333334]
      5%|▌         | 6/110 [00:01<00:35,  2.93trial/s, best loss: -0.9533333333333334]
      6%|▋         | 7/110 [00:02<00:27,  3.81trial/s, best loss: -0.9533333333333334]
      7%|▋         | 8/110 [00:02<00:30,  3.38trial/s, best loss: -0.9533333333333334]
      9%|▉         | 10/110 [00:02<00:26,  3.80trial/s, best loss: -0.9533333333333334]
     10%|█         | 11/110 [00:03<00:23,  4.26trial/s, best loss: -0.9533333333333334]
     11%|█         | 12/110 [00:03<00:28,  3.47trial/s, best loss: -0.9533333333333334]
     12%|█▏        | 13/110 [00:03<00:23,  4.05trial/s, best loss: -0.9533333333333334]
     13%|█▎        | 14/110 [00:03<00:21,  4.45trial/s, best loss: -0.9533333333333334]
     14%|█▎        | 15/110 [00:04<00:25,  3.78trial/s, best loss: -0.9533333333333334]
     15%|█▍        | 16/110 [00:04<00:27,  3.42trial/s, best loss: -0.9533333333333334]
     15%|█▌        | 17/110 [00:04<00:26,  3.55trial/s, best loss: -0.9533333333333334]
     16%|█▋        | 18/110 [00:05<00:25,  3.60trial/s, best loss: -0.9533333333333334]
     17%|█▋        | 19/110 [00:05<00:25,  3.58trial/s, best loss: -0.9533333333333334]
     18%|█▊        | 20/110 [00:05<00:25,  3.50trial/s, best loss: -0.9533333333333334]
     19%|█▉        | 21/110 [00:05<00:21,  4.14trial/s, best loss: -0.9533333333333334]
     20%|██        | 22/110 [00:05<00:20,  4.35trial/s, best loss: -0.9533333333333334]
     21%|██        | 23/110 [00:06<00:18,  4.71trial/s, best loss: -0.9533333333333334]
     22%|██▏       | 24/110 [00:06<00:15,  5.39trial/s, best loss: -0.9533333333333334]
     23%|██▎       | 25/110 [00:06<00:19,  4.42trial/s, best loss: -0.9533333333333334]
     24%|██▎       | 26/110 [00:06<00:20,  4.12trial/s, best loss: -0.9533333333333334]
     25%|██▍       | 27/110 [00:07<00:19,  4.31trial/s, best loss: -0.9533333333333334]
     25%|██▌       | 28/110 [00:07<00:22,  3.69trial/s, best loss: -0.9533333333333334]
     26%|██▋       | 29/110 [00:07<00:19,  4.19trial/s, best loss: -0.9533333333333334]
     27%|██▋       | 30/110 [00:07<00:22,  3.57trial/s, best loss: -0.9533333333333334]
     28%|██▊       | 31/110 [00:08<00:20,  3.77trial/s, best loss: -0.9533333333333334]
     29%|██▉       | 32/110 [00:08<00:18,  4.29trial/s, best loss: -0.9533333333333334]
     30%|███       | 33/110 [00:08<00:20,  3.72trial/s, best loss: -0.9533333333333334]
     31%|███       | 34/110 [00:09<00:23,  3.20trial/s, best loss: -0.9533333333333334]
     32%|███▏      | 35/110 [00:09<00:21,  3.48trial/s, best loss: -0.9533333333333334]
     33%|███▎      | 36/110 [00:09<00:22,  3.35trial/s, best loss: -0.9533333333333334]
     34%|███▎      | 37/110 [00:10<00:23,  3.16trial/s, best loss: -0.9533333333333334]
     35%|███▍      | 38/110 [00:10<00:21,  3.40trial/s, best loss: -0.9533333333333334]
     35%|███▌      | 39/110 [00:10<00:20,  3.40trial/s, best loss: -0.9533333333333334]
     36%|███▋      | 40/110 [00:10<00:22,  3.12trial/s, best loss: -0.9533333333333334]
     37%|███▋      | 41/110 [00:11<00:19,  3.48trial/s, best loss: -0.9533333333333334]
     38%|███▊      | 42/110 [00:11<00:20,  3.37trial/s, best loss: -0.9533333333333334]
     39%|███▉      | 43/110 [00:11<00:17,  3.81trial/s, best loss: -0.9533333333333334]
     40%|████      | 44/110 [00:11<00:16,  4.08trial/s, best loss: -0.9533333333333334]
     41%|████      | 45/110 [00:12<00:17,  3.67trial/s, best loss: -0.9533333333333334]
     42%|████▏     | 46/110 [00:12<00:14,  4.42trial/s, best loss: -0.9533333333333334]
     43%|████▎     | 47/110 [00:12<00:13,  4.71trial/s, best loss: -0.9533333333333334]
     44%|████▎     | 48/110 [00:12<00:15,  3.91trial/s, best loss: -0.9533333333333334]
     45%|████▍     | 49/110 [00:13<00:15,  3.95trial/s, best loss: -0.9533333333333334]
     45%|████▌     | 50/110 [00:13<00:15,  3.84trial/s, best loss: -0.9533333333333334]
     46%|████▋     | 51/110 [00:13<00:15,  3.79trial/s, best loss: -0.9533333333333334]
     47%|████▋     | 52/110 [00:14<00:17,  3.31trial/s, best loss: -0.9533333333333334]
     48%|████▊     | 53/110 [00:14<00:17,  3.30trial/s, best loss: -0.9533333333333334]
     49%|████▉     | 54/110 [00:14<00:14,  3.88trial/s, best loss: -0.9533333333333334]
     50%|█████     | 55/110 [00:14<00:16,  3.40trial/s, best loss: -0.9533333333333334]
     51%|█████     | 56/110 [00:15<00:15,  3.51trial/s, best loss: -0.9533333333333334]
     52%|█████▏    | 57/110 [00:15<00:17,  2.96trial/s, best loss: -0.9533333333333334]
     53%|█████▎    | 58/110 [00:15<00:18,  2.80trial/s, best loss: -0.9533333333333334]
     54%|█████▎    | 59/110 [00:16<00:15,  3.39trial/s, best loss: -0.9533333333333334]
     55%|█████▍    | 60/110 [00:16<00:11,  4.18trial/s, best loss: -0.9533333333333334]
     55%|█████▌    | 61/110 [00:16<00:11,  4.37trial/s, best loss: -0.9533333333333334]
     56%|█████▋    | 62/110 [00:16<00:12,  3.73trial/s, best loss: -0.9533333333333334]
     57%|█████▋    | 63/110 [00:16<00:10,  4.45trial/s, best loss: -0.9533333333333334]
     58%|█████▊    | 64/110 [00:17<00:11,  4.00trial/s, best loss: -0.9533333333333334]
     59%|█████▉    | 65/110 [00:17<00:11,  3.80trial/s, best loss: -0.9533333333333334]
     60%|██████    | 66/110 [00:17<00:13,  3.29trial/s, best loss: -0.9533333333333334]
     61%|██████    | 67/110 [00:18<00:12,  3.48trial/s, best loss: -0.9533333333333334]
     62%|██████▏   | 68/110 [00:18<00:13,  3.21trial/s, best loss: -0.9533333333333334]
     63%|██████▎   | 69/110 [00:18<00:13,  3.15trial/s, best loss: -0.9533333333333334]
     64%|██████▎   | 70/110 [00:19<00:12,  3.32trial/s, best loss: -0.9533333333333334]
     65%|██████▍   | 71/110 [00:19<00:09,  4.10trial/s, best loss: -0.9533333333333334]
     65%|██████▌   | 72/110 [00:19<00:11,  3.20trial/s, best loss: -0.9533333333333334]
     66%|██████▋   | 73/110 [00:20<00:12,  3.08trial/s, best loss: -0.9533333333333334]
     67%|██████▋   | 74/110 [00:20<00:09,  3.65trial/s, best loss: -0.9533333333333334]
     68%|██████▊   | 75/110 [00:20<00:09,  3.83trial/s, best loss: -0.9533333333333334]
     69%|██████▉   | 76/110 [00:20<00:08,  4.00trial/s, best loss: -0.9533333333333334]
     70%|███████   | 77/110 [00:20<00:08,  3.74trial/s, best loss: -0.9533333333333334]
     71%|███████   | 78/110 [00:21<00:08,  3.65trial/s, best loss: -0.9533333333333334]
     72%|███████▏  | 79/110 [00:21<00:10,  2.98trial/s, best loss: -0.9533333333333334]
     73%|███████▎  | 80/110 [00:21<00:08,  3.60trial/s, best loss: -0.9533333333333334]
     74%|███████▎  | 81/110 [00:22<00:08,  3.37trial/s, best loss: -0.9533333333333334]
     75%|███████▍  | 82/110 [00:22<00:08,  3.40trial/s, best loss: -0.9533333333333334]
     75%|███████▌  | 83/110 [00:22<00:08,  3.32trial/s, best loss: -0.9533333333333334]
     76%|███████▋  | 84/110 [00:23<00:08,  3.06trial/s, best loss: -0.9533333333333334]
     77%|███████▋  | 85/110 [00:23<00:08,  2.90trial/s, best loss: -0.9533333333333334]
     78%|███████▊  | 86/110 [00:23<00:07,  3.28trial/s, best loss: -0.9533333333333334]
     79%|███████▉  | 87/110 [00:24<00:08,  2.82trial/s, best loss: -0.9533333333333334]
     80%|████████  | 88/110 [00:24<00:06,  3.25trial/s, best loss: -0.9533333333333334]
     81%|████████  | 89/110 [00:24<00:05,  3.65trial/s, best loss: -0.9533333333333334]
     82%|████████▏ | 90/110 [00:25<00:05,  3.54trial/s, best loss: -0.9533333333333334]
     83%|████████▎ | 91/110 [00:25<00:05,  3.28trial/s, best loss: -0.9533333333333334]
     84%|████████▎ | 92/110 [00:25<00:05,  3.13trial/s, best loss: -0.9533333333333334]
     85%|████████▍ | 93/110 [00:25<00:04,  3.52trial/s, best loss: -0.9533333333333334]
     85%|████████▌ | 94/110 [00:26<00:04,  3.82trial/s, best loss: -0.9533333333333334]
     86%|████████▋ | 95/110 [00:26<00:04,  3.11trial/s, best loss: -0.9533333333333334]
     87%|████████▋ | 96/110 [00:26<00:03,  3.65trial/s, best loss: -0.9533333333333334]
     88%|████████▊ | 97/110 [00:26<00:03,  3.97trial/s, best loss: -0.9533333333333334]
     89%|████████▉ | 98/110 [00:27<00:03,  3.55trial/s, best loss: -0.9533333333333334]
     90%|█████████ | 99/110 [00:27<00:03,  3.35trial/s, best loss: -0.9533333333333334]
     91%|█████████ | 100/110 [00:28<00:03,  3.05trial/s, best loss: -0.9533333333333334]
     92%|█████████▏| 101/110 [00:28<00:03,  2.92trial/s, best loss: -0.9533333333333334]
     93%|█████████▎| 102/110 [00:28<00:03,  2.63trial/s, best loss: -0.9533333333333334]
     94%|█████████▎| 103/110 [00:29<00:02,  2.84trial/s, best loss: -0.9533333333333334]
     95%|█████████▍| 104/110 [00:29<00:02,  2.95trial/s, best loss: -0.9533333333333334]
     95%|█████████▌| 105/110 [00:29<00:01,  3.53trial/s, best loss: -0.9533333333333334]
     96%|█████████▋| 106/110 [00:29<00:01,  3.97trial/s, best loss: -0.9533333333333334]
     97%|█████████▋| 107/110 [00:30<00:00,  3.70trial/s, best loss: -0.9533333333333334]
     98%|█████████▊| 108/110 [00:30<00:00,  4.35trial/s, best loss: -0.9533333333333334]
     99%|█████████▉| 109/110 [00:30<00:00,  4.12trial/s, best loss: -0.9533333333333334]
    100%|██████████| 110/110 [00:30<00:00,  3.57trial/s, best loss: -0.96]              
    100%|██████████| 110/110 [00:30<00:00,  3.56trial/s, best loss: -0.96]
    Time of the bayesian search 30.95 s
    Bayesian Search optimization has run 110 algorithm executions




.. GENERATED FROM PYTHON SOURCE LINES 352-386

Summary Table
-------------
The summary table below compares the optimization methods based on their best metric,
the time taken for optimization, and the number of evaluations performed:


Overview of Optimization Methods:

- **Genetic Optimization**:

  - Achieved the highest metric (0.961004) with 110 evaluations.
  - Completed in the shortest time (25.87 seconds).
  - Utilizes evolutionary algorithms to perform the search in the hyperparameter space.
  - Efficient in finding optimal hyperparameters.

- **Grid Search**:

  - Achieved a best metric of 0.960000 with 108 evaluations.
  - Took the longest time to complete (69.11 seconds).
  - Performs an exhaustive search over a predefined grid of hyperparameters.
  - Computationally expensive, particularly for large search spaces.

- **Random Search**:

  - Achieved a best metric of 0.960000 with 110 evaluations.
  - Took less time than Grid Search (58.89 seconds).
  - Samples random combinations of hyperparameters from a predefined space.
  - Less computationally expensive than Grid Search and more efficient for large spaces.

Conclusions:

- Genetic Optimization provided the best performance in terms of both metric and time.
- Grid Search is the most computationally expensive method.
- Random Search is a good compromise between computational cost and performance.

.. GENERATED FROM PYTHON SOURCE LINES 386-400

.. code-block:: default


    row_gen = {'Method': "Genetic",
               'Best Metric': float(population_df.sort_values(by="fitness", ascending=False)['fitness'].iloc[0]),
               'Time': execution_time_gen, 'Evaluated': population_df.shape[0]}
    row_gs = {'Method': "Grid Search", 'Best Metric': clf_gs.best_score_,
              'Time': execution_time_gs, 'Evaluated': synth_population_gs.shape[0]}
    row_rs = {'Method': "Random Search", 'Best Metric': clf_rs.best_score_,
              'Time': execution_time_rs, 'Evaluated': synth_population_rs.shape[0]}
    row_bay = {'Method': "Bayesian Search",
               'Best Metric': max(scores_list),
               'Time': execution_time_bay, 'Evaluated': num_eval}

    df = pd.DataFrame([row_gen, row_gs, row_rs, row_bay])
    df





.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>Method</th>
          <th>Best Metric</th>
          <th>Time</th>
          <th>Evaluated</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>Genetic</td>
          <td>0.946667</td>
          <td>18.76</td>
          <td>110</td>
        </tr>
        <tr>
          <th>1</th>
          <td>Grid Search</td>
          <td>0.960000</td>
          <td>29.29</td>
          <td>108</td>
        </tr>
        <tr>
          <th>2</th>
          <td>Random Search</td>
          <td>0.960000</td>
          <td>31.12</td>
          <td>110</td>
        </tr>
        <tr>
          <th>3</th>
          <td>Bayesian Search</td>
          <td>0.960000</td>
          <td>30.95</td>
          <td>110</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />


.. rst-class:: sphx-glr-timing

   **Total running time of the script:** (1 minutes 59.645 seconds)


.. _sphx_glr_download_auto_examples_plot_xgboost_hyperparam_opt_comparison.py:

.. only:: html

  .. container:: sphx-glr-footer sphx-glr-footer-example




    .. container:: sphx-glr-download sphx-glr-download-python

      :download:`Download Python source code: plot_xgboost_hyperparam_opt_comparison.py <plot_xgboost_hyperparam_opt_comparison.py>`

    .. container:: sphx-glr-download sphx-glr-download-jupyter

      :download:`Download Jupyter notebook: plot_xgboost_hyperparam_opt_comparison.ipynb <plot_xgboost_hyperparam_opt_comparison.ipynb>`


.. only:: html

 .. rst-class:: sphx-glr-signature

    `Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_