Hyper-Parameter Tuning with Cross-Validation

Importance of Hyper-Parameter Tuning

Hyperparameter tuning is crucial for optimizing machine learning (ML) algorithms. Effective tuning results in improved real-world performance. Cross-validation (CV) plays a vital role in this, especially in the finance sector, where conventional approaches often fall short. This blog focuses on utilizing the Purged k-fold CV method for hyper-parameter optimization.

Purged-Kfold Integration into MLJBase

For hyperparameter tuning, grid search is often an initial step to understand the data's underlying structure. In MLJBase, GridSearchcV uses a CV generator, and to avoid overfitting, our PurgedKFold class can be passed as an argument.

def clf_hyper_fit(
    feature_data: pd.DataFrame,
    label: pd.DataFrame,
    times: pd.Series,
    pipe_clf: Pipeline,
    param_grid: dict,
    validator_type: str = 'purgedkfold',
    validator_params: dict = None,
    bagging: list = [0, -1, 1.],
    rnd_search_iter: int = 0,
    n_jobs: int = -1,
    **fit_params
) -> MyPipeline:
    if set(label.values) == {0, 1}:
        scoring = 'f1'  # F1-score for meta-labeling
    else:
        scoring = 'neg_log_loss'  # Symmetric towards all cases

    if validator_params is None:
        validator_params = {
            'times' : times,
            'n_splits' : 5,
            'embargo' : 0.01,
        }

    # Hyperparameter search on train data
    inner_cv =  CrossValidatorController(
        validator_type,
        **validator_params
    ).cross_validator
    
    if rnd_search_iter == 0:
        gs = GridSearchCV(estimator=pipe_clf, param_grid=param_grid, scoring=scoring, cv=inner_cv, n_jobs=n_jobs)
    else:
        gs = RandomizedSearchCV(estimator=pipe_clf, param_distributions=param_grid, scoring=scoring,
                                cv=inner_cv, n_jobs=n_jobs, n_iter=rnd_search_iter)
    gs = gs.fit(feature_data, label, **fit_params).best_estimator_  # Pipeline

    # Fit validated model on the entirety of the data
    if bagging[1] > 0:
        gs = BaggingClassifier(base_estimator=MyPipeline(gs.steps), n_estimators=int(bagging[0]),
                               max_samples=float(bagging[1]), max_features=float(bagging[2]), n_jobs=n_jobs)
        gs = gs.fit(feature_data, label, sample_weight=fit_params[gs.base_estimator.steps[-1][0] + '__sample_weight'])
        gs = Pipeline([('bag', gs)])

    return gs

Non-Negative Parameters

Non-negative hyperparameters are common in some ML algorithms, such as the SVC classifier and RBF kernel. Rather than using a uniform distribution for sampling, using a log-uniform distribution is often more effective for such parameters.

For a variable $x$ to have a log-uniform distribution between $a>0$ and $b>a$, its CDF and PDF can be defined as:

Limitations of Accuracy as a Measure

Accuracy alone doesn't provide a meaningful evaluation in finance-related ML, particularly in investment strategies. It fails to account for the probabilities associated with predictions. Cross-entropy loss, or $log loss$, is a better performance metric as it incorporates prediction probabilities.

The formula for log loss is:

Accuracy doesn't suffice for hyperparameter tuning in financial applications. It should ideally be supplemented or replaced with metrics that better capture the complexities of financial decision-making.

Note: All these functionalities are available in both Python and Julia in the RiskLabAI library. You can view more here for Python and here for Julia.

References

1. De Prado, M. L. (2018). Advances in financial machine learning. John Wiley & Sons.
2. De Prado, M. M. L. (2020). Machine learning for asset managers. Cambridge University Press.