# Automated machine learning

This notebook contains some basics for autoML using CapyMOA:

* Implement a custom model selection procedure using CapyMOA.
* Perform hyperparameter optimisation, and model selection using CapyMOA’s AutoML features.

---

*More information about CapyMOA can be found at* https://www.capymoa.org.

**last updated on 28/11/2025**

```ipython3
import contextlib
import io

from moa.streams import ConceptDriftStream

from capymoa.automl import (
    AutoClass,
    BanditClassifier,
    EpsilonGreedy,
    SuccessiveHalvingClassifier,
)
from capymoa.classifier import (
    KNN,
    AdaptiveRandomForestClassifier,
    HoeffdingAdaptiveTree,
    HoeffdingTree,
    LeveragingBagging,
    NaiveBayes,
    StreamingRandomPatches,
)
from capymoa.datasets import Electricity
from capymoa.evaluation import prequential_evaluation
from capymoa.evaluation.visualization import plot_windowed_results
from capymoa.stream.drift import AbruptDrift, DriftStream, GradualDrift
from capymoa.stream.generator import SEA

# Setup the data streams for our experiments:
stream = Electricity()
drift_stream = DriftStream(
    moa_stream=ConceptDriftStream(),
    CLI="""
    -s (generators.AgrawalGenerator -f 1) 
    -d (ConceptDriftStream 
        -s (generators.AgrawalGenerator -f 2)
        -d (ConceptDriftStream -s (generators.AgrawalGenerator -f 3) 
        -d (generators.AgrawalGenerator -f 4) -p 30000 -w 0)
        -p 20000 -w 0) 
    -p 10000 
    -w 0
""",
)
```

```ipython3
def print_summary(label: str, results):
    print(
        f"{label.ljust(10)} Cumulative accuracy = {results.accuracy():.2f}, "
        f"wall-clock time: {results.wallclock():.3f}"
    )
```

## Custom model selection

In this section we manually define a method to select the best model based on its accuracy on a given stream. First we define a list of models to be tried and then we iterate over the stream to train and evaluate each model. The model with the highest accuracy is selected as the best model.

```ipython3
# Define a generic adaptive learning function
def stream_learning_model_selection(model_list, window_size, max_instances):
    best_accuracy = 0  # The best accuracy score
    all_results = {}
    for model in model_list:
        results = prequential_evaluation(
            stream=stream,
            learner=model,
            window_size=window_size,
            max_instances=max_instances,
        )
        all_results[model] = results
        if results.cumulative.accuracy() > best_accuracy:
            best_accuracy = results.cumulative.accuracy()
            model_b = model

    print_summary(f"Best ({model_b})", all_results[model_b])
    return all_results
```

```ipython3
# Code to select the best performing model
schema = stream.get_schema()
ht = HoeffdingTree(schema)
knn = KNN(schema)
arf = AdaptiveRandomForestClassifier(schema, ensemble_size=5)
srp = StreamingRandomPatches(schema, ensemble_size=5)
nb = NaiveBayes(schema)
model_list = [ht, knn, arf, srp, nb]

all_res = stream_learning_model_selection(model_list, 1000, 5000)
plot_windowed_results(*all_res.values(), metric="accuracy")
```

```myst-ansi
Best (StreamingRandomPatches) Cumulative accuracy = 90.56, wall-clock time: 0.632
```

## AutoML

The following example shows how to use the AutoClass algorithm using CapyMOA.

* `base_classifiers` is a list of classifier class types that will be candidates for the AutoML algorithm. The AutoML algorithm will select the best classifier based on its performance on the stream.
* `configuration_json` is a json file that contains the configuration for the AutoML algorithm. An example of the configuration file is shown below:

*Maroua Bahri, Nikolaos Georgantas. Autoclass: Automl for data stream
classification. In BigData, IEEE, 2023. https://ieeexplore.ieee.org/document/10386362*

```ipython3
with open("./settings_autoclass.json", "r") as f:
    settings = f.read()
    print(settings)
```

```myst-ansi
{

  "windowSize" :  1000,
  "ensembleSize" : 10,
  "newConfigurations" : 10,
  "keepCurrentModel" : true,
  "lambda" : 0.05,
  "preventAlgorithmDeath" : true,
  "keepGlobalIncumbent" : true,
  "keepAlgorithmIncumbents" : true,
  "keepInitialConfigurations" : true,
  "useTestEnsemble" : true,
  "resetProbability" : 0.01,
  "numberOfCores" : 1,
  "performanceMeasureMaximisation": true,

  "algorithms": [
    {
      "algorithm": "moa.classifiers.lazy.kNN",
      "parameters": [
        {"parameter": "k", "type":"integer", "value":10, "range":[2,30]}
      ]
    }
  ,
    {
      "algorithm": "moa.classifiers.trees.HoeffdingTree",
      "parameters": [
        {"parameter": "g", "type":"integer", "value":200, "range":[10, 200]},
       {"parameter": "c", "type":"float", "value":0.01, "range":[0, 1]}
      ]
    }
  ,
    {
      "algorithm": "moa.classifiers.lazy.kNNwithPAWandADWIN",
      "parameters": [
        {"parameter": "k", "type":"integer", "value":10, "range":[2,30]}
      ]
    }
    ,
    {
      "algorithm": "moa.classifiers.trees.HoeffdingAdaptiveTree",
      "parameters": [
        {"parameter": "g", "type":"integer", "value":200, "range":[10, 200]},
        {"parameter": "c", "type":"float", "value":0.01, "range":[0, 1]}
      ]
    }

  ]
}

```

```ipython3
max_instances = 20000
window_size = 2500

schema = stream.get_schema()
autoclass = AutoClass(
    schema=schema,
    configuration_json="./settings_autoclass.json",
    base_classifiers=[KNN, HoeffdingAdaptiveTree, HoeffdingTree],
)
results_autoclass = prequential_evaluation(
    stream=stream,
    learner=autoclass,
    window_size=window_size,
    max_instances=max_instances,
)
print_summary("AutoClass", results_autoclass)
```

```myst-ansi
AutoClass  Cumulative accuracy = 88.35, wall-clock time: 24.237
```

We compare the performance of the AutoML algorithm against basic classifiers:

```ipython3
ht = HoeffdingTree(schema)
hat = HoeffdingAdaptiveTree(schema)
knn = KNN(schema)
nb = NaiveBayes(schema)

results_ht = prequential_evaluation(
    stream, ht, window_size=window_size, max_instances=max_instances
)
results_hat = prequential_evaluation(
    stream, hat, window_size=window_size, max_instances=max_instances
)
results_knn = prequential_evaluation(
    stream, knn, window_size=window_size, max_instances=max_instances
)
results_nb = prequential_evaluation(
    stream, nb, window_size=window_size, max_instances=max_instances
)

print_summary("HT", results_ht)
print_summary("HAT", results_hat)
print_summary("KNN", results_knn)
print_summary("NB", results_nb)
plot_windowed_results(
    results_ht,
    results_knn,
    results_hat,
    results_nb,
    results_autoclass,
    metric="accuracy",
)
```

```myst-ansi
HT         Cumulative accuracy = 85.61, wall-clock time: 0.083
HAT        Cumulative accuracy = 87.08, wall-clock time: 0.157
KNN        Cumulative accuracy = 85.47, wall-clock time: 1.466
NB         Cumulative accuracy = 77.23, wall-clock time: 0.055
```

## AutoML with concept drift

We test the same algorithms on a data stream with simulated concept drift:

```ipython3
from capymoa.automl import AutoClass
from capymoa.evaluation import prequential_evaluation

max_instances = 40000
window_size = 4000

ht = HoeffdingTree(schema=drift_stream.get_schema())
hat = HoeffdingAdaptiveTree(schema=drift_stream.get_schema())
knn = KNN(schema=drift_stream.get_schema())
nb = NaiveBayes(schema=drift_stream.get_schema())
autoclass = AutoClass(
    schema=drift_stream.get_schema(),
    configuration_json="./settings_autoclass.json",
    base_classifiers=[KNN, HoeffdingAdaptiveTree, HoeffdingTree],
)

results_ht = prequential_evaluation(
    drift_stream, ht, window_size=window_size, max_instances=max_instances
)
results_hat = prequential_evaluation(
    drift_stream, hat, window_size=window_size, max_instances=max_instances
)
results_knn = prequential_evaluation(
    drift_stream, knn, window_size=window_size, max_instances=max_instances
)
results_nb = prequential_evaluation(
    drift_stream, nb, window_size=window_size, max_instances=max_instances
)
results_autoclass = prequential_evaluation(
    drift_stream, autoclass, window_size=window_size, max_instances=max_instances
)

print_summary("HT", results_ht)
print_summary("HAT", results_hat)
print_summary("KNN", results_knn)
print_summary("NB", results_nb)
print_summary("AutoClass", results_autoclass)
plot_windowed_results(
    results_ht,
    results_knn,
    results_hat,
    results_nb,
    results_autoclass,
    metric="accuracy",
)
```

```myst-ansi
HT         Cumulative accuracy = 78.82, wall-clock time: 0.309
HAT        Cumulative accuracy = 78.24, wall-clock time: 0.349
KNN        Cumulative accuracy = 67.14, wall-clock time: 4.581
NB         Cumulative accuracy = 60.29, wall-clock time: 0.189
AutoClass  Cumulative accuracy = 85.10, wall-clock time: 73.102
```

## Analysis of further AutoML methods

### **Bandit Classifier**

It is a model selection for classification in streaming scenarios.

Each model is associated with an arm. At each train call, the policy decides which arm/model to pull.
The reward is the performance of the model on the provided sample. The predict and predict_proba
methods use the current best model.

### **Successive Halving Classifier**

The algorithm progressively eliminates poorly performing models while allocating more resources to promising ones.

Successive halving is a method for performing model selection without having to train each model on all the dataset.
At certain points in time (called “rungs”), the worst performing models will be discarded and the best ones
will keep competing between each other. The rung values are designed so that at most ‘budget’ model updates
will be performed in total.

### Testing whether SuccessiveHalvingClassifier and BanditClassifier work correctly

```ipython3
f = io.StringIO()


def test_successive_halving_and_bandit(
    stream, max_instances=20000, window_size=2500, budget=None, bandit_eps=0.1
):
    """Test SuccessiveHalving and BanditClassifier for parameter optimization of a single model type."""
    print("\n" + "=" * 80)
    print("TESTING SUCCESSIVE HALVING AND BANDIT CLASSIFIER FOR PARAMETER OPTIMIZATION")
    print("=" * 80)

    schema = stream.get_schema()
    if budget is None:
        budget = max_instances * 2

    # Create a wide range of HoeffdingTree configurations with different parameters
    ht_models = []

    # Test different grace periods
    for grace_period in [50, 100, 200, 300, 400]:
        # Test different split confidences
        for confidence in [0.001, 0.01, 0.05, 0.1, 0.2]:
            # Test different tie thresholds
            for tie_threshold in [0.05, 0.1, 0.2]:
                ht_models.append(
                    HoeffdingTree(
                        schema=schema,
                        grace_period=grace_period,
                        confidence=confidence,
                        tie_threshold=tie_threshold,
                    )
                )

    print(f"Created {len(ht_models)} different HoeffdingTree configurations")

    # Use SuccessiveHalving to find the best HoeffdingTree configuration
    with contextlib.redirect_stdout(f):
        shc_ht = SuccessiveHalvingClassifier(
            schema=schema,
            base_classifiers=ht_models,
            budget=budget,
            eta=2.0,
            min_models=1,
            verbose=False,
        )

    # Use BanditClassifier to find the best HoeffdingTree configuration
    bandit_ht = BanditClassifier(
        schema=schema,
        base_classifiers=ht_models,
        metric="accuracy",
        policy=EpsilonGreedy(epsilon=bandit_eps, burn_in=150),
        verbose=False,
    )

    # Default HoeffdingTree for comparison
    default_ht = HoeffdingTree(schema=schema)

    print("\nRunning prequential evaluation...")
    results_shc_ht = prequential_evaluation(
        stream=stream,
        learner=shc_ht,
        window_size=window_size,
        max_instances=max_instances,
    )
    results_bandit_ht = prequential_evaluation(
        stream=stream,
        learner=bandit_ht,
        window_size=window_size,
        max_instances=max_instances,
    )
    results_default_ht = prequential_evaluation(
        stream=stream,
        learner=default_ht,
        window_size=window_size,
        max_instances=max_instances,
    )

    # Print results
    print("\nEvaluation Results:")
    print(
        f"[SuccessiveHalving with {len(ht_models)} HT configs] Accuracy = {results_shc_ht.accuracy():.3f}, "
        f"Time: {results_shc_ht.wallclock():.3f}s"
    )
    print(
        f"[BanditClassifier with {len(ht_models)} HT configs] Accuracy = {results_bandit_ht.accuracy():.3f}, "
        f"Time: {results_bandit_ht.wallclock():.3f}s"
    )
    print(
        f"[Default HoeffdingTree] Accuracy = {results_default_ht.accuracy():.3f}, "
        f"Time: {results_default_ht.wallclock():.3f}s"
    )

    # Calculate improvements
    improvement_shc = results_shc_ht.accuracy() - results_default_ht.accuracy()
    improvement_bandit = results_bandit_ht.accuracy() - results_default_ht.accuracy()
    print("Improvement over default parameters:")
    print(f"SuccessiveHalving: {improvement_shc:.2f}% absolute")
    print(f"BanditClassifier: {improvement_bandit:.2f}% absolute")

    # Plot results
    plot_windowed_results(
        results_shc_ht, results_bandit_ht, results_default_ht, metric="accuracy"
    )

    # Display final model info
    model_info_shc = shc_ht.get_model_info()
    print("\nSuccessive Halving Final Status:")
    print(
        f"Active models: {model_info_shc['active_models']} / {model_info_shc['total_models']}"
    )
    print(f"Total rungs: {model_info_shc['current_rung']}")
    print(
        f"Budget used: {model_info_shc['budget_used']} / {model_info_shc['total_budget']}"
    )
```

```ipython3
stream = Electricity()
max_instances = 20000
window_size = 2500
budget = max_instances * 2

test_successive_halving_and_bandit(
    stream, max_instances=max_instances, window_size=window_size, budget=budget
)
```

```myst-ansi

================================================================================
TESTING SUCCESSIVE HALVING AND BANDIT CLASSIFIER FOR PARAMETER OPTIMIZATION
================================================================================
Created 75 different HoeffdingTree configurations

Running prequential evaluation...
```

```myst-ansi

Evaluation Results:
[SuccessiveHalving with 75 HT configs] Accuracy = 87.225, Time: 132.661s
[BanditClassifier with 75 HT configs] Accuracy = 84.555, Time: 78.409s
[Default HoeffdingTree] Accuracy = 85.615, Time: 0.083s
Improvement over default parameters:
SuccessiveHalving: 1.61% absolute
BanditClassifier: -1.06% absolute
```

```myst-ansi

Successive Halving Final Status:
Active models: 1 / 75
Total rungs: 7
Budget used: 39999 / 40000
```

In the initial test, SuccessiveHalvingClassifier delivers the best accuracy (87.2%, +1.6% over HoeffdingTree) but takes 140s, while BanditClassifier is much faster (81s) with similar accuracy to the default, trading precision for speed.

The results highlight a clear accuracy–time trade-off between thorough configuration search and faster, less precise optimization.

## Comparing AutoClass with SuccessiveHalving and BanditClassifier

```ipython3
def base_models_configuration():
    # Create base classifiers with same parameters for SuccessiveHalving and BanditClassifier
    base_models = []

    # Add Hoeffding Trees
    for grace_period in [100, 200]:
        for confidence in [0.001, 0.01]:
            base_models.append(
                HoeffdingTree(
                    schema=schema, grace_period=grace_period, confidence=confidence
                )
            )

    # Add KNN variations
    for k in [3, 5]:
        base_models.append(KNN(schema=schema, k=k))

    # Add Random Forest models
    for ensemble_size in [60, 80]:
        base_models.append(
            AdaptiveRandomForestClassifier(schema=schema, ensemble_size=ensemble_size)
        )

    # Add Hoeffding Adaptive Trees
    for grace_period in [100, 200]:
        base_models.append(
            HoeffdingAdaptiveTree(schema=schema, grace_period=grace_period)
        )

    # Add Leveraging Bagging models
    for ensemble_size in [60, 80]:
        base_models.append(
            LeveragingBagging(schema=schema, ensemble_size=ensemble_size)
        )

    # Add Streaming Random Patches
    for ensemble_size in [60, 80]:
        base_models.append(
            StreamingRandomPatches(schema=schema, ensemble_size=ensemble_size)
        )

    return base_models
```

```ipython3
def autoclass(stream, max_instances, window_size, budget):
    schema = stream.get_schema()
    if budget is None:
        budget = max_instances * 3

    # Initialize AutoClass with the enhanced configuration
    autoclass_enhanced = AutoClass(
        schema=schema,
        configuration_json="./enhanced_autoclass_config.json",
        base_classifiers=[
            KNN,
            HoeffdingTree,
            HoeffdingAdaptiveTree,
            AdaptiveRandomForestClassifier,
            LeveragingBagging,
            StreamingRandomPatches,
        ],
    )

    results_autoclass_enhanced = prequential_evaluation(
        stream=stream,
        learner=autoclass_enhanced,
        window_size=window_size,
        max_instances=max_instances,
    )

    return results_autoclass_enhanced


def successive_halving(stream, max_instances, window_size):
    # Create base classifiers with same parameters for SuccessiveHalving and BanditClassifier
    base_models = base_models_configuration()

    # Create the SuccessiveHalvingClassifier
    with contextlib.redirect_stdout(f):
        shc_direct = SuccessiveHalvingClassifier(
            schema=schema,
            base_classifiers=base_models,
            budget=budget,
            eta=2.0,
            min_models=2,
            verbose=True,
        )

    results_shc = prequential_evaluation(
        stream=stream,
        learner=shc_direct,
        window_size=window_size,
        max_instances=max_instances,
    )

    return shc_direct, results_shc


def bandit_classifier(stream, max_instances, window_size, bandit_eps):
    # Create base classifiers with same parameters for SuccessiveHalving and BanditClassifier
    base_models = base_models_configuration()

    # Create the BanditClassifier
    bandit_clf = BanditClassifier(
        schema=schema,
        base_classifiers=base_models,
        metric="accuracy",
        policy=EpsilonGreedy(epsilon=bandit_eps, burn_in=150),
        verbose=True,
    )

    results_bandit = prequential_evaluation(
        stream=stream,
        learner=bandit_clf,
        window_size=window_size,
        max_instances=max_instances,
    )

    return bandit_clf, results_bandit
```

```ipython3
def test_autoclass_vs_successive_halving_and_bandit(
    stream,
    results_autoclass_enhanced,
    shc_direct,
    results_shc,
    bandit_clf,
    results_bandit,
    max_instances=20000,
    window_size=2500,
):
    """Compare AutoClass with enhanced configuration against SuccessiveHalvingClassifier and BanditClassifier."""
    print("\n" + "=" * 80)
    print("COMPARING AUTOCLASS WITH CONFIG, SUCCESSIVE HALVING, AND BANDIT CLASSIFIER")
    print("=" * 80)

    # Initialize default models for comparison
    print("\nInitializing default models for comparison...")
    default_ht = HoeffdingTree(schema=schema)
    default_knn = KNN(schema=schema)
    default_arf = AdaptiveRandomForestClassifier(schema=schema)
    default_hat = HoeffdingAdaptiveTree(schema=schema)
    default_lb = LeveragingBagging(schema=schema)
    default_srp = StreamingRandomPatches(schema=schema)

    # Evaluate default models
    print("\nEvaluating default models...")
    results_ht = prequential_evaluation(
        stream=stream,
        learner=default_ht,
        window_size=window_size,
        max_instances=max_instances,
    )
    results_knn = prequential_evaluation(
        stream=stream,
        learner=default_knn,
        window_size=window_size,
        max_instances=max_instances,
    )
    results_arf = prequential_evaluation(
        stream=stream,
        learner=default_arf,
        window_size=window_size,
        max_instances=max_instances,
    )
    results_hat = prequential_evaluation(
        stream=stream,
        learner=default_hat,
        window_size=window_size,
        max_instances=max_instances,
    )
    results_lb = prequential_evaluation(
        stream=stream,
        learner=default_lb,
        window_size=window_size,
        max_instances=max_instances,
    )
    results_srp = prequential_evaluation(
        stream=stream,
        learner=default_srp,
        window_size=window_size,
        max_instances=max_instances,
    )

    # Print results
    print("\nEvaluation Results:")
    print(
        f"[Enhanced AutoClass] Accuracy = {results_autoclass_enhanced.accuracy():.3f}, "
        f"Time: {results_autoclass_enhanced.wallclock():.3f}s"
    )
    print(
        f"[SuccessiveHalving] Accuracy = {results_shc.accuracy():.3f}, "
        f"Time: {results_shc.wallclock():.3f}s"
    )
    print(
        f"[BanditClassifier] Accuracy = {results_bandit.accuracy():.3f}, "
        f"Time: {results_bandit.wallclock():.3f}s"
    )
    print(
        f"[Default HoeffdingTree] Accuracy = {results_ht.accuracy():.3f}, "
        f"Time: {results_ht.wallclock():.3f}s"
    )
    print(
        f"[Default KNN] Accuracy = {results_knn.accuracy():.3f}, "
        f"Time: {results_knn.wallclock():.3f}s"
    )
    print(
        f"[Default AdaptiveRandomForest] Accuracy = {results_arf.accuracy():.3f}, "
        f"Time: {results_arf.wallclock():.3f}s"
    )
    print(
        f"[Default HoeffdingAdaptiveTree] Accuracy = {results_hat.accuracy():.3f}, "
        f"Time: {results_hat.wallclock():.3f}s"
    )
    print(
        f"[Default LeveragingBagging] Accuracy = {results_lb.accuracy():.3f}, "
        f"Time: {results_lb.wallclock():.3f}s"
    )
    print(
        f"[Default StreamingRandomPatches] Accuracy = {results_srp.accuracy():.3f}, "
        f"Time: {results_srp.wallclock():.3f}s"
    )

    # Plot results
    plot_windowed_results(
        results_autoclass_enhanced,
        results_shc,
        results_bandit,
        results_ht,
        results_knn,
        results_arf,
        results_hat,
        results_lb,
        results_srp,
        metric="accuracy",
    )

    # Display final model info for SuccessiveHalving
    model_info_shc = shc_direct.get_model_info()
    print("\nSuccessive Halving Final Status:")
    print(
        f"Active models: {model_info_shc['active_models']} / {model_info_shc['total_models']}"
    )
    print(f"Total rungs: {model_info_shc['current_rung']}")
    print(
        f"Budget used: {model_info_shc['budget_used']} / {model_info_shc['total_budget']}"
    )

    # Display final model info for BanditClassifier
    model_info_bandit = bandit_clf.get_model_info()
    print("\nBandit Classifier Final Status:")
    print(f"Total models: {model_info_bandit['total_models']}")
    print(f"Best model accuracy: {model_info_bandit['best_model_accuracy']:.4f}")

    print("\nTop performing models:")
    print("SuccessiveHalving:")
    for i, model_info in enumerate(model_info_shc["top_models"]):
        print(
            f"  {i + 1}. {model_info['model']} - Accuracy: {model_info['accuracy']:.4f}"
        )
```

### Test on Electricity Stream

```ipython3
stream = Electricity()
max_instances = 8000
window_size = 2000
budget = max_instances
```

```ipython3
results_autoclass = autoclass(
    stream, window_size=window_size, max_instances=max_instances, budget=budget
)
```

```ipython3
shc_direct, results_shc = successive_halving(stream, max_instances, window_size)
```

```myst-ansi
[Rung 1]	7 models removed	7 models left	142 instances per model	budget used: 1988	budget left: 6012	best accuracy: 95.0704
Top models:
  1. StreamingRandomPatches - Accuracy: 95.0704
  2. StreamingRandomPatches - Accuracy: 95.0704
  3. Leveraging OnlineBagging - Accuracy: 93.6620
```

```myst-ansi
[Rung 2]	3 models removed	4 models left	286 instances per model	budget used: 3990	budget left: 4010	best accuracy: 88.3178
Top models:
  1. StreamingRandomPatches - Accuracy: 88.3178
  2. StreamingRandomPatches - Accuracy: 88.3178
  3. Leveraging OnlineBagging - Accuracy: 87.3832
```

```myst-ansi
[Rung 3]	2 models removed	2 models left	501 instances per model	budget used: 5994	budget left: 2006	best accuracy: 90.9580
Top models:
  1. StreamingRandomPatches - Accuracy: 90.9580
  2. StreamingRandomPatches - Accuracy: 90.6351
```

```ipython3
bandit_clf, results_bandit = bandit_classifier(
    stream, max_instances, window_size, bandit_eps=0.3
)
```

```myst-ansi
Using 14 provided base classifiers
```

```myst-ansi

Chosen model: StreamingRandomPatches
Current accuracy: 90.7145
```

```ipython3
test_autoclass_vs_successive_halving_and_bandit(
    stream,
    window_size=window_size,
    max_instances=max_instances,
    results_autoclass_enhanced=results_autoclass,
    shc_direct=shc_direct,
    results_shc=results_shc,
    bandit_clf=bandit_clf,
    results_bandit=results_bandit,
)
```

```myst-ansi

================================================================================
COMPARING AUTOCLASS WITH CONFIG, SUCCESSIVE HALVING, AND BANDIT CLASSIFIER
================================================================================

Initializing default models for comparison...

Evaluating default models...
```

```myst-ansi

Evaluation Results:
[Enhanced AutoClass] Accuracy = 91.425, Time: 70.704s
[SuccessiveHalving] Accuracy = 91.300, Time: 68.928s
[BanditClassifier] Accuracy = 89.925, Time: 35.776s
[Default HoeffdingTree] Accuracy = 86.462, Time: 0.029s
[Default KNN] Accuracy = 84.775, Time: 0.559s
[Default AdaptiveRandomForest] Accuracy = 90.112, Time: 11.219s
[Default HoeffdingAdaptiveTree] Accuracy = 87.525, Time: 0.051s
[Default LeveragingBagging] Accuracy = 90.188, Time: 4.467s
[Default StreamingRandomPatches] Accuracy = 91.325, Time: 12.048s

Successive Halving Final Status:
Active models: 2 / 14
Total rungs: 3
Budget used: 5994 / 8000
```

```myst-ansi

Bandit Classifier Final Status:
Total models: 14
Best model accuracy: 90.2970

Top performing models:
SuccessiveHalving:
  1. StreamingRandomPatches - Accuracy: 91.5125
  2. StreamingRandomPatches - Accuracy: 91.4375
```

On Electricity, AutoClass achieves the highest accuracy (91.6%) but is slowest (134s).

SuccessiveHalvingClassifier nearly matches it (91.3%) while running faster, offering the best balance.

BanditClassifier (82.1%, 36s) is fastest but underperforms, even below the default HoeffdingTree (86.5%).

Overall, thorough model selection proves more effective here, with SuccessiveHalving striking the optimal trade-off.

### Test on custom stream with drift

```ipython3
window_size = 500
max_instances = 8000
budget = max_instances * 3
drift_stream = DriftStream(
    stream=[
        SEA(function=1),
        AbruptDrift(position=3000),
        SEA(function=3),
        GradualDrift(position=6000, width=2000),
        SEA(function=1),
    ]
)
```

```ipython3
results_autoclass = autoclass(
    drift_stream, window_size=window_size, max_instances=max_instances, budget=budget
)
```

```ipython3
shc_direct, results_shc = successive_halving(drift_stream, max_instances, window_size)
```

```myst-ansi
[Rung 1]	7 models removed	7 models left	428 instances per model	budget used: 5992	budget left: 18008	best accuracy: 80.8411
Top models:
  1. HoeffdingAdaptiveTree - Accuracy: 80.8411
  2. HoeffdingAdaptiveTree - Accuracy: 80.8411
  3. kNN - Accuracy: 80.3738
```

```myst-ansi
[Rung 2]	3 models removed	4 models left	857 instances per model	budget used: 11991	budget left: 12009	best accuracy: 85.9922
Top models:
  1. AdaptiveRandomForest - Accuracy: 85.9922
  2. kNN - Accuracy: 84.8249
  3. HoeffdingTree - Accuracy: 84.5914
```

```myst-ansi
[Rung 3]	2 models removed	2 models left	1501 instances per model	budget used: 17995	budget left: 6005	best accuracy: 87.2936
Top models:
  1. AdaptiveRandomForest - Accuracy: 87.2936
  2. Leveraging OnlineBagging - Accuracy: 85.9296
```

```ipython3
bandit_clf, results_bandit = bandit_classifier(
    drift_stream, max_instances, window_size, bandit_eps=0.3
)
```

```myst-ansi
Using 14 provided base classifiers
```

```myst-ansi

Chosen model: kNN
Current accuracy: 85.0684
```

```ipython3
test_autoclass_vs_successive_halving_and_bandit(
    drift_stream,
    window_size=window_size,
    max_instances=max_instances,
    results_autoclass_enhanced=results_autoclass,
    shc_direct=shc_direct,
    results_shc=results_shc,
    bandit_clf=bandit_clf,
    results_bandit=results_bandit,
)
```

```myst-ansi

================================================================================
COMPARING AUTOCLASS WITH CONFIG, SUCCESSIVE HALVING, AND BANDIT CLASSIFIER
================================================================================

Initializing default models for comparison...

Evaluating default models...
```

```myst-ansi

Evaluation Results:
[Enhanced AutoClass] Accuracy = 87.087, Time: 34.469s
[SuccessiveHalving] Accuracy = 86.775, Time: 72.587s
[BanditClassifier] Accuracy = 85.400, Time: 30.074s
[Default HoeffdingTree] Accuracy = 85.088, Time: 0.266s
[Default KNN] Accuracy = 84.362, Time: 0.673s
[Default AdaptiveRandomForest] Accuracy = 86.913, Time: 5.019s
[Default HoeffdingAdaptiveTree] Accuracy = 84.800, Time: 0.278s
[Default LeveragingBagging] Accuracy = 86.375, Time: 2.088s
[Default StreamingRandomPatches] Accuracy = 86.638, Time: 5.395s
```

```myst-ansi

Successive Halving Final Status:
Active models: 2 / 14
Total rungs: 3
Budget used: 17995 / 24000

Bandit Classifier Final Status:
Total models: 14
Best model accuracy: 85.5615

Top performing models:
SuccessiveHalving:
  1. AdaptiveRandomForest - Accuracy: 86.8750
  2. Leveraging OnlineBagging - Accuracy: 86.6000
```

On the drift stream, Enhanced AutoClass reaches the top accuracy (86.9%) with moderate runtime (68s). BanditClassifier (85.2%, 37s) offers the best efficiency-accuracy balance, being more than 2x faster than SuccessiveHalving while losing only 1.4% accuracy.

AdaptiveRandomForest (86.7%) confirms that strong default ensembles can rival AutoML methods.

### Overall Analysis

Across tested streams, the three model selection approaches show consistent accuracy–efficiency trade-offs:

- AutoClass: highest (or near-highest) accuracy, especially under concept drift, but slower than BanditClassifier.
- SuccessiveHalvingClassifier: accuracy almost the one of AutoClass, but faster.
- BanditClassifier: fastest but less accurate.

Guidelines based on these results:

1. Max accuracy & high resources → AutoClass.
2. Balance accuracy/efficiency → SuccessiveHalvingClassifier.
3. Time/resource-constrained → BanditClassifier.