evaluation#

Evaluate online continual learning in classification tasks.

Modules#

events

Event definitions for OCL evaluation loops.

Classes#

OCLMetrics

A collection of metrics evaluating an online continual learner.

Functions#

ocl_train_eval_loop

Run the OCL training loop with periodic continual evaluation.
capymoa.ocl.evaluation.ocl_train_eval_loop(
learner: Classifier,
train_streams: Sequence[DataLoader[tuple[Tensor, Tensor]]],
test_streams: Sequence[DataLoader[tuple[Tensor, Tensor]]],
continual_evaluations: int = 1,
progress_bar: bool = False,
eval_window_size: int = 1000,
epochs: int = 1,
dispatcher: Dispatcher | None = None,
) → OCLMetrics[source]#

Run the OCL training loop with periodic continual evaluation.

Parameters:

  • learner – The classifier to train and evaluate.

  • train_streams – Sequence of task-wise training data loaders.

  • test_streams – Sequence of task-wise test data loaders. Must have the same number of tasks as train_streams.

  • continual_evaluations – Number of evaluation passes performed during each training task, defaults to 1.

  • progress_bar – Whether to enable a progress-bar, defaults to False.

  • eval_window_size – Window size used by the default metrics handler to compute rolling metrics, defaults to 1000.

  • epochs – Number of epochs to train each task stream, defaults to 1.

  • dispatcher – Optional event dispatcher. If None, a new dispatcher is created.

Returns:

Aggregated OCL metrics collected by the default metrics handler.

Raises:

  • ValueError – If train/test task counts differ, continual_evaluations < 1, or a train stream has fewer batches than requested evaluations.

  • TypeError – If learner is not a classifier.