evaluation#

Evaluate online continual learning in classification tasks.

Classes#

OCLMetrics

A collection of metrics evaluating an online continual learner.

Functions#

capymoa.ocl.evaluation.ocl_train_eval_loop(
learner: TrainTaskAware | TestTaskAware | Classifier,
train_streams: Sequence[DataLoader[Tuple[Tensor, Tensor]]],
test_streams: Sequence[DataLoader[Tuple[Tensor, Tensor]]],
continual_evaluations: int = 1,
progress_bar: bool = False,
eval_window_size: int = 1000,
epochs: int = 1,
) → OCLMetrics[source]#

Train and evaluate a learner on a sequence of tasks.

Parameters:

  • learner – A classifier that is possibly train task aware and/or test task aware.

  • train_streams – A sequence of streams containing the training tasks.

  • test_streams – A sequence of streams containing the testing tasks.

  • continual_evaluations – The number of times to evaluate the learner during each task. If 1, the learner is only evaluated at the end of each task.

  • progress_bar – Whether to display a progress bar. The bar displayed will show the progress over all training and evaluation steps including the continual evaluations.

  • epochs – The number of times to repeat the training stream for each task. This violates the online learning experimental setting, since each instance is seen multiple times. However, it can be useful for offline continual learning experiments.

Returns:

A collection of metrics evaluating the learner’s performance.