evaluation#

Evaluate online continual learning in classification tasks.

Classes#

OCLMetrics

A collection of metrics evaluating an online continual learner.

Functions#

capymoa.ocl.evaluation.ocl_train_eval_loop(
learner: TrainTaskAware | TestTaskAware | Classifier,
train_streams: Sequence[DataLoader[Tuple[Tensor, Tensor]]],
test_streams: Sequence[DataLoader[Tuple[Tensor, Tensor]]],
continual_evaluations: int = 1,
progress_bar: bool = False,
eval_window_size: int = 1000,
) → OCLMetrics[source]#

Train and evaluate a learner on a sequence of tasks.

Parameters:

  • learner – A classifier that is possibly train task aware and/or test task aware.

  • train_streams – A sequence of streams containing the training tasks.

  • test_streams – A sequence of streams containing the testing tasks.

  • continual_evaluations – The number of times to evaluate the learner during each task. If 1, the learner is only evaluated at the end of each task.

  • progress_bar – Whether to display a progress bar. The bar displayed will show the progress over all training and evaluation steps including the continual evaluations.

Returns:

A collection of metrics evaluating the learner’s performance.