# `OCLMetrics`

### *class* capymoa.ocl.evaluation.OCLMetrics[[source]](https://github.com/adaptive-machine-learning/CapyMOA/blob/3e255b1/src/capymoa/ocl/evaluation/_metrics.py#L12)

Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object)

A collection of metrics evaluating an online continual learner.

We define some metrics in terms of a matrix $R\in\mathbb{R}^{T \times T}$
([`accuracy_matrix`](#capymoa.ocl.evaluation.OCLMetrics.accuracy_matrix)) where each element $R_{i,j}$ contains the
the test accuracy on task $j$ after sequentially training on tasks
$1$ through $i$.

Online learning make predictions continuously during training, so we also
provide “anytime” versions of the metrics. These metrics are collected
periodically during training. Specifically, $H$ times per task.
The results of this evaluation are stored in a matrix
$A\in\mathbb{R}^{T \times H \times T}$ ([`anytime_accuracy_matrix`](#capymoa.ocl.evaluation.OCLMetrics.anytime_accuracy_matrix))
where each element $A_{i,h,j}$ contains the test accuracy on task
$j$ after sequentially training on tasks $1$ through $i-1$
and step $h$ of task $i$.

#### \_\_init_\_(anytime_accuracy_all: [ndarray](https://numpy.org/doc/stable/reference/generated/numpy.ndarray.html#numpy.ndarray), anytime_accuracy_all_avg: [float](https://docs.python.org/3/builtins/functions.html#float), anytime_accuracy_seen: [ndarray](https://numpy.org/doc/stable/reference/generated/numpy.ndarray.html#numpy.ndarray), anytime_accuracy_seen_avg: [float](https://docs.python.org/3/builtins/functions.html#float), anytime_task_index: [ndarray](https://numpy.org/doc/stable/reference/generated/numpy.ndarray.html#numpy.ndarray), accuracy_all: [ndarray](https://numpy.org/doc/stable/reference/generated/numpy.ndarray.html#numpy.ndarray), accuracy_all_avg: [float](https://docs.python.org/3/builtins/functions.html#float), accuracy_seen: [ndarray](https://numpy.org/doc/stable/reference/generated/numpy.ndarray.html#numpy.ndarray), accuracy_seen_avg: [float](https://docs.python.org/3/builtins/functions.html#float), accuracy_final: [float](https://docs.python.org/3/builtins/functions.html#float), task_index: [ndarray](https://numpy.org/doc/stable/reference/generated/numpy.ndarray.html#numpy.ndarray), forward_transfer: [float](https://docs.python.org/3/builtins/functions.html#float), backward_transfer: [float](https://docs.python.org/3/builtins/functions.html#float), accuracy_matrix: [ndarray](https://numpy.org/doc/stable/reference/generated/numpy.ndarray.html#numpy.ndarray), class_cm: [ndarray](https://numpy.org/doc/stable/reference/generated/numpy.ndarray.html#numpy.ndarray), anytime_accuracy_matrix: [ndarray](https://numpy.org/doc/stable/reference/generated/numpy.ndarray.html#numpy.ndarray), n_classes: [int](https://docs.python.org/3/builtins/functions.html#int), n_tasks: [int](https://docs.python.org/3/builtins/functions.html#int), n_continual_evaluations: [int](https://docs.python.org/3/builtins/functions.html#int), ttt: [PrequentialResults](capymoa.evaluation.results.PrequentialResults.md#capymoa.evaluation.results.PrequentialResults), boundaries: [ndarray](https://numpy.org/doc/stable/reference/generated/numpy.ndarray.html#numpy.ndarray), ttt_windowed_task_index: [ndarray](https://numpy.org/doc/stable/reference/generated/numpy.ndarray.html#numpy.ndarray)) → [None](https://docs.python.org/3/builtins/constants.html#None)

#### accuracy_all *: [ndarray](https://numpy.org/doc/stable/reference/generated/numpy.ndarray.html#numpy.ndarray)*

The accuracy on all tasks after training on each task.

Is a ndarray of shape (n_tasks,), dtype=np.float32

$$
a_\text{all}(t) = \frac{1}{T} \sum_{i=1}^{T} R_{t,i}
$$

Use [`task_index`](#capymoa.ocl.evaluation.OCLMetrics.task_index) to get the corresponding task index for plotting.

#### accuracy_all_avg *: [float](https://docs.python.org/3/builtins/functions.html#float)*

The average of [`accuracy_all`](#capymoa.ocl.evaluation.OCLMetrics.accuracy_all) over all tasks.

$$
\bar{a}_\text{all} = \frac{1}{T}\sum_{t=1}^T a_\text{all}(t)
$$

#### accuracy_final *: [float](https://docs.python.org/3/builtins/functions.html#float)*

The accuracy on all tasks after training on the final task.

$$
a_\text{final} = a_\text{all}(T)
$$

#### accuracy_matrix *: [ndarray](https://numpy.org/doc/stable/reference/generated/numpy.ndarray.html#numpy.ndarray)*

A matrix measuring the accuracy on each task after training on each task.

Is a ndarray of shape (n_tasks, n_tasks), dtype=np.float32.

`R[i, j]` is the accuracy on task $j$ after training on tasks
$1$ through $i$.

#### accuracy_seen *: [ndarray](https://numpy.org/doc/stable/reference/generated/numpy.ndarray.html#numpy.ndarray)*

The accuracy on **seen** tasks after training on each task.

Is a ndarray of shape (n_tasks,), dtype=np.float32.

$$
a_\text{seen}(t) = \frac{1}{t}\sum^t_{i=1} R_{t,i}
$$

Use [`task_index`](#capymoa.ocl.evaluation.OCLMetrics.task_index) to get the corresponding task index for plotting.

#### accuracy_seen_avg *: [float](https://docs.python.org/3/builtins/functions.html#float)*

The average of [`accuracy_seen`](#capymoa.ocl.evaluation.OCLMetrics.accuracy_seen) over all tasks.

$$
\bar{a}_\text{seen} = \frac{1}{T}\sum_{t=1}^T a_\text{seen}(t)
$$

#### anytime_accuracy_all *: [ndarray](https://numpy.org/doc/stable/reference/generated/numpy.ndarray.html#numpy.ndarray)*

The accuracy on all tasks after training on each step in each task.

Is a ndarray of shape (n_tasks \* n_continual_evaluations,), dtype=np.float32.

$$
a_\text{any all}(t, h) = \frac{1}{T}\sum^T_{i=1} A_{t,h,i}
$$

We flatten the $t,h$ dimensions to a 1D array. Use
[`anytime_task_index`](#capymoa.ocl.evaluation.OCLMetrics.anytime_task_index) to get the corresponding task index for plotting.

#### anytime_accuracy_all_avg *: [float](https://docs.python.org/3/builtins/functions.html#float)*

The average of [`anytime_accuracy_all`](#capymoa.ocl.evaluation.OCLMetrics.anytime_accuracy_all) over all tasks.

$$
\bar{a}_\text{any all} = \frac{1}{T}\sum_{t=1}^T \frac{1}{H}\sum_{h=1}^H a_\text{any all}(t, h)
$$

#### anytime_accuracy_matrix *: [ndarray](https://numpy.org/doc/stable/reference/generated/numpy.ndarray.html#numpy.ndarray)*

A matrix measuring the accuracy on each task after training on each task and step.

Is a ndarray of shape (n_tasks \* n_continual_evaluations, n_tasks), dtype=np.float32.

This matrix is $A$ with the first two dimensions flattened to a 2D array.

#### anytime_accuracy_seen *: [ndarray](https://numpy.org/doc/stable/reference/generated/numpy.ndarray.html#numpy.ndarray)*

The accuracy on **seen** tasks after training on each step in each task.

$$
a_\text{any seen}(t, h) = \frac{1}{t}\sum^t_{i=1} A_{t,h,i}
$$

We flatten the $t,h$ dimensions to a 1D array. Use
[`anytime_task_index`](#capymoa.ocl.evaluation.OCLMetrics.anytime_task_index) to get the corresponding task index for plotting.

#### anytime_accuracy_seen_avg *: [float](https://docs.python.org/3/builtins/functions.html#float)*

The average of [`anytime_accuracy_seen`](#capymoa.ocl.evaluation.OCLMetrics.anytime_accuracy_seen) over all tasks.

$$
\bar{a}_\text{any seen} = \frac{1}{T}\sum_{t=1}^T \frac{1}{H}\sum_{h=1}^H a_\text{any seen}(t, h)
$$

#### anytime_task_index *: [ndarray](https://numpy.org/doc/stable/reference/generated/numpy.ndarray.html#numpy.ndarray)*

The position in each task where the anytime accuracy was measured.

Is a ndarray of shape (n_tasks \* n_continual_evaluations,), dtype=np.integer.

#### backward_transfer *: [float](https://docs.python.org/3/builtins/functions.html#float)*

A scalar measuring the impact learning had on past tasks.

$$
r_\text{BWT} = \frac{2}{T(T-1)} \sum_{i=2}^{T} \sum_{j=1}^{i-1} (R_{i,j} - R_{j,j})
$$

#### boundaries *: [ndarray](https://numpy.org/doc/stable/reference/generated/numpy.ndarray.html#numpy.ndarray)*

Instance index for the boundaries.

Used to map online evaluation to specific tasks.

Is a ndarray of shape (n_tasks + 1,), dtype=np.integer.

#### class_cm *: [ndarray](https://numpy.org/doc/stable/reference/generated/numpy.ndarray.html#numpy.ndarray)*

A confusion matrix of shape `(task, true_class, predicted_class)`.

#### forward_transfer *: [float](https://docs.python.org/3/builtins/functions.html#float)*

A scalar measuring the impact learning had on future tasks.

$$
r_\text{FWT} = \frac{2}{T(T-1)}\sum_{i=1}^{T} \sum_{j=i+1}^{T} R_{i,j}
$$

#### n_classes *: [int](https://docs.python.org/3/builtins/functions.html#int)*

The number of classes $C$.

#### n_continual_evaluations *: [int](https://docs.python.org/3/builtins/functions.html#int)*

The number of continual evaluations per task $H$.

#### n_tasks *: [int](https://docs.python.org/3/builtins/functions.html#int)*

The number of tasks $T$.

#### task_index *: [ndarray](https://numpy.org/doc/stable/reference/generated/numpy.ndarray.html#numpy.ndarray)*

The position of each task in the metrics.

#### ttt *: [PrequentialResults](capymoa.evaluation.results.PrequentialResults.md#capymoa.evaluation.results.PrequentialResults)*

Test-then-train/prequential results.

#### ttt_windowed_task_index *: [ndarray](https://numpy.org/doc/stable/reference/generated/numpy.ndarray.html#numpy.ndarray)*

The position of each window within each task.

Useful as the `x` axis for
[`capymoa.evaluation.results.PrequentialResults.windowed`](capymoa.evaluation.results.PrequentialResults.md#capymoa.evaluation.results.PrequentialResults.windowed).