Evaluation#

CapyMOA provides comes with some evaluation methods to evaluate the performance of learners.

class capymoa.evaluation.evaluation.ClassificationEvaluator[source]#

Bases: object

Wrapper for the Classification Performance Evaluator from MOA. By default, it uses the BasicClassificationPerformanceEvaluator

__init__(
schema: Schema = None,
window_size=None,
allow_abstaining=True,
moa_evaluator=None,
)[source]#
get_instances_seen()[source]#
update(y_target_index: int, y_pred_index: int | None)[source]#

Update the evaluator with the ground-truth and the prediction.

Parameters:
  • y_target_index – The ground-truth class index. This is NOT the actual class value, but the index of the class value in the schema.

  • y_pred_index – The predicted class index. If the classifier abstains from making a prediction, this value can be None.

Raises:

ValueError – If the values are not valid indexes in the schema.

metrics_header()[source]#
metrics()[source]#
metrics_dict()[source]#
metrics_per_window()[source]#
accuracy()[source]#
kappa()[source]#
kappa_temporal()[source]#
kappa_M()[source]#
class capymoa.evaluation.evaluation.RegressionEvaluator[source]#

Bases: object

Wrapper for the Regression Performance Evaluator from MOA. By default, it uses the MOA BasicRegressionPerformanceEvaluator as moa_evaluator.

__init__(schema=None, window_size=None, moa_evaluator=None)[source]#
get_instances_seen()[source]#
update(y, y_pred: float | None)[source]#
metrics_header()[source]#
metrics()[source]#
metrics_dict()[source]#
metrics_per_window()[source]#
predictions()[source]#
ground_truth_y()[source]#
MAE()[source]#
RMSE()[source]#
RMAE()[source]#
R2()[source]#
adjusted_R2()[source]#
class capymoa.evaluation.evaluation.ClassificationWindowedEvaluator[source]#

Bases: ClassificationEvaluator

Uses the ClassificationEvaluator to perform a windowed evaluation.

IMPORTANT: The results for the last window are always through `metrics()`, if the window_size does not perfectly divide the stream, the metrics corresponding to the last remaining instances in the last window can be obtained by invoking `metrics()`

__init__(schema=None, window_size=1000)[source]#
accuracy()[source]#
get_instances_seen()[source]#
kappa()[source]#
kappa_M()[source]#
kappa_temporal()[source]#
metrics()[source]#
metrics_dict()[source]#
metrics_header()[source]#
metrics_per_window()[source]#
update(y_target_index: int, y_pred_index: int | None)[source]#

Update the evaluator with the ground-truth and the prediction.

Parameters:
  • y_target_index – The ground-truth class index. This is NOT the actual class value, but the index of the class value in the schema.

  • y_pred_index – The predicted class index. If the classifier abstains from making a prediction, this value can be None.

Raises:

ValueError – If the values are not valid indexes in the schema.

class capymoa.evaluation.evaluation.RegressionWindowedEvaluator[source]#

Bases: RegressionEvaluator

Uses the RegressionEvaluator to perform a windowed evaluation.

IMPORTANT: The results for the last window are always through `metrics()`, if the window_size does not perfectly divide the stream, the metrics corresponding to the last remaining instances in the last window can be obtained by invoking `metrics()`

__init__(schema=None, window_size=1000)[source]#
MAE()[source]#
R2()[source]#
RMAE()[source]#
RMSE()[source]#
adjusted_R2()[source]#
get_instances_seen()[source]#
ground_truth_y()[source]#
metrics()[source]#
metrics_dict()[source]#
metrics_header()[source]#
metrics_per_window()[source]#
predictions()[source]#
update(y, y_pred: float | None)[source]#
capymoa.evaluation.evaluation.start_time_measuring()[source]#
capymoa.evaluation.evaluation.stop_time_measuring(start_wallclock_time, start_cpu_time)[source]#
capymoa.evaluation.evaluation.test_then_train_evaluation(
stream,
learner,
max_instances=None,
sample_frequency=None,
evaluator=None,
optimise=True,
)[source]#

Test-then-train evaluation. Returns a dictionary with the results.

capymoa.evaluation.evaluation.windowed_evaluation(stream, learner, max_instances=None, window_size=1000)[source]#

Windowed evaluation. Returns a dictionary with the results.

capymoa.evaluation.evaluation.prequential_evaluation(
stream,
learner,
max_instances=None,
window_size=1000,
optimise=True,
store_predictions=False,
store_y=False,
)[source]#

Calculates the metrics cumulatively (i.e. test-then-train) and in a window-fashion (i.e. windowed prequential evaluation). Returns both evaluators so that the caller has access to metric from both evaluators.

capymoa.evaluation.evaluation.test_then_train_SSL_evaluation(
stream,
learner,
max_instances=None,
sample_frequency=None,
initial_window_size=0,
delay_length=0,
label_probability=0.01,
random_seed=1,
evaluator=None,
optimise=True,
)[source]#

Test-then-train SSL evaluation. Returns a dictionary with the results.

capymoa.evaluation.evaluation.prequential_ssl_evaluation(
stream,
learner,
max_instances=None,
window_size=1000,
initial_window_size=0,
delay_length=0,
label_probability=0.01,
random_seed=1,
optimise=True,
)[source]#

If the learner is not a SSL learner, then it will just train on labeled instances.

capymoa.evaluation.evaluation.prequential_evaluation_multiple_learners(
stream,
learners,
max_instances=None,
window_size=1000,
)[source]#

Calculates the metrics cumulatively (i.e., test-then-train) and in a windowed-fashion for multiple streams and learners. It behaves as if we invoked prequential_evaluation() multiple times, but we only iterate through the stream once. This function is useful in situations where iterating through the stream is costly, but we still want to assess several learners on it. Returns the results in a dictionary format. Infers whether it is a Classification or Regression problem based on the stream schema.

class capymoa.evaluation.evaluation.PredictionIntervalEvaluator[source]#

Bases: RegressionEvaluator

__init__(schema=None, window_size=None, moa_evaluator=None)[source]#
update(y, y_pred)[source]#
metrics_header()[source]#
metrics()[source]#
MAE()[source]#
R2()[source]#
RMAE()[source]#
RMSE()[source]#
adjusted_R2()[source]#
get_instances_seen()[source]#
ground_truth_y()[source]#
metrics_dict()[source]#
metrics_per_window()[source]#
predictions()[source]#
coverage()[source]#
average_length()[source]#
NMPIW()[source]#
class capymoa.evaluation.evaluation.PredictionIntervalWindowedEvaluator[source]#

Bases: PredictionIntervalEvaluator

MAE()[source]#
NMPIW()[source]#
R2()[source]#
RMAE()[source]#
RMSE()[source]#
adjusted_R2()[source]#
average_length()[source]#
coverage()[source]#
get_instances_seen()[source]#
ground_truth_y()[source]#
metrics()[source]#
metrics_dict()[source]#
metrics_header()[source]#
metrics_per_window()[source]#
predictions()[source]#
update(y, y_pred)[source]#
__init__(schema=None, window_size=1000)[source]#