functional

A collection of functional utilities for OCL.

Functions

hinton_distillation_loss

Hinton's distillation loss [1] .

capymoa.ocl.util.functional.hinton_distillation_loss(

teacher_logits: Tensor,

student_logits: Tensor,

temperature: float = 1.0,

) → Tensor

Hinton’s distillation loss [1] .

\[L_{KD} = T^2 KL(softmax(z_t / T), softmax(z_s / T))\]

where \(T\) is the temperature, \(z_s\) are the student logits, and \(z_t\) are the teacher logits.

[1] (1,2)

Hinton, G., Vinyals, O., & Dean, J. (2015). Distilling the Knowledge in a Neural Network. arXiv:1503.02531 [Cs, Stat]. http://arxiv.org/abs/1503.02531

Parameters:

• teacher_logits – Teacher logits of shape (batch_size, num_classes).

• student_logits – Student logits of shape (batch_size, num_classes).

• temperature – Temperature for distillation. Higher values produce softer probability distributions.

Returns:

The distillation loss as a scalar tensor.