IEWC#

Arguments#

Options

--e_lambdafloat

Help: lambda weight for EWC

  • Default: None

--gammafloat

Help: gamma parameter for EWC online

  • Default: None

--kappafloat

Help: Relative damping: tau = kappa * median of the nonzero output-gradient scales of the task.

  • Default: 0.01

--sample_weightingstr

Help: Per-sample aggregation weights: uniform or residual-diversity weighting (rdw).

  • Default: uniform

  • Choices: uniform, rdw

--rdw_max_basisint

Help: Maximum number of earlier gradient directions kept by rdw. The basis is stored densely on CPU (rdw_max_basis x num_params floats, ~2.9 GB for ResNet-18 with the default).

  • Default: 64

Implementation of Improved Elastic Weight Consolidation (IEWC) from the paper “Improved Elastic Weight Consolidation as an Optimization Constraint for Continual Learning” Paper: https://arxiv.org/abs/2611.16039 Code: https://github.com/Axym-Labs/iewc

IEWC keeps the online EWC penalty of ewc_on and only changes the diagonal importance. Each old sample contributes its squared parameter gradient divided by its own squared output-gradient norm, so well-fitted samples are not down-weighted by their small loss gradient:

F = sum_n h_n * g_n^2 / (s_n + tau) / sum_n h_n

where g_n is the per-sample gradient of the loss w.r.t. the parameters, s_n = ||dl_n / df_n||^2 (see IEWC.output_scale), tau = kappa * median{s_n : s_n > 0} and h_n is either 1 (uniform), the residual-diversity weight (rdw), or given by a custom function (see IEWC.get_sample_weighting).

As in ewc_on, the penalty is e_lambda * sum_i F_i (theta_i - theta*_i)^2, so e_lambda corresponds to lambda / 2 in the paper. The importance has a different scale than in ewc_on, so e_lambda has to be tuned separately. Unlike ewc_on, samples are not weighted by p(y|x) and the importance is computed with the backbone in eval mode.

Note: the ImageNet-R gain reported in the paper (+2.00 pts) is measured against the observed-label empirical Fisher with uniform weighting, not against the p(y|x)-weighted estimate used by ewc_on.

Classes#

class models.iewc.IEWC(backbone, loss, args, transform, dataset=None)[source]#

Bases: EwcOn

Continual learning via Improved Elastic Weight Consolidation (IEWC).

COMPATIBILITY: List[str] = ['class-il', 'domain-il', 'task-il']#
NAME: str = 'iewc'#
end_task(dataset)[source]#
static get_parser(parser)[source]#
Return type:

ArgumentParser

get_sample_weighting()[source]#

Returns the aggregation weights used by iewc_importance: uniform or rdw (from –sample_weighting). Override this method to return a function fn(x, y, flat_grad) -> float with custom non-negative weights.

output_scale(output_grads)[source]#

Maps the output gradients r_n (one per row) to the scales s_n = r_n^T G_n^{-1} r_n. The default is the identity metric. Override this method to use a different output geometry.

class models.iewc.ResidualBasis(dim, max_size=64, eps=1e-12)[source]#

Bases: object

Streaming residual-diversity weights (RDW).

Keeps an orthonormal basis of at most max_size earlier residual directions (on CPU) and returns, for each new gradient, the squared norm of its unit direction outside that span.

weight(grad)[source]#
Return type:

float

Functions#

models.iewc.iewc_importance(net, loss_fn, batches, kappa=0.01, sample_weighting='uniform', rdw_max_basis=64, output_scale=<function squared_norm>)[source]#

Computes the diagonal IEWC importance.

Parameters:
  • net – the backbone (a MammothBackbone)

  • loss_fn – the per-sample loss, called as loss_fn(outputs, labels) on single-sample batches

  • batches – a list of (inputs, labels) batches; both passes use exactly these inputs

  • kappa – relative damping, tau = kappa * median{s_n : s_n > 0}

  • sample_weighting – uniform, rdw, or a function fn(x, y, flat_grad) -> float called once per sample in order, returning a non-negative (unnormalized) weight

  • rdw_max_basis – maximum number of directions kept by RDW

  • output_scale – maps the output gradients (B, …) to the scales s_n (B,)

Returns:

the importance (same shape as net.get_params()) and the mean-normalized sample weights

models.iewc.squared_norm(output_grads)[source]#

Default output geometry (identity metric): s_n = ||r_n||^2 for each row r_n.

Return type:

Tensor

models.iewc.iewc_importance(net, loss_fn, batches, kappa=0.01, sample_weighting='uniform', rdw_max_basis=64, output_scale=<function squared_norm>)[source]#

Computes the diagonal IEWC importance.

Parameters:
  • net – the backbone (a MammothBackbone)

  • loss_fn – the per-sample loss, called as loss_fn(outputs, labels) on single-sample batches

  • batches – a list of (inputs, labels) batches; both passes use exactly these inputs

  • kappa – relative damping, tau = kappa * median{s_n : s_n > 0}

  • sample_weighting – uniform, rdw, or a function fn(x, y, flat_grad) -> float called once per sample in order, returning a non-negative (unnormalized) weight

  • rdw_max_basis – maximum number of directions kept by RDW

  • output_scale – maps the output gradients (B, …) to the scales s_n (B,)

Returns:

the importance (same shape as net.get_params()) and the mean-normalized sample weights

models.iewc.squared_norm(output_grads)[source]#

Default output geometry (identity metric): s_n = ||r_n||^2 for each row r_n.

Return type:

Tensor