IEWC#
Arguments#
Options
- --e_lambdafloat
Help: lambda weight for EWC
Default:
None
- --gammafloat
Help: gamma parameter for EWC online
Default:
None
- --kappafloat
Help: Relative damping: tau = kappa * median of the nonzero output-gradient scales of the task.
Default:
0.01
- --sample_weightingstr
Help: Per-sample aggregation weights: uniform or residual-diversity weighting (rdw).
Default:
uniformChoices:
uniform, rdw
- --rdw_max_basisint
Help: Maximum number of earlier gradient directions kept by rdw. The basis is stored densely on CPU (rdw_max_basis x num_params floats, ~2.9 GB for ResNet-18 with the default).
Default:
64
Implementation of Improved Elastic Weight Consolidation (IEWC) from the paper “Improved Elastic Weight Consolidation as an Optimization Constraint for Continual Learning” Paper: https://arxiv.org/abs/2611.16039 Code: https://github.com/Axym-Labs/iewc
IEWC keeps the online EWC penalty of ewc_on and only changes the diagonal importance. Each old sample contributes its squared parameter gradient divided by its own squared output-gradient norm, so well-fitted samples are not down-weighted by their small loss gradient:
F = sum_n h_n * g_n^2 / (s_n + tau) / sum_n h_n
where g_n is the per-sample gradient of the loss w.r.t. the parameters, s_n = ||dl_n / df_n||^2 (see IEWC.output_scale), tau = kappa * median{s_n : s_n > 0} and h_n is either 1 (uniform), the residual-diversity weight (rdw), or given by a custom function (see IEWC.get_sample_weighting).
As in ewc_on, the penalty is e_lambda * sum_i F_i (theta_i - theta*_i)^2, so e_lambda corresponds to lambda / 2 in the paper. The importance has a different scale than in ewc_on, so e_lambda has to be tuned separately. Unlike ewc_on, samples are not weighted by p(y|x) and the importance is computed with the backbone in eval mode.
Note: the ImageNet-R gain reported in the paper (+2.00 pts) is measured against the observed-label empirical Fisher with uniform weighting, not against the p(y|x)-weighted estimate used by ewc_on.
Classes#
- class models.iewc.IEWC(backbone, loss, args, transform, dataset=None)[source]#
Bases:
EwcOnContinual learning via Improved Elastic Weight Consolidation (IEWC).
- class models.iewc.ResidualBasis(dim, max_size=64, eps=1e-12)[source]#
Bases:
objectStreaming residual-diversity weights (RDW).
Keeps an orthonormal basis of at most max_size earlier residual directions (on CPU) and returns, for each new gradient, the squared norm of its unit direction outside that span.
Functions#
- models.iewc.iewc_importance(net, loss_fn, batches, kappa=0.01, sample_weighting='uniform', rdw_max_basis=64, output_scale=<function squared_norm>)[source]#
Computes the diagonal IEWC importance.
- Parameters:
net – the backbone (a MammothBackbone)
loss_fn – the per-sample loss, called as loss_fn(outputs, labels) on single-sample batches
batches – a list of (inputs, labels) batches; both passes use exactly these inputs
kappa – relative damping, tau = kappa * median{s_n : s_n > 0}
sample_weighting – uniform, rdw, or a function fn(x, y, flat_grad) -> float called once per sample in order, returning a non-negative (unnormalized) weight
rdw_max_basis – maximum number of directions kept by RDW
output_scale – maps the output gradients (B, …) to the scales s_n (B,)
- Returns:
the importance (same shape as net.get_params()) and the mean-normalized sample weights
- models.iewc.squared_norm(output_grads)[source]#
Default output geometry (identity metric): s_n = ||r_n||^2 for each row r_n.
- Return type:
Tensor
- models.iewc.iewc_importance(net, loss_fn, batches, kappa=0.01, sample_weighting='uniform', rdw_max_basis=64, output_scale=<function squared_norm>)[source]#
Computes the diagonal IEWC importance.
- Parameters:
net – the backbone (a MammothBackbone)
loss_fn – the per-sample loss, called as loss_fn(outputs, labels) on single-sample batches
batches – a list of (inputs, labels) batches; both passes use exactly these inputs
kappa – relative damping, tau = kappa * median{s_n : s_n > 0}
sample_weighting – uniform, rdw, or a function fn(x, y, flat_grad) -> float called once per sample in order, returning a non-negative (unnormalized) weight
rdw_max_basis – maximum number of directions kept by RDW
output_scale – maps the output gradients (B, …) to the scales s_n (B,)
- Returns:
the importance (same shape as net.get_params()) and the mean-normalized sample weights