ConstrainedDense

class selfeeg.models.layers.ConstrainedDense(in_features, out_features, bias=True, device=None, dtype=None, max_norm=2.0, min_norm=None, axis_norm=1, minmax_rate=1.0)[source]

nn.Linear layer with norm constraints.

This is a Pytorch implementation of the Dense layer with the possibility of adding a MaxNorm, MinMaxNorm, or a UnitNorm constraint. Most of the parameters are the same as described in torch.nn.Linear help.

Parameters:

  • in_features (int) – Number of input features.

  • out_channels (int) – Number of output features.

  • bias (bool, optional) –

    If True, adds a learnable bias to the output.

    Default = True

  • device (torch.device or str, optional) – The torch device.

  • dtype (torch dtype, optional) – layer dtype, i.e., the data type of the torch.Tensor defining the layer weights.

  • max_norm (float, optional) –

    The maximum norm each hidden unit can have. If None no constraint will be added.

    Default = 2.0

  • min_norm (float, optional) –

    The minimum norm each hidden unit can have. Must be a float lower than max_norm. If given, MinMaxNorm will be applied in the case max_norm is also given. Otherwise, it will be ignored.

    Default = None

  • axis_norm (Union[int, list, tuple], optional) –

    The axis along weights are constrained. It behaves like Keras. So, considering that a Conv2D layer has shape (output_depth, input_depth), set axis to 1 will constrain the weights of each filter tensor of size (input_depth,).

    Default = 1

  • minmax_rate (float, optional) –

    A constraint for MinMaxNorm setting how weights will be rescaled at each step. It behaves like Keras rate argument of MinMaxNorm contraint. So, using minmax_rate = 1 will set a strict enforcement of the constraint, while rate<1.0 will slowly rescale layer’s hidden units at each step.

    Default = 1.0

Note

To Apply a MaxNorm constraint, set only max_norm. To apply a MinMaxNorm constraint, set both min_norm and max_norm. To apply a UnitNorm constraint, set both min_norm and max_norm to 1.0.

Example

>>> from selfeeg.models import ConstrainedDense
>>> import torch
>>> x = torch.randn(4,64)
>>> mdl = ConstrainedDense(64,32)
>>> out = mdl(x)
>>> norms = torch.sqrt(torch.sum(torch.square(mdl.weight), axis=1))
>>> print(out.shape) # shoud return torch.Size([4, 32])
>>> print(torch.isnan(out).sum()) # shoud return 0
>>> print(torch.sum(norms>(1.4+1e-3)).item() == 0) # should return True
scale_norm(eps=1e-09)[source]

applies the desired constraint on the Layer. It is highly based on the Keras implementation, but here MaxNorm, MinMaxNorm and UnitNorm are all implemented inside this function.