ConstrainedConv1d

class selfeeg.models.layers.ConstrainedConv1d(in_channels, out_channels, kernel_size, stride=1, padding='same', dilation=1, groups=1, bias=True, padding_mode='zeros', device=None, dtype=None, max_norm=2.0, min_norm=None, axis_norm=[1, 2], minmax_rate=1.0)

nn.Conv1d layer with norm constraints.

This is a Pytorch implementation of the 1D Convolutional layer with the possibility to add a MaxNorm, MinMaxNorm, or UnitNorm constraint along the given axis. Most of the parameters are the same as described in pytorch Conv1d help.

Parameters:

• in_channels (int) – Number of input channels.

• out_channels (int) – Number of output channels.

• kernel_size (int or tuple) – Size of the convolving kernel.

• stride (int or tuple, optional) –

  Stride of the convolution.

  Default = 1

• padding (int, tuple or str, optional) –

  Padding added to all four sides of the input. This class also accepts the string ‘causal’, which triggers causal convolution like in Wavenet.

  Default = 0

• dilation (int or tuple, optional) –

  Spacing between kernel elements.

  Default = 1

• groups (int, optional) –

  Number of blocked connections from input channels to output channels.

  Default = 1

• bias (bool, optional) –

  If True, adds a learnable bias to the output.

  Default = True

• padding_mode (str, optional) –

  Any of ‘zeros’, ‘reflect’, ‘replicate’ or ‘circular’.

  Default = ‘zeros’

• device (torch.device or str, optional) – The torch device.

• dtype (torch.dtype, optional) – Layer dtype, i.e., the data type of the torch.Tensor defining the layer weights.

• max_norm (float, optional) –

  The maximum norm each hidden unit can have. If None no constraint will be added.

  Default = 2.0

• min_norm (float, optional) –

  The minimum norm each hidden unit can have. Must be a float lower than max_norm. If given, MinMaxNorm will be applied in the case max_norm is also given. Otherwise, it will be ignored.

  Default = None

• axis_norm (Union[int, list, tuple], optional) –

  The axis along weights are constrained. It behaves like Keras. So, considering that a Conv1D layer has shape (output_depth, input_depth, length), set axis to [1, 2] will constrain the weights of each filter tensor of size (input_depth, length).

  Default = [1,2]

• minmax_rate (float, optional) –

  A constraint for MinMaxNorm setting how weights will be rescaled at each step. It behaves like Keras rate argument of MinMaxNorm contraint. So, using minmax_rate = 1 will set a strict enforcement of the constraint, while rate<1.0 will slowly rescale layer’s hidden units at each step.

  Default = 1.0

Note

To Apply a MaxNorm constraint, set only max_norm. To apply a MinMaxNorm constraint, set both min_norm and max_norm. To apply a UnitNorm constraint, set both min_norm and max_norm to 1.0.

Note

When setting padding to "causal", padding will be internally changed to an integer equal to (kernel_size - 1) * dilation. Then, during forward, the extra features are removed. This is preferable over F.pad, which can lead to memory allocation or even non-deterministic operations during the backboard pass. Additional information can be found at the following link: https://github.com/pytorch/pytorch/issues/1333

Example

>>> from import selfeeg.models import ConstrainedConv1d
>>> import torch
>>> x = torch.randn(4, 8, 64)
>>> mdl = ConstrainedConv1d(8, 16, 15, max_norm = 1.4, min_norm = 0.3)
>>> mdl.weight = torch.nn.Parameter(mdl.weight*10)
>>> out = mdl(x)
>>> norms = torch.sqrt(torch.sum(torch.square(mdl.weight), axis=[1,2]))
>>> print(out.shape) # shoud return torch.Size([4, 16, 64])
>>> print(torch.isnan(out).sum()) # shoud return 0
>>> print(torch.sum(norms>(1.4+1e-3)).item() == 0) # should return True

scale_norm(eps=1e-09)

applies the desired constraint on the Layer. It is highly based on the Keras implementation, but here MaxNorm, MinMaxNorm and UnitNorm are all implemented inside this function.