ConstrainedConv2d
- class selfeeg.models.layers.ConstrainedConv2d(in_channels, out_channels, kernel_size, stride=1, padding='same', dilation=1, groups=1, bias=True, padding_mode='zeros', device=None, dtype=None, max_norm=2.0, min_norm=None, axis_norm=[1, 2, 3], minmax_rate=1.0)[source]
nn.conv2d layer with norm constraints.
This is a Pytorch implementation of the 2D Convolutional layer with the possibility to add a MaxNorm, MinMaxNorm, or UnitNorm constraint along the given axis. Most of the parameters are the same as described in pytorch Conv2D help.
- Parameters:
in_channels (int) – Number of input channels.
out_channels (int) – Number of output channels.
kernel_size (int or tuple) – Size of the convolving kernel.
stride (int or tuple, optional) –
Stride of the convolution.
Default = 1
padding (int, tuple or str, optional) –
Padding added to all four sides of the input.
Default = 0
dilation (int or tuple, optional) –
Spacing between kernel elements.
Default = 1
groups (int, optional) –
Number of blocked connections from input channels to output channels.
Default = 1
bias (bool, optional) –
If True, adds a learnable bias to the output.
Default = True
padding_mode (str, optional) –
Any of ‘zeros’, ‘reflect’, ‘replicate’ or ‘circular’.
Default = ‘zeros’
device (torch.device or str, optional) – The torch device.
dtype (torch.dtype, optional) – Layer dtype, i.e., the data type of the torch.Tensor defining the layer weights.
max_norm (float, optional) –
The maximum norm each hidden unit can have. If None no constraint will be added.
Default = 2.0
min_norm (float, optional) –
The minimum norm each hidden unit can have. Must be a float lower than max_norm. If given, MinMaxNorm will be applied in the case max_norm is also given. Otherwise, it will be ignored.
Default = None
axis_norm (Union[int, list, tuple], optional) –
The axis along weights are constrained. It behaves like Keras. So, considering that a Conv2D layer has shape (output_depth, input_depth, rows, cols), set axis to [1, 2, 3] will constrain the weights of each filter tensor of size (input_depth, rows, cols).
Default = [1,2,3]
minmax_rate (float, optional) –
A constraint for MinMaxNorm setting how weights will be rescaled at each step. It behaves like Keras rate argument of MinMaxNorm contraint. So, using minmax_rate = 1 will set a strict enforcement of the constraint, while rate<1.0 will slowly rescale layer’s hidden units at each step.
Default = 1.0
Note
To Apply a MaxNorm constraint, set only max_norm. To apply a MinMaxNorm constraint, set both min_norm and max_norm. To apply a UnitNorm constraint, set both min_norm and max_norm to 1.0.
Example
>>> from import selfeeg.models import ConstrainedConv2d >>> import torch >>> x = torch.randn(4, 1, 8, 64) >>> mdl = ConstrainedConv2d(1, 4, (1, 15), max_norm = 0.5) >>> mdl.weight = torch.nn.Parameter(mdl.weight*10) >>> out = mdl(x) >>> norms = torch.sqrt(torch.sum(torch.square(mdl.weight), axis=[1,2,3])) >>> print(out.shape) # shoud return torch.Size([4, 2, 8, 64]) >>> print(torch.isnan(out).sum()) # shoud return 0 >>> print(torch.sum(norms>(0.5+1e-3)).item() == 0) # should return True