Regularizers

Implementation of new regularizers. The package is also compatible with all the regularizer functions from tensorflow as well as custom regularizer functions. There are two main ways to use loss functions in tensorflow :

  1. Use the string identifiant :

    >>> dense = tf.keras.layers.Dense(3, kernel_regularizer='orthonormality')

  2. Use the class to specify parameters

    >>> dense = tf.keras.layers.Dense(3, kernel_regularizer=Orthonormality(lambda_coeff=0.01))

Here is the list of the new regularizers :

Orthonormality

purestochastic.common.regularizers.Orthonormality(miso=True, lambda_coeff=1)[source]

A regularizer that applies a Orthonormality penalty 1 to 2D and 3D kernel. The penalty is computed as :

\[\text{loss} = \text{lambda_coeff} \frac{1}{n} \sum_{i=1}^{K} \sum_{j=1}^{K} ((C^TC)_{i,j} - I_{i,j})\]

with C, the kernel matrix. If C is a 3D tensor the operation is the mean over the new dimension.

The penalty forces the kernel to be an orthonormal matrix.

Parameters

  • miso (boolean (default: True)) – Tell the regularizer if the kernel is a 2D or 3D matrix.

  • lambda_coeff (float (default : 1)) – The regularization factor.

References

1

Tagasovska, Natasa and Lopez-Paz, David. « Single-model uncertainties for deep learning ». In : Advances in Neural Information Processing Systems 2019.Nips (2019), p. 1-12. issn : 10495258. arXiv : 1811.00908.