Custom Activations

Implementation of new stochastic activations. At present, there is only one activation functions but it will be easy in the future to add more possibility.

There are often used with the last layer of a network. For example, the MeanVarianceActivation is especially used with the Dense2Dto3D or Dense3Dto4D layers :

>>> inputs = Input(shape=(input_dim,))
>>> x = Dense2Dto3D(100, activation="relu")(inputs)
>>> x = Dense3Dto3D(100, activation="relu")(x)
>>> outputs = Dense3Dto4D(4, 2, activation=MeanVarianceActivation)(x)
>>> model = Model(inputs=inputs, outputs=outputs)

Here is the list of the new activations :

Tip

In order to specify the parameters of a custom activation, you can use the following syntaxes :

  • model.add(Activation(lambda x: MeanVarianceActivation(x, activation="relu")))

  • Dense3Dto4D(4, 2, activation=lambda x: MeanVarianceActivation(x, activation="relu"))

MeanVarianceActivation

purestochastic.model.activations.MeanVarianceActivation(x, activation=None)[source]

An activation function used for the mean and variance pair.

This activation function supposes that the last dimension of x is of length 2 with the first part for the mean and the second part for the variance :

  • \(\hat{\mu} = x[ : , \cdots , : , 0]\)

  • \(\hat{\sigma}^2 = x[ : , \cdots , : , 1]\)

It applies the activation given in argument for the mean and the exponential activation function for the variance so that it’s positive.

It’s especially used when it’s necessary to output the mean and the variance of a gaussian distribution at the end of the network like in the Mean Variance Estimation Method 1 2 .

Parameters

  • x (ndarray or Tensor) – Input tensors.

  • activation (func or str, optional) – Activation function to use. If you don’t specify anything, no activation is applied (ie. “linear” activation: a(x) = x).

Return type

A new tensor to which the activation function has been applied.

References

1

Abbas Khosravi et al. « Comprehensive Review of Neural Network-Based Prediction Intervals and New Advances ». In : IEEE Transactions on Neural Networks 22.9 (2011), p. 1341-1356. doi : 10.1109/TNN.2011.2162110.

2

D.A. Nix et A.S. Weigend. « Estimating the mean and variance of the target probability distribution ». In : Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN94). T. 1. 1994, 55-60 vol.1. doi : 10.1109/ICNN.1994.374138.