Custom Layers

Dense layer in Tensorflow uses a 2D kernel of shape (n_input, n_output). This module adds new Dense layers with 2D, 3D and 4D kernel.

The names of the new Dense layers are defined as Dense<Input_dim>to<Output_dim> with the batch size dimension included. For example, the layer Dense2Dto3D takes as input a 2D tensor of shape (batch_size, n_input) and outputs a tensor of shape (batch_size, units_dim1, units_dim2).

The new layers can be used as usual tensorflow layers. They are useful when the model outputs parameters of a distribution. For instance, if the model predicts the mean and the variance of a gaussian distribution for 4 variables, it is interesting to have an output shape equal to (batch_size, 4, 2). It is then possible using this piece of code :

>>> inputs = Input(shape=(input_dim,))
>>> x = Dense(100, activation="relu")(inputs)
>>> outputs = Dense2Dto3D(4, 2, activation=MeanVarianceActivation)(x)
>>> model = Model(inputs=inputs, outputs=outputs)
>>> model.output_shape
(None, 4, 2)

Here is the list of the new layers :

A detailed presentation of each layer is available below along with an image describing the operations performed by each layer.

Dense2Dto3D

class purestochastic.model.layers.Dense2Dto3D(*args, **kwargs)[source]

An adaptation of the densely-connected NN layer that outputs a 3D tensor from a 2D tensor.

Dense2Dto3D is a change of the Dense layer when the kernel is a tensor of order 3. It implements the dot product between the inputs and the kernel along the last axis of the inputs and axis 0 of the kernel :

>>> output = activation(tensordot(input, kernel) + bias).

It’s like having a Dense layer with units_dim1*units_dim2 units followed by a Reshape layer with a target shape of (units_dim1, units_dim2).

Parameters

  • units_dim1 (int) – Dimensionality of the first dimension of the output space.

  • units_dim2 (int) – Dimensionality of the second dimension of the output space.

  • activation (func or str, default: None) – Activation function to use. If you don’t specify anything, no activation is applied (ie. “linear” activation: a(x) = x).

  • use_bias (boolean, default:True) – Indicates whether the layer uses a bias matrix.

  • kernel_initializer (str or dict or func, default:’glorot_uniform’) – Initializer for the kernel weights tensor.

  • bias_initializer (str or dict or func, default:’zeros’) – Initializer for the bias matrix.

  • kernel_regularizer (str or dict or func, optional) – Regularizer function applied to the kernel weights tensor.

  • bias_regularizer (str or dict or func, optional) – Regularizer function applied to the bias matrix.

  • activity_regularizer (str or dict or func, optional) – Regularizer function applied to the output of the layer (its “activation”).

  • kernel_constraint (str or dict or func, optional) – Constraint function applied to the kernel weights tensor.

  • bias_constraint (str or dict or func, optional) – Constraint function applied to the bias matrix.

Input shape

2D tensor with shape: (batch_size, input_dim).

Output shape

3D tensor with shape: (batch_size, units_dim1, units_dim2).

The following figure represents the linear operation performed by the layer Dense2Dto3D. If activation is specified, the activation function is applied to the output of the linear operation described below.

../_images/Dense2Dto3D.drawio.svg

Dense3Dto3D

class purestochastic.model.layers.Dense3Dto3D(*args, **kwargs)[source]

An adaptation of the densely-connected NN layer that outputs a 3D tensor from a 3D tensor.

Dense3Dto3D is a change of the Dense2Dto3D layer when the input is a tensor of order 3. It implements the dot product between the inputs and the kernel along the last axis of the inputs and axis 1 of the kernel for each element in axis 1 of inputs and axis 0 of kernel :

>>> for d in range(nb_dense):
>>>     output[:,d,:] = activation(tensordot(input[:,d,:], kernel[d,:,:]) + bias[d,:])

It’s like having several Dense layers that have different inputs and which function independently.

Parameters

  • units (int) – Dimensionality of the second dimension of the output space.

  • activation (func or str, default:None) – Activation function to use. If you don’t specify anything, no activation is applied (ie. “linear” activation: a(x) = x).

  • use_bias (boolean, default:True) – Indicates whether the layer uses a bias matrix.

  • kernel_initializer (str or dict or func, default:’glorot_uniform’) – Initializer for the kernel weights tensor.

  • bias_initializer (str or dict or func, default:’zeros’) – Initializer for the bias matrix.

  • kernel_regularizer (str or dict or func, optional) – Regularizer function applied to the kernel weights tensor.

  • bias_regularizer (str or dict or func, optional) – Regularizer function applied to the bias matrix.

  • activity_regularizer (str or dict or func, optional) – Regularizer function applied to the output of the layer (its “activation”).

  • kernel_constraint (str or dict or func, optional) – Constraint function applied to the kernel weights tensor.

  • bias_constraint (str or dict or func, optional) – Constraint function applied to the bias matrix.

Input shape

3D tensor with shape: [batch_size, nb_dense, input_dim].

Output shape

3D tensor with shape: [batch_size, nb_dense, units].

Dense3Dto2D

class purestochastic.model.layers.Dense3Dto2D(*args, **kwargs)[source]

An adaptation of the densely-connected NN layer that outputs a 2D tensor from a 3D tensor.

Dense3Dto2D is the inverse of the Dense2Dto3D layer. It implements the dot product between the inputs and the kernel along the two last axis of the inputs and the two first axis of the kernel so that the inputs is projected in a 2D space

>>> output = activation(tensordot(input, kernel, axes=[[-2,-1], [0, 1]]) + bias).

It’s like having Reshape layer with a target shape of (input_dim1*input_dim2) followed by a Dense layer with units units.

Parameters

  • units (int) – Dimensionality of the dimension of the output space.

  • activation (func or str, default:None) – Activation function to use. If you don’t specify anything, no activation is applied (ie. “linear” activation: a(x) = x).

  • use_bias (boolean, default:True) – Indicates whether the layer uses a bias vector.

  • kernel_initializer (str or dict or func, default:’glorot_uniform’) – Initializer for the kernel weights tensor.

  • bias_initializer (str or dict or func, default:’zeros’) – Initializer for the bias vector.

  • kernel_regularizer (str or dict or func, optional) – Regularizer function applied to the kernel weights tensor.

  • bias_regularizer (str or dict or func, optional) – Regularizer function applied to the bias vector.

  • activity_regularizer (str or dict or func, optional) – Regularizer function applied to the output of the layer (its “activation”).

  • kernel_constraint (str or dict or func, optional) – Constraint function applied to the kernel weights tensor.

  • bias_constraint (str or dict or func, optional) – Constraint function applied to the bias vector.

Input shape

3D tensor with shape: (batch_size, input_dim1, input_dim2).

Output shape

2D tensor with shape: (batch_size, units).

Dense3Dto4D

class purestochastic.model.layers.Dense3Dto4D(*args, **kwargs)[source]

An adaptation of the densely-connected NN layer that outputs a 4D tensor from a 3D tensor.

Dense3Dto4D is the same adaptation from Dense to Dense2Dto3D layer but from the layer Dense3Dto3D this time with a kernel of order 4. It implements the dot product between the inputs and the kernel along the last axis of the inputs and axis 1 of the kernel for each element in axis 1 of inputs and axis 0 of kernel :

>>> for d in range(nb_dense):
>>>     output[:,d,:,:] = activation(tensordot(input[:,d,:,:], kernel[d,:,:,:]) + bias[d,:,:])

It’s like having several Dense and Reshape layers that have different inputs and which function independently with units_dim1*units_dim2 units followed by a Reshape layer with a target shape of (units_dim1, units_dim2).

Parameters

  • units_dim1 (int) – Dimensionality of the first dimension of the output space.

  • units_dim2 (int) – Dimensionality of the second dimension of the output space.

  • activation (func or str, default:None) – Activation function to use. If you don’t specify anything, no activation is applied (ie. “linear” activation: a(x) = x).

  • use_bias (boolean, default:True) – Indicates whether the layer uses a bias tensor.

  • kernel_initializer (str or dict or func, default:’glorot_uniform’) – Initializer for the kernel weights tensor.

  • bias_initializer (str or dict or func, default:’zeros’) – Initializer for the bias tensor.

  • kernel_regularizer (str or dict or func, default:None) – Regularizer function applied to the kernel weights tensor.

  • bias_regularizer (str or dict or func, default:None) – Regularizer function applied to the bias tensor.

  • activity_regularizer (str or dict or func, default:None) – Regularizer function applied to the output of the layer (its “activation”).

  • kernel_constraint (str or dict or func, default:None) – Constraint function applied to the kernel weights tensor.

  • bias_constraint (str or dict or func, default:None) – Constraint function applied to the bias tensor.

Input shape

3D tensor with shape: (batch_size, nb_dense, input_dim).

Output shape

4D tensor with shape: (batch_size, nb_dense, units_dim1, units_dim2).