Metrics

Implementation of new stochastic metrics. The metrics assume that the predictions are the parameters of a gaussian distribution. These metrics have to be specified with the arguments stochastic_metrics or stochastic_weighted_metrics when calling the method compile in a subclass of StochasticModel :

>>> from src.common.metrics import PredictionIntervalCoverageProbability
>>> from src.model.base_uncertainty_models import StochasticModel
>>> issubclass(model, StochasticModel)
True
>>> model.compile(stochastic_metrics=[PredictionIntervalCoverageProbability()])

The metrics PICP, PINAW and CWC suppose predictions given in update_state are different according the argument input_type :

  1. If input_type="gaussian", the predictions need to be the means and the variances of a gaussian distribution defined as :

  • \(\hat{\mu} = \text{predictions}[ : , \cdots , : , 0]\)

  • \(\hat{\sigma}^2 = \text{predictions}[ : , \cdots , : , 1]\)

  1. If input_type="pi", the predictions need to be the lower and upper bounds of the predictions defined as :

  • \(\hat{y}_{lower} = \text{predictions}[ : , \cdots , : , 0]\)

  • \(\hat{y}_{upper} = \text{predictions}[ : , \cdots , : , 1]\)

Here is the list of the new metrics :

PICP

class purestochastic.common.metrics.PredictionIntervalCoverageProbability(*args, **kwargs)[source]

Prediction Interval Coverage Probability metric.

A prediction interval \([\hat{\underline{y_i}}, \hat{\bar{y_i}}]\) is constructed so that with probability p, \(y_i\) is included in the interval. The PICP 1 (Prediction Interval Coverage Probability) aims at computing the true percentage of values included in the interval. Mathematically, it is defined as :

\[\begin{split}PICP=\frac{1}{n} \displaystyle\sum_{i=1}^{n} c_j ~~\text{ avec } c_j = \left\{ \begin{array}{ll} 1 &\text{ si } y_i\in [\underline{\hat{y}_i} ,\overline{\hat{y}_i}]\\ 0 &\text{ si } y_i \not \in [\underline{\hat{y}_i} ,\overline{\hat{y}_i}] \end{array} \right.\end{split}\]

The best PICP is a percentage that is equal to p. If y is not a 1d array, the batch dimension needs to be the first and the output value is the mean over all other dimensions.

Parameters

  • name (str, default: ‘picp’) – String name of the metric instance.

  • p (float, default: 0.95) – Probability of values included in the interval associated with the predicted parameters of the gaussian distribution.

  • input_type ({“gaussian”, “pi”}, default: “gaussian”) – Type of input to the metric. If “gaussian”, the metric assumes the predictions are given in the mean and the variance of a gaussian distribution. If “pi”, the metric assumes the predictions are given in the lower and upper bound of the prediction interval.

References

1

Abbas Khosravi, Saeid Nahavandi et Doug Creighton. « A prediction interval-based ap- proach to determine optimal structures of neural network metamodels ». In : Expert Syst. Appl. 37 (mars 2010), p. 2377-2387. doi : 10.1016/j.eswa.2009.07.059.

update_state(y_true, predictions, sample_weight=None)[source]

Accumulates picp statistics.

Parameters

  • y_true (shape= [batch size, d_0, ...,  d_N]) – The ground truth values.

  • predictions (shape= [batch size, d_0, ...,  d_N, 2]) – The predicted values.

  • sample_weight (optional) – Optional weighting of each example.

Return type

Update op.

PINAW

class purestochastic.common.metrics.PredictionIntervalNormalizedAverageWidth(*args, **kwargs)[source]

Prediction Interval Normalized Average Width metric.

The PINAW 2 (Prediction Interval Normalized Average Width) computes the average width of prediction intervals (\([\hat{\underline{y_i}}, \hat{\bar{y_i}}]\)) normalized by a the distance between the maximum and minimum value of y. Mathematically, it is defined as :

\[PINAW=\frac{1}{R*n}\displaystyle\sum_{i=1}^n[\overline{\hat{y}_i}- \underline{\hat{y}_i}]\]

avec \(R=\max(y)-\min(y)\).

The best PINAW is the minimum value. If y is not a 1d array, the batch dimension needs to be the first and the output value is the mean over all other dimensions.

Parameters

  • name (str, default: ‘pinaw’) – String name of the metric instance.

  • p (float, default: 0.95) – Probability of values included in the interval associated with the predicted parameters of the gaussian distribution.

  • input_type ({“gaussian”, “pi”}, default: “gaussian”) – Type of input to the metric. If “gaussian”, the metric assumes the predictions are given in the mean and the variance of a gaussian distribution. If “pi”, the metric assumes the predictions are given in the lower and upper bound of the prediction interval.

References

2

Abbas Khosravi et al. « Comprehensive Review of Neural Network-Based Prediction Intervals and New Advances ». In : IEEE Transactions on Neural Networks 22.9 (2011), p. 1341-1356. doi : 10.1109/TNN.2011.2162110.

update_state(y_true, predictions, sample_weight=None)[source]

Accumulates pinaw statistics.

Parameters

  • y_true (shape= [batch size, d_0, ...,  d_N]) – The ground truth values.

  • predictions (shape= [batch size, d_0, ...,  d_N, 2]) – The predicted values.

  • sample_weight (optional) – Optional weighting of each example.

Return type

Update op.

CWC

class purestochastic.common.metrics.CoverageWidthBasedCriterion(*args, **kwargs)[source]

Coverage Width Based Criterion metric.

The CWC 3 (Coverage Width Based Criterion) was defined to find a trade-off between the PICP and the PINAW. Mathematically, it is defined as :

\[\begin{split}CWC = PINAW(1+\gamma(PICP)e^{\eta(\mu-PICP)} ) ~~ \text{ avec } \gamma(PICP) = \left\{ \begin{array}{ll} 1 &\text{ si } ~~ PICP < \mu\\ 0 &\text{ si }~~ PICP \geq \mu \end{array} \right.\end{split}\]

The operation can be explained in two parts :

  • If the PICP is higher than p, the CWC is equaled to the PINAW metric so that the size of the prediction interval decreases.

  • If the PICP is lower than p, the exponential term becomes very large and the most influent criterion is the PICP that the size of the prediction interval increases.

The best CWC is the minimum value. If y is not a 1d array, the batch dimension needs to be the first and the output value is the mean over all other dimensions.

Parameters

  • name (str, default: ‘cwc’) – String name of the metric instance.

  • p (float, default: 0.95) – Probability of values included in the interval associated with the predicted parameters of the gaussian distribution.

  • eta (positive float, default: 50) – Penalty value when the PICP is lower than p.

  • input_type ({“gaussian”, “pi”}, default: “gaussian”) – Type of input to the metric. If “gaussian”, the metric assumes the predictions are given in the mean and the variance of a gaussian distribution. If “pi”, the metric assumes the predictions are given in the lower and upper bound of the prediction interval.

Warning

If input_type="pi", it’s important that the p specified in the constructor is the same as the one used to construct the prediction interval. Otherwise, the CWC will be wrong.

References

3

Abbas Khosravi et al. « Comprehensive Review of Neural Network-Based Prediction Intervals and New Advances ». In : IEEE Transactions on Neural Networks 22.9 (2011), p. 1341-1356. doi : 10.1109/TNN.2011.2162110.

update_state(y_true, predictions, sample_weight=None)[source]

Accumulates cwc statistics.

Parameters

  • y_true (shape= [batch size, d_0, ... ,  d_N]) – The ground truth values.

  • predictions (shape= [batch size, d_0, ... ,  d_N, 2]) – The predicted values.

  • sample_weight (optional) – Optional weighting of each example.