Validation of a system prediction model consists of comparing the predictions of the model with observed values of the actual system to determine its predictive capacity by means of some method. In this stage of the mathematical modeling process, accuracy and precision of the model are assessed. Accuracy refers to the proximity of the predictions (z) to observed values (y), for example, their differences (d=y-z) from zero. Precision refers to dispersion of the points (z,y). However, in the presence of accuracy, precision is measured by quantifying the dispersion of the points with respect to a reference, for example, the deterministic line y=z, or by evaluating the variances of the differences ( σ D 2 ) around zero ( μ D = 0 ).

In the literature, different techniques for validating models designed for purposes of prediction have been proposed; see Tedeschi (1): linear regression analysis, fitted error analysis, concordance correlation coefficient, diverse measures of deviation, mean square error of prediction, non-parametric analyses, and comparing the distribution of observed and predicted data. Medina-Peralta et al (2) point out that some deviation measures to validate models contradict graphic methods when the model predictions are biased. They recommend the joint use of deviation measurement and graphic methods for model validation.

Among the statistical inference techniques dealing with validation of models with no bias (NB), and with constant (CB), or proportional bias (PB) in their predictions, the procedure given by Freese (3) stands out. This method consists in determining whether accuracy and precision of a prediction model, or technique meets the requirements of the model developer or user. Medina et al (4) extended the Freese method when the model has CB in its predictions: hypothesis tests and maximum anticipated error for the alternative proposal, and the confidence interval for a quantile of the error distribution.

This paper presents extensions to the statistical method for model validation proposed by Freese (3), Rennie and Wiant (5), Reynolds (6), Barrales et al (7), and Medina et al (4) when the model has PB in its predictions: 1) maximum anticipated error for the original proposal, 2) hypothesis testing, and maximum anticipated error for the alternative proposal, and 3) the interval of confidence for a quantile of error distribution. The method is illustrated with published data of Barrales et al (7) corresponding to a model that simulates grassland growth.

Basic concepts. Assume that we have “n” pairs to compare (y_i, z_i) i=1,2,…,n, where for the ith pair, yi is the observed value, z_i the corresponding predicted value for the deterministic model to be validated, and d_i=y_i–z_i the difference between the two values. In the development of extensions of the method, the observed values are considered as realizations of random variables Y_i , and the predicted values are deterministic, thus D_i=Y_i-z_i.

In Freese’s (3) approach applied to determining the required accuracy and precision, the “e” values (maximum admitted error of the deviations |y_i–z_i|=|d_i|) and α (1-α represents the required certainty) specified by the model developer or user, it is needed that D be normally distributed with zero mean and P(|D|≤e)≥1-α, in order to accept the model and consider it as sufficiently reliable for system prediction. Therefore, a model is exact, or with no bias (NB), when the differences (d_i) fit a normal distribution with mean zero.

Medina, Vargas-Villamil (4) indicate that CB is recognized by an average value of the differences ( d − ) distant from zero and that the graph of the points (z_i, d_i=y_i–z_i) forms a horizontal band centered at with systematic distribution either positive or negative (points above and below the line d − d − ). When the samples are related, a t-test applied to determine if the mean of the differences is significantly different from zero, would prove the presence of CB in the model predictions.

Determination of proportional bias. PB is recognized in the graph of points (z_i, d_i=y_i–z_i) whenever a positive or negative linear trend is found; the magnitude of the bias (d_i=y_i-z_i) increases or decreases in direct proportion to the predicted values (z_i). A simple linear regression analysis of the bias vs predicted (Di=β₀+β1z_i+ε_i) contributes to the detection of PB more objectively (2,8).

Validation of models with proportional bias. Given that we have PB, the points (z_i, D_i) i=1,2,…,n are related by E(D_i)=β₀+β₁z_i and Var(D_i)=σ²D=σ² with Di~NI(β₀+β₁z_i,σ²). That is, Di=β₀+β₁z_i+ε_i, where E(ε_i)=0, Var(ε_i)=σ² and ε_i=Di-(β₀+β₁z_i)=[D_i-E(D_i)]~NI(0,σ²) and thus, the model will be accurate when carrying out the correction ε_i=D_i-(β₀+β₁z_i). In practice, a regression model is fit and the errors (ε_i): are estimated, where a and b are the least squares estimates of the intercept (β₀) and the slope (β₁), respectively. Thus, we would need only a statistical test for the required precision P(|D|≤e)≥1-α, which is translated to σ ≤ e 2 / x 1 , 1 - α 2 , once PB is removed by the referred correction.

If ε~N(0,σ2), then. P ε 2 σ 2 ≤ x 1 , 1 - α 2 = P x 1 2 ≤ x 1 , 1 - α 2 = 1 - α

Moreover, for e and α satisfying P(|ε|≤e)≥1-α it follows that P ε 2 σ 2 ≤ e 2 σ 2 ≥ 1 - α ; thus, e 2 σ 2 ≥ x 1 , 1 - α 2 and σ 2 ≤ e 2 x 1 , 1 - α 2 where, in general, x κ , y 2 represents the quantile γ of the chi-square distribution with “k” degrees of freedom ( x k 2 ); that is, P x κ 2 ≤ x κ , y 2 = γ .

The indicated correction results in an accurate model, and only the statistical test for precision is needed. Thus, the following step is to test the hypotheses with the original proposal (OP) H 0 : σ D 2 ≤ e 2 / x 1 , 1 - α 2 vs H 1 : σ D 2 > e 2 / x 1 , 1 - α 2 (3) or the alternative proposal (AP) H 0 A P : σ D 2 > e 2 / x 1 , 1 - α 2 vs H 1 A P : σ D 2 ≤ e 2 / x 1 , 1 - α 2 (8). The test statistic with ε~ N I 0 σ 2 = σ D 2 and under true null hypotheses H₀ or H₀ ^AP is: V = x 1 , 1 - α 2 e 2 n - 2 (CME)_D~ x n - 2 2 . H₀ or H₀ ^AP is rejected with a significance level α’ if V c > x n - 2 , 1 - α ' 2 or V c ≤ x n - 2 , α ' 2 , where V_c corresponds to the calculated value of the test statistic (V), and (CME)_D to the mean square of the error for the estimated model d_i=a+bz_i Therefore, if H₀ is not rejected or is rejected, then the model is considered acceptable for prediction under OP and AP, respectively.

Another approach for evaluating precision once PB is corrected, is to use confidence intervals (CI), similar to the procedure presented by Medina et al (4) when the model has CB in its predictions. This approach is motivated by Freese’s proposal: different users of the model may have different requirements of precision, leading to different values of the maximum admitted error of the deviations (e).

From the rejection region for H_0' V c = ( x 1 , 1 - α 2 / e 2 ) ( n - 2 ) ( C M E ) D > x n - 2 , 1 - α ' 2 , and solving for “e”, the critical error is found (8): e < ( x 1 , 1 - α 2 ( n - 2 ) ( C M E ) D / x n - 2 , 1 - α ' 2 ) 1 / 2 = e S P * )

Therefore,H₀ will be rejected if e < e P B * , and it will not be rejected if e ≥ e P B * ; that is, if the model user specifies a value for “e” such that e ≥ e P B * , then the model is considered acceptable in the prediction of the system under OP.

Using an analogous procedure for AP, the maximum anticipated error, or critical error was obtained (8): e P B * * = ( x 1 , 1 - α 2 ( n - 2 ) ( C M E ) D x n - 2 , α ' 2 ) 1 2

Thus, if the model user specifies a value of “e” such that e ≥ e P B * * , then the model is considered acceptable in predicting the system under AP. In a previous study (9), it can be checked that hypothesis tests and confidence intervals are related; this relationship enables to construct one from the other.

Other published works from Reynolds (6) and Barrales et al (7) do not present the confidence interval (CI) for ε P B = ( σ 2 x 1 , 1 - α 2 ) 1 2 , the quantile 1-α of the distribution of |ε| (in practice, the errors (e) once PB is corrected) or equivalently ε P B 2 , is the quantile 1-α of the distribution of ε² since, if ε~N(0,σ²), then.

1 - α - P ( ε 2 σ 2 ≤ x 1 , 1 - α 2 ) - P ( ε 2 ≤ σ 2 x 1 , 1 - α 2 ) = P ( ε 2 ≤ ε P B 2 )

A (1-α’)100% CI for ε_PB (8) based on CI (1-α’)100% for σ² (where τ σ 2 = ( σ 2 x 1 , 1 - α 2 ) 1 2 = ε P B is an increasing monotonic function) is given by

τ ( ( n - 2 ) ( C M E ) D x n - 2 , 1 α ' 2 2 ) < τ σ 2 < τ ( ( n - 2 ) ( C M E ) D x n - 2 , α ' 2 2 )

( x 1 , 1 - α 2 ( n - 2 ) ( C M E ) D x n - 2 , 1 - α ' 2 2 ) 1 2 < ε P B < ( x 1 - 1 - α ( n - 2 ) ( C M E ) D 2 x n - 2 , α ' 2 2 ) 1 2

e I P B * < ε P B < e I P B * * where e I P B * and e I P B * * correspond to the critical errors e P B * and e P B * * , with the difference that α’ is substituted by α’/2. The estimated CI ( e I P B * , e I P B * * ) means that we have a confidence of 100(1-α’)% that the value of the distribution for |ε|=|D_i-(β₀+β₁z_i)| having below the 100(1-α) percent of the absolute errors is located in some location of the referred interval. This enables us to determine with certain probability an upper bound for the magnitude of the prediction error (e), and to use it in the evaluation of the evolution of the model for system prediction.

Figure 1

To illustrate the application of the methodology, the CI approach (4) was used with a published data set from Barrales et al (7), which corresponds to a model of grassland growth simulation. In this example, we applied the method of calculating the maximum anticipated errors or critical errors of the OP and AP, when the model has PB. In addition, 95% CI was calculated for, the 1-α quantile of the error (e) distribution, once PB is corrected with d_i-(α+bz_i).

The distribution of the points in Figure 1 indicates that the model has PB in its predictions; the magnitude of the bias (d_i=y_i-z_i) decreases directly with the predicted values (z_i) and the linear relationship was significant (d=912.838-0.396z: F=13.665, p=0.001). With α=α’=0.05, the critical errors were e P B * = 6 9 7 . 3 1 1 k g h a - 1 and e P B * * = 1 1 5 6 . 2 7 3 k g h a - 1 . Therefore, if the modeler or the model user specifies a value “e” such that e ≥ e P B * or e ≥ e P B * * , then the model is considered sufficiently reliable in predicting the system based on OP and AP, respectively. For example, if e=1157 kg ha^-1, the model is considered acceptable for predicting with both proposals (OP: e>697.311 kg ha^-1; AP: e>1156.273 kg ha^-1).

The 1-α’=95% CI for ε_PB is 669.689 kg ha^-1 <ε_PB<1225.564 kg ha^-1, and thus, we are 95% confident that the distribution value |ε|=|Di-(β₀+β₁z_i)| found below 1-α=95% of the absolute errors is located in some part of the interval (669.689 kg ha^-1 – 1225.564 kg ha^-1); that is, there is a 95% of probability that the magnitude of the prediction error does not surpass 1225.564 kg ha^-1.

With 95% of confidence, the critical error with AP and CI for ε_PB would have to allow 1156.273 kg ha^-1 as the minimum prediction error, which should not surpass 1225.564 kg ha^-1. In this way, the validated model could be used to predict grassland growth. However, it would require a fit in its structure based on the presence PB in its predictions.

Identifying the type of bias may allow to improve the structure of the model and its evaluation, the data and methods used in all the processes of its construction, and validation (4). McCarthy et al have pointed out that testing a model helps to identify its weaknesses so that its predictive performance can be improved (10), since identification and acceptance of inaccuracies of a model is a step towards the evolution of a more accurate, more reliable model (1).

The extensions we present for validating deterministic model possessing PB in their predictions are applied with no modifications in the model structure because this type of bias is removed from the available data (z_i, y_i) by means of the bias values (d_i), namely, by the correction d_i-(a+bz_i).

The test statistic and critical error with AP require a larger maximum admissible error (e) value than that with OP, in order to infer whether the model is acceptable for predicting the system or not. A previous study by Medina et al (4) points out that the inconvenience of applying the OP is the ambiguity occurring when the null hypothesis is not rejected (H₀) since what it can be inferred is that the data do not provide sufficient evidence to reject it, and that the affirmation stated in H₀ is not accepted. Moreover, as the research hypothesis is posed in the alternative hypothesis, we recommend using the AP in validating a system prediction model.

Validation by means of critical errors or by the confidence limits approach (the latter being a procedure equivalent to the hypothesis testing approach, as mentioned above), is reduced to calculating the maximum anticipated error, or critical error, in which the model developer or the user decide whether the model is acceptable in predicting the system. This is carried out by comparing the critical error with the required accuracy (e) under the values α and α’ specified beforehand. This requires that the model developer or the user have good understanding of the system to establish, a priori, the maximum admissible error (e). Medina et al (4) recommend the CI approach to validate a model for several reasons. It provides the range of possible values of the parameter under consideration thus, it is more informative than hypothesis testing in the sense that it permits determining the a posteriori maximum admissible prediction error.

It should be pointed out that the critical error, or a posteriori maximum admissible error, can be used to compare several models for a single system, in such a way that the best model would be the one with the lowest critical error. To assess the improvement of a model in system prediction it is necessary to observe whether a posteriori maximum admissible error decreases at each process of improvement.

The estimated CI for the quantile 1-α of the error distribution, once PB is corrected, allows us to determine an upper bound for the magnitude of the prediction error with a certain probability, and to use it in the evaluation of the evolving model whenever an improvement is required. This is similar to the case of validating a model with CB in its predictions (4). Thus, based on the critical error with AP and the aforementioned CI, we can establish the minimum permitted prediction error, and what amount of error of prediction would not surpass.

In conclusion, the extensions to Freese’s statistical method presented here to validate models in the presence of proportional bias are applied without modification in the model structure.

In validating a model, we recommend using the confidence interval approach under the alternative proposal.

The confidence interval for the 1-α quantile of error distribution, once the proportional bias is corrected, allows determination of an upper bound for the magnitude of the prediction error.

Both a posteriori maximum admissible error, and the upper bound for the magnitude of the prediction error can be used to evaluate the evolution of a model in predicting the system in the face of a modification of the modeling process.

We thank the Subsecretaría de Educación Superior e Investigación Científica, Programa de Mejoramiento del Profesorado (PROMEP) Secretaría de Educación Pública of México for funding this study.