Scheme of uncertainty and sensitivity analysis performed with LHS and PRCC methods. The mathematical model is represented as an ordinary differential equations system, where x is the vector of state variables in a n-dimensional space ¡ ⁿ (n=2 in this example and θ is the parameter vector in ¡ ^k (k=3 in this example). For ease of notation, the output y is unidimensional and it is a function of x and θ.
Panel A: mathematical model specification (dynamical system, parameters, output) and the corresponding LHS scheme. Probability density functions (pdfs) are assigned to the parameters of the model (e.g. a, b, c). We show an example with sample size N equal to 5. Each interval is divided into 5 equiprobable subintervals, and independent samples are drawn from each pdf (uniform and normal). The subscript represents the sampling sequence. Panel B: the LHS matrix (X) is then built by assembling the samples from each pdf. Each row of the LHS matrix represents a unique combination of parameter values sampled without replacement. The model x = g(x, θ) is then solved, the corresponding output generated, and stored in the matrix Y. Each matrix is then rank-transformed (X_R and Y_R).
Panel C: The LHS matrix (X) and the output matrix (Y) are used to calculate Pearson correlation coefficient (CC_Pearson). The rank-transformed LHS matrix (X_R) and output matrix (Y_R) are used to calculate the Spearman or rank correlation coefficient (RCC) and the Partial Rank Correlation Coefficient (PRCC) (see section 3.1)

general LHS scheme and PRCC performed on the Lotka-Volterra model (model equations and parameters are as described in Section 3.1).
Panel A: Cumulative Distribution Functions – CDFs of 1000 samples independently drawn from normal pdfs (initialized following ) for the four parameters (α, β, σ, δ) of the Lotka-Volterra model described in Section 3.1, -.
Panel B: outputs of the Lotka-Volterra model over time (days) corresponding to the parameter combinations of the LHS scheme illustrated in Panel A.
Panel C: Example of sampling-based correlation indexes calculated on the LHS matrix resulting from Panel A and from the output matrix resulting from Panel B. The reference output is the variable Q(t)-prey at time t=9 and it is shown on the ordinate. Parameter σ is taken as the parameter of interest and it is represented on the abscissa. Each dot represents the output value Q(9) for a specific sampled value of parameter σ.
Note that at each step of processing (correlation, rank correlation, partial rank correlation), the linear relationship between parameter and output variations becomes more apparent.

eFAST performed on the Lotka-Volterra model. Model equations and parameters are as described in Section 3.1. Panel A: Input Parameter Sampling: Parameters are varied according to a sinusoidal function based on run number. Note that the shapes of these search curves result in normal distribution of parameter values (see for details) when sampled (horizontal histograms). σ is assigned a frequency of 31, and β is assigned a frequency of 2. α and δ are sampled at frequencies 1 and 3, respectively (not shown). These frequencies are chosen automatically to meet the criteria described in Appendix A.1. Panel B: Model Output: The model is solved for each parameter combination from Panel A. The number of prey at t=9 is taken as the model output. Note that both high- and low-frequency components are evident by inspection. Panel C: Fourier analysis and Variance Spectrum: The variance at each frequency is calculated from the model ouput (see text) and normalized to total variance. The variance at frequency 2, 31, and higher harmonics of frequency 31 are indicated (arrows). Panel D: Sensitivity Indexes: Taking parameter σ (varied at frequency 31) as our parameter of interest, first-order sensitivity index (S_i) is calculated by the sum of variance at frequency 31 and higher harmonics, normalized to total variance (white pie slice). The sensitivity of the complementary set of parameters is calculated similarly (black pie slice). The remaining variance is assumed to be the result of non-linear higher-order interaction between parameters (gray pie slice). The total-order sensitivity index (S_Ti) is calculated by the sum of S_i and higher-order effects (white + gray pie slices).

Dummy parameter on eFAST and PRCC performed on the Lotka-Volterra model. Model equations and parameters are as described in Section 4.1 and . The reference output is Q(t)-prey , at t=9. Panel A: eFAST results with resampling and significance testing. Search curves were resampled 5 times (N_R=5), for a total of 1285 model evaluations (N_S=257). First-order S_i and total-order S_Ti are shown for each parameter, including a dummy parameter, as described in the text. Error bars indicate +/- 2 S.D on the mean of resamples. Parameters with first- or total-order indexes significantly different (p<0.01) from those of the dummy parameter are indicated with asterisks (*). Panel B: PRCC results. Sample size N=1000. (*) denotes PRCCs that are significantly different from zero.

PRCCs of the two compartmental model US analysis performed in section 4.3, plotted over a time course. The gray area represents PRCC values that are not statistically significant. Note how the sensitivity of parameters change as system dynamics progress.

Panel A: plot of the output (1000 runs) of the Lotka-Volterra model. The y axis represents variable P(t), x axis represents time (days). Panel B: scatter plot (linear scale) of parameter σ (x axis) and the “slice” of the output of Panel A at time=9 (r_Pearson =-0.15257, p<0.001). Panel C: linear-linear plot of the rank-transformed data of Panel B (r_Spearman =-0.0638, p>0.04, where r_Spearman is the r_Pearson coefficient calculated on the data of Panel C.). Panel D: linear-linear plot of the residuals of the linear regressions of the parameter σ versus all the other parameters of the model (x-axis) and the residuals of the linear regression of the output versus parameter σ (y-axis) (PRCC =-0.0575, p>0.06). PRCC is the Pearson correlation coefficient calculated on the data of Panel D. Scatter plots of Panel B, C and D remains qualitatively invariant for different LHS simulations. The input parameter (and its rank-transformed values) shown on the x-axis is σ, although all four parameters are varied simultaneously.

PRCC scatter plots of parameters τ₁, τ₂, α₁ α₃, α₄ and α₂₀ (calculated at day 200, all 8 parameters are varied simultaneously). Sample size N=1000. The abscissa represents the residuals of the linear regression between the rank-transformed values of the parameter under investigation versus the rank-transformed values of all the other parameters. The ordinate represents the residuals of the linear regression between the rank-transformed values of the output versus the rank-transformed values of all the parameter under investigation The title of each plot represents the PRCC value with the corresponding p-value (see )

Variation in ABM output due to epistemic uncertainty and aleatory uncertainty. Differences in parameter values lead to either containment of extracellular bacterial load (solid lines), or dissemination of bacteria (dotted lines), as described previously in (). Within each parameter combination, inherent stochasticity within the model causes aleatory uncertainty. At time points earlier than approximately day 150, aleatory uncertainty masks the epistemic difference that USA methodologies seek to measure.

Effect of aleatory uncertainty on LHS/PRCC and the benefit of averaging replicates. Model parameters were sampled 300 (black bars) or 100 times (gray bars). For the 100 sample condition, the 3 replicate model simulations were performed for each sample and averaged. Parameters with a PRCC significantly (p<0.005) different from zero are indicated with (*). Parameters with discordant statistical significance between the two experimental conditions are indicated (arrow).

Performance of eFAST on the stochastic agent-based model of Segovia-Juarez et al. () Panel A: eFAST first-order sensitivity index S_i. Replication and averaging (gray bars) has a small effect on S_i, allowing additional parameters (indicated by arrow) to be identified as statistically significant (*). Panel B: eFAST total-order sensitivity index S_Ti. eFAST artifactually assigns aleatory variance to the total-order index, resulting in a high dummy parameter background value (dummy parameter, black bar). Replication and averaging (gray bars) partially reduces this artifact, allowing an additional parameter (indicated by arrow) to be found significant (*).