← Back to Use Cases

Sandia Verification & Validation Challenge

Name Affiliation Phone Number Email Address
Sandia Labs Sandia Labs matt.butcher@ktn-uk.org
Industrial Sectors:



Taken from https://share.sandia.gov.vvcw

MysteryLiquid Co. maintains a large number of liquid storage tanks. Standard operating procedures limit the liquid level to below a certain fraction of the tank's height, and the remaining space is filled with pressurised gas.

The tanks are placed all over the world, and are used to store Mystery Liquid. The weight of the contents plus the pressurisation causes deformation of the tank walls.

Tank Information

The Tanks are cylinders with two half-sphere end caps. They are supported by rings around the circumference, located at the junction of the cylinder and end caps. Locations on the tank surface are described by axial distance from centerline and circumferential length, from straight down. This is shown in Figure 1.

Fig. 1 Side view and axial view of tanks Fig. 1 Side view and axial view of tanks

Tank Inspections

Each year, several tanks are inspected at random. This year, one tank failed to meet a required safety criterion when a large load was applied. This margin has been established from historical data, but is not a regulatory requirement. It has never before been violated during an inspection. The tank in question, Tank 0, did not physically fail, but the consequences of a failure would be significant. Given that the tank is out of spec, we wish to know if there is a real chance of physical failure. \(\bigstar\) See Appendix 4.8 for more information about testing, the tanks and Tank 0.

The out-of-spec tank and its two neighboring tanks were taken out of service and underwent testing. In addition, four tanks, in four different locations, each underwent multiple tests while still in service.

The company has commissioned a modeling study to complement these experimental tests. The assumption is that the historical safety margins are being violated, and we need to better understand the margin to failure. The goal is to determine whether the remaining tanks must be retired, or if they can be kept in service for a few years while replacements are ordered. The decision will be based on calculation of Probability of Failure.

Challenge Problem

The 2014 Verification and Validation Challenge Problem consists of three parts:

  • Prediction: the ultimate product of this study will be prediction of Probability of Failure for two scenarios. In addition to a best estimate of Probability of Failure, we expect to produce uncertainty estimates.
  • Credibility Assessment: In addition to the predictions, we need to know the credibility the predicted Probability of Failure
  • V&V Strategy: The key to providing a good credibility assessment is a logical and clearly defined strategy to gather evidence that the predictions are accurate

All data and models will be provided. No model development will be necessary or accepted, and no additional data can be generated (this year).


Modeling & Simulation will be used to make a decision on whether to remove all the tanks from service, or modify operating limits. The specific model predictions of interest will be Probability of Failure under two scenarios, listed below.

Simulation at the nominal conditions of the out-of-spec tank
  • In this scenario, the environmental state is specified at the nominal test conditions:
    • \(P = 73.5 psig\)
    • \(\chi = 1 \bigstar\)
    • H = 50 in
  • Participants should compute the Probability of Failure and uncertainty, with these input values fixed.

Understand the limits of the operating space
  • Here, the Probability of Failure is set at a threshold, \(P (Fail) \leq 10^{−3}\), and the participants must determine the loading levels which will violate the threshold.
  • Standard operating procedures put limits on the pressure, composition, and liquid height.
    • Pressures must be within \(P = [15,75] psig\)
    • Composition: \( \chi = [0.1,1]\)
    • Liquid height should be \(H \leq 55 in\)
  • These limits are strictly followed, but the measured operating conditions are not completely accurate \(\bigstar\) - meaning that operators ensure that the measured values are within limits.
  • What is the range of “safe” operating condition measurements, such that \(P (Fail) \leq 10^{−3}\)? Are current operating procedures enough to ensure safety?

These calculations will require both model predictions and some failure criterion. To simplify this exercise and ensure some level of consistency, “Failure” is strictly defined based on stress. More explicitly, failure occurs when the von Mises stress exceeds the yield stress at any point on the tank’s surface. See Section 3.4 for more discussion about Quantities of Interest.

V&V Strategy

The V&V strategy is the overall approach to making predictions AND assessing the uncertainty and credibility of those predictions. The implemented approach takes the form of a series of tasks to incorporate the experimental and Modeling & Simulation results.

The requirement for this part of the Challenge Problem is:

Develop and communicate a strategy for how data and models will be used to make the requested predictions AND assess both uncertainty and credibility of those predictions

The specific predictions of interest were listed in Section 3.1. The data and models are described in Sections 4 and 4.3. In this section, we first give an overview of the available data and models, and then give a list of possible tasks that might make up a V&V strategy, and finally discuss a V&V hierarchy as a possible way of communicating the strategy.


See ASME V&V 10 - Guide for Verification and Validation in Computational Solid Mechanics for standard definitions and introduction to Verification and Validation (V&V)
  • \(\sigma\) - von Mises stress
  • \(d\) - Tank wall displacement, normal to the surface
  • \(x\) - Axial location
  • \(\varphi\) - Circumferential angle
  • \(P\) - Gauge pressure
  • \(\gamma\) - Liquid specific weight
  • \(\chi\) - Liquid Composition (mass fraction)
  • \(H\) - Liquid height
  • \(E\) - Young's Modulus
  • \(v\) - Poisson's ration
  • \(L\) - Length
  • \(R\) - Radius
  • \(T\) - Wall thickness
  • \(m\) - Mesh ID

2.1 Process Inputs

Summary of available Data

The experimental study includes legacy data and five test series:

  1. Legacy data from the manufacturer. Documented nominal material properties and tank dimensions
  2. Coupon tests in a controlled, lab environment Measure material properties and Tank wall thickness
  3. Liquid characterization tests in a controlled, lab environment Specific weight & composition measurements on Mystery Liquid
  4. Full Tank tests in a controlled, lab environment full tank indicates the complete system, not that the tank is filled w/ liquid. No loading on the tank – measure dimensions (length and radius)
  5. Full Tank tests in a controlled, lab environment Pressure loading, measure displacements at four locations
  6. Full Tank tests in a production environment Measured loading – both pressure and liquid, measured displacement at 20 locations

2.2 Propagation

Tank Model

In addition, a mathematical model has been created for an idealized Tank under pressure & liquid loading. The pressure only loading is a special case. Datasets 5) and 6) above are collected from experimental conditions that are subsets of the scenarios that this model can simulate. The final predictions of interest are an extrapolation from the experimental conditions of dataset 6). The participant must determine if the model is adequate to simulate these more extreme conditions.

Additional information about V&V Strategies is included in Section 4.1. This includes some potential V&V related tasks that might be beneficial in this project, and an introduction to the concept of a V&V hierarchy. The relationship between the datasets and the model can be visualized with the V&V hierarchy in Figure 2. This is, of course, an incomplete picture but it does help to identify uses for the data and models. Details about the data and models are available in the Appendix.

2.3 Interpretation and Communication of Results

Credibility Assessment At the end of the project, the ultimate goal would be to make a decision regarding viability of the tanks. Such a decision would require knowledge of many external factors, like company finances, economics, and consequences of tank failure. This is too broad a scope for the challenge problem. Instead, participants are asked to comment – qualitatively or quantitatively – on the credibility of their predictions. Some guiding questions include:

  • How do you communicate the results, uncertainty, and credibility?
  • How does each V&V task contribute to the credibility of the predictions of interest?
  • Does the V&V strategy as a whole add credibility?
  • What is the impact of extrapolation from the validation domain?
  • Would you feel comfortable making decisions based on your analysis?
  • How would you improve the analysis?

Quantities of Interest

We will use Quantities of Interest (QoIs) to refer to: model predictions of a specific quantity, quantities derived from model predictions, quantities measured experimentally, OR quantities derived from measurements.

The experimental and modeling studies must be coordinated, so that the experiments produce QoIs that will be useful for the modeling activity. In order to reduce the scope of this challenge, several QoI decisions have been made and cannot be modified.
  1. The first type of QoI is displacement normal to the tank surface, at various, specified locations. This is the quantity that is directly measured during tests, and is simulated in the model. It is directly available from the Python code. Displacements were used because they are easy to understand, visualize, and compute. This is not intended to be a completely realistic scenario.
  2. The second type of QoI is the von Mises stress at arbitrary locations on the Tank walls. The material is observed to fail very quickly after reaching its yield stress. Therefore, the decision has been made to correlate tank failure to the event where von Mises stress exceeds yield stress. This is the only available failure criterion and must be used to estimate Probability of Failure.

Note that displacement data is available from the tanks, but no stresses are ever measured. This means that the prediction of interest is based on a quantity that is never really observed. However, there is a strong relationship between these two quantities of interest.


This Challenge was created by Sandia National Laboratories as an open problem, Results from the community response, and full appendices referenced in the above text can be found here: https://share.sandia.gov.vvcw . Appendix Additional Information about V&V Strategies A V&V Strategy is very problem dependent and is influenced by: the intended use of the predictions, the computational & experimental resources, schedule, etc. Most of these constraints are not applicable for this Challenge Problem. As examples:

  • The intended use is described, but lacks context and consequence
  • The models are very inexpensive, so computational budget is not a factor
It is up to participants to self-impose constraints – realizing this is a learning experience, not a real project. The primary concern here is function evaluations. Keep in mind the code provided is a proxy for an expensive finite element model, and so the number of function evaluations is quite important and should be tracked. A list of potential tasks

To help focus the activities, we have created a list of potential tasks to include in a V&V strategy. This is not exhaustive or exclusive, but is meant as a starting point:
  • Characterization of input uncertainties (material parameters, dimensions, etc.)
  • Characterization of environmental variability/uncertainty (loading conditions / model inputs)
  • Calibration of model parameters to match experimental data
  • Elicitation and/or treatment of epistemic vs. aleatoric uncertainty
  • Solution verification / estimate of numerical uncertainty
  • Sensitivity analysis
  • Uncertainty quantification
  • Validation of models against experimental data
  • Aggregation of uncertainty
  • Assess relevancy of information
  • Predictions plus uncertainty
  • Qualitative credibility assessment
Note that some critical tasks are not listed because we have made assumptions and restrictions. The biggest example is the choice of quantities of interest. For more information see Section 3.4. Also, it is not required that all tasks be performed, and not all data must be taken into account.