L9_UQ_Framework Design Document
he Design Document page provides a description of the algorithms, implementation and planned testing including unit, verification, validation and performance testing. Please read Step 1.3 Performance Expectations that explains feature documentation requirements from the performance group point of view.
Design Document
In the table below 4.Equ means Equations and Algorithms, 5.Ver means Verification, 6.Perf - Performance, 7. Val - Validation, 
- competed, 
- in progress, 
- not done
Overview table for the owner and an approver of this feature
1.Description | Uncertainty quantification framework |
| 2.Owner | Khachik Sargsyan, Daniel Ricciuto |
| 3.Created | |
| 4.Equ | |
| 5.Ver | |
| 6.Perf | |
| 7.Val | |
| 8.Approver | Peter Thornton, William Riley (Unlicensed) |
| 9.Approved Date | |
| V1.0 | Accepted |
Title: Uncertainty Quantification Framework
Requirements and Design
ACME Land Group
Date:
Summary
A set of Python scripts is being included in the ACME Land Model v1 (ALMv1) to perform basic forward uncertainty quantification tasks, such as surrogate model construction and variance-based sensitivity analysis. As pre- and post-processing steps, the scripts heavily utilize functionalities of Uncertainty Quantification Toolkit (UQTk, http://www.sandia.gov/uqtoolkit/) being developed at SNL-CA as a lightweight C++/Python UQ library. UQTk is an external library and should be downloaded and installed separately. The general workflow consists of three steps (a) pre-processing: input parameter sampling, (b) ensemble simulation of ALM, (c) post-processing: for each output quantity of interest, build a polynomial surrogate and compute Sobol sensitivity indices.
Requirements
Requirement: Generate input parameter ensemble
Date last modified:
Contributors: Khachik Sargsyan, Daniel Ricciuto
One needs a selection of ALM parameters with associated value ranges for each parameter in order to generate ensemble input. This information is provided in the file Parameters_TitanUQ_ACME.csv. Note that some inputs are dependent making the sampling slightly less trivial.
Requirement: Perform ALM ensemble simulations
Date last modified:
Contributors: Daniel Ricciuto
A scripting framework to clone an existing case for a given number of ensemble members, and write parameter files for each according to a given set of samples.
Requirement: Polynomial Chaos surrogate model construction
Date last modified:
Contributors: Khachik Sargsyan
Given the ensemble of simulations corresponding to varied high-dimensional input, build a Polynomial Chaos (PC) – i.e. essentially a polynomial fit – surrogate for the input-to-output map.
Requirement: Variance-based uncertainty decomposition
Date last modified:
Contributors: Khachik Sargsyan
Given the polynomial fit (PC surrogate), one can extract global sensitivity indices, also known as Sobol sensitivity or variance-based decomposition. This is output variance fractional attribution to the input parameters used for surrogate construction.
Algorithmic Formulations
Design solution: Generate input parameter ensemble
Date last modified:
Contributors: Khachik Sargsyan, Daniel Ricciuto
The MaxEnt principle requires uniform sampling on the feasible input parameter space that respects the input constraints. Rosenblatt transformation (RT) is employed. For linear constraints such as a_1+a_2+a_3=1, one can avoid using RT by simply removing a_3 from the varying parameter list, and enforcing a_3=1-a_1-a_2 in order to prepare the input set for ALM. There are two triples with such constraints, lf_lab+lf_fcel+lf_flig=1 and fr_lab+fr_fcel+fr_flig=1.
Design solution: Perform ALM ensemble simulations
Date last modified:
Contributors: Daniel Ricciuto
To perform and manage large ensembles in parallel, a pbs script is used. The script must take into account different requirements and limitations on supported machines. The standard create/setup/build utilities are used to create a case with the desired compset and configuration, and we will provide fortran code to clone this case, set the correct environment and modify parameter files. Output from ensemble is then post-processed using fortran code into a single .nc file for variables and time periods of interest.
Design solution: Polynomial Chaos surrogate model construction
Date last modified:
Contributors: Khachik Sargsyan
For high-dimensional input space, one necessarily operates in sparse-data regime. For example, 10000 simulations of varying 65 model parameters cover a small fraction of the 65-dimensional space only. Thus, it is necessary to a) estimate the surrogate uncertainty due to lack-of-data, and b) enforce adaptive polynomial selection in order to avoid overfitting. Bayesian compressive sensing (BCS) provides both capabilities. Details of the initial form of the algorithm can be found in the paper Sargsyan_IJUQ.pdf. Remark: we have since enhanced the algorithm to include weighted regularization.
Design solution: Variance-based uncertainty decomposition
Date last modified:
Contributors: Khachik Sargsyan
The major advantage of PC-based surrogate is the access to Sobol sensitivity information without extra computational effort, via collecting the appropriate polynomial terms together.
Design and Implementation
Date last modified:
Contributors: Khachik Sargsyan, Daniel Ricciuto
The samples are generated as a preprocessing step, via Python script prepr.py. The syntax is prepr.py <csv file> <number of samples> [parameters to exclude]. For example,
prepr.py Parameters_TitanUQ_ACME.csv 10000 lf_flig fr_flig generates the following output files:
- pdomain_x.dat - input ranges for the unconstrained parameters
- pnames_clm.dat, pnames_x.dat - parameter names, the full list, and the unconstrained list, respectively.
- input_clm.dat, input_x.dat - input ensembles, the full input for ALM, and the 'computational' unconstrained input in [-1,1] for surrogate construction.
Implementation: Perform ALM ensemble simulations
Date last modified:
Contributors: Daniel Ricciuto
New code is located under <scripts>/acme/landuq.
In the input file ccsm_opt.txt, the user provides input about the desired case to clone, parameter sample file and length of simulation.
The fortran code psens.f90 and associated code psens_setup_rundir.f90 and psensmodf90 are used to create run subdirectories for each ensemble member, including specific parameter files.
The .pbs script UQ_ensemble.pbs is then used to submit and manage the ensemble.
The program postproc.f90 is used to process the large number of output file into a single NetCDF file containing the variables of interest.
One can run postp.py 0 <observable> to read the NetCDF file for a selected observable (e.g. GPP, TLAI, TOTSOMC, TOTVEGC, EFLX_LH_TOT) and store it in Python's numpy format for further use.
Furthermore, postp.py 1 <observable> <site_id> generates several diagnostic plots for a selected observable and site.
Implementation: Polynomial Chaos surrogate model construction
Date last modified:
Contributors: Khachik Sargsyan
PC surrogate for specific observable and site can be obtained by running postp.py 2 <observable> <site_id> which generates python results_*.pk files for each surrogate information.
One can use the multi-threading routine uqmulti.py in order to launch such construction in parallel for all observables and sites. First, create tasks by running create_tasks.x 2, then execute uqmulti.py tasks to launch a set of surrogate construction tasks in parallel.
Implementation: Variance-based uncertainty decomposition
Date last modified:
Contributors: Khachik Sargsyan
Sensitivity information for specific observable and site can be obtained by running postp.py 3 <observable> <site_id> which reads the results_*.pk file and generates text files with sensitivity information as well as plots main and joint sensitivities for visual inspection.
One can use the multi-threading routine uqmulti.py in order to launch sensitivity computation in parallel for all observables and sites. First, create tasks by running create_tasks.x 3, then execute uqmulti.py tasks to launch a set of sensitivity computation tasks in parallel.
Furthermore, to bar-plot sensitivity information for all sites simultaneously for a given observable, execute postp.py 4 <observable>.
The full workflow, barring the actual ALM computation as black-box, is implemented in a single script uq_workflow.x.
Planned Verification and Unit Testing
Verification and Unit Testing: UQ framework
Date last modified:
Contributors: Daniel Ricciuto
We will verify that the methods used to produce ALM ensemble members produce bfb results with simulations performed using the standard model configure/build/submit procedure, given the same input data, environment and parameters.
Planned Validation Testing
Validation Testing: UQ framework
Date last modified:
Contributors: Khachik Sargsyan, Daniel Ricciuto
Site-level UQ results will be compared with flux tower and experimental observations. While the BCS algorithm internally validates the surrogate, we will further test the accuracy of the surrogate model by creating a validation ensemble that is distinct from the training one, with the same mechanism as in the first step (i.e. sample generation) of the UQ workflow.
Planned Performance Testing
Performance Testing: UQ framework
Date last modified:
Contributors: Daniel Ricciuto
The standard ACME timing/performance tools will be used to evaluate ensemble performance. Although ensemble members run as independent executables within a large pbs job, performance may be degraded by large numbers of file operations in a large ensemble. We will play close attention to I/O performance for large ensembles.