Workflow Group

Owned by
Dean N. Williams (Unlicensed)
Last updated: Nov 02, 2015
2 min read

12 Month Roadmap divided into two six month periods (v1: 01 Jan 2016 through 30 Jun 2016; and v2: 1 Jul 2016 through 31 Dec 2016)

  • v1 (manual process for the running of the model)
    • Hardware Infrastructure (new task):
      • Specific LCF hardware:
        • OLCF (Titian, Rhea, DTN nodes)
          • ORNL CADES (VMs, Pilaus)
        • ALCF (Mira, Cooley, DTN nodes)
      • Support ACME workflow software stack at OLCF and ALCF; software stack includes: ESGF, UV-CDAT Velo, Pegasus, and Globus)
      • Storage capacity for each LCF (OLCF and ALCF) as we grow ACME's data archive
      • Operating systems running on the LCF platforms
      • Supporting UV-CDAT at PNNL and LLNL
    • Process Flow
      • Manually run and maintain process flow steps
      • Work within the Request Hub process to request and log ESGF data publication (new)
      • Work with science teams for input data and observational SVN process (How does this process work?)
      • Repository backup process (HPSS)
      • Provenance capture tool development (work with Worley to capture his provenance into the workflow)
    • Data Management
      • Installation and maintenance of ESGF v2.x at LCFs (ORNL and ALCF)
      • Replication of important data and other ESGF sites
      • Test scientists remote publication
      • HPSS connection with ESGF (ALCF and NERSC) (new)
    • Analysis
      • Port UV-CDAT to LCFs at OLCF, ALCF, LLNL, and PNNL
      • test suite and performance testing
      • teaching scripts and automatic Python code generation
      • User documentation and tutorials
    • Diagnostics
      • Port UV-CDAT to LCFs at OLCF, ALCF, LLNL, PNNL, and NERSC
      • Continue to improve the AMWG and LMWG diagnostics to scientists satisfaction
      • Allow scientists to include their own diagnostics (loosely coupled or tightly coupled)
      • Teaching scripts
      • User documentation and tutorials
  • v2 (automated process for running of the model)
    • Hardware Infrastructure (continuation of new task)
      • Specific LCF hardware:
        • OLCF (Titian, Rhea, DTN nodes)
        • ORNL CADES (VMs and Pilaus)
        • ALCF (Mira, Cooley)
        • NERSC (Edison, DTN nodes)
      • Support ACME workflow software stack at OLCF and ALCF; software stack includes: ESGF, UV-CDAT Velo, Pegasus, and Globus, ProvEn (new), and ACME Dashboard (new) )
      • Storage capacity for each LCF (OLCF, ALCF, NERSC) as we grow ACME's data archive
      • Supporting UV-CDAT at PNNL and LLNL
    • Process Flow
      • Automate the process process at LCFs (OLCF, ALCF, and NERSC)
    • Rule Engine (new task)
      • Requirements gathering
      • Baseline prototype
      • Implementation of Rule Engine API
    • Data Management
      • Installation and maintenance of ESGF v2.x at LCFs (ORNL, ALCF, and NERSC)
      • Maintain scientist remote publication process
      • HPSS connection with ESGF (OLCF) (new)
    • Analysis
      • Port UV-CDAT to LCFs at OLCF, ALCF, LLNL, PNNL, and NERSC
      • Parallelism
      • teaching scripts and automatic Python code generation
      • User documentation and tutorials
    • Diagnostics
      • Port UV-CDAT to LCFs at OLCF, ALCF, LLNL, PNNL, and NERSC
      • Parallelism
      • Allow scientists to include their own diagnostics (loosely coupled or tightly coupled)
      • Teaching scripts
      • User documentation and tutorials
    • Provenance Capture (new task)
      • Provenance tool integrated into the automated process
    • Dashboard User Interface (new task)
      • Baseline dashboard implementation

General group confusing as to which working group is responsible for which process or operation.