Data Publication Requests

Owned by Renata McCoy
Last updated: Aug 12, 2024 by Anthony Bartoletti
6 min read

Additional table for https://acme-climate.atlassian.net/wiki/spaces/EIDMG/pages/2234549238

Instructions to teams on preparing simulation data for access: Simulation Data Management | 3. Document HPSS locations

For v2, more detailed simulation information can be found https://acme-climate.atlassian.net/wiki/spaces/ED/pages/2766340117 :

Campaign

Experiment/dataset(s) [IMPROVE THESE DESCRIPTIONS!]

Request (Archive|Stage|Publish)

Date: Status (+ notes)

Campaign

Experiment/dataset(s) [IMPROVE THESE DESCRIPTIONS!]

Request (Archive|Stage|Publish)

Date: Status (+ notes)

v2_1.LR

1pctCO4

abrupt-4xCO2

amip (1 ensemble: _0101 only)

historical (5 ensembles: _0101,_0151,_0201,_0251,_0301)

piControl

Archive, Stage Publish to ESGF

2024-08-07: Transferred CMIP6 datasets to ANL:/lcrc/group/e3sm2/DSM/Staging/Data (5.18 TB) for ANL publication to ESGF.

2024-08-05: Updated CMIP6 metadata references with recent publication specifics (997 datasets, 10800+ files).

2024-06-14: CMIP6 generation completed (997 datasets)

2024-04-12: CMIP6 generation at 96%. Random MPAS sets are balking

2024-03-01: CMIP6 generation at 84%. “amip”, “day” variables and random MPAS sets are balking.

2024-02-06: Native data validation completed. CMIP6 generation commenced.

2024-01-10: extraction completed except for amip (96/120 datasets to warehouse)

2024-01-10: Archive completed except for amip (and piControl tarfile 000010.tar replaced)

v2.NARRM

v2. NARRM:

1pctCO4

abrupt-4xCO2

amip (3 ensembles: _0101,_0201,_0301)

historical (5 ensembles: _0101,_0151,_0201,_0251,_0301)

piControl

Archive, Stage Publish to E3SM

2023-05-04: Published (120 native datasets published and verified).

2023-04-18: Validation complete (120 datasets, required repair to 7 sets with missing file)

2023-03-23: Extraction complete (120 datasets in 11 ensembles to warehouse)

 

 

Stage (Cmorize), Publish to CMIP

2023-07-18: Published 1073 datasets

2023-07-14: Cmorizing complete (1073 datasets complete, 37 deferred bad data (historical ensemble 2).

v2.LR

v2.LR.1pctCO2 (1 ensemble)

Archive, Stage Publish to E3SM

2022-08-03: Published (12 datasets)

 

 

Stage (Cmorize), Publish to CMIP

2022-12-16: Cmorized and Published (114 datasets)

v2.LR

v2.LR.abrupt-4xCO2 (2 ensembles)

Archive, Stage Publish to E3SM

2022-08-29: Published (24 datasets)

 

 

Stage (Cmorize), Publish to CMIP

2022-12-17: Cmorized and Published (228 datasets)

v2.LR

v2.LR.amip (3 ensembles: _0101,_0201,_0301)

Archive, Stage Publish to E3SM

2022-08-04: Published (24 datasets)

2022-06-14: Archive at LLNL:/p/user_pub/e3sm/archive/2_0

 

 

Stage (Cmorize), Publish to CMIP

2022-12-17: Cmorized and Published (231 datasets)

v2.LR

v2.LR.hist-GHG (5 ensembles: _0101,_0151,_0201,_0251,_0301)

Archive, Stage Publish to E3SM

2022-08-10: Published (60 datasets)

2022-06-13: Archive at LLNL:/p/user_pub/e3sm/archive/2_0

 

 

Stage (Cmorize), Publish to CMIP

2022-12-17: Cmorized and Published (570 datasets)

v2.LR

v2.LR.hist-aer (5 ensembles: _0101,_0151,_0201,_0251,_0301)

Archive, Stage Publish to E3SM

2022-08-10: Published (60 datasets)

2022-06-15: Archive at LLNL:/p/user_pub/e3sm/archive/2_0

 

 

Stage (Cmorize), Publish to CMIP

2022-12-18: Cmorized and Published (570 datasets)

v2.LR

v2.LR.hist-nat[*] (5 ensembles: _0101,_0151,_0201,_0251,_0301)

[*] aka, “hist-all-xGHG-xaer"

Archive, Stage Publish to E3SM

2022-12-21: Published (60 datasets)

2022-06-15: Archive at LLNL:/p/user_pub/e3sm/archive/2_0

 

 

Stage (Cmorize), Publish to CMIP

2023-01-04: Cmorized and Published (530 datasets)

(this experiment doesn’t follow exact CMIP protocol, which we added a note in the metadata when submitting to CMIP)

v2.LR

v2.LR.historical (5 ensembles: _0101,_0151,_0201,_0251,_0301)

Archive, Stage Publish to E3SM

2022-08-10: Published (60 datasets)

2022-06-13: Archive at LLNL:/p/user_pub/e3sm/archive/2_0

 

 

Stage (Cmorize), Publish to CMIP

2022-12-17: Cmorized and Published (570 datasets)

v2.LR

v2.LR.piClim-Control (1 ensemble)

Archive, Stage Publish to E3SM

2022-08-07: Published (8 datasets)

2022-06-15: Archive at LLNL:/p/user_pub/e3sm/archive/2_0

 

 

Stage (Cmorize), Publish to CMIP

2022-12-18: Cmorized and Published (77 datasets)

v2.LR

v2.LR.piClim-histaer (3 ensembles: _0021,_0031,_0041)

Archive, Stage Publish to E3SM

2022-08-07: Published (24 datasets)

2022-06-15: Archive at LLNL:/p/user_pub/e3sm/archive/2_0

 

 

Stage (Cmorize), Publish to CMIP

2022-12-18: Cmorized and Published (231 datasets)

v2.LR

v2.LR.piClim-histall (3 ensembles: _0021,_0031,_0041)

Archive, Stage Publish to E3SM

2022-08-07: Published (24 datasets)

2022-06-15: Archive at LLNL:/p/user_pub/e3sm/archive/2_0

 

 

Stage (Cmorize), Publish to CMIP

2022-12-18: Cmorized and Published (231 datasets)

v2.LR

v2.LR.piControl (1 ensemble)

Archive, Stage Publish to E3SM

2022-08-10: Published (8 datasets)

2022-06-13: Archive at LLNL: /p/user_pub/e3sm/archive/2_0

 

 

Stage (Cmorize), Publish to CMIP

2022-12-17: Cmorized and Published (114 datasets)

 

 

 

 

 

Campaign

Experiment/dataset(s) [IMPROVE THESE DESCRIPTIONS!]

Request (Archive|Stage|Publish)

Date: Status (+ notes)

Campaign

Experiment/dataset(s) [IMPROVE THESE DESCRIPTIONS!]

Request (Archive|Stage|Publish)

Date: Status (+ notes)

HR v1

1950 control standard volume output (56-136/09) from Theta

theta.20190910.branch_noCNT.n825def.unc06.A_WCYCL1950S_CMIP6_HR.ne120_oRRS18v3_ICG

Archive, Stage Publish to E3SM

2022-03-03: Native Publication Completed (9 native datasets, derivatives remain)

2021-11-19: Data transfer completed.

2021-11-01: Archive are in transfer from Theta to LLNL

HR v1

1950 control standard volume output (136/10-150 skip MPAS files for now) from Chrysalis

20210112.A_WCYCL1950S_CMIP6_HR.ne120_oRRS18v3_ICG.unc06

Archive, Stage Publish to E3SM

2022-03-03: Native Publication Completed (6 native datasets non-MPAS, derivatives remain)

2022-01-18: Data transfer completed

2021-11-01: Archive in creation on Theta

HR v1

Transient run from Theta 1951-1971

20200517-maint-1.0-tro.A_WCYCL20TRS_CMIP6_HR.ne120_oRRS18v3_ICG.unc11-20years

Archive, Stage Publish to E3SM

2023-04-25: Native data published to ESGF

2022-01-18: Data transfer completed

2021-11-01: Archive are in transfer from theta to LLNL

2021-08-11: We will publish the datasets upon the publication of US water cycle metrics paper. The paper is being submitted.

HR v1

Transient run from Chrysalis 1971-2015

2021/202101027-maint-1.0-tro.A_WCYCL20TRS_CMIP6_HR.ne120_oRRS18v3_ICG.unc12

Archive, Stage Publish to E3SM

2023-04-25: Native data published to ESGF

2022-01-18: Data transfer completed

HR-v1

1950-Transient-LRtunedHR

Archive, Stage Publish to E3SM

2023-04-27: Publication Completed (11 native datasets)

2021-11-05: Datasets Pass Validation, awaiting authorization to publish
2021-11-01: Archive was transferred to LLNL,

CRYO-v1

Revision to experiments B-1850-DIB-ISMF and B-1850-DIB-ISMF-3dGM

All native and derivative datasets

Archive, Stage Publish to E3SM

2021-11-30: Publication Completed (12 native datasets, 4 timeseries, 2 climos)

2021-11-22: Publication Completed (12 native datasets). Generation of 6 derivatives in progress.
2021-11-21: (revised) datasets (12) passed Validation, publication in progress.

2021-11-17: Archives in place, 12 (revised) native datasets extracted to warehouse, pending Validation.

DECK-v1

 

DAMIP hist-GHG H1, H2, H3

Stage (Cmorize), Publish to CMIP

2022-06-13 Updates:

For all 8 ensembles (3 DAMIP hist-GHG and 5 ssp585):

  • Amon:       All 352 are published

  • AERmon:     All  16 are published

  • day:        All  48 are published

  • 3hr:        All   8 are published

  • CFmon:       64 of 72 are published (8 await a clcalipso handler)

  • Omon:       208 of 240 are published (16 pbo, pso are deferred, 14 thkcello, volcello and 2 zhalfo need investigation)

  • fx:          24 not yet engaged

  • Ofx:          8 not yet engaged

Lmon, LImon, SImon (168 sets), deferred for Charlie’s NCO updates.

2022-05-09: All Amon datasets are completed.

Publication authorized Sep 9, 2021

DAMIP hist-GHG H1, H2, H3

Stage Publish to E3SM

2021-09-20: Publication Completed (42 datasets, each ensemble 11 native and 3 derivative datasets)

ssp585 projection P1, P2, P3, P4, P5

Stage (Cmorize), Publish to CMIP

2022-05-09: All Amon datasets are completed.

Publication authorized Sep 9, 2021

ssp585 projection P1, P2, P3, P4, P5

Stage Publish to E3SM

2021-09-20: Publication Completed (70 datasets, each ensemble 11 native and 3 derivative datasets)

SSP5-8.5-GHG P1

Stage Publish to E3SM

2021-09-20: Publication Completed (42 datasets, each ensemble 11 native and 3 derivative datasets)

GHG-only simulations in SSP585 are not part of CMIP6, will only be published to E3SM project

BGC-v1

CTC piControl-ext85yr

Stage Publish to E3SM

2021-09-27: Publication Completed (85 year extensions as separate datasets)

BGC-v1

ECA piControl-ext85yr

Stage Publish to E3SM

2021-09-27: Publication Completed (85 year extensions as separate datasets)

BGC-v1

1-1(CTC): piControl with 85yr extension

 

Extend, Stage (Cmor)
Publish to CMIP

2020-11-23: Second version with 85yrs extension was published

2019-10-29: First version was published

The caveat is that two versions are published from two different esgf nodes. Both versions show up during search, ex:

Data Node: aims3.llnl.gov
Version: 20191029
Total Number of Files (for all variables): 7

 

Data Node: esgf-data2.llnl.gov
Version: 20201123
Total Number of Files (for all variables): 8

BGC-v1

1-1(ECA): piControl with 85yr extension

 

Extend, Stage (Cmor)
Publish to CMIP

2020-10-15: extension published with original data

BGCv1:

 

 

1-1(CTC):ssp585 BCRC

Archive, Stage, Publish to E3SM

2021-05-10: Publication Completed

note: 20191107.BCRC_CNPCTC_SSP585_OIBGC.ne30_oECv3.compy

1-1(CTC):ssp585 BCRD

Archive, Stage, Publish to E3SM

2021-05-10: Publication Completed

note: 20191204.BCRD_CNPCTC_SSP585_OIBGC.ne30_oECv3.compy

1-1(CTC):ssp585 BDRC

Archive, Stage, Publish to E3SM

2021-05-10: Publication Completed

note: 20191107.BDRC_CNPCTC_SSP585_OIBGC.ne30_oECv3.compy

1-1(CTC):ssp585 BDRC

Stage (Cmorize), Publish to CMIP

2020-11-25: Publication Completed

1-1(CTC):ssp585 BDRD

Archive, Stage, Publish to E3SM

2021-05-10: Publication Completed

note: 20191204.BDRD_CNPCTC_SSP585_OIBGC.ne30_oECv3.compy

1-1(CTC):ssp585 BDRD

Stage (Cmorize), Publish to CMIP

2020-12-01: Publication Completed

BGC-v1:

 

 

 

 

1-1(ECA):ssp585 BCRC

Archive, Stage, Publish to E3SM

2021-09-25: Publication Completed

note: BCRC_CNPECACNT_SSP85.ne30_oECv3.cori-knl

1-1(ECA):ssp585 BCRD

Archive, Stage, Publish to E3SM

2021-09-23: Publication Completed

note: BCRD_CNPECACNT_SSP85.ne30_oECv3.cori-knl

1-1(ECA):ssp585 BDRC

Archive, Stage, Publish to E3SM

2021-09-23: Publication Completed

note: 20191108.BDRC_CNPECACNT_SSP585.ne30_oECv3.cori-knl (2015-2076), and 20200326.BDRC_CNPECACNT_SSP585.ne30_oECv3.cori-knl (2077-2100)

1-1(ECA):ssp585 BDRC

Stage (Cmorize), Publish to CMIP

2020-12-09: Publication Completed

1-1(ECA):ssp585 BDRD

Archive, Stage, Publish to E3SM

2021-05-10: Publication Completed (2021-07: Agreed on to use an non-bfb rerun for year 2076 atmospheric)

note: 20191108.BDRD_CNPECACNT_SSP585.ne30_oECv3.cori-knl (2015-2076), and 20200326.BDRD_CNPECACNT_SSP585.ne30_oECv3.cori-knl (2077-2100)

1-1(ECA):ssp585 BDRD

Stage (Cmorize), Publish to CMIP

2022-04-04: 97 out of 106 datasets completed and published

All that remains are these 9: fx.areacella, fx.orog, fx.sftlf, Ofx.areacello, Omon.pbo, Omon.pso,Omon.thkcello, Omon.volcello ,Omon.zhalfo

BGC-v1

 

 

 

 

1-1(CTC):hist-BCRC

Archive, Stage, Publish to E3SM

2021-02-10: Publication Completed

note: 20181217.BCRC_CNPCTC20TR_OIBGC.ne30_oECv3.edison

1-1(CTC):hist-BCRD

Archive, Stage, Publish to E3SM

2021-02-10: Publication Completed

note: 20181217.BCRD_CNPCTC20TR_OIBGC.ne30_oECv3.edison + 20190812.BCRD_CNPCTC20TR_OIBGC.ne30_oECv3.compy

1-1(CTC):hist-BDRC

Archive, Stage, Publish to E3SM

2021-04-16: Publication Completed

note: 20181217.BDRC_CNPCTC20TR_OIBGC.ne30_oECv3.edison

1-1(CTC):hist-BDRD

Archive, Stage, Publish to E3SM

2021-04-16: Publication Completed

note: 20181217.BDRD_CNPCTC20TR_OIBGC.ne30_oECv3.edison

1-1(ECA):hist-BCRC

Archive, Stage, Publish to E3SM

2021-02-10: Publication Completed

1-1(ECA):hist-BCRD

Archive, Stage, Publish to E3SM

2021-04-16: Publication Completed

1-1(ECA):hist-BDRC

Archive, Stage, Publish to E3SM

2021-04-16: Publication Completed

1-1(ECA):hist-BDRD

Archive, Stage, Publish to E3SM

2021-04-16: Publication Completed

CRYO-v1

all native and derivative datasets

Archive, Stage Publish to E3SM

2020-10-20: Publication Completed

DECK-v1

All experiments, all datasets (I guess - nothing specific was named)

Archive, Stage, Publish

https://acme-climate.atlassian.net/wiki/spaces/ED/pages/668959422/v1+DECK+output+HPSS+archive

2021-02-12: Completed. DECK-v1 Low Res

note: (excluding future scenario simulations)

 

 

 

 

 

[Jill: Old Descriptions Below - review and incorporate or toss)

 

 

 

 

 

 

HR-v1

Pull High res v1 zstash to LLNL, Stage and Publish

(theta.20180906.branch_noCNT.A_WCYCL1950S_CMIP6_HR.ne120_oRRS18v3_ICG)

Archive, Stage, Publish

Need HPSS locations:

On Theta: /home/azamatm/2018/E3SM_simulations/theta.20180906.branch_noCNT.A_WCYCL1950S_CMIP6_HR.ne120_oRRS18v3_ICG/

Completed: https://acme-climate.atlassian.net/wiki/pages/createpage.action?spaceKey=ED&title=HR-v1%3A%20Control-HR%20-%20e1

BGC-v1

Pull BGCv1 zstash to LLNL, Stage and Publish

 

CBGC v1 output HPSS archive . All data is now properly in zstash format and has been transferred to LLNL E3SM Archive.

BGC-v1 -- Archive Source and Transfer Status

Update Sep 22, 2020 Rectifying last bit of high-frequency output.BGC-v1

Update Oct 21, 2020 Note: All high-frequency output published, except for those from piContol simulations. Those will be published together with the 85 year pi-control extension.