Production of the Linoz_v3 data

Owned by Jinbo Xie
Last updated: Mar 29, 2024 by Qi Tang
7 min read

Metadata

Origin: May 2023
Documentation author(s): Jinbo Xie (LLNL)
E3SMv3 Linoz_v3 collaborators and contributors: @Qi Tang, @Michael J Prather, @Juno Hsu

Overview

Disclaimer

The information that follows is an outline of the steps that were taken to develop the Linov-v3 input forcing data for SSP component sets ("compsets") for E3SM version 3. These same steps may or may not work for previous or future versions of E3SM.  The instructions are accurate to the best of the author's knowledge. If errors are found in this document, please notify @Jinbo Xie or add a comment to this page. A high-level overview of compset input forcing creation (from 2022, some information may be obsolete!) can be found in https://acme-climate.atlassian.net/wiki/spaces/DOC/pages/3413737487; the information outlined below is intended to supplement this workflow overview by providing greater detail and is specific to the Linoz-v3 input forcing specific to the historical, SSP370 and SSP585 compsets for E3SMv3. The example below is for SSP370, but the process is the same for all other SSPs with the minimum change in several instances (the latest version of c20231207 includes Linoz file for components of Historical, SSP119, SSP126, SSP245, SSP370, SSP434, SSP460, SSP534-over, and SSP585).

The output files are:

linv3_1849-2101_CMIP6_Hist_SSP370_10deg_58km_c20230430.nc
linv3_1849-2101_CMIP6_Hist_SSP585_10deg_58km_c20230430.nc

Creation of E3SM SSP370 input files for Linoz-v3

Linoz-v3 files (SSP370)

For SSP370, the Linoz-v3 forcing data must be produced before Linoz-v3 can be run on E3SMv3.

For E3SMv2, these files are associated with the following EAM namelist entry

  1. linoz_data_file

One additional namelist entry, linoz_data_path, defines the path to linoz_data_file.  Below are instructions on how to convert raw input4mips data files to Linoz-v3 input files for E3SM.  These instructions assume the user has access to IDL and Fortran compilers, and the workflow was invoked on Livermore Computing server of SYRAH.

linoz_data_file

For linoz_data_file, the UCI chemistry box model (we’ll denote this BOXMODEL) provided by @Michael J Prather and @Juno Hsu was used to process the raw input4mips files into the netcdf form usable by E3SMv3.

It is recommended that the user download the BOXMODEL directory from the weblink using either http downloads or wget (see, e.g., this note on wget). We will create the linoz_data_file following instructions in $BOXMODEL/Linoz_Input/CMIP6_derived_files/README.txt.

  • NOTE:  The steps outlined below have already been completed as part of the process to create the SSP370 compset.  The user may wish to repeat the steps independently, or leverage what already exists in the $BOXMODEL workspace and customize existing files as needed.

  • Step 1: Create a combined GHG concentration file. The creation of these files are explicitly described by https://acme-climate.atlassian.net/wiki/spaces/DOC/pages/3413737487/Creating+SSP370+and+SSP585+compsets+for+E3SM#chlorine_loading_file-and-linoz_data_file “ step 1 for details.“

  • Step 2: Generate input concentration file for PRATMO

  • The generation of input concentration for PRATMO is similar to that in https://acme-climate.atlassian.net/wiki/spaces/DOC/pages/3413737487/Creating+SSP370+and+SSP585+compsets+for+E3SM#chlorine_loading_file-and-linoz_data_file, except the equation is slightly modified to add halon2402 to Bry calculation. The new modified IDL files are under $BOXMODEL/PRATMO/PRATMO_Input:

    module load idl idl Extract_new_GHG_for_PRATMO.pro

    Output to: ./LINOZ_E3SM/data/CMIP6_ghg.dat

    • Modify file Extract_new_SSP585_GHG_for_PRATMO.pro:

      cp Extract_new_SSP585_GHG_for_PRATMO.pro Extract_new_SSP370_GHG_for_PRATMO.pro

      (Change "SSP585" to "SSP370" in several instances.)

    • Run "SSP370" of script:

      idl Extract_new_SSP370_GHG_for_PRATMO.pro

      Output to: ./LINOZ_E3SM/data/CMIP6_ghg_SSP370.dat

  • Step 3: Run PRATMO first for the historical period and then for SSP370.  Detailed instructions for running PRATMO as part of step 3 only can be found in $BOXMODEL/PRATMO_Input/LINOZ_E3SM/README.txt -- note:  this is a separate file from the previous README.txt mentioned above.

    • For part (1) in $BOXMODEL/PRATMO/PRATMO_Input/LINOZ_E3SM/README.txt, no action is needed, assuming that the listed FORTRAN executable have been properly set.

    • For part (2), it is recommended to create multiple copies of bctmx.f, one for each configuration.  For example, the user could create bctmx.f.Historical containing settings for the historical GHG configuration, bctmx.f.SSP370 containing settings for the SSP370 configuration, and so on.  Whenever PRATMO is run, simply copy the desired configuration file to bctmx.f, which is the only version that PRATMO will use.  Note: It is recommended to use the same procedure for batmo.f as well (see “part 3” below).  First, check bctmx.f.Historical near L376 to ensure that the following line is being used:

      Then copy the "Historical" version of the file to the "active" bctmx.f version to be compiled:

    • For part (3), make copies of batmo.f (see note in part 2 above):  batmo.f.Historical, batmo.f.SSP370, etc.  Check batmo.f.Historical to ensure that:

      • L7:  character*25 FNAME

      • L13:  LINOZ=.FALSE. !(set linoz_v2 as false)

      • L16:  LDERIVS=.TRUE. !(set linoz_v3 taylor expansion)

      • L29:  iyear spans 1849,2017

      • L35:  fname='init_fspecies_0000'
        Copy batmo.f.Historical to batmo.f, the version of the file to be compiled.

    • For part (4), compile the executable using:

      You may get a few warnings, but these should not be fatal.  Then, run the script.  NOTE:  The script takes ~3.5 hours to finish, so one option is to run the script on a login node using tmux, which is supported on most supercomputing centers and otherwise is freely available:

    • Output is:  $BOXMODEL/PRATMO/PRATMO_output/historical/cmip6/... 
      ...init_fspecies_1849 to init_fspecies_2017  (linoz v3 table)

      • ...CLIM_LINCHEM_1849 to CLIMCHEM_2017

      • ...lnoy_1849 to lnoy_2017 and lnoy_1849.bin to lnoy_2017.bin

      • ...no3x_1849 to no3x_2017 and no3x_1849.bin to no3x_2017.bin

      • ...pnoy_1849 to pnoy_2017 and pnoy_1849.bin to pnoy_2017.bin

      • ...ln2o_1849 to ln2o_2017 and ln2o_1849.bin to ln2o_2017.bin

      • ...nch4_1849 to nch4_2017 and nch4_1849.bin to nch4_2017.bin

      • ...pn2o_1849 to pn2o_2017 and pn2o_1849.bin to pn2o_2017.bin

      • ...lnoy_1849 to lnoy_2017 and lnoy_1849.bin to lnoy_2017.bin

      • The results are linoz v3 table, Climatology and production/loss derivatives for tracers noy, o3x, noy, n2o, ch4.

      • The outputs are in txt/bin format. To ensure a netcdf output, we’ll have to use the TxttoNetcdf folder’s scripts to turn these file into Netcdf format and combine the data:

    • REPEAT parts (2-4) but for SSP370:

      • SSP370 part (2): Check bctmx.f.SSP370 near L376 to ensure that the following line is used:

        Copy bctmx.f.SSP370 to bctmx.f.

      • SSP370 part (3): Check batmo.f.SSP370 to ensure that:

        • L7:  character*25 FNAME

        • L13:  LINOZ=.FALSE. !(set linoz_v2 as false)

        • L16:  LDERIVS=.TRUE. !(set linoz_v3 taylor expansion)

        • L29:  iyear spans 1849,2017

        • L35:  fname='init_fspecies_0000'
          Copy batmo.f.SSP370 to batmo.f.

      • SSP370 part (4):  Compile the code...

        You may get a few warnings, but these should not be fatal.  Run the script.  NOTE:  The script takes ~8 hours to finish, so it is highly recommended that the user either submit a batch job (sample batch script for NERSC-Cori is available at $BOXMODEL/Linoz_Input/CMIP6_derived_files/LINOZ_E3SM/linoz_file_generator_SSP370_BATCH) or use a tmux window (see 'historical' above for example). The output is:
        $BOXMODEL/PRATMO/PRATMO_output/SSP370/cmip6/...
        ...init_fspecies_2018 to init_fspecies_2503  (linoz v3 table)

        • ...CLIM_LINCHEM_2018 to CLIMCHEM_2017

        • ...lnoy_2018 to lnoy_2053 and lnoy_2018.bin to lnoy_2053.bin

        • ...no3x_2018 to no3x_2053 and no3x_2018.bin to no3x_2053.bin

        • ...pnoy_2018 to pnoy_2053 and pnoy_2018.bin to pnoy_2053.bin

        • ...ln2o_2018 to ln2o_2053 and ln2o_2018.bin to ln2o_2053.bin

        • ...nch4_2018 to nch4_2053 and nch4_2018.bin to nch4_2053.bin

        • ...pn2o_2018 to pn2o_2053 and pn2o_2018.bin to pn2o_2053.bin

        • ...lnoy_2018 to lnoy_2053 and lnoy_2018.bin to lnoy_2053.bin

        • The results are linoz v3 table, Climatology and production/loss derivatives for tracers noy, o3x, noy, n2o, ch4.

    • PARALLELISM of the PRATMO execution:

      • As noted above, the execution of the PRATMO for historical or SSP may take from 3~8 hours for single execution. In order speed up the execution, we implement a simple parallelism tool to the PRATMO execution, so as to accelerate the PRATMO execution for whole period to 20~40 mins.

      • The problems is based on the fact that PRATMO is run on a single loop through time.

        • Output on each loop is only based on the current year of inputs;

        • Thus one execution each year to form 100 or more parallel executed PRATMO calculations can be executed using scripts batch.sh and sed.sh in $BOXMODEL/scripts.

        • Take input into $BOXMODEL/PRATMO/PRATMO_output/historical/cmip6 for example:

      • The batch.sh and sed.sh is designed to form batch scripts to concurrently run the process from compilation to execution of PRATMO for every year.

      • The batch is designed to run 10 years per node to ensure computer load balance, it can be changed simply with L1 in batch.sh

        • L1: g=10

      • The loop goes through four scenarios, historical, SSP245, SSP370, SSP585, we can change to loop to fit our need

        • L8: for i in {1..4..1}; do

      • The idea for the concurrent parallelism

        • To create a batmo.f for each year and bctmx.f for each scenario and separately compile and execute.

        • $BOXMODEL/scripts/batch.sh creates 4 sed.sh scripts-- sed_hist.sh, sed_SSP245.sh, sed_370.sh, sed_585.sh for each scenario.

        • Each scenario sed_*.sh creates concurrently running PRATMO executable files and runs for each year concurrently.

        • Outputs files:

          • batmo.f like batmo-1849.f, batmo-1850.f.....batmo-2017.f;

          • btcmx-1849.f, btcmx-1850.f......btcmx-2017.f;

          • Executables cmip6a_historical-1849.out......cmip6a_historical-2017.out;

          • PRATMO outputs each year shown above.

          • It is recommended to put the batch.sh and sed.sh files in a single clean folder since the output files may be of large number.

  • Step 4:  Convert PRATMO output to E3SM input format

    • This process is slightly different from the Fortran files above because we're not swapping in/out modified scripts.  Instead, there are independent scripts here for both historical and SSP* configurations in $BOXMODEL/TxttoNetcdf.

    • Run the following for PRATMO txt/bin output in $BOXMODEL/PRATMO/PRATMO_output/historical/cmip6

      • Outputs files:

        • ...$BOXMODEL/PRATMO/PRATMO_output/historical/netcdf.....

        • ...linv3_1849.nc to linv3_2017.nc

        • ...Clim_LinozV3_1849.nc to Clim_LinozV3_2017.nc

        • ...lnoy_1849.nc to lnoy_2017.nc

        • ...no3x_1849.nc to no3x_2017.nc

        • ...pnoy_1849.nc to pnoy_2017.nc

        • ...ln2o_1849.nc to ln2o_2017.nc

        • ...nch4_1849.nc to nch4_2017.nc

        • ...pn2o_1849.nc to pn2o_2017.nc

        • ...lnoy_1849.nc to lnoy_2017.nc

      • To combine the output, a separate combinenc.sh in $BOXMODEL/TxttoNetcdf should be used to combine the data:

      • Outputs files:

        • ...$BOXMODEL/PRATMO/PRATMO_output/historical/netcdf.....

        • ...linv3_1849-1850.nc to linv3_1849-2017.nc

    • Final adjustment

      • Before forming the final data, the data will goes through an adjustment process in $BOXMODEL/E3SM_Linozv3_forcing.

      • This includes four parts

        • Adding a cariolle_pscs for the whole time period for the data.

        • Modifying the lowest model level to 985.5hPa.

        • Adding the variable long_name to the output files.

        • Concatenate the historical and SSP files together to form files from 1849-2101.

      • To execute the process, we'll go through the following process in $BOXMODEL/E3SM_Linozv3_forcing

      • Before execution, be sure to check the following setup in connect.sh:

        • L2 fil_in1=$BOXMODEL/PRATMO_output/historical/netcdf/linv3_1849-2017.nc

        • L4 fil_in3=$BOXMODEL/PRATMO_output/SSP379/netcdf/linv3_2018-2101.nc

        • L5 fil_in4=$BOXMODEL/PRATMO_output/SSP585/netcdf/linv3_2018-2101.nc

        • fil_ou1=linv3_1849-2101_CMIP6_Hist_SSP370_10deg_58km_c20230430.nc

        • fil_ou2=linv3_1849-2101_CMIP6_Hist_SSP585_10deg_58km_c20230430.nc

  • Step 5:  Plot data from E3SM files to check for oddities.

 

 

Notes for updated version c20231207 and c20231222 (E90):

  • The Linoz-v3 data for all Scenarios (Historical, SSP119, SSP126, SSP245, SSP370, SSP434, SSP460, SSP534-over, and SSP585) are now produced on Chrysalis for version c20231207 following the same ways documented here. Filepath for the data are on the E3SM input data server (or Chrysalis) documented below:

    • SSP forcing:

      • https://web.lcrc.anl.gov/public/e3sm/inputdata/atm/cam/chem/trop_mozart/ub/*linv3*c20231207*nc

      • /lcrc/group/acme/public_html/inputdata/atm/cam/chem/trop_mozart/ub/*linv3*c20231207*nc

    • E90:

      • https://web.lcrc.anl.gov/public/e3sm/inputdata/atm/cam/chem/trop_mozart/ub/emissions_E90_surface_1750-2101_1.9x2.5_c20231222.nc

      • lcrc/group/acme/public_html/inputdata/atm/cam/chem/trop_mozart/ub/emissions_E90_surface_1750-2101_1.9x2.5_c20231222.nc

  • There are roundoff differences for Hist, SSP370 and SSP585 of this c20231207 version with the original c20230705 version. This is due to machine difference (difference documented here ) – the c20230730 version is originally produced on SYRAH of LC-Computing that’s now retired, while the new ones are now produced on Chrysalis. Future data will be produced on Chrysalis for continuous support.

  • Correspondent E90 file is also produced in the latest version for all scenarios emissions_E90_surface_1750-2101_1.9x2.5_c20231222.nc.