L7_UQ_efficiency_improvements Performance assessment phase 1

This page should describe Performance Assessment Tests performed for this stand alone feature and should provide links to all the result pages.

Summary

Performance testing for coupler bypass offline land simulations vs. standard offline land simulations using DATM.

Performance test 1 - Global test

For the standard simulations with DATM:

Compset:  I1850CLM45CN, Resolution:  f09_f09, Machine:  Titan, simulation length:  3 years

For the coupler bypass simulation:

Compset:  I1850CLM45CBCN, Resolution:  f09_f09, Machine:  Titan, simulation length:  3 years

For simulations using 512 cores, the coupler bypass speeds up the simulation by about 30%, primarily by eliminating the time spent in DATM.  The coupler bypass option improves the scalability of offline runs significantly.  When using 2048 cores, the walltime used in the standard offline simulation using DATM does not change, while the coupler bypass simulation is about 3x faster than coupler bypass with 512 cores and nearly 5x faster than DATM with 512 or 2048 cores.

Performance test 2 - Single point test

For the standard simulations with DATM:

Compset:  I1850CLM45CN, Resolution:  CLM1PT, Machine:  OICphase2, simulation length:  100 years

For the coupler bypass simulation:

Compset:  I1850CLM45CBCN, Resolution:  CLM1PT, Machine:  OICphase2, simulation length:  100 years

The coupler bypass simulation is slightly under 2.5x faster than the standard DATM simulation on a single processor.  More detailed performance data are available below.  Even excluding DATM, there is a significant performance benefit when using the coupler bypass option within the land model.  This is because the coupler bypass option also avoids the use of stream files for nitrogen deposition and fire input data, which are responsible for nearly half of the run time within the land model loop.  Note that these simulations were conducted with a 1-hour timestep and annual output data for a subset of about 30 output variables.

Performance Test 1

Performance Test 1:  Coupler bypass:  Global test on Titan

Date last modified:  

Contributors: Daniel Ricciuto

Provenance:

ACME git hash:  4532ebff59a9ba93332e4c024e0728918e5a84d4

Results: 

Case directory location on Titan:

/lustre/atlas/proj-shared/cli112/zdr/models/ACME/cime/scripts/TEST_GLOBPERF/   (standard run with DATM)

/lustre/atlas/proj-shared/cli112/zdr/models/ACME/cime/scripts/TEST_GLOBPERF_CB/ (coupler bypass run)

 

 

CPL BYPASS         DATM        
512 cores         512 cores        
TOTRunTime:3960.409seconds3.617seconds/mday65.45myears/wday TOTRunTime:5875.978seconds5.366seconds/mday44.11myears/wday
LNDRunTime:3787.495seconds3.459seconds/mday68.44myears/wday LNDRunTime:4115.522seconds3.758seconds/mday62.98myears/wday
ROFRunTime:0seconds0seconds/mday0myears/wday ROFRunTime:0seconds0seconds/mday0myears/wday
ICERunTime:0seconds0seconds/mday0myears/wday ICERunTime:0seconds0seconds/mday0myears/wday
ATMRunTime:0seconds0seconds/mday0myears/wday ATMRunTime:1640.854seconds1.498seconds/mday157.97myears/wday
OCNRunTime:0seconds0seconds/mday0myears/wday OCNRunTime:0seconds0seconds/mday0myears/wday
GLCRunTime:0seconds0seconds/mday0myears/wday GLCRunTime:0seconds0seconds/mday0myears/wday
WAVRunTime:0seconds0seconds/mday0myears/wday WAVRunTime:0seconds0seconds/mday0myears/wday
CPLRunTime:4.985seconds0.005seconds/mday51995.99myears/wday CPLRunTime:85.33seconds0.078seconds/mday3037.62myears/wday
CPLCOMMTime:1207.525seconds1.103seconds/mday214.65myears/wday CPLCOMMTime:1243.335seconds1.135seconds/mday208.47myears/wday
2048 cores         2048 cores        
TOTRunTime:1334.271seconds1.219seconds/mday194.26myears/wday TOTRunTime:5829.534seconds5.324seconds/mday44.46myears/wday
LNDRunTime:1121.122seconds1.024seconds/mday231.2myears/wday LNDRunTime:2108.362seconds1.925seconds/mday122.94myears/wday
ROFRunTime:0seconds0seconds/mday0myears/wday ROFRunTime:0seconds0seconds/mday0myears/wday
ICERunTime:0seconds0seconds/mday0myears/wday ICERunTime:0seconds0seconds/mday0myears/wday
ATMRunTime:0seconds0seconds/mday0myears/wday ATMRunTime:3534.852seconds3.228seconds/mday73.33myears/wday
OCNRunTime:0seconds0seconds/mday0myears/wday OCNRunTime:0seconds0seconds/mday0myears/wday
GLCRunTime:0seconds0seconds/mday0myears/wday GLCRunTime:0seconds0seconds/mday0myears/wday
WAVRunTime:0seconds0seconds/mday0myears/wday WAVRunTime:0seconds0seconds/mday0myears/wday
CPLRunTime:2.491seconds0.002seconds/mday104054.62myears/wday CPLRunTime:74.085seconds0.068seconds/mday3498.68myears/wday
CPLCOMMTime:581.574seconds0.531seconds/mday445.69myears/wday CPLCOMMTime:1020.487seconds0.932seconds/mday254myears/wday


Performance Test 2

Performance Test 2:  Coupler bypass:  Single-point test on OIC

Date last modified:  

Contributors: Daniel Ricciuto

Provenance: 

ACME git hash:  4532ebff59a9ba93332e4c024e0728918e5a84d4

Results: 

Location on OIC:

/home/zdr/models/ACME/cime/scripts/FULL_US-UMB_I1850CLM45CBCN_ad_spinup (with DATM)

/home/zdr/models/ACME/cime/scripts/FULL_US-UMB_I1850CLM45CN_ad_spinup (cpl bypass)

 

nameCPL BYPASS with DATM
CPL:INIT0.1470.725
CPL:cesm_pre_init20.0070.007
cesm_init0.140.717
CPL:RUN_LOOP_BSTART00
CPL:RUN_LOOP923.7572202.374
CPL:CLOCK_ADVANCE80.01576.334
CPL:RUN6.4224.262
CPL:LNDPREP2.30717.871
CPL:COMM37.52755.808
CPL:C2L18.46417.894
CPL:LND_RUN792.311444.823
lc_lnd_import3.4720.623
clm_run697.9831357.342
shr_orb_decl1.9172.114
decomp_vert11.69511.645
begcnbal14.59913.727
dyn_subgrid57.58957.35
begwbal0.6420.613
ndep_interpN/A507.868
ndepdyn_strd_adv_readLBUBN/A2.836
ndepdyn_readLBUB_setupN/A0.415
ndepdyn_readLBUB_filemgtN/A0.309
ndepdyn_strd_adv_tintN/A124.211
hdmdyn_strd_adv_readLBUBN/A2.28
hdmdyn_readLBUB_setupN/A0.389
hdmdyn_readLBUB_filemgtN/A0.249
hdmdyn_strd_adv_tintN/A63.261
lnfmdyn_strd_adv_readLBUBN/A296.355
lnfmdyn_readLBUB_setupN/A0.364
lnfmdyn_readLBUB_filemgtN/A0.414
lnfmdyn_strd_adv_tintN/A3.673
pdnep_interpN/A0
pdepdyn_strd_adv_readLBUBN/A2.356
pdepdyn_readLBUB_setupN/A0.366
pdepdyn_readLBUB_filemgtN/A0.294
pdepdyn_strd_adv_tintN/A122.987
drvinit2.3362.191
canhydro2.9012.686
surfrad5.3435.535
bgp12.0542.172
bgflux2.4622.555
canflux60.0160.758
can_iter45.26145.74
uflux6.446.896
bgplake27.75928.612
bgc15.80915.817
soiltemperature37.17738.237
SoilThermProp4.5294.635
SoilTempBandDiag6.3126.65
PhaseChangeH2osfc0.4140.421
PhaseChangebeta2.3052.307
bgp23.4083.465
bgp2_loop_10.3880.384
bgp2_loop_20.3010.314
bgp2_loop_30.4990.496
bgp2_loop_40.3720.336
patch2col1.8691.877
hydro without drainage32.52333.459
hylake7.4228.481
snow_init0.3260.317
ecosysdyn200.648203.608
CNZero25.43525.542
CNDeposition0.150.171
CNFixation2.1712.161
CNMResp2.7542.733
PDeposition0.1360.135
CNAllocation3.4743.597
CNDecompAlloc53.61154.663
CNAllocation9.2729.74
CNAllocation2.8953.04
CNPhenology8.1348.341
CNGResp0.4680.455
CNRootDyn5.8655.513
CNUpdate00.2040.184
CNUpdate19.87910.134
CNSoilLittVertTransp43.63543.744
CNGapMortality6.0316.137
CNUpdate225.28525.855
depvel0.9620.961
ch453.654.427
hydro24.0394.195
PWeathering1.6381.629
PAdsorption1.9932.04
PDesorption1.8681.83
POcclusion1.8611.818
PBiochemMin12.12111.668
CNUpdate31.5671.625
PUpdate34.3354.461
CNPsum25.22526.307
balchk1.982.203
lnd2atm12.90112.685
wrtdiag0.10710.195
hbuf29.41329.942
clm_drv_io7.9768.557
clm_drv_io_htapes7.2667.892
hist_htapes_wrapup_define3.7224.161
hist_htapes_wrapup_tconst0.0850.056
hist_htapes_wrapup_write0.1730.185
lc_lnd_exportN/A0.393
lc_clm2_adv_timestep0.5150.434
accum15.816.486
CPL:L2C18.40218.329
CPL:LNDPOST0.2060.27
CPL:FRACSET1.7721.895
CPL:ATM_RUNN/A589.837
DATM_RUNN/A554.634
datm_run1N/A9.802
datmN/A533.357
datm_strdata_advanceN/A470.285
datm_strd_adv_readLBUBN/A337.459
datm_readLBUB_setupN/A0.854
datm_readLBUB_filemgtN/A0.778
datm_strd_adv_tintN/A126.035
datm_scatterN/A53.158
datm_modeN/A0.72
datm_run2N/A10.739
CPL:A2CN/A18.6
CPL:ATMPOSTN/A0.25
CPL:HISTORY0.2250.242
CPL:TSTAMP_WRITE0.3591.557
CPL:TPROF_WRITE0.3760.544
lnfmdyn_readLBUB_fboundN/A4.668
lnfmdyn_readLBUB_bcastN/A0.181
lnfmdyn_readLBUB_LB_copyN/A20.906
lnfmdyn_readLBUB_UB_setupN/A8.125
lnfmdyn_readLBUB_UB_readpioN/A254.055
lnfmdyn_strd_adv_mapN/A4.98
cnbalchk1.2641.318
surfalb15.77316.172
urbsurfalb11.17611.566
datm_readLBUB_fboundN/A4.572
datm_readLBUB_bcastN/A0.309
datm_readLBUB_LB_setupN/A13.269
datm_readLBUB_LB_readpioN/A144.641
datm_readLBUB_UB_setupN/A12.614
datm_readLBUB_UB_readpioN/A141.815
datm_strd_adv_rearrN/A1.858
datm_readLBUB_LB_copyN/A0.011
datm_strd_adv_fillN/A0.081
datm_strd_adv_mapN/A0
hdmdyn_readLBUB_fboundN/A0
hdmdyn_readLBUB_bcastN/A0
hdmdyn_readLBUB_LB_copyN/A0.001
hdmdyn_readLBUB_UB_setupN/A0
hdmdyn_readLBUB_UB_readpioN/A0.015
hdmdyn_strd_adv_mapN/A0
sync1_tprof00
t_prf0.3290.485
sync2_tprof00
clm_drv_io_wrest0.3580.289
datm_restartN/A0.186
CPL:RESTART0.0320.08
CPL:RUN_LOOP_BSTOP00
CPL:FINAL0.0010.008
DATM_FINALN/A0