This page should describe Verification and Unit Tests performed for this stand alone feature and should provide links to all the result pages.
Summary
Short summary of what was done and what was the result.
Unit Testing
Unit Testing:
Date last modified:
Contributors:Anthony Craig; Hongyi Li; Ruby Leung
Provenance: (Run provenance Link, Code Tag, etc:)
Results: (link to results, data and plots)
Verification Test 1
Verification Test 1: Testing exact restart
Date last modified:
Contributors: Anthony Craig; Hongyi Li
Provenance:
Testing of exact restart (the “IM” compsets included mosart, and “I” compsets are with rtm)
Results:
PASS ERS.T31_g37.ICLM45.constance_intel
CFAIL ERS.T31_g37.ICLM45.constance_pgi.mos70
PASS ERS.T31_g37.ICN.constance_intel
PASS ERS.T31_g37.ICN.constance_pgi
PASS ERS.T31_g37.IMCLM45.constance_intel
CFAIL ERS.T31_g37.IMCLM45.constance_pgi.mos70
PASS ERS.T31_g37.IMCN.constance_intel
PASS ERS.T31_g37.IMCN.constance_pgi
PASS ERS.T31_g37.X.constance_intel
PASS ERS.T31_g37.X.constance_pgi
PASS NCK.T31_g37.IMCN.constance_intel
PASS NCK.T31_g37.IMCN.constance_pgi
PASS SMS.T31_g37.IMCN.constance_intel
PASS SMS.T31_g37.IMCN.constance_pgi
Verification Test 2
Verification Test 2: Testing of DEBUG, esmf, and multi-instance support
Date last modified:
Contributors: Anthony Craig; Hongyi Li
Provenance:
Results:
PASS CME_D_Ld5.f45_f45.IMCRUCLM50BGC.yellowstone_intel.clm-default.clm2.h0.nc: test compare clm2.h0 (.base and .esmf files)
PASS CME_D_Ld5.f45_f45.IMCRUCLM50BGC.yellowstone_intel.clm-default.clm2.h1.nc: test compare clm2.h1 (.base and .esmf files)
PASS CME_D_Ld5.f45_f45.IMCRUCLM50BGC.yellowstone_intel.clm-default.cpl.hi.nc: test compare cpl.hi (.base and .esmf files)
PASS CME_Ld5.f45_f45.IMCRUCLM50BGC.yellowstone_intel.clm-default.clm2.h0.nc: test compare clm2.h0 (.base and .esmf files)
PASS CME_Ld5.f45_f45.IMCRUCLM50BGC.yellowstone_intel.clm-default.clm2.h1.nc: test compare clm2.h1 (.base and .esmf files)
PASS CME_Ld5.f45_f45.IMCRUCLM50BGC.yellowstone_intel.clm-default.cpl.hi.nc: test compare cpl.hi (.base and .esmf files)
PASS ERR_N2.f19_g16.B1850.yellowstone_intel.allactive-defaultio.cam.h0.nc: test compare cam.h0 (.base for _0001 and .rest for _0001)
PASS ERR_N2.f19_g16.B1850.yellowstone_intel.allactive-defaultio.cam.h0.nc: test compare cam.h0 (.base for _0002 and .rest for _0002)
PASS ERR_N2.f19_g16.B1850.yellowstone_intel.allactive-defaultio.cice.h.nc: test compare cice.h (.base for _0001 and .rest for _0001)
PASS ERR_N2.f19_g16.B1850.yellowstone_intel.allactive-defaultio.cice.h.nc: test compare cice.h (.base for _0002 and .rest for _0002)
PASS ERR_N2.f19_g16.B1850.yellowstone_intel.allactive-defaultio.clm2.h0.nc: test compare clm2.h0 (.base for _0001 and .rest for _0001)
PASS ERR_N2.f19_g16.B1850.yellowstone_intel.allactive-defaultio.clm2.h0.nc: test compare clm2.h0 (.base for _0002 and .rest for _0002)
PASS ERR_N2.f19_g16.B1850.yellowstone_intel.allactive-defaultio.pop.h.nc: test compare pop.h (.base for _0001 and .rest for _0001)
PASS ERR_N2.f19_g16.B1850.yellowstone_intel.allactive-defaultio.pop.h.nc: test compare pop.h (.base for _0002 and .rest for _0002)
PASS ERR_N2.f19_g16.B1850.yellowstone_intel.allactive-defaultio.memleak
PASS ERS_Ld5.f45_f45.IMCRUCLM50BGC.yellowstone_intel.clm-default.clm2.h0.nc: test compare clm2.h0 (.base and .rest files)
PASS ERS_Ld5.f45_f45.IMCRUCLM50BGC.yellowstone_intel.clm-default.clm2.h1.nc: test compare clm2.h1 (.base and .rest files)
PASS ERS_Ld5.f45_f45.IMCRUCLM50BGC.yellowstone_intel.clm-default.cpl.hi.nc: test compare cpl.hi (.base and .rest files)
PASS ERS_Ld5.f45_f45.IMCRUCLM50BGC.yellowstone_intel.clm-default.memleak
PASS ERS_Ld7.f19_g16.B1850.yellowstone_intel.allactive-defaultio.cam.h0.nc: test compare cam.h0 (.base and .rest files)
PASS ERS_Ld7.f19_g16.B1850.yellowstone_intel.allactive-defaultio.cice.h.nc: test compare cice.h (.base and .rest files)
PASS ERS_Ld7.f19_g16.B1850.yellowstone_intel.allactive-defaultio.clm2.h0.nc: test compare clm2.h0 (.base and .rest files)
PASS ERS_Ld7.f19_g16.B1850.yellowstone_intel.allactive-defaultio.pop.h.nc: test compare pop.h (.base and .rest files)
PASS ERS_Ld7.f19_g16.B1850.yellowstone_intel.allactive-defaultio.cpl.hi.nc: test compare cpl.hi (.base and .rest files)
PASS ERS_Ld7.f19_g16.B1850.yellowstone_intel.allactive-defaultio.memleak
PASS NCK_Ld3.f45_f45.IMCRUCLM50BGC.yellowstone_intel.clm-default.clm2.h0.nc: test compare clm2.h0 (.base and .multiinst for _0001)
PASS NCK_Ld3.f45_f45.IMCRUCLM50BGC.yellowstone_intel.clm-default.clm2.h0.nc: test compare clm2.h0 (.base and .multiinst for _0002)
PASS NCK_Ld3.f45_f45.IMCRUCLM50BGC.yellowstone_intel.clm-default.clm2.h1.nc: test compare clm2.h1 (.base and .multiinst for _0001)
PASS NCK_Ld3.f45_f45.IMCRUCLM50BGC.yellowstone_intel.clm-default.clm2.h1.nc: test compare clm2.h1 (.base and .multiinst for _0002)
PASS SMS_D_E_Ld5.f45_f45.IMCRUCLM50BGC.yellowstone_intel.clm-default: successful coupler log
PASS SMS_D_E_Ld5.f45_f45.IMCRUCLM50BGC.yellowstone_intel.clm-default.memleak
PASS SMS_D_Ld5.f45_f45.IMCRUCLM50BGC.yellowstone_gnu.clm-default: successful coupler log
PASS SMS_D_Ld5.f45_f45.IMCRUCLM50BGC.yellowstone_gnu.clm-default.memleak
PASS SMS_D_Ld5.f45_f45.IMCRUCLM50BGC.yellowstone_intel.clm-default: successful coupler log
PASS SMS_D_Ld5.f45_f45.IMCRUCLM50BGC.yellowstone_intel.clm-default.memleak
PASS SMS_D_Ld5.f45_f45.IMCRUCLM50BGC.yellowstone_pgi.clm-default: successful coupler log
PASS SMS_D_Ld5.f45_f45.IMCRUCLM50BGC.yellowstone_pgi.clm-default.memleak
PASS SMS_Ld5.f19_g16.B1850.yellowstone_intel.allactive-defaultio: successful coupler log
PASS SMS_Ld5.f19_g16.B1850.yellowstone_intel.allactive-defaultio.memleak
PASS SMS_Ld5.f45_f45.IMCRUCLM50BGC.yellowstone_gnu.clm-default: successful coupler log
PASS SMS_Ld5.f45_f45.IMCRUCLM50BGC.yellowstone_gnu.clm-default.memleak
PASS SMS_Ld5.f45_f45.IMCRUCLM50BGC.yellowstone_pgi.clm-default: successful coupler log
PASS SMS_Ld5.f45_f45.IMCRUCLM50BGC.yellowstone_pgi.clm-default.memleak
Verification Test 3
Verification Test 3: Testing parallelization
Date last modified:
Contributors: Anthony Craig; Hongyi Li
Provenance:
Results:
Testing parallelization including pe count and decomposition in mosart alone and demonstrated bit for bit results when doing so. This required cpl_decomp = 3 set in the coupler to ensure the coupler decomposition was identical for different mosart pe counts and decompositions.
Verification Test 4
Verification Test 4: Testing of model timing
Date last modified:
Contributors: Anthony Craig; Hongyi Li
Provenance:
Results:
Testing of model timing, on constance, 90 day tests with restart turned off, the intel compiler, comp_run_barriers turned on (to isolate component timing), in I cases (IMCLM45 and ICLM45), with mosart running on the half degree global grid, on 24 tasks, the rof run time (multiple runs shown) is
- rtm: 0.06, 0.06 seconds/day
- mosart: 0.47, 0.52 seconds/day
Independent testing in CESM on yellowstone, suggests a difference in cost between
rtm and mosart of 3x-5x. This is expected given the differences in complexity of the rtm and mosart models.
Even with the 3x-8x increase in mosart cost compared to rtm, the mosart model is still going to be relatively inexpensive compared to other models in the system. In the tests on constance, clm was run at T31 resolution which is 56x lower resolution than mosart (at half degree) and the clm run time was 1.2 seconds/day (or 2.5x higher). On a per grid point basis, mosart is therefore about 140x less expensive than clm45.
Scaling tests were also done on constance for 90 days, no restarts, intel compiler, comp_run_barriers, IMCLM45, mosart on the half degree grid, the rof run time (multiple runs shown) vs pe count is
pe counts | Run1 (s/day) | Run2 (s/day) |
12 | 1.08 | 1.09 |
23 | 0.47 | 0.52 |
48 | 0.25 | 0.25 |
96 | 0.14 | 0.14 |
192 | 0.11 | 0.11 |
384 | 0.11 |
|
Additional information:
MOSART currently supports three different decompositions; basin, 1d, and roundrobin. These are set by a namelist option, decomp_option. The basin decomposition ensures there is no off pe communication in downstream advection, the roundrobin decomp generally provides a well load balanced decomposition, and the 1d decomp is an alternative. mosart cost varies by gridcell because the subcycling timestep depends on input parameters on a gridcell by gridcell basis. In testing, the roundrobin performed the best because it provides the best internal load balance and communication cost is generally about 10% of the total cost at moderate pe counts in the half degree resolution. roundrobin is the default decomposition. All three decompositions result in bit for bit results. Additional decompositions could be implemented that trade off load balance and communication cost further by providing a basin-driven decomposition that estimates cost on a gridcell by gridcell basis and allows for basins to be subdivided to balance cost on pes and minimize communication. This would be most beneficial at higher pe counts where the relative cost of communication is higher. This would almost certainly improve scaling at higher pe counts.
The downstream communication of water in mosart has been implemented using an mct sparse matrix multiply. To ensure bit-for-bit results in the sparse matrix multiply, the sparse matrix multiply is performed with an "Xonly" strategy to eliminate partial sums. This option is controllable via a namelist variable, smat_option.
Tests were performed with settings of "Xonly", "Yonly", and "opt" (which is "XandY") and performance varied minimally with "Xonly" performing best. There are actually two sparse matrix multiplies in mosart. The first is for downstream advection, the second is for transport of water directly to outlet points. Both are derived at initialization from the mosart input file "downstream index" field. Water that is passed into mosart can either be tranported directly to the outlet points immediately or it can be transported downstream incrementally via the mosart solver.