Difference between revisions of "Advanced Use of the GCM"

Revision as of 16:39, 22 July 2022

Running in parallel

For large simulation (long run, high resolution etc...), the computational cost can be huge and hence the run time very long. To overcome this issue, the model can be run in parallel. This however requires a few extra steps (compared to compiling and running the serial version of the code). For all the details see the dedicated page.

Disambiguation between ifort, mpif90, etc.

For users not used to compilers and/or compiling and running codes in parallel, namely in MPI mode, there is often some confusion which hopefully the following paragraph might help clarify:

• the compiler (typically gfortran, ifort, pgfortran, etc.) is the required tool to compile the Fortran source code and generate an executable. It is strongly recommended that libraries used by a program are also compiled using the same compiler. Thus if you plan to use different compilers to compile the model, note that you should also have at hand versions of the libraries it uses also compiled with these compilers.
• the MPI (Message Passing Interface) library is a library used to solve problems using multiple processes by enabling message-passing between the otherwise independent processes. There are a number of available MPI libraries out there, e.g. OpenMPI, MPICH or IntelMPI to name a few. The important point here is that on a given machine the MPI library is related to a given compiler and that it provides related wrappers to compile and run with. Typically (but not always) the compiler wrapper is mpif90 and the execution wrapper is mpirun. If you want to know which compiler is wrapped in the mpif90 compiler wrapper, check out the output of:

mpif90 --version

• In addition a second type of parallelism, shared memory parallelism known as OpenMP, is also implemented in the code. In contradistinction to MPI, OpenMP does not require an external library but is instead implemented as a compiler feature. At run time one must then specify some dedicated environment variables (such as OMP_NUM_THREADS and OMP_STACKSIZE) to specify the number of threads to use per process.
• In practice one should favor compiling and running with both MPI and OPenMP enabled.
• For much more detailed information about compiling and running in parallel, check out the the page dedicated to Parallelism.

A word about the IOIPSL and XIOS libraries

• The IOIPSL (Input Output IPSL) library is a library that has developed by the IPSL community to handle input and outputs of (mostly terrestrial) climate models. For the Generic PCM only a small part of this library is actually used, related to reading and processing the input run.def file. For more details check out the The IOIPSL Library page.
• The XIOS (Xml I/O Server) library is based on client-server principles where the server manages the outputs asynchronously from the client (the climate model) so that the bottleneck of writing data in a parallel environment is alleviated. All aspects of the outputs (name, units, file, post-processing operations, etc.) are then controlled by dedicated XML files which are read at run-time. Using XIOS is currently optional (and requires compiling the GCM with the XIOS library). More about the XIOS library, how to install and use it, etc. here.

Playing with the output files

TBD by Martin

Changing the output temporal resolution

in run.def

Changing the output variable

diagfi.def

add a list of all variables

hard coded in physiq_mod.F90

Spectral outputs

diagspec_VI.nc diagspec_IR.nc

Statistical outputs

stats.nc

How to Change Vertical and Horizontal Resolutions

When you are using the regular longitude/latitude horizontal grid

To run at a different grid resolution than available initial conditions files, one needs to use the tools newstart.e and start2archive.e

For example, to create initial states at grid resolution 32×24×25 from NetCDF files start and startfi at grid resolution 64×48×32 :

• Create file start_archive.nc with start2archive.e compiled at grid resolution 64×48×32 using old file z2sig.def used previously
• Create files restart.nc and restartfi.nc with newstart.e compiled at grid resolution 32×24×25, using a new file z2sig.def (more details below on the choice of the z2sig.def).
• While executing newstart.e, you need to choose the answer '0 - from a file start_archive' and then press enter to all other requests.

What you need to know about the z2sig.def file

For a model with Nlay layers, the z2sig.def file must contain at least Nlay+1 lines (the other not being read).

The first line is a scale height ($$H$$). The following lines are the target pseudo-altitudes for the model from the bottom up ($$z_i$$). The units do not matter as long as you use the same ones for both.

The model will use these altitudes to compute a target pressure grid ($$p_i$$ ) as follows: \begin{align} \label{def:pseudoalt} p_i &= p_s \exp(-z_i/H), \end{align} where $$p_s$$ is the surface pressure.

As you can see, the scale height and pseudo altitudes enter the equation only through their ratio. So they do not have to to be the real scale-height and altitudes of the atmosphere you are simulating. So you can use the same z2sig.def for different planets.

There is no hard rule to follow to determine the altitude/pressure levels you should use. As a rule of thumb, layers should be thiner near the surface to properly resolve the surface boundary layer. Then they should gradually increase in size over a couple scale heights and transition to constant thickness above that. Of course, some specific applications may require thinner layers in some specific parts of the atmospheres.

A little trick for those who prefer to think in terms of (log)pressure: if you use $$H= \ln 10 \approx 2.30259$$, then $$z_i=x$$ corresponds to a pressure difference with the surface of exactly x pressure decades. This is particularly useful for giant-planet applications.

When you are using the DYNAMICO icosahedral horizontal grid

The horizontal resolution for the DYNAMICO dynamical core is managed from several setting files, online during the execution. To this purpose, each part of the GCM managing the in/output fields (ICOSAGCM, ICOSA_LMDZ, XIOS) requires to know the input and output grids:

1. context_lmdz_physics.xml:

You can find several grid setup already defined:

 1 <domain_definition>
 2     <domain id="dom_96_95" ni_glo="96" nj_glo="95" type="rectilinear"  >
 3       <generate_rectilinear_domain/>
 4       <interpolate_domain order="1"/>
 5     </domain>
 6 
 7     <domain id="dom_144_142" ni_glo="144" nj_glo="142" type="rectilinear"  >
 8       <generate_rectilinear_domain/>
 9       <interpolate_domain order="1"/>
10     </domain>
11 
12     <domain id="dom_512_360" ni_glo="512" nj_glo="360" type="rectilinear"  >
13       <generate_rectilinear_domain/>
14       <interpolate_domain order="1"/>
15     </domain>
16 
17     <domain id="dom_720_360" ni_glo="720" nj_glo="360" type="rectilinear">
18       <generate_rectilinear_domain/>
19       <interpolate_domain order="1"/>
20     </domain>
21 
22     <domain id="dom_out" domain_ref="dom_720_360"/>
23 </domain_definition>

In this example, the output grid for the physics fields is defined by

<domain id="dom_out" domain_ref="dom_720_360"/>

which is an half-degree horizontal resolution. To change this resolution, you have to change name of the domain_ref grid, for instance:

<domain id="dom_out" domain_ref="dom_96_95"/>

2. run_icosa.def: setting file to execute a simulation

In this file, regarding of the horizontal resolution intended, you have to set the number of subdivision on the main triangle. For reminder, each hexagonal mesh is divided in several main triangles and each main triangles are divided in suitable number of sub-triangles according the horizontal resolution

1 #nbp --> number of subdivision on a main triangle: integer (default=40)
2 #              nbp = sqrt((nbr_lat x nbr_lon)/10)
3 #              nbp:                 20   40   80  160
4 #              T-edge length (km): 500  250  120   60
5 #              Example: nbp(128x96) = 35 -> 40
6 #                       nbp(256x192)= 70 -> 80
7 #                       nbp(360x720)= 160 -> 160
8 nbp = 160

If you have chosen the 96_95 output grid in context_lmdz_physics.xml, you have to calculate $$nbp = \sqrt(96x95) / 10 = 10$$ and in this case

nbp = 20

After the number of subdivision of the main triangle, you have to define the number subdivision over each direction. At this stage you need to be careful as the number of subdivisions on each direction:

• needs to be set according to the number of subdivisions on the main triangle nbp
• will determine the number of processors on which the GCM will be most effective

 1 ## sub splitting of main rhombus : integer (default=1)
 2 #nsplit_i=1
 3 #nsplit_j=1
 4 #omp_level_size=1
 5 ###############################################################
 6 ## There must be less MPIxOpenMP processes than the 10 x nsplit_i x nsplit_j tiles
 7 ## typically for pure MPI runs, let nproc = 10 x nsplit_i x nsplit_j
 8 ## it is better to have nbp/nsplit_i  > 10 and nbp/nplit_j > 10
 9 ###############################################################
10 #### 40 noeuds de 24 processeurs = 960 procs
11 nsplit_i=12
12 nsplit_j=8
13 
14 #### 50 noeuds de 24 processeurs = 1200 procs
15 #nsplit_i=10
16 #nsplit_j=12

With the same example as above, the 96_95 output grid requires: $$nsplit_i < 2$$ and $$nsplit_j < 2$$ We advise you to select:

## sub splitting of main rhombus : integer (default=1)
nsplit_i=1
nsplit_j=1

and using 10 processors.

How to Change the Topography (or remove it)

The generic model can use in principle any type of surface topography, provided that the topographic data file is available in the right format, and put in the right place. The information content on the surface topography is contained in the startfi.nc, and we do have developed tools (see below) to modify the startfi.nc to account for a new surface topography.

To change the surface topography of a simulation, we recommend to follow the procedure detailed below:

• Create file start_archive.nc with start2archive.e compiled at the same (horizontal and vertical) resolution than the start.nc and startfi.nc files.
• Create files restart.nc and restartfi.nc with newstart.e compiled again at the same (horizontal and vertical) resolution.
• While executing newstart.e, you need to choose the answer '0 - from a file start_archive' and then press enter to all other requests.
• At some point, the script newstart.e asks you to chose the surface topography you want from the list of files available in your 'datagcm/surface_data/' directory.

We do have a repository of for Venus, Earth and Mars through time available at https://web.lmd.jussieu.fr/~lmdz/planets/LMDZ.GENERIC/datagcm/surface_data/. You can download the surface topography files and place them in your 'datagcm/surface_data/' directory.

Special note: How to remove the topography?

TBD by Gwenael?

How to Change the Stellar Spectrum

TO BE COMPLETED by Martin (once Martin has updated the code with the modifications regarding stellar spectrum input)

Black Body Stellar Spectra

Pre-Tabulated spectra

How to Change the Opacity Tables

To change the opacity tables (generated "offline" for a given set of pressures, temperatures, a given composition and a specific spectral decomposition), follow these steps:

• copy your directory containing your opacity tables in .... It has to contain...
• change corrkdir = ... in callphys.def with the name of that directory.
• change gases.def: it has to be consistent with Q.dat. A special case concerns...
• change -b option when compiling the model with makelmdz_fcm: it has to correspond to the number of bands (in the IR x in the visible) of the new opacity tables. For instance, compile with -b 38x26 if you used 38 bands in the infrared and 26 in the visible to generate the opacity tables.

How to Manage Tracers

Tracers are managed thanks to the traceur.def file.

Specific treatment of some tracers (e.g., water vapor cycle) can be added directly in the model and an option added in callphys.def file.

Use the Z of LMDz : Zoomed version

Do we need this? Has anyone already made use of the zoom module?