LMDZ Setup

LMDZ_Setup is a set of scripts that allows a light automatic setup of LMDZ long chained climate simulations, including the installation of the model itself.

LMDZ_Setup can only be used on the Jean-Zay computer at Idris so far.

Coming VERY soon (summer 2024) : extension to other computing centers ("spirit", adastra) and Linux PCs

Documentation (in French): LMDZ_Setup_HowTo

To extract LMDZ_Setup :

a/ Recommended : extraction from svn repository :

   svn co https://svn.lmd.jussieu.fr/LMDZ/BOL/LMDZ_Setup

b/ Download of the "tar" archive :

   wget https://lmdz.lmd.jussieu.fr/pub/Training/LMDZ_Setup.tar

c/ For those used with the old name "tutorial_prod.tar" : a link with this name still exists, pointing to LMDZ_Setup.tar :

   wget https://lmdz.lmd.jussieu.fr/pub/Training/tutorial_prod.tar

(Text extracted from LMDZ_Setup/README0_HowTo):

LMDZ_Setup consists in running LMDZ coupled to Orchidee for land surface (optional), but with imposed sea surface temperature.

The basic default configuration makes use of the IPSL-CM6A grid configuration and tuning, and runs a multi annual simulation on "climatological" amip sea surface temperature (with a mean annual cycle) using a calendar with 360 days.

Optionally : the configuration includes the ability to use interannually varying SST and to activate "nudging" by a reanalysis. In both cases, the calendar is then a real one. When nudging is activated, the simulation must be run on a monthly basis, while otherwise, it can be either monthly or yearly.

Since LMDZ_Setup automatically generates its own initial files, it can be run with zoom configurations by only changing the number of grid points ("resol" in main.sh) and the DEF/gcm.def file (see bellow).

Aerosols can be read for the year 2000 (weighted average over 1999-2001 cf Lurton et al 2020), and instantaneous forcing with respect to 1850 can be computed as well.

LMDZ_Setup can also run LMDZ coupled with the SPLA model (SimPLe Aerosol, activated with option aerosols=spla in setup.sh). Emissions of dust and sea salt are then computed interactively. For the time being, SPLA-specific input files (anthropic aerosol emissions)are only available on the grid zoomed over N Africa used by J. Escribano in his PhD thesis, and by B. Diallo and H. Senghor in subsequent work on N-African dust (file DEF/gcm.def_zNAfrica_BiJe).

A configuration with isotopes is also available since 2023-04.

Below: new LMDZ_Setup docs

Old documentation (French): LMDZ_Setup_HowTo

User guide

 /!\ WARNING 06/2024 - (ignore if installing a new version)
The JeanZay calculator, as well as Adastra, do not allow (anymore) communication from/to $STORE for running jobs on the main computing partitions.
As a result, make sure to replace $STORE by $WORK in lmdz_env.sh wherever relevant.

READ comments at the top of scripts ! Often your questions are answered there.

Note: LMDZ_Setup has been tested on Adastra and Spirit, and to a lesser extent on JeanZay (legacy support only). No guarantee is made on other platforms, although it should be easy to adapt it. It can also be ran locally - for expert use only.

Downloading LMDZ_Setup

/!\ Note: below is for when the new version will replace the old one - for now, use svn co https://svn.lmd.jussieu.fr/LMDZ/BOL/LMDZ_Setup_amaury

(recommended) Using subversion: svn co https://svn.lmd.jussieu.fr/LMDZ/BOL/LMDZ_Setup
(alternative) wget https://lmdz.lmd.jussieu.fr/pub/Training/LMDZ_Setup.tar
(deprecated) The old tutorial_prod version: wget https://lmdz.lmd.jussieu.fr/pub/Training/tutorial_prod.tar

Structure

Once you have downloaded LMDZ_Setup, you will find the following main files:

main.sh: the main script. It contains the basic settings you will modify.
setup.sh: this script contains the internal workings of LMDZ_Setup. It's where you can edit expert-level settings.
lmdz_env.sh: this file contains platform-specific configuration, such as where to install the model, where to run the simulations, etc.

How to run a simulation

Edit lmdz_env.sh. In lmdz_env.sh, in the function set_env, find the case corresponding to your supercomputer: jean-) for JeanZay, spiri) for Spirit, adast) for Adastra. Edit the variables required for your use case. All the variables are documented in the last *) case. In particular:
Edit main.sh. In main.sh, edit all relevant settings.
[optional] Edit setup.sh. In setup.sh, in the function define_expert_options, edit all relevant settings.
[optional] Edit DEF/*. Edit the DEF/*.def or DEF/XMLfiles* files, such as config.def, to your liking. The options you specify in main.sh/setup.sh don't require any adjustment here; such adjustments are automatically performed at runtime.
Launch main.sh.

Miscellanous notes

Simulations run in a subdirectory of $SIMRUNBASEDIR with the same name as the LMDZ_Setup folder (if you have extracted LMDZ_Setup and renamed it to My_Setup then simulations will run in $SIMRUNBASEDIR/My_Setup). Each simulation is contained in its own folder, named after the simulation's name + a process number. As a result, you don't need to clean $SIMRUNBASEDIR before relaunching a simulation.
If your simulation crashes, look directly in $SIMRUNBASEDIR for the relevant log.
If you need a version of ORCHIDEE different from the one provided by default in the model (such as CMIP6), a password is required - ask orchidee-help at listes.ipsl.fr if you need help.

XIOS

/!\ Although XIOS is not required for ORCHIDEE >=2.2, not using it will disable some features.

Different sets of *.xml files are available in LMDZ_Setup/DEF/XMLFiles*. However, only some common files (histf, histday, histmth) have been adapted for LMDZ_Setup. If you need other outputs, be sure to edit/check the relevant *.xml:

replace the _AUTO_ for output_level and enabled by actual values;
use compression_level=0 instead of the default values (2 or 4) (since LMDZ_Setup uses XIOS in attached mode rather than client-server mode, using non-zero compression levels would require running XIOS on 1+ dedicated MPI).

Note: After installing the model, two files are automatically initialized in DEF/XMLfiles*: context_lmdz.xml and field_def_lmdz.xml. Those files don't require user intervention, and must match the exact LMDZ revision used.

Compilation

When launching main.sh, before running the simulation, we first download and compile the required models. Although we call the compilation script every time, if the model has already been compiled before, the script should detect that and skip the actual compilation, and thus execute very quickly.

Note: on supported clusters, compilation is launched in jobs rather than on the login node. This speeds up the compilation.

/!\ As written in the comments of main.sh/setup.sh, if you change the veget version or the aerosols, it is highly recommended to do so in a NEW clean LMDZ_Setup folder rather than recompiling in the same folder.

Nudging

Nudging is set via the nudging variable in main.sh. If activated, then after installing the model, LMDZ_Setup will prompt you to run era2gcm_tuto.sh. This script fetches reanalysis data and puts it in a GUIDE/ folder. Once this is done, simply set init=0 in main.sh and relaunch it.

/!\ On Adastra, the reanalysis files are not yet available.

Aerosols

/!\ As written in setup.sh, it is highly recommended to install LMDZ_Setup in a new clean directory when changing aerosols options. Otherwise i.e. the DEF/config.def may not get modified accordingly.

When aerosols is set to anything other than "n" in setup.sh, aerosols files will be downloaded in LIMIT/, and will be interpolated to the horizontal grid resolution (vertical interpolation is performed at runtime by the GCM.)

More detail

The aerosols files used for the options "n" and "clim" are those used in CMIP6 with the LMDZORINCA configuration, from here.

For "n", we only use aerosols1850_from_inca.nc (natural aerosols), interpolated as aerosols.nat.nc
For "clim", we also use aerosols9999_from_inca.nc with the mean aerosols on 1995-2014, interpolated as aerosols.clim.nc.

ORCHIDEE

The officially supported versions of ORCHIDEE supported by LMDZ_Setup are:

the CMIP6 version (the last version where the use of the old two-layer hydrology is still possible)
version 2.2 r7983
version 4 (trunk) r7994 (corresponding to libIGCM's LMDZOR_6.4work configuration)

Note: The relevant ORCHIDEE def file DEF/orchidee.def_* gets copied as DEF/orchidee.def at initialisation.

Activating sechiba outputs

Sechiba outputs are written to $SIMRUNBASEDIR, but are deleted by tmp_SIM before they're rebuilt. To counter that, remove the sec* on the relevant line set +e ; \rm out* sec* sta* list* rest* gcm.e aer* ; set -e.

Simple routing

From orch 2.2 r7597 onwards, the "simple" routing scheme by Y. Meurdesoif can be activated.

In DEF/orchidee.def_6.*, set RIVER_ROUTING=y, ROUTING_METHOD = simple, RIVER_DESC=y (Note: RIVER_DESC is automatically set to "n" after the first simulation period.)
In DEF/XMLfilesOR7***/iodef.xml, add at the end of context_orchidee.xml: <context id="orchidee" src="./context_routing_orchidee.xml"/>.

See also:User Guide/Routage Simple (French).

Isotopes

To activate isotopes, simply change the lmd_phys parameter to "lmdiso" in main.sh. This will automatically

use "phylmdiso" instead of "phylmd";
copy DEF/PHYS/physiq.def_NPiso to DEF/physiq.def.

Note: for isotopes, iflag_ice_thermo is automatically set to 0 instead of 1.

FAQ

How to relaunch a failed simulation ?

If the simulation ran for at least one successful period, simplys set init=0 in main.sh and rerun it. Otherwise, see the following question.

I have ran a simulation, but I messed up the parameters. I want to run a new clean simulation with the same name (e.g. NPv6.3.)

Remove INIT, LIMIT, NPv6.3 and relaunch main.sh with init=1.

How can I follow the state of a running simulation ?

Check the file LMDZ_Setup/$SIM_NAME/etat. After each period (mo/yr), it gets updated like so:

200001 a faire
200001 OK
200002 a faire

To check if the simulation is still running, query your cluster's queue (e.g. squeue --me on SLURM clusters). If nothing is running, read the next question.

How to investigate a crash ?

The job error log is in $SIMRUNBASEDIR/LMDZ_Setup/out*.

The simulation log is in $SIMRUNBASEDIR/LMDZ_Setup/$SIM_NAME*/listing. (Note: to know which folder is the latest, run ls -lat).

Cluster-specific information

Generic

ORCH 4 (trunk)

ORCHIDEE 4 is currently not able to compile with gfortran, and as a result can't be used. This has been reported to the relevant team, and hopefully should be fixed soon (06/2024).

CDO & aerosols

The current version of cdo on Adastra (2.4.0) and Spirit (2.3.0) contain a bug which interferes with the script interpolating aerosols. This bug is supposedly solved in version 2.4.1+. This bug corrupts the aerosols files, and results in a hgardfou error when running the simulation.

JeanZay

JeanZay was the main supported cluster of the old LMDZ_Setup. Although the new version can be ran there, it is discouraged.

Slowness

Compilation of the various components of the model, in particular of LMDZ and XIOS, is extremely slow on JeanZay compared to other clusters (for unknown reasons) (~15 minutes for LMDZ+rrtm). Moreover, it seems that for uninvestigated reasons, the LMDZ model is systematically recompiled on each call to main.sh (expected behavior, as on other clusters: the fcm system detects that it's already compiled and avoids compiling again). As a result, the whole process of launching a simulation is much slower.

LMDZ Setup

Sommaire