LMDZ Setup
LMDZ_Setup is a set of scripts that allows a light automatic setup of LMDZ long chained climate simulations, including the installation of the model itself.
LMDZ_Setup can only be used on the Jean-Zay computer at Idris so far.
Coming VERY soon (summer 2024) : extension to other computing centers ("spirit", adastra) and Linux PCs
Documentation (in French):
LMDZ_Setup_HowTo
To extract LMDZ_Setup :
a/ Recommended : extraction from svn repository :
svn co https://svn.lmd.jussieu.fr/LMDZ/BOL/LMDZ_Setup
b/ Download of the "tar" archive :
wget https://lmdz.lmd.jussieu.fr/pub/Training/LMDZ_Setup.tar
c/ For those used with the old name "tutorial_prod.tar" : a link with this name still exists, pointing to LMDZ_Setup.tar :
wget https://lmdz.lmd.jussieu.fr/pub/Training/tutorial_prod.tar
(Text extracted from LMDZ_Setup/README0_HowTo):
LMDZ_Setup consists in running LMDZ coupled to Orchidee for land surface (optional), but with imposed sea surface temperature.
The basic default configuration makes use of the IPSL-CM6A grid configuration and tuning, and runs a multi annual simulation on "climatological" amip sea surface temperature (with a mean annual cycle) using a calendar with 360 days.
Optionally : the configuration includes the ability to use interannually varying SST and to activate "nudging" by a reanalysis. In both cases, the calendar is then a real one. When nudging is activated, the simulation must be run on a monthly basis, while otherwise, it can be either monthly or yearly.
Since LMDZ_Setup automatically generates its own initial files, it can be run with zoom configurations by only changing the number of grid points ("resol" in main.sh) and the DEF/gcm.def file (see bellow).
Aerosols can be read for the year 2000 (weighted average over 1999-2001 cf Lurton et al 2020), and instantaneous forcing with respect to 1850 can be computed as well.
LMDZ_Setup can also run LMDZ coupled with the SPLA model (SimPLe Aerosol, activated with option aerosols=spla in setup.sh). Emissions of dust and sea salt are then computed interactively. For the time being, SPLA-specific input files (anthropic aerosol emissions)are only available on the grid zoomed over N Africa used by J. Escribano in his PhD thesis, and by B. Diallo and H. Senghor in subsequent work on N-African dust (file DEF/gcm.def_zNAfrica_BiJe).
A configuration with isotopes is also available since 2023-04.
Below: new LMDZ_Setup docs
Old documentation (French): LMDZ_Setup_HowTo
User guide
/!\ WARNING 06/2024 - (ignore if installing a new version)
The JeanZay calculator, as well as Adastra, do not allow (anymore) communication from/to $STORE for running jobs on the main computing partitions.
As a result, make sure to replace $STORE by $WORK in lmdz_env.sh
wherever relevant.
READ comments at the top of scripts ! Often your questions are answered there.
Note: LMDZ_Setup
has been tested on Adastra and Spirit, and to a lesser extent on JeanZay (legacy support only). No guarantee is made on other platforms, although it should be easy to adapt it. It can also be ran locally - for expert use only.
Downloading LMDZ_Setup
- (recommended) Using subversion:
svn co https://svn.lmd.jussieu.fr/LMDZ/BOL/LMDZ_Setup
- (alternative)
wget https://lmdz.lmd.jussieu.fr/pub/Training/LMDZ_Setup.tar
- (deprecated) The old
tutorial_prod
version:wget https://lmdz.lmd.jussieu.fr/pub/Training/tutorial_prod.tar
Structure
Once you have downloaded LMDZ_Setup
, you will find the following main files:
-
main.sh
: the main script. It contains the basic settings you will modify. -
setup.sh
: this script contains the internal workings ofLMDZ_Setup
. It's where you can edit expert-level settings. -
lmdz_env.sh
: this file contains platform-specific configuration, such as where to install the model, where to run the simulations, etc.
How to run a simulation
- Edit
lmdz_env.sh
. Inlmdz_env.sh
, in the functionset_env
, find the case corresponding to your supercomputer:jean-)
for JeanZay,spiri)
for Spirit,adast)
for Adastra. Edit the variables required for your use case. All the variables are documented in the last*)
case. In particular:- you must set
root_dir
to the path where you extractedLMDZ_Setup
.
- you must set
- Edit
main.sh
. Inmain.sh
, edit all relevant settings. - [optional] Edit
setup.sh
. Insetup.sh
, in the functiondefine_expert_options
, edit all relevant settings. - [optional] Edit
DEF/*
. Edit theDEF/*.def
orDEF/XMLfiles*
files, such asconfig.def
, to your liking. The options you specify in main.sh/setup.sh don't require any adjustment here; such adjustments are automatically performed at runtime. - Launch
main.sh
.
Miscellanous notes
- Simulations run in a subdirectory of
$SIMRUNBASEDIR
with the same name as theLMDZ_Setup
folder (if you have extractedLMDZ_Setup
and renamed it toMy_Setup
then simulations will run in$SIMRUNBASEDIR/My_Setup
). Each simulation is contained in its own folder, named after the simulation's name + a process number. As a result, you don't need to clean $SIMRUNBASEDIR before relaunching a simulation. - If your simulation crashes, look directly in
$SIMRUNBASEDIR
for the relevant log. - If you need a version of ORCHIDEE different from the one provided by default in the model (such as
CMIP6
), a password is required - ask orchidee-help at listes.ipsl.fr if you need help.
XIOS
/!\ Although XIOS is not required for ORCHIDEE >=2.2, not using it will disable some features.
Different sets of *.xml
files are available in LMDZ_Setup/DEF/XMLFiles*
. However, only some common files (histf, histday, histmth
) have been adapted for LMDZ_Setup
. If you need other outputs, be sure to edit/check the relevant *.xml
:
- replace the
_AUTO_
foroutput_level
andenabled
by actual values; - use
compression_level=0
instead of the default values (2
or4
) (sinceLMDZ_Setup
uses XIOS in attached mode rather than client-server mode, using non-zero compression levels would require running XIOS on 1+ dedicated MPI).
Note: After installing the model, two files are automatically initialized inDEF/XMLfiles*
:context_lmdz.xml
andfield_def_lmdz.xml
. Those files don't require user intervention, and must match the exact LMDZ revision used.
Compilation
When launching main.sh
, before running the simulation, we first download and compile the required models. Although we call the compilation script every time, if the model has already been compiled before, the script should detect that and skip the actual compilation, and thus execute very quickly.
Note: on supported clusters, compilation is launched in jobs rather than on the login node. This speeds up the compilation.
/!\ As written in the comments ofmain.sh/setup.sh
, if you change theveget
version or theaerosols
, it is highly recommended to do so in a NEW cleanLMDZ_Setup
folder rather than recompiling in the same folder.
Nudging
Nudging is set via the nudging
variable in main.sh
. If activated, then after installing the model, LMDZ_Setup
will prompt you to run era2gcm_tuto.sh
. This script fetches reanalysis data and puts it in a GUIDE/
folder.
Once this is done, simply set init=0
in main.sh
and relaunch it.
/!\ On Adastra, the reanalysis files are not yet available.
Aerosols
/!\ As written insetup.sh
, it is highly recommended to installLMDZ_Setup
in a new clean directory when changing aerosols options. Otherwise i.e. theDEF/config.def
may not get modified accordingly.
When aerosols
is set to anything other than "n"
in setup.sh
, aerosols files will be downloaded in LIMIT/
, and will be interpolated to the horizontal grid resolution (vertical interpolation is performed at runtime by the GCM.)
More detail
The aerosols files used for the options "n"
and "clim"
are those used in CMIP6 with the LMDZORINCA configuration, from here.
- For
"n"
, we only useaerosols1850_from_inca.nc
(natural aerosols), interpolated asaerosols.nat.nc
- For
"clim"
, we also useaerosols9999_from_inca.nc
with the mean aerosols on 1995-2014, interpolated asaerosols.clim.nc
.
ORCHIDEE
The officially supported versions of ORCHIDEE supported by LMDZ_Setup are:
- the CMIP6 version (still using the two-layer hydrology channel)
- version 2.2 r7983
- version 4 (trunk) r7994 (corresponding to libIGCM's LMDZOR_6.4work configuration)
Note: The relevant ORCHIDEE def fileDEF/orchidee.def_*
gets copied asDEF/orchidee.def
at initialisation.
Activating sechiba outputs
Sechiba outputs are written to $SIMRUNBASEDIR
, but are deleted by tmp_SIM
before they're rebuilt. To counter that, remove the sec*
on the relevant line set +e ; \rm out* sec* sta* list* rest* gcm.e aer* ; set -e
.
Simple routing
From orch 2.2 r7597 onwards, the "simple" routing scheme by Y. Meurdesoif can be activated.
- In
DEF/orchidee.def_6.*
, setRIVER_ROUTING=y
,ROUTING_METHOD = simple
,RIVER_DESC=y
(Note:RIVER_DESC
is automatically set to "n" after the first simulation period.) - In
DEF/XMLfilesOR7***/iodef.xml
, add at the end ofcontext_orchidee.xml
:<context id="orchidee" src="./context_routing_orchidee.xml"/>
.
See also:User Guide/Routage Simple (French).
Isotopes
To activate isotopes, simply change the lmd_phys
parameter to "lmdiso"
in main.sh
. This will automatically
- use
"phylmdiso"
instead of"phylmd"
; - copy
DEF/PHYS/physiq.def_NPiso
toDEF/physiq.def
.
Note: for isotopes, iflag_ice_thermo
is automatically set to 0
instead of 1
.
FAQ
- How to relaunch a failed simulation ?
If the simulation ran for at least one successful period, simplys set init=0
in main.sh
and rerun it. Otherwise, see the following question.
- I have ran a simulation, but I messed up the parameters. I want to run a new clean simulation with the same name (e.g.
NPv6.3
.)
Remove INIT, LIMIT, NPv6.3
and relaunch main.sh
with init=1
.
- How can I follow the state of a running simulation ?
Check the file LMDZ_Setup/$SIM_NAME/etat
. After each period (mo/yr), it gets updated like so:
200001 a faire 200001 OK 200002 a faire
To check if the simulation is still running, query your cluster's queue (e.g. squeue --me
on SLURM clusters). If nothing is running, read the next question.
- How to investigate a crash ?
The job error log is in $SIMRUNBASEDIR/LMDZ_Setup/out*
.
The simulation log is in $SIMRUNBASEDIR/LMDZ_Setup/$SIM_NAME*/listing
. (Note: to know which folder is the latest, run ls -lat
).
Cluster-specific information
Generic
ORCH 4 (trunk)
ORCHIDEE 4 is currently not able to compile with gfortran
, and as a result can't be used. This has been reported to the relevant team, and hopefully should be fixed soon (06/2024).
CDO & aerosols
The current version of cdo
on Adastra (2.4.0
) and Spirit (2.3.0
) contain a bug which interferes with the script interpolating aerosols. This bug is supposedly solved in version 2.4.1+
. This bug corrupts the aerosols files, and results in a hgardfou
error when running the simulation.
JeanZay
JeanZay was the main supported cluster of the old LMDZ_Setup
. Although the new version can be ran there, it is discouraged.
Slowness
Compilation of the various components of the model, in particular of LMDZ and XIOS, is extremely slow on JeanZay compared to other clusters (for unknown reasons) (~15 minutes for LMDZ+rrtm). Moreover, it seems that for uninvestigated reasons, the LMDZ model is systematically recompiled on each call to main.sh
(expected behavior, as on other clusters: the fcm
system detects that it's already compiled and avoids compiling again). As a result, the whole process of launching a simulation is much slower.