Branche Amaury dev

2024-08-01T19:59:09Z

Abarral :

Branche Amaury dev

2024-08-01T19:05:06Z

Abarral :

Abarral : (WIP) Add information regarding new LMDZ_Setup

LMDZ_Setup is a set of scripts that allows a light automatic setup of LMDZ long chained climate simulations, including the installation of the model itself.

LMDZ_Setup can only be used on the Jean-Zay computer at Idris so far.

Coming VERY soon (summer 2024) : extension to other computing centers ("spirit", adastra) and Linux PCs

'''Documentation (in French):'''
[https://docs.google.com/document/d/1OLZG6e-86NiXuv5-aALxKIh-QPkp4BdCwWtiBFot-6c/edit LMDZ_Setup_HowTo]

'''To extract LMDZ_Setup :'''

a/ Recommended : extraction from svn repository :

svn co https://svn.lmd.jussieu.fr/LMDZ/BOL/LMDZ_Setup

b/ Download of the "tar" archive :

wget https://lmdz.lmd.jussieu.fr/pub/Training/LMDZ_Setup.tar

c/ For those used with the old name "tutorial_prod.tar" : a link with this name still exists, pointing to LMDZ_Setup.tar :

wget https://lmdz.lmd.jussieu.fr/pub/Training/tutorial_prod.tar

''(Text extracted from LMDZ_Setup/README0_HowTo):''

LMDZ_Setup consists in running '''LMDZ coupled to Orchidee''' for land surface (optional), but with '''imposed sea surface temperature'''.

The '''basic default configuration''' makes use of the IPSL-CM6A grid configuration and tuning, and runs a multi annual simulation on "climatological"
amip sea surface temperature (with a mean annual cycle) using a calendar with 360 days.

Optionally : the configuration includes the '''ability to use interannually varying SST''' and to activate '''"nudging"''' by a reanalysis.
In both cases, the calendar is then a real one.
When nudging is activated, the simulation must be run on a monthly basis, while otherwise, it can be either monthly or yearly.

Since LMDZ_Setup automatically generates its own initial files, it can be run with '''zoom configurations''' by only changing the number of grid points ("resol" in main.sh) and the DEF/gcm.def file (see bellow).

'''Aerosols''' can be read for the year 2000 (weighted average over 1999-2001 cf Lurton et al 2020), and instantaneous forcing with respect to 1850 can be computed as well.

LMDZ_Setup can also run LMDZ coupled with the '''SPLA model''' (SimPLe Aerosol, activated with option aerosols=spla in setup.sh).
Emissions of dust and sea salt are then computed interactively.
For the time being, SPLA-specific input files (anthropic aerosol emissions)are only available on the grid zoomed over N Africa used by J. Escribano in his PhD thesis, and by B. Diallo and H. Senghor in subsequent work on N-African dust (file DEF/gcm.def_zNAfrica_BiJe).

A configuration with '''isotopes''' is also available since 2023-04.

----

== WIP - Migration de LMDZ_Setup_HowTo + inclusion nouveau LMDZ_Setup ==

''TODO Amaury: reprendre à "SSTs et SIC" + vérifier réponse Adriana pour xios+omp>2 + rajouter commentaire compilation dans jobs dans section "compilation"

''Old documentation (French): [https://docs.google.com/document/d/1OLZG6e-86NiXuv5-aALxKIh-QPkp4BdCwWtiBFot-6c LMDZ_Setup_HowTo]''

== User guide ==

''' /!\ WARNING''' 06/2024 - (ignore if installing a new version)
The JeanZay calculator, as well as Adastra, do not allow (anymore) communication from/to $STORE for running jobs on the main computing partitions.
As a result, make sure to replace $STORE by $WORK in <code>lmdz_env.sh</code> wherever relevant.

'''READ comments at the top of scripts ! Often your questions are answered there.'''

''Note'': <code>LMDZ_Setup</code> has been tested on Adastra and Spirit, and to a lesser extent on JeanZay (legacy support only). No guarantee is made on other platforms, although it should be easy to adapt it. It can also be ran locally - for expert use only.

=== Downloading LMDZ_Setup ===

* ''(recommended)'' Using ''subversion'': <code>svn co https://svn.lmd.jussieu.fr/LMDZ/BOL/LMDZ_Setup</code>
* ''(alternative)'' <code>wget https://lmdz.lmd.jussieu.fr/pub/Training/LMDZ_Setup.tar</code>
* ''(deprecated)'' The old <code>tutorial_prod</code> version: <code>wget https://lmdz.lmd.jussieu.fr/pub/Training/tutorial_prod.tar</code>

=== Structure ===

Once you have downloaded <code>LMDZ_Setup</code>, you will find the following main files:
* <code>main.sh</code>: the main script. It contains the basic settings you will modify.
* <code>setup.sh</code>: this script contains the internal workings of <code>LMDZ_Setup</code>. It's where you can edit expert-level settings.
* <code>lmdz_env.sh</code>: this file contains platform-specific configuration, such as where to install the model, where to run the simulations, etc.

=== How to run a simulation ===

# '''Edit <code>lmdz_env.sh</code>.''' In <code>lmdz_env.sh</code>, in the function <code>set_env</code>, find the case corresponding to your supercomputer: <code>jean-)</code> for JeanZay, <code>spiri)</code> for Spirit, <code>adast)</code> for Adastra. Edit the variables required for your use case. All the variables are documented in the last <code>*)</code> case. In particular:
#* you must set <code>root_dir</code> to the path where you extracted <code>LMDZ_Setup</code>.
# '''Edit <code>main.sh</code>.''' In <code>main.sh</code>, edit all relevant settings.
# '''''[optional]'' Edit <code>setup.sh</code>.''' In <code>setup.sh</code>, in the function <code>define_expert_options</code>, edit all relevant settings.
# '''''[optional]'' Edit <code>DEF/*</code>.''' Edit the <code>DEF/*.def</code> or <code>DEF/XMLfiles*</code> files, such as <code>config.def</code>, to your liking. ''The options you specify in main.sh/setup.sh don't require any adjustment here; such adjustments are automatically performed at runtime.''
# '''Launch <code>main.sh</code>'''.

=== Miscellanous notes ===

* Simulations run in a subdirectory of <code>$SIMRUNBASEDIR</code> with the same name as the <code>LMDZ_Setup</code> folder (if you have extracted <code>LMDZ_Setup</code> and renamed it to <code>My_Setup</code> then simulations will run in <code>$SIMRUNBASEDIR/My_Setup</code>). Each simulation is contained in its own folder, named after the simulation's name + a process number. As a result, '''you don't need to clean $SIMRUNBASEDIR before relaunching a simulation'''.
* If your simulation crashes, look directly in <code>$SIMRUNBASEDIR</code> for the relevant log.
* If you need a version of ORCHIDEE different from the one provided by default in the model (such as <code>CMIP6</code>), a password is required - ask ''orchidee-help at listes.ipsl.fr'' if you need help.

==== XIOS ====

'''/!\''' Although XIOS is not ''required'' for ORCHIDEE >=2.2, not using it will disable some features.
Different sets of <code>*.xml</code> files are available in <code>LMDZ_Setup/DEF/XMLFiles*</code>. However, only some common files (<code>histf, histday, histmth</code>) have been adapted for <code>LMDZ_Setup</code>. ''If you need other outputs, '''be sure to edit/check the relevant <code>*.xml</code>''''':
* replace the <code>_AUTO_</code> for <code>output_level</code> and <code>enabled</code> by actual values;
* use <code>compression_level=0</code> instead of the default values (<code>2</code> or <code>4</code>) (since <code>LMDZ_Setup</code> uses XIOS in attached mode rather than client-server mode, using non-zero compression levels would require running XIOS on 1+ dedicated MPI).

''Note:'' After installing the model, two files are automatically initialized in <code>DEF/XMLfiles*</code>: <code>context_lmdz.xml</code> and <code>field_def_lmdz.xml</code>. Those files don't require user intervention, and must match the exact LMDZ revision used.

==== Compilation ====

When launching <code>main.sh</code>, before running the simulation, we first download and compile the required models. Although we call the compilation script every time, if the model has already been compiled before, the script should detect that and skip the actual compilation, and thus execute very quickly.

''Note: on supported clusters, compilation is launched in jobs rather than on the login node. This speeds up the compilation.''

'''/!\''' As written in the comments of <code>main.sh/setup.sh</code>, '''if you change the <code>veget</code> version or the <code>aerosols</code>, it is highly recommended to do so in a NEW clean <code>LMDZ_Setup</code> folder''' rather than recompiling in the same folder.

=== Nudging ===

Nudging is set via the <code>nudging</code> variable in <code>main.sh</code>. If activated, then after installing the model, <code>LMDZ_Setup</code> will prompt you to run <code>era2gcm_tuto.sh</code>. This script fetches reanalysis data and puts it in a <code>GUIDE/</code> folder.
Once this is done, simply set <code>init=0</code> in <code>main.sh</code> and relaunch it.

'''/!\''' On Adastra, the reanalysis files are not yet available.

=== Aerosols ===

'''/!\ As written in <code>setup.sh</code>, it is highly recommended to install <code>LMDZ_Setup</code> in a new clean directory when changing aerosols options. Otherwise i.e. the <code>DEF/config.def</code> may not get modified accordingly.'''

When <code>aerosols</code> is set to anything other than <code>"n"</code> in <code>setup.sh</code>, aerosols files will be downloaded in <code>LIMIT/</code>, and will be interpolated to the horizontal grid resolution (vertical interpolation is performed at runtime by the GCM.)

==== More detail ====

The aerosols files used for the options <code>"n"</code> and <code>"clim"</code> are those used in CMIP6 with the LMDZORINCA configuration, from [https://www.lmd.jussieu.fr/~lmdz/pub/3DInputData/AerChem/ here].
* For <code>"n"</code>, we only use <code>aerosols1850_from_inca.nc</code> (natural aerosols), interpolated as <code>aerosols.nat.nc</code>
* For <code>"clim"</code>, we also use <code>aerosols9999_from_inca.nc</code> with the mean aerosols on 1995-2014, interpolated as <code>aerosols.clim.nc</code>.

=== ORCHIDEE ===

The officially supported versions of ORCHIDEE supported by LMDZ_Setup are:
* the CMIP6 version (still using the two-layer hydrology channel)
* version 2.2 r7983
* version 4 (trunk) r7994 (corresponding to libIGCM's LMDZOR_6.4work configuration)

''Note:'' The relevant ORCHIDEE def file <code>DEF/orchidee.def_*</code> gets copied as <code>DEF/orchidee.def</code> at initialisation.

==== Activating sechiba outputs ====

Sechiba outputs are written to <code>$SIMRUNBASEDIR</code>, but are deleted by <code>tmp_SIM</code> before they're rebuilt. To counter that, remove the <code>sec*</code> on the relevant line <code>set +e ; \rm out* sec* sta* list* rest* gcm.e aer* ; set -e</code>.

==== Simple routing ====

From orch 2.2 r7597 onwards, the "simple" routing scheme by Y. Meurdesoif can be activated.

* In <code>DEF/orchidee.def_6.*</code>, set <code>RIVER_ROUTING=y</code>, <code>ROUTING_METHOD = simple</code>, <code>RIVER_DESC=y</code> (''Note: <code>RIVER_DESC</code> is automatically set to "n" after the first simulation period.)
* In <code>DEF/XMLfilesOR7***/iodef.xml</code>, add at the end of <code>context_orchidee.xml</code>: <code><context id="orchidee" src="./context_routing_orchidee.xml"/></code>.

See also:[https://forge.ipsl.jussieu.fr/orchidee/wiki/Documentation/UserGuide/RoutageSimple User Guide/Routage Simple] (French).

=== Isotopes ===

To activate isotopes, simply change the <code>lmd_phys</code> parameter to <code>"lmdiso"</code> in <code>main.sh</code>. This will automatically
* use <code>"phylmdiso"</code> instead of <code>"phylmd"</code>;
* copy <code>DEF/PHYS/physiq.def_NPiso</code> to <code>DEF/physiq.def</code>.

''Note: for isotopes, <code>iflag_ice_thermo</code> is automatically set to <code>0</code> instead of <code>1</code>.''

== Cluster-specific information ==

=== Generic ===

==== ORCH 4 (trunk) ====

ORCHIDEE 4 is currently not able to compile with <code>gfortran</code>, and as a result can't be used. This has been reported to the relevant team, and hopefully should be fixed soon (06/2024).

==== CDO & aerosols ====

The current version of <code>cdo</code> on Adastra (<code>2.4.0</code>) and Spirit (<code>2.3.0</code>) contain [https://code.mpimet.mpg.de/boards/1/topics/15752 a bug] which interferes with the script interpolating aerosols. This bug is supposedly solved in version <code>2.4.1+</code>. This bug corrupts the aerosols files, and results in a <code>hgardfou</code> error when running the simulation.

=== JeanZay ===

JeanZay was the main supported cluster of the old <code>LMDZ_Setup</code>. Although the new version can be ran there, it is discouraged.

==== Slowness ====

Compilation of the various components of the model, in particular of LMDZ and XIOS, is extremely slow on JeanZay compared to other clusters (for unknown reasons) (~15 minutes for LMDZ+rrtm). Moreover, it seems that for uninvestigated reasons, the LMDZ model is systematically recompiled on each call to <code>main.sh</code> (expected behavior, as on other clusters: the <code>fcm</code> system detects that it's already compiled and avoids compiling again). As a result, the whole process of '''launching a simulation is much slower'''.

LMDZ Setup

2024-07-08T14:52:07Z

Abarral : add todo

LMDZ Setup

2024-07-08T14:51:06Z

Abarral : typo

LMDZ Setup

2024-07-08T14:48:13Z

Abarral : (WIP) Add information regarding new LMDZ_Setup

LMDZ Setup

2024-07-08T13:12:28Z

Abarral : (WIP) Add information regarding new LMDZ_Setup