Revision as of 10:44, 20 June 2024

Venus - DYNAMICO

Dynamico is the recently developed Dynamical core, enabling better performance and solving some issues of the LMDZ model. To know more about it, see this page: The DYNAMICO dynamical core

Installation - DYNAMICO

Before installing DYNAMICO, you should have previously installed LMDZ (and everything that refers to it), see this page: Quick Install and Run Venus PCM

Then, you should have XIOS too, do this:

cd /your/path/trunk/
svn co -r 2319 -q http://forge.ipsl.jussieu.fr/ioserver/svn/XIOS/trunk XIOS

Or see this page: The XIOS Library

It is more practical for what comes next, to have each package (LMDZ.COMMON, LMDZ.VENUS, XIOS, etc.) installed alongside each other.

Then you should compile XIOS (only once):

cd your/path/trunk/XIOS
./make_xios --prod --arch YOUR_ARCH --job 8

(You have to replace “YOUR_ARCH” by your architecture. Every architecture is listed in /XIOS/arch)

For each architecture, there will be 3 files: arch-YOUR_ARCH.env, arch-YOUR_ARCH.path, and arch-YOUR_ARCH.fcm.

No need to specify everything in the command line, just the name. For example, if my architecture is “ifort_MESOIPSL”, there will be the 3 files arch-ifort_MESOIPSL.env, arch-ifort_MESOIPSL.path, and arch-ifort_MESOIPSL.fcm, but my command line will be:

./make_xios --prod --arch ifort_MESOIPSL --job 8

This will be the same thing every time you have to specify your architecture.

To install DYNAMICO, you should clone the GitLab repository (once more, alongside XIOS, etc.):

cd /your/path/trunk/
git clone https://gitlab.in2p3.fr/ipsl/projets/dynamico/dynamico.git ICOSAGCM

A new folder named “ICOSAGCM” will now appear, it contains the model.

Compilation - DYNAMICO

Go to the repository /ICOSAGCM, then compile the model:

cd /your/path/trunk/ICOSAGCM/
./make_icosa -parallel mpi_omp -with_xios -arch YOUR_ARCH -job 8 -debug

You can use the “-debug” option to compile in debug mode, but it will be slower to run.

The executable “icosa_gcm.exe” will be in ICOSAGCM/bin.

Execution - Test_Case - DYNAMICO

Now, we will run a testCase “without the physics”, to verify that the Dynamical Core works alone.

To do this, make a new folder “test_HELD_SUAREZ”, alongside ICOSAGCM and XIOS.

cd /your/path/trunk/
mkdir test_HELD_SUAREZ

Then, we need to copy the specific .def files for this testCase. (they are in /ICOSAGCM)

# Go where the .def files are
cd /your/path/trunk/ICOSAGCM/TEST_CASE/HELD_SUAREZ

# Copy the .def files in the repository test_HELD_SUAREZ
cp *def ../../../test_HELD_SUAREZ

Do the same for the .xml files:

cd /your/path/trunk/ICOSAGCM/xml/DYNAMICO_XML
cp *xml ../../../test_HELD_SUAREZ

cd ..
cp iodef.xml ../../test_HELD_SUAREZ

Then, you should modify the stack size to avoid any segmentation fault when running. Change your ~/.bashrc:

gedit ~/.bashrc

# This option will unlimit the stack size
ulimit -s unlimited

source ~/.bashrc

If gedit doesn’t work, use nano:

nano ~/.bashrc

# This option will unlimit the stack size
ulimit -s unlimited

source ~/.bashrc

Then, copy the executable “icosa_gcm.exe” in the test directory (it is placed in ICOSAGCM/bin):

cd /your/path/trunk/ICOSAGCM/bin
cp icosa_gcm.exe ../../test_HELD_SUAREZ

If, when running the model, you want a NetCDF file (.nc) with all the data, you should modify the .xml file “file_def_dynamico.xml”, line 70, changing the “false” to “true” for “enabled”. This will enable the “dynamico.nc” file to be created, it is already a re-interpolation of the dynamico-grid into long-lat, which makes it usable directly with Ferret/Panoply.

Everything is ready to run the model. Go to test_HELD_SUAREZ, then use the slurm command “sbatch” to submit a job to the cluster.

cd /your/path/trunk/test_HELD_SUAREZ
sbatch script_d_execution.slurm

Slurm script (example):

#!/bin/bash
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=8
#SBATCH --cpus-per-task=1
#SBATCH --partition=zen4 # zen4: 64cores/node and 240GB of memory
##SBATCH --partition=zen16 # zen16: 32 cores/node core and 496GB of memory
#SBATCH -J job_mpi_omp
#SBATCH --time=0:20:00
#SBATCH --output %x.%j.out

source /your/path/trunk/ICOSAGCM/arch/arch-YOUR_ARCH.env

export OMP_NUM_THREADS=1
export OMP_STACKSIZE=400M

mpirun icosa_gcm.exe > icosa_gcm.out 2>&1

In this script, you should modify the path and “YOUR_ARCH” with your architecture (for the source command). Note that we are not using OpenMP here, it is not functional for now (TO UPDATE).

To verify that the code is properly running, you can show directly the “icosa_gcm.out” file.

tail -f icosa_gcm.out

Once the code is finished running, something like this should appear (at the end of the icosa_gcm.out):

GETIN restart_file_name = restart
      masse     advec mass     rmsdpdt      energie   enstrophie     entropie     rmsv     mt.ang
GLOB  -0.999E-15 0.000E+00  0.79047E+01    0.110E-02    0.261E+00    0.155E-02    0.743E+01    0.206E-01

Time elapsed :    601.628763000000    

Connection Venus - DYNAMICO

Now that we verified that the testCase HELD_and_SUAREZ is working, we will be able to “plug” the DYNAMICO dynamical core to some real Physics. For this, you already need LMDZ, alongside XIOS and DYNAMICO.

In addition, you should have ICOSA_LMDZ:

cd /your/path/trunk/
svn update -r 2655 -q ICOSA_LMDZ

You can also make a file named ARCH, and put your arch-YOUR_ARCH.env and arch-YOUR_ARCH.path files.

cd /your/path/trunk/
mkdir ARCH
cd /your/path/trunk/ICOSAGCM/arch
cp YOUR_MACHINE.env /your/path/trunk/ARCH
cp YOUR_MACHINE.path /your/path/trunk/ARCH

Once more, it is more practical to install every new package alongside the others. Here’s what you should have after all the previous steps:

cd /your/path/trunk/
ls
ARCH ICOSAGCM ICOSA_LMDZ LMDZ.COMMON LMDZ.VENUS IOIPSL XIOS test_HELD_SUAREZ  

If you are missing one of these folders, repeat the previous sections for ICOSAGCM.

Or follow the documentation for installing the Venus GCM (which will give you LMDZ.COMMON, LMDZ.VENUS, IOIPSL): Quick Install and Run Venus PCM

For XIOS: The XIOS Library

For ICOSA_LMDZ: we just installed it before.

Compilation Venus - DYNAMICO

Everything needs to be properly compiled before you can run the model. Here’s how to do it:

(some compilation has already been done, but this recaps everything)

!! Everything has to be compiled in the right order !!

Compile IOIPSL:

cd /your/path/trunk/LMDZ.COMMON/ioipsl
./install_ioipsl_YOUR_ARCH.bash

Compile XIOS:

cd ../../XIOS/
./make_xios --prod --arch YOUR_ARCH --arch_path ../ARCH --job 8

Compile LMDZ.COMMON (the Physics packages):

cd ../LMDZ.COMMON/
./makelmdz_fcm -arch YOUR_ARCH -io xios -p venus -d 48x32x50 -j 8 gcm -parallel mpi

“-d 48x32x50” is the model dimension, it can be changed.

Compile ICOSAGCM (the Dynamical Core):

./make_icosa -parallel mpi -with_xios -arch YOUR_ARCH -job 8

Eventually, compile ICOSA_LMDZ:

./make_icosa_lmdz -p venus -parallel mpi -arch YOUR_ARCH -arch_path ../ARCH -job 8 -nodeps

The executable “icosa_lmdz.exe” will be in /ICOSA_LMDZ/bin/.

Running Venus - DYNAMICO

After compiling everything in the right order, we need to prepare the directory. Make a new one alongside the others:

cd /your/path/trunk/
mkdir Venus_DYNAMICO

See the README.md in /ICOSA_LMDZ to know what files to take where :

organization of XML files and synchronization with code
-------------------------------------------------------

from ICOSAGCM/xml [DYNAMICO dynamical core]
- context_input_dynamico.xml
- field_def_dynamico.xml

from ICOSA_LMDZ/xml [INTERFACE]
- iodef.xml

from LMDZ.VENUS/deftank [LMDZ physics]
- field_def_physics.xml
- context_lmdz_physics.xml

-----

to be created and adapted from ICOSAGCM/xml
>> check compatibility when changing ICOSAGCM version 
- context_dynamico.xml

to be created and adapted from ICOSAGCM test cases
>> check compatibility when changing ICOSAGCM version
- file_def_dynamico.xml

From ICOSAGCM/xml/DYNAMICO/ :

cd /your/path/ICOSAGCM/xml/DYNAMICO
cp context_input_dynamico.xml field_def_dynamico.xml dynamico.xml nudging_dynamico.xml sponge_dynamico.xml ../../Venus_DYNAMICO

From ICOSA_LMDZ :

cd /your/path/ICOSA_LMDZ/xml
cp iodef.xml ../../Venus/DYNAMICO

From LMDZ.VENUS :

cd /your/path/LMDZ.VENUS/deftank
cp field_def_physics.xml context_lmdz_physics.xml ../../Venus_DYNAMICO