Venus - DYNAMICO
Contents
Venus - DYNAMICO
Dynamico is the recently developed Dynamical core, enabling better performance and solving some issues of the LMDZ model. To know more about it, see this page: The DYNAMICO dynamical core
Installation - DYNAMICO
Before installing DYNAMICO, you should have previously installed LMDZ (and everything that refers to it), see this page: Quick Install and Run Venus PCM
Then, you should have XIOS too, do this:
cd /your/path/trunk/ svn co http://forge.ipsl.jussieu.fr/ioserver/svn/XIOS/trunk XIOS
Or see this page: The XIOS Library
It is more practical for what comes next, to have each package (LMDZ.COMMON, LMDZ.VENUS, XIOS, etc.) installed alongside each other.
Then you should compile XIOS (only once):
cd your/path/trunk/XIOS ./make_xios --prod --arch YOUR_ARCH --job 8
(You have to replace “YOUR_ARCH” by your architecture. Every architecture is listed in /XIOS/arch)
For each architecture, there will be 3 files: arch-YOUR_ARCH.env, arch-YOUR_ARCH.path, and arch-YOUR_ARCH.fcm.
No need to specify everything in the command line, just the name. For example, if my architecture is “ifort_MESOIPSL”, there will be the 3 files arch-ifort_MESOIPSL.env, arch-ifort_MESOIPSL.path, and arch-ifort_MESOIPSL.fcm, but my command line will be:
./make_xios --prod --arch ifort_MESOIPSL --job 8
This will be the same thing every time you have to specify your architecture.
To install DYNAMICO, you should clone the GitLab repository (once more, alongside XIOS, etc.):
cd /your/path/trunk/ git clone https://gitlab.in2p3.fr/ipsl/projets/dynamico/dynamico.git ICOSAGCM
A new folder named “ICOSAGCM” will now appear, it contains the model.
Compilation - DYNAMICO
Go to the repository /ICOSAGCM, then compile the model:
cd /your/path/trunk/ICOSAGCM/ ./make_icosa -parallel mpi_omp -with_xios -arch YOUR_ARCH -job 8 -debug
You can use the “-debug” option to compile in debug mode, but it will be slower to run.
The executable “icosa_gcm.exe” will be in ICOSAGCM/bin.
Execution - Test_Case Held&Suarez - DYNAMICO
Now, we will run a testCase “without the physics”, to verify that the Dynamical Core works alone.
To do this, make a new folder “test_HELD_SUAREZ”, alongside ICOSAGCM and XIOS.
cd /your/path/trunk/ mkdir test_HELD_SUAREZ
Then, we need to copy the specific .def files for this testCase. (they are in /ICOSAGCM)
# Go where the .def files are cd /your/path/trunk/ICOSAGCM/TEST_CASE/HELD_SUAREZ # Copy the .def files in the repository test_HELD_SUAREZ cp *def ../../../test_HELD_SUAREZ
Do the same for the .xml files:
cd /your/path/trunk/ICOSAGCM/xml/DYNAMICO_XML cp *xml ../../../test_HELD_SUAREZ cd .. cp iodef.xml ../../test_HELD_SUAREZ
Then, you should modify the stack size to avoid any segmentation fault when running. Change your ~/.bashrc:
gedit ~/.bashrc # This option will unlimit the stack size ulimit -s unlimited source ~/.bashrc
If gedit doesn’t work, use nano:
nano ~/.bashrc # This option will unlimit the stack size ulimit -s unlimited source ~/.bashrc
Then, copy the executable “icosa_gcm.exe” in the test directory test_HELD_SUAREZ (it is placed in ICOSAGCM/bin):
cd /your/path/trunk/ICOSAGCM/bin cp icosa_gcm.exe ../../test_HELD_SUAREZ
If, when running the model, you want a NetCDF file (.nc) with all the data, you should modify the .xml file “file_def_dynamico.xml”, line 70, changing the “false” to “true” for “enabled”. This will enable the “dynamico.nc” file to be created, it is already a re-interpolation of the dynamico-grid into long-lat, which makes it usable directly with Ferret/Panoply.
Everything is ready to run the model. Go to test_HELD_SUAREZ, then use the slurm command “sbatch” to submit a job to the cluster.
cd /your/path/trunk/test_HELD_SUAREZ sbatch script_d_execution.slurm
Slurm script (example for spirit1):
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=8
#SBATCH --cpus-per-task=1
#SBATCH --partition=zen4 # zen4: 64cores/node and 240GB of memory
##SBATCH --partition=zen16 # zen16: 32 cores/node core and 496GB of memory
#SBATCH -J job_mpi_omp
#SBATCH --time=0:20:00
#SBATCH --output %x.%j.out
source /your/path/trunk/ICOSAGCM/arch/arch-YOUR_ARCH.env
export OMP_NUM_THREADS=1
export OMP_STACKSIZE=400M
mpirun icosa_gcm.exe > icosa_gcm.out 2>&1
In this script, you should modify the path and “YOUR_ARCH” with your architecture (for the source command). Note that we are not using OpenMP here, it is not functional for now (TO UPDATE).
To verify that the code is properly running, you can show directly the “icosa_gcm.out” file.
tail -f icosa_gcm.out
Once the code is finished running, something like this should appear (at the end of the icosa_gcm.out):
GETIN restart_file_name = restart
masse advec mass rmsdpdt energie enstrophie entropie rmsv mt.ang
GLOB -0.999E-15 0.000E+00 0.79047E+01 0.110E-02 0.261E+00 0.155E-02 0.743E+01 0.206E-01
Time elapsed : 601.628763000000
Connection Venus - DYNAMICO
Now that we verified that the testCase HELD_and_SUAREZ is working, we will be able to “plug” the DYNAMICO dynamical core to some real Physics. For this, you already need LMDZ, alongside XIOS and DYNAMICO.
In addition, you should have ICOSA_LMDZ:
cd /your/path/trunk/ svn update -r 2655 -q ICOSA_LMDZ
You can also make a file named ARCH, and put your arch-YOUR_ARCH.env and arch-YOUR_ARCH.path files.
cd /your/path/trunk/ mkdir ARCH cd /your/path/trunk/ICOSAGCM/arch cp YOUR_MACHINE.env /your/path/trunk/ARCH cp YOUR_MACHINE.path /your/path/trunk/ARCH
Once more, it is more practical to install every new package alongside the others. Here’s what you should have after all the previous steps:
cd /your/path/trunk/ ls ARCH ICOSAGCM ICOSA_LMDZ LMDZ.COMMON LMDZ.VENUS IOIPSL XIOS test_HELD_SUAREZ
If you are missing one of these folders, repeat the previous sections for ICOSAGCM.
Or follow the documentation for installing the Venus GCM (which will give you LMDZ.COMMON, LMDZ.VENUS, IOIPSL): Quick Install and Run Venus PCM
For XIOS: The XIOS Library
For ICOSA_LMDZ: we just installed it before.
Compilation Venus - DYNAMICO
Everything needs to be properly compiled before you can run the model. Here’s how to do it:
(some compilation has already been done, but this recaps everything)
!! Everything has to be compiled in the right order !!
Compile IOIPSL:
cd /your/path/trunk/LMDZ.COMMON/ioipsl ./install_ioipsl_YOUR_ARCH.bash
Compile XIOS:
cd ../../XIOS/ ./make_xios --prod --arch YOUR_ARCH --arch_path ../ARCH --job 8
Compile LMDZ.COMMON (the Physics packages):
cd ../LMDZ.COMMON/ ./makelmdz_fcm -arch YOUR_ARCH -io xios -p venus -d 48x32x50 -j 8 gcm -parallel mpi
“-d 48x32x50” is the model dimension, it can be changed.
Compile ICOSAGCM (the Dynamical Core):
./make_icosa -parallel mpi -with_xios -arch YOUR_ARCH -job 8
Eventually, compile ICOSA_LMDZ:
./make_icosa_lmdz -p venus -parallel mpi -arch YOUR_ARCH -arch_path ../ARCH -job 8 -nodeps
The executable “icosa_lmdz.exe” will be in /ICOSA_LMDZ/bin/.
Running Venus - DYNAMICO
After compiling everything in the right order, we need to prepare the directory. Make a new one alongside the others:
cd /your/path/trunk/ mkdir Venus_DYNAMICO
See the README.md in /ICOSA_LMDZ to know what files to take where :
organization of XML files and synchronization with code
-------------------------------------------------------
from ICOSAGCM/xml [DYNAMICO dynamical core]
- context_input_dynamico.xml
- field_def_dynamico.xml
from ICOSA_LMDZ/xml [INTERFACE]
- iodef.xml
from LMDZ.VENUS/deftank [LMDZ physics]
- field_def_physics.xml
- context_lmdz_physics.xml
-----
to be created and adapted from ICOSAGCM/xml
>> check compatibility when changing ICOSAGCM version
- context_dynamico.xml
to be created and adapted from ICOSAGCM test cases
>> check compatibility when changing ICOSAGCM version
- file_def_dynamico.xml
From ICOSAGCM/xml/DYNAMICO/ :
cd /your/path/ICOSAGCM/xml/DYNAMICO cp context_input_dynamico.xml field_def_dynamico.xml dynamico.xml nudging_dynamico.xml sponge_dynamico.xml ../../Venus_DYNAMICO
From ICOSA_LMDZ :
cd /your/path/ICOSA_LMDZ/xml cp iodef.xml ../../Venus/DYNAMICO
From LMDZ.VENUS :
cd /your/path/LMDZ.VENUS/deftank cp field_def_physics.xml context_lmdz_physics.xml ../../Venus_DYNAMICO
TO CONTINUE
