Difference between revisions of "Installing Mars mesoscale model on spirit"

From Planets
Jump to: navigation, search
(→‎Troubleshooting (in case this happens to you))
 
(11 intermediate revisions by one other user not shown)
Line 1: Line 1:
 
== Set up environment ==
 
== Set up environment ==
−
 
+
=== Option 1 ===
 
Add this in your ~/.bashrc then source the file
 
Add this in your ~/.bashrc then source the file
  
Line 30: Line 30:
 
In the end, source the .bashrc file!
 
In the end, source the .bashrc file!
  
−
=== Extra technical details===
+
=== Option 2 ===
−
* Note that you can decide to not modify your .bashrc file but instead put all the lines in a "mesoscale.env file and add a
+
If you prefer to not modify your .bashrc file, you should instead put all the lines in a "mesoscale.env" file and add a
 
<syntaxhighlight lang="bash">
 
<syntaxhighlight lang="bash">
 
source mesoscale.env
 
source mesoscale.env
 
</syntaxhighlight>
 
</syntaxhighlight>
 
at the beginning of the meso_install.sh script.
 
at the beginning of the meso_install.sh script.
 +
 +
Note that you will still need to have "." in your PATH and unlimited stacksize. So you will definitely need to have:
 +
<syntaxhighlight lang="Bash">
 +
declare -x PATH=./:$PATH
 +
ulimit -s unlimited
 +
</syntaxhighlight>
 +
in your .bashrc file
 +
 +
=== Extra technical details===
 
* WRF needs a NETCDF environment variable where everything is in the same dir (Fortran and C stuff). If this is not available one needs to create a uniquer dire with links to all the C and Fortran stuff. NCDFLIB and NCDFINC are for the physics.
 
* WRF needs a NETCDF environment variable where everything is in the same dir (Fortran and C stuff). If this is not available one needs to create a uniquer dire with links to all the C and Fortran stuff. NCDFLIB and NCDFINC are for the physics.
 
* WHERE_MPI env variable is used by some scripts to make sure we use the right one
 
* WHERE_MPI env variable is used by some scripts to make sure we use the right one
Line 76: Line 85:
 
./meso_install.sh -n DESCRIBE_YOUR_RESEARCH_PROJECT
 
./meso_install.sh -n DESCRIBE_YOUR_RESEARCH_PROJECT
 
</syntaxhighlight>
 
</syntaxhighlight>
 +
''Important'': This will only work if you have access (i.e. an account) to the IN2P3 Gitlab project "La communauté des modèles atmosphériques planétaires" AND if you have added your personal ssh key there. Otherwise proceed as indicated below.
  
 
''Special case'': In case you do not have a gitlab account, ask for an archive (tar.gz) of the code.
 
''Special case'': In case you do not have a gitlab account, ask for an archive (tar.gz) of the code.
Line 85: Line 95:
 
</syntaxhighlight>
 
</syntaxhighlight>
  
−
''Weird!'' fcm is not working interactively. It only works if being declared in the environment file by adding
+
 
 +
=== Troubleshooting (in case this happens to you) ===
 +
If GCM compilation fails with error message "Can't locate Fcm/Config.pm in @INC (you may need to install the Fcm::Config module)", this is because "." is in your PATH before "FCM_V1.2/bin".  
 +
You can fix it by either adapting the PATH in the environment file by adding
 
<syntaxhighlight lang="Bash">
 
<syntaxhighlight lang="Bash">
 
declare -x PATH=/homedata/_MY_LOGIN_/DESCRIBE_YOUR_RESEARCH_PROJECT/code/LMDZ.COMMON/FCM_V1.2/bin/:$PATH
 
declare -x PATH=/homedata/_MY_LOGIN_/DESCRIBE_YOUR_RESEARCH_PROJECT/code/LMDZ.COMMON/FCM_V1.2/bin/:$PATH
 
</syntaxhighlight>
 
</syntaxhighlight>
−
then recompile the GCM compiling (which had failed previously) by doing
+
or by removing the symbolic link to "fcm" in "/homedata/_MY_LOGIN_/DESCRIBE_YOUR_RESEARCH_PROJECT/code/LMDZ.COMMON/".
 +
 
 +
Once you have fixed the problem you can recompile the GCM (which had failed previously) by doing:
 
<syntaxhighlight lang="Bash">
 
<syntaxhighlight lang="Bash">
 
cd /homedata/_MY_LOGIN_/DESCRIBE_YOUR_RESEARCH_PROJECT/
 
cd /homedata/_MY_LOGIN_/DESCRIBE_YOUR_RESEARCH_PROJECT/
Line 98: Line 113:
 
./mini_startbase.sh
 
./mini_startbase.sh
 
</syntaxhighlight>
 
</syntaxhighlight>
 +
 +
=== Extra technical stuff ===
 +
 +
* in meso_install.sh is hard coded the git tag of the version that will be installed e.g. version="tags/mesoscale-compile-run_MESOIPSL_exploration". the tags point to specific versions which have been tested and are thus reference. e.g. mesoscale_compile-run_MESOIPSL on the git-trunk
 +
 +
* To recompile WRF (see "readme" file)
 +
<syntaxhighlight lang="Bash">
 +
cd code/MESOSCALE/LMD_MM_MARS
 +
makemeso -p mars_lmd_new
 +
</syntaxhighlight>
 +
To recompile WRF in debug mode:
 +
<syntaxhighlight lang="Bash">
 +
cd code/MESOSCALE/LMD_MM_MARS
 +
makemeso -p mars_lmd_new -g
 +
</syntaxhighlight>
 +
The compilation script (and options) is in MESOSCALE/LMD_MM_MARS/SRC/WRFV2/mars_lmd_new/makegcm_mpifort for physics
 +
And in MESOSCALE/LMD_MM_MARS/makemeso for WRF
  
 
== Run the full workflow GCM + initialization + mesoscale model ==
 
== Run the full workflow GCM + initialization + mesoscale model ==
  
−
Have a look in the script named launch just for info or to change the number of processors or the step at which you start. Send the launch job to the cluster by typing (note that you can change the number of core to run with in the header of the script, e.g. ''#SBATCH --ntasks=24'' to use 24 cores)
+
Have a look in the script named launch just for info or to change the number of processors or the step at which you start. Send the launch job to the cluster by typing (note that you can change the number of core to run with in the header of the script, e.g. ''#SBATCH --ntasks=24'' to use 24 cores, but this doesn't work for now...)
 
<syntaxhighlight lang="Bash">
 
<syntaxhighlight lang="Bash">
 
sbatch launch
 
sbatch launch
 
</syntaxhighlight>
 
</syntaxhighlight>
 +
IMPORTANT: if you have used "Option 2" in the setup/environment, i.e. creating a dedicated "mesoscale.env" file, then you should source it in the "launch" script with something of the likes of:
 +
<syntaxhighlight lang="Bash">
 +
source ../mesoscale.env
 +
</syntaxhighlight>
 +
 +
 
You can check the status of the run by typing
 
You can check the status of the run by typing
 
<syntaxhighlight lang="Bash">
 
<syntaxhighlight lang="Bash">
−
squeueme
+
squeue -u $USER
 +
</syntaxhighlight>
 +
 
 +
=== Extra technical stuff ===
 +
* The namelist.input file contains inputs for WRF. One can change output rate by specifying "history_interval_s" in seconds (instead of "history_interval"), e.g. set to the timestep for an output at every step to debug.
 +
<syntaxhighlight lang="Bash">
 +
history_interval_s=20
 +
</syntaxhighlight>
 +
* to run in hydrostatic mode: it is a parameter for "&dynamics" namelist
 +
<syntaxhighlight lang="Bash">
 +
non_hydrostatic=F
 
</syntaxhighlight>
 
</syntaxhighlight>
 +
* Note that the full list of options for namelist.in can be found in "SIMU/namelist.input_full"
  
 
== A more detailed run ==
 
== A more detailed run ==

Latest revision as of 14:37, 15 May 2024

Set up environment

Option 1

Add this in your ~/.bashrc then source the file 

module purge
module load intel/2021.4.0
module load intel-mkl/2020.4.304
module load openmpi/4.0.7
module load hdf5/1.10.7-mpi
module load netcdf-fortran/4.5.3-mpi
module load netcdf-c/4.7.4-mpi
declare -x WHERE_MPI=/net/nfs/tools/u20/22.3/PrgEnv/intel/linux-ubuntu20.04-zen2/openmpi/4.0.7-intel-2021.4.0-43fdcnab3ydwu7ycrplnzlp6xieusuz7/bin/
declare -x NETCDF=/scratchu/spiga/les_mars_project_spirit/netcdf_hacks/SPIRIT
declare -x NCDFLIB=$NETCDF/lib
declare -x NCDFINC=$NETCDF/include

Do not forget to declare the local directory in PATH by adding this to your ~/.bashrc 

declare -x PATH=./:$PATH

It is necessary to unlimit the stacksize to avoit unwanted segmentation faults by adding this to your ~/.bashrc 

ulimit -s unlimited

In the end, source the .bashrc file!

Option 2

If you prefer to not modify your .bashrc file, you should instead put all the lines in a "mesoscale.env" file and add a 

source mesoscale.env

at the beginning of the meso_install.sh script.

Note that you will still need to have "." in your PATH and unlimited stacksize. So you will definitely need to have: 

declare -x PATH=./:$PATH
ulimit -s unlimited

in your .bashrc file

Extra technical details

  • WRF needs a NETCDF environment variable where everything is in the same dir (Fortran and C stuff). If this is not available one needs to create a uniquer dire with links to all the C and Fortran stuff. NCDFLIB and NCDFINC are for the physics.
  • WHERE_MPI env variable is used by some scripts to make sure we use the right one


Set up the installer

Go to your 'data' directory (cd /homedata/_MY_LOGIN_ or cd /data/_MY_LOGIN_) and download the installer with the following command 

svn co https://svn.lmd.jussieu.fr/Planeto/trunk/MESOSCALE/LMD_MM_MARS/SIMU

Make sure the installer can be executed 

chmod 755 SIMU/meso_install.sh

Make a link (e.g. where you are, in the parent directory of /data) to the main script in this SIMU directory 

ln -sf SIMU/meso_install.sh .

Run the installer with a simple display of possible options 

./meso_install.sh -h

Install the code

Update to the latest version of installer 

cd SIMU
svn update
chmod 755 meso_install.sh
cd ..

Run the installer by providing a name for your specific directory 

./meso_install.sh -n DESCRIBE_YOUR_RESEARCH_PROJECT

Important: This will only work if you have access (i.e. an account) to the IN2P3 Gitlab project "La communauté des modèles atmosphériques planétaires" AND if you have added your personal ssh key there. Otherwise proceed as indicated below.

Special case: In case you do not have a gitlab account, ask for an archive (tar.gz) of the code. Let us assume the name is git-trunk-mesoscale-compile-run-spirit.tar.gz and the file is in the current directory (for instance in /homedata/_MY_LOGIN_) then run the installer with the following command 

./meso_install.sh -n DESCRIBE_YOUR_RESEARCH_PROJECT -a git-trunk-mesoscale-compile-run-spirit


Troubleshooting (in case this happens to you)

If GCM compilation fails with error message "Can't locate Fcm/Config.pm in @INC (you may need to install the Fcm::Config module)", this is because "." is in your PATH before "FCM_V1.2/bin". You can fix it by either adapting the PATH in the environment file by adding 

declare -x PATH=/homedata/_MY_LOGIN_/DESCRIBE_YOUR_RESEARCH_PROJECT/code/LMDZ.COMMON/FCM_V1.2/bin/:$PATH

or by removing the symbolic link to "fcm" in "/homedata/_MY_LOGIN_/DESCRIBE_YOUR_RESEARCH_PROJECT/code/LMDZ.COMMON/".

Once you have fixed the problem you can recompile the GCM (which had failed previously) by doing: 

cd /homedata/_MY_LOGIN_/DESCRIBE_YOUR_RESEARCH_PROJECT/
./compile_gcm.sh

and recreate start files from a sample start_archive.nc by doing (in subdirectory gcm/newstart) 

./mini_startbase.sh

Extra technical stuff

  • in meso_install.sh is hard coded the git tag of the version that will be installed e.g. version="tags/mesoscale-compile-run_MESOIPSL_exploration". the tags point to specific versions which have been tested and are thus reference. e.g. mesoscale_compile-run_MESOIPSL on the git-trunk
  • To recompile WRF (see "readme" file)
cd code/MESOSCALE/LMD_MM_MARS
makemeso -p mars_lmd_new

To recompile WRF in debug mode: 

cd code/MESOSCALE/LMD_MM_MARS
makemeso -p mars_lmd_new -g

The compilation script (and options) is in MESOSCALE/LMD_MM_MARS/SRC/WRFV2/mars_lmd_new/makegcm_mpifort for physics And in MESOSCALE/LMD_MM_MARS/makemeso for WRF

Run the full workflow GCM + initialization + mesoscale model

Have a look in the script named launch just for info or to change the number of processors or the step at which you start. Send the launch job to the cluster by typing (note that you can change the number of core to run with in the header of the script, e.g. #SBATCH --ntasks=24 to use 24 cores, but this doesn't work for now...) 

sbatch launch

IMPORTANT: if you have used "Option 2" in the setup/environment, i.e. creating a dedicated "mesoscale.env" file, then you should source it in the "launch" script with something of the likes of: 

source ../mesoscale.env


You can check the status of the run by typing 

squeue -u $USER

Extra technical stuff

  • The namelist.input file contains inputs for WRF. One can change output rate by specifying "history_interval_s" in seconds (instead of "history_interval"), e.g. set to the timestep for an output at every step to debug.
history_interval_s=20
  • to run in hydrostatic mode: it is a parameter for "&dynamics" namelist
non_hydrostatic=F
  • Note that the full list of options for namelist.in can be found in "SIMU/namelist.input_full"

A more detailed run

The best is probably to use a more complete startbase than the minimal one that was created. For instance, to get a more complete startbase, link 'data_gcm' to point towards '/data/spiga/2021_STARTBASES_rev2460/MY35'

To use the tyler cap setting in namelist.wps, download the tylerall archive here 'https://web.lmd.jussieu.fr/~aslmd/mesoscale_model/data_static/' and extract the content in the folder named data_static

Retrieved from "http://lmdz-forge.lmd.jussieu.fr/mediawiki/Planets/index.php?title=Installing_Mars_mesoscale_model_on_spirit&oldid=2055"