Running the Venus PCM in parallel - Revision history

Emillour at 07:14, 5 June 2023

2023-06-05T07:14:11Z

Emillour at 16:26, 2 June 2023

2023-06-02T16:26:57Z

Emillour at 10:33, 31 May 2023

2023-05-31T10:33:00Z

Emillour at 15:29, 29 May 2023

2023-05-29T15:29:28Z

Emillour at 15:01, 29 May 2023

2023-05-29T15:01:56Z

Emillour at 11:49, 27 May 2023

2023-05-27T11:49:05Z

Emillour: Created page with " == Level of parallelism available in the Venus PCM == Currently the Venus physics package is only adapted to MPI parallelism == Compiling the Venus PCM - LMDZ model with MPI..."

2023-05-27T11:45:00Z

Created page with " == Level of parallelism available in the Venus PCM == Currently the Venus physics package is only adapted to MPI parallelism == Compiling the Venus PCM - LMDZ model with MPI..."

New page

== Level of parallelism available in the Venus PCM ==
Currently the Venus physics package is only adapted to MPI parallelism

== Compiling the Venus PCM - LMDZ model with MPI enabled ==
This requires as a prerequisite that:
* the Venus PCM is alerady installed and working in serial (i.e. as discussed in [[Quick Install and Run Venus PCM]])
* an MPI library is installed and available ()
* The relevant architecture "arch" files (more about these [[The Target Architecture ("arch") Files|here]] have been adapted as need be. In practice this usually means specifying in the arch.fcm that the compiler is '''mpif90''' and also adding paths to the MPI library '''include''' and '''lib''' directories to the ''%MPI_FFLAGS'' and ''%MPI_LD'' lines. e,g:
<pre>
%COMPILER mpif90
%LINK mpif90
%AR ar
%MAKE make
%FPP_FLAGS -P -traditional
%FPP_DEF NC_DOUBLE
%BASE_FFLAGS -c -fdefault-real-8 -fdefault-double-8 -ffree-line-length-none -fno-align-commons
%PROD_FFLAGS -O3
%DEV_FFLAGS -O
%DEBUG_FFLAGS -ffpe-trap=invalid,zero,overflow -fbounds-check -g3 -O0 -fstack-protector-all -finit-real=snan -fbacktrace
%MPI_FFLAGS -I/usr/openmpi/include
%OMP_FFLAGS
%BASE_LD
%MPI_LD -L/usr/openmpi/lib -lmpi
%OMP_LD
</pre>
Where in the example above the MPI library is installed in '''/usr/openmpi'''.

Once all the prerequisites are met, then one merely needs to compile the PCM with the usual makelmdz_fcm script (located in ''LMDZ.COMMON'') with the '''-parallell mpi''' option, e.g.:
<syntaxhighlight lang="bash">
./makelmdz_fcm -arch local -p venus -d 48x32x50 -parallel mpi -j 8 gcm
</syntaxhighlight>
If the compilation went well then the executable will be generated in ''LMDZ.COMMON/bin'' and, with the options from the example above, be called '''gcm_48x32x50_phyvenus_para.e'''

== running in parallel ==
For the sake of simply we assume here that we are following up on the case described in [[Quick Install and Run Venus PCM]] which we now want to run in parallel with MPI. For that, once '''gcm_48x32x50_phyvenus_para.e''' created, one simply needs to copy it from '''LMDZ.COMMON/bin''' to the directory containing the initial conditions and parameter files, e.g. '''bench_48x32x50''' and then:

1. (optional but recommended step) source the environment architecture file (the very same that was used to compile the model), e.g.,:
<syntaxhighlight lang="bash">
source ../LMDZ.COMMON/arch.env
</syntaxhighlight>
2. execute the model on a given number of processors, for instance 4:
<syntaxhighlight lang="bash">
mpirun -np 4 gcm_48x32x50_phyvenus_para.e > gcm.out 2>&1
</syntaxhighlight>
With this command line, the (text) outputs messages are redirected into a text file, '''gcm.out'''. It is convenient to keep this file for later inspection (e.g., to track a bug). If there is no redirection (only '''mpirun -np 4 gcm_48x32x50_phyvenus_seq.e'''), then the outputs will be directly on the screen.

== Miscellaneous comments ==
* There is a limitation to the number of MPI processes one may run with. Each process needs to handle at least 2 bands of latitude. Put another way, for a given number of latitude intervals jjm one may use at most jjm/2 processes (in the example above the grid is 48x32 in lonxlat so one could use at most 32/2=16 processes).
* if running with the IOIPSL library then there will as many output (histmnth and histins) files as there are processes. One will need to recombine them (using the dedicated ''rebuild'' tool) in single global files.
* PCM start and restart files are insensitive to whether the model was run in serial or parallel. In fact results should be identical (at least in debug mode, where there are no optimisations) between a serial run or a parallel one (regardless of the number of cores used).

[[Category:Venus-LMDZ]]

@@ Line 49: / Line 49: @@
 For what it's worth here are some performance results when running 0.05Vday (96x96x78 grid, with chemistry) with different number of processes, on the Irene-Rome supercomputer:
 <pre>
-Number of processes | Run time       | Speedup wrt 8 cores           | Speedup wrt 16 cores
+Number of processes | Run time       | Speedup wrt 4 cores           | Speedup wrt 8 cores           | Speedup wrt 16 cores
-           |  6h15 = 375mn  |                               |
+           |  9h53 = 593mn  |                               |                               |
-           |  4h05 = 245mn  | ~0.765 (=375.*(8./16.)/245.)  |
+           |  6h15 = 375mn  | ~0.791 (=593.*(4./8.)/375.)   |                               |
-           |  3h02 = 182mn  | ~0.686 (=375.*(8./24.)/182.)  | ~0.897 (=245.*(16./24.)/182.)
+           |  4h05 = 245mn  | ~0.605 (=593.*(4./16.)/245.)  | ~0.765 (=375.*(8./16.)/245.)  |
-           |  2h29 = 149mn  | ~0.629 (=375.*(8./32.)/149.)  | ~0.822 (=245.*(16./32.)/149.)
+           |  3h02 = 182mn  | ~0.543 (=593.*(4./24.)/182.)  | ~0.686 (=375.*(8./24.)/182.)  | ~0.897 (=245.*(16./24.)/182.)
-           |  1h56 = 116mn  | ~0.539 (=375.*(8./48.)/116.)  | ~0.704 (=245.*(16./48.)/116.)
+           |  2h29 = 149mn  | ~0.497 (=593.*(4./32.)/149.)  | ~0.629 (=375.*(8./32.)/149.)  | ~0.822 (=245.*(16./32.)/149.)
 </pre>
 Note the reasonable speedup up to ~32 cores, but limited improvement when going to the maximum of 48 usable cores (since the model includes 96 latitude intervals).

@@ Line 49: / Line 49: @@
 For what it's worth here are some performance results when running 0.05Vday (96x96x78 grid, with chemistry) with different number of processes, on the Irene-Rome supercomputer:
 <pre>
-Number of processes | Run time       | Speedup wrt 16 cores
+Number of processes | Run time       | Speedup wrt 8 cores           | Speedup wrt 16 cores
-           |  4h05 = 245mn  |
+           |  6h15 = 375mn  |                               |
-           |  3h02 = 182mn  | ~0.897 (=245.*(16./24.)/182.)
+           |  4h05 = 245mn  | ~0.765 (=375.*(8./16.)/245.)  |
-           |  2h29 = 149mn  | ~0.822 (=245.*(16./32.)/149.)
+           |  3h02 = 182mn  | ~0.686 (=375.*(8./24.)/182.)  | ~0.897 (=245.*(16./24.)/182.)
-           |  1h56 = 116mn  | ~0.704 (=245.*(16./48.)/116.)
+           |  2h29 = 149mn  | ~0.629 (=375.*(8./32.)/149.)  | ~0.822 (=245.*(16./32.)/149.)
 </pre>
 Note the reasonable speedup up to ~32 cores, but limited improvement when going to the maximum of 48 usable cores (since the model includes 96 latitude intervals).

@@ Line 49: / Line 49: @@
 For what it's worth here are some performance results when running 0.05Vday (96x96x78 grid, with chemistry) with different number of processes, on the Irene-Rome supercomputer:
 <pre>
-Number of processes | Run time
+Number of processes | Run time       | Speedup wrt 16 cores
-           |  4h05
+           |  4h05 = 245mn  |
-           |  3h02
+           |  3h02 = 182mn  | ~0.897 (=245.*(16./24.)/182.)
-           |  2h29
+           |  2h29 = 149mn  | ~0.822 (=245.*(16./32.)/149.)
-           |  1h56
+           |  1h56 = 116mn  | ~0.704 (=245.*(16./48.)/116.)
 </pre>
 Note the reasonable speedup up to ~32 cores, but limited improvement when going to the maximum of 48 usable cores (since the model includes 96 latitude intervals).

@@ Line 44: / Line 44: @@
 mpirun -np 4 gcm_48x32x50_phyvenus_para.e > gcm.out 2>&1
 </syntaxhighlight>
-With this command line, the (text) outputs messages are redirected into a text file, '''gcm.out'''. It is convenient to keep this file for later inspection (e.g., to track a bug). If there is no redirection (only '''mpirun -np 4 gcm_48x32x50_phyvenus_seq.e'''), then the outputs will be directly on the screen.
+With this command line, the (text) outputs messages are redirected into a text file, '''gcm.out'''. It is convenient to keep this file for later inspection (e.g., to track a bug). If there is no redirection (only '''mpirun -np 4 gcm_48x32x50_phyvenus_para.e'''), then the outputs will be directly on the screen.
 == Illustration of model speedup with MPI ==
@@ Line 60: / Line 60: @@
 == Miscellaneous comments ==
 * There is a limitation to the number of MPI processes one may run with. Each process needs to handle at least 2 bands of latitude. Put another way, for a given number of latitude intervals jjm one may use at most jjm/2 processes (in the example above the grid is 48x32 in lonxlat so one could use at most 32/2=16 processes).
-* if running with the IOIPSL library then there will as many output (histmnth and histins) files as there are processes. One will need to recombine them (using the dedicated ''rebuild'' tool) in single global files.
+* if running with [[The IOIPSL Library|the IOIPSL library]] then there will as many output (histmnth and histins) files as there are processes. One will need to recombine them (using the dedicated ''rebuild'' tool) in single global files.
 * PCM start and restart files are insensitive to whether the model was run in serial or parallel. In fact results should be identical (at least in debug mode, where there are no optimisations) between a serial run or a parallel one (regardless of the number of cores used).
 [[Category:Venus-LMDZ]]

@@ Line 6: / Line 6: @@
 This requires as a prerequisite that:
 * the Venus PCM is alerady installed and working in serial (i.e. as discussed in [[Quick Install and Run Venus PCM]])
-* an MPI library is installed and available ()
+* an MPI library is installed and available (if none is already available you'll have to [[Building an MPI library|build one]])
-* The relevant architecture "arch" files (more about these [[The Target Architecture ("arch") Files|here]] have been adapted as need be. In practice this usually means specifying in the arch.fcm that the compiler is '''mpif90''' and also adding paths to the MPI library '''include''' and '''lib''' directories to the ''%MPI_FFLAGS'' and ''%MPI_LD'' lines. e,g:
+* The relevant architecture "arch" files (more about these [[The Target Architecture ("arch") Files|here]]) have been adapted as need be. In practice this usually means specifying in the arch.fcm that the compiler is '''mpif90''' and also adding paths to the MPI library '''include''' and '''lib''' directories to the ''%MPI_FFLAGS'' and ''%MPI_LD'' lines. e,g:
 <pre>
 %COMPILER            mpif90