Parallelism - Revision history

Emillour at 13:42, 4 February 2026

2026-02-04T13:42:39Z

Emillour: /* How to compile the parallel version of the PCM */

2025-10-06T08:29:53Z

‎How to compile the parallel version of the PCM

Emillour at 09:51, 3 October 2025

2025-10-03T09:51:44Z

Emillour: /* Run interactively */

2024-01-30T16:56:46Z

‎Run interactively

Emillour at 16:55, 30 January 2024

2024-01-30T16:55:11Z

Emillour at 08:14, 26 May 2023

2023-05-26T08:14:01Z

Martin.turbet at 08:54, 20 March 2023

2023-03-20T08:54:04Z

Martin.turbet at 10:15, 9 March 2023

2023-03-09T10:15:59Z

Romain.vande: /* How parallelism is implemented in the model */

2022-05-11T09:38:44Z

‎How parallelism is implemented in the model

Romain.vande: /* How to run in parallel */

2022-05-11T09:13:12Z

‎How to run in parallel

@@ Line 38: / Line 38: @@
 So the command line to generate the Generic PCM to run in mixed MPI and OpenMP mode will be for example :
 <pre>
-./makelmdz_fcm -s XX -d LONxLATxALT -b IRxVI -p std -arch archFile -parallel mpi_omp gcm
+./makelmdz_fcm -s XX -d LONxLATxALT -b IRxVI -p generic -arch archFile -parallel mpi_omp gcm
 </pre>

@@ Line 36: / Line 36: @@
 </pre>
-So the command line to run in mix MPI_OMP will be for example :
+So the command line to generate the Generic PCM to run in mixed MPI and OpenMP mode will be for example :
 <pre>
-./makelmdz_fcm -s XX -t XX -d LONxLATxALT -b IRxVI -p physicSuffix -arch archFile -parallel mpi_omp gcm
+./makelmdz_fcm -s XX -d LONxLATxALT -b IRxVI -p std -arch archFile -parallel mpi_omp gcm
 </pre>

@@ Line 6: / Line 6: @@
 Large problems can often be divided into smaller ones, which can then be solved at the same time.
-In short : '''Parallelism can help you save time'''.
+In short : '''Parallelism can help you save time''': One will get the same results when running on more cores than if running the serial version of the code, but sooner.
-Indeed, as the problem is cut into smaller part that are solved simultaneously, the waiting time for the user is reduced.
+Indeed, as the problem is cut into smaller part that are solved simultaneously, the waiting (wall clock) time for the user is reduced.
-However this usually comes with a counterpart, it can '''increase the total computation time'''.
+However this usually comes with a counterpart (overheads and extra computations due to the parallelization, along with some inherent inefficiences concerning some computations which must be done sequentially), it can '''increase the total computational cost'''.
 == How parallelism is implemented in the model ==
-* MPI tiling
+=== Parallelism with the lon-lat (LMDZ) dynamical core ===
-* Open MP (OMP)
+* MPI tiling: In the lon-lat dynamics the globe is tiled in regions covering all longitudes and a few latitudes. In practice these latitude bands must contain at least 2 points. There is therefore  a limitation to the number of MPI processes one may run with: for a given number of latitude intervals jjm one may use at most jjm/2 processes (for example if the horizontal grid is 64x48 in lonxlat so one could use at most 48/2=24 MPI processes.
-== How to compile in parallel ==
+* Open MP (OMP): In the dynamics this parallelism is implemented on the loops along the vertical. One could thus use as many OpenMP threads are there are model levels. In practice however the speedup breaks down with much less and it is recommended to have OpenMP chunks of at least ten vertical levels each. Therefore for simulation with llm altitude levels one should target using at most llm/10 OpenMP threads (e.g. for a 64x48x54 grid target using at most 5 OpenMP threads).
-To compile the model in parallel use the same command as in sequential (CITE HOW TO COMPILE) and add the following option :
+* In practice: One will want to use both MPI and OpenMP for simulations, with as many MPI processes as possible, combined to a good number of OpenMP threads (for each MPI process). Depending on the cluster used, the speedup as a function of number of MPI processes and OpenMP threads can vary a lot. It is therefore recommended to test it to find the "optimal" setup for a given grid.
 <pre>
   -parallel
 </pre>
-Then there is three choices for parallelism MPI, OMP and mix MPI_OMP:
+Then there is three choices for parallelism MPI, OMP and mixed (i.e. combined) MPI_OMP:
 <pre>
   -parallel mpi

@@ Line 48: / Line 48: @@
 IMPORTANT: one MUST use the mpirun command corresponding to the mpif90 compiler specified in the arch file.
-IMPORTANT: One can use at most one MPI process for every 2 points along the latitude (e.g. a maximum of 24 processes for a horizontal grid of 64x48).
+IMPORTANT: One can use at most one MPI process for every 2 points along the latitude (e.g. a maximum of 24 processes for a horizontal grid of 64x48). If you try to use too many MPI processes you will get the following error message (in French!!):
 Output files (restart.nc, diagfi.nc ,etc.) are just as when running in serial.

← Older revision		Revision as of 16:55, 30 January 2024
Line 47:		Line 47:

	IMPORTANT: one MUST use the mpirun command corresponding to the mpif90 compiler specified in the arch file.		IMPORTANT: one MUST use the mpirun command corresponding to the mpif90 compiler specified in the arch file.
		+
		+	IMPORTANT: One can use at most one MPI process for every 2 points along the latitude (e.g. a maximum of 24 processes for a horizontal grid of 64x48).

	Output files (restart.nc, diagfi.nc ,etc.) are just as when running in serial.		Output files (restart.nc, diagfi.nc ,etc.) are just as when running in serial.

@@ Line 38: / Line 38: @@
 == How to run in parallel ==
-* Run interactively
+=== Run interactively ===
-** MPI only :
+* MPI only :
 <pre>
@@ Line 55: / Line 55: @@
-** Mix MPI_OMP :
+* Mix MPI_OMP :
 <pre>
@@ Line 64: / Line 64: @@
 In this exemple, each of the 2 process MPI have 2 OpenMP tasks with a 2500MB memor.
-* Run with a job scheduler
+=== Run with a job scheduler ===
 This will be different for each machine.
-Some example are provided here but will need to be adapted for each configuration and machine.
+Some example are provided here but will need to be adapted for each configuration and machine; see also pages dedicated to some clusters we use, such as [[Using the MESOIPSL cluster]] or [[Using Irene Rome]] or [[Using Adastra]]
-** MPI only :
+* MPI only :
 <pre>
@@ Line 120: / Line 120: @@
 </pre>
-** Mix MPI_OMP :
+* Mix MPI_OMP :
 <pre>
@@ Line 168: / Line 168: @@
 IMPORTANT: In this case, each core needs 2.5gb and we are using 2 OpenMP tasks for each MPI process so as_limit = 2 × 2.5.
-[[Category:Generic-LMDz]]
+[[Category:Generic-LMDZ]]

← Older revision		Revision as of 08:54, 20 March 2023
Line 168:		Line 168:
	IMPORTANT: In this case, each core needs 2.5gb and we are using 2 OpenMP tasks for each MPI process so as_limit = 2 × 2.5.		IMPORTANT: In this case, each core needs 2.5gb and we are using 2 OpenMP tasks for each MPI process so as_limit = 2 × 2.5.

−	[[Category:Generic-~~Model]]~~	+	[[Category:Generic-LMDz]]
−	~~[[Category:Mars-Model]]~~
−	~~[[Category:Venus-Model]]~~
−	~~[[Category:Titan-Model~~]]

← Older revision		Revision as of 10:15, 9 March 2023
Line 167:		Line 167:
	</pre>		</pre>
	IMPORTANT: In this case, each core needs 2.5gb and we are using 2 OpenMP tasks for each MPI process so as_limit = 2 × 2.5.		IMPORTANT: In this case, each core needs 2.5gb and we are using 2 OpenMP tasks for each MPI process so as_limit = 2 × 2.5.
		+
		+	[[Category:Generic-Model]]
		+	[[Category:Mars-Model]]
		+	[[Category:Venus-Model]]
		+	[[Category:Titan-Model]]

@@ Line 51: / Line 51: @@
 But standard output messages are written by each process.
 If using chained simulations (run mcd/run0 scripts), then the command line to run the gcm in run0 must be adapted for local settings.
 NB: LMDZ.COMMON dynamics set to run in double precision, so keep NC_DOUBLE declaration (and real to double precision promotion) in the arch files.