The XIOS Library

The XIOS (Xml I/O Server) library is based on client-server principles where the server manages the outputs asynchronously from the client (the climate model) so that the bottleneck of writing data is alleviated.

Installing the XIOS library

Prerequisites

There are a couple of prerequisites to installing and using the XIOS library:

1. An MPI library must be available
2. A NetCDF4-HDF5 library, preferably compiled with MPI enabled, must be available

The rest of this page assume all prerequisites are met. People interested in building an appropriate NetCDF library on their Linux machine might be interested in the following installation script: https://web.lmd.jussieu.fr/~lmdz/pub/import/install_netcdf4_hdf5.bash (which might need some adaptations to work in your specific case).

Downloading and compiling the XIOS library

The XIOS source code is available for download using svn (subversion), e.g.:

svn co http://forge.ipsl.jussieu.fr/ioserver/svn/XIOS/trunk XIOS

• To compile the library, one must first have adequate architecture "arch" files at hand, just like for the GCM (see The_Target Architecture_("arch")_Files). In principle both arch.env and arch.path files could be the same as for the GCM; arch.fcm will of course differ, as XIOS source code is in C++ (along with a Fortran interface). If using a "known" machine (e.g. Occigen, Irene-Rome, Ciclad) then ready-to-use up-to-date arch files for that machine should be present in the arch directory. If not you will have to create your own (it is advised to use the existing ones as templates!)
• Assuming some_machine arch files (i.e. files arch-some_machine.env, arch-some_machine.path, arch-some_machine.fcm) are present in the arch subdirectory, compiling the XIOS is done using the dedicated make_xios script, e.g.:

./make_xios --prod --arch some_machine --job 8

If the compilation steps went well then the lib directory should contain file libxios.a and the bin directory should contain

fcm_env.ksh  generic_testcase.exe  xios_server.exe

XIOS documentation

Note that the downloaded XIOS distribution includes some documentation in the doc subdirectory:

reference_xml.pdf  XIOS_reference_guide.pdf  XIOS_user_guide.pdf

Definitely worth checking out!

Compiling the GCM with the XIOS library

To compile with XIOS enabled, one must specify the option

 -io xios

to the makelmdz_fcm script.

XIOS output controls

All aspects of the outputs (name, units, file, post-processing operations, etc.) are controlled by dedicated XML files which are read at run-time. Samples of xml files are provided in the "deftank" directory.

In a nutshell

• the master file read by XIOS is iodef.xml; and contains specific XIOS parameters such as using_server to dictate whether XIOS is run in client-server mode (true) or attached (false) mode, info_level to set the verbosity of XIOS messages (0: none, 100: very verbose), print_file to set whether XIOS messages will be sent to standard output (false) or dedicated xios_*.out and xios_*.err files (true).

                        <variable id="using_server" type="bool">false</variable>
                        <variable id="info_level" type="int">0</variable>
                        <variable id="print_file" type="bool"> false </variable>

• It is common practice to have LMDZ-related definitions and outputs in separate XML files, e.g. context_lmdz.xml which are included in iodef.xml via the src attribute, e.g.

       <context id="LMDZ" src="./context_lmdz_physics.xml"/>

The context_lmdz_physics.xml file must then contain all fields/grid/file output definitions, which may be split into multiple XML files, for instance the definition of model variables (i.e. all fields that may be outputed) is often put in a separate file field_def_physics.xml which is referenced within context_lmdz_physics.xml as:

       <field_definition src="./field_def_physics.xml" />

Concerning output files, the current recommended practice is to use separate file_def_histsomething_lmdz.xml files, one for each histsomething.nc file to generate, and include these in context_lmdz.xml using the file_definition key. e.g.:

  <!-- Define output files
              Each file contains the list of variables and their output levels -->
  <file_definition src="./file_def_histins.xml"/>
  <file_definition src="./file_def_specIR.xml"/>

Thorougher description with illustrative examples

TODO: PUT SOME SIMPLE ILLUSTRATIVE EXAMPLES HERE

Saving or loading interpolation weights

With the XIOS library one can define output domains (grid) which are different from input domains (grids), and XIOS does the necessary interpolation.

This requires, once source and destination grids are known, to compute some interpolation weights (during the initialization step). For large grids, this can take some time. One can however tell XIOS to save the interpolation weights in a file and use that file (if it is present) rather than recompute them when a new simulation is ran.

In practice one must add extra keys to the "interpolate_domain" tag, e.g.:

    <domain id="dom_256_192" type="rectilinear" ni_glo="256" nj_glo="192" >
      <generate_rectilinear_domain/>
      <interpolate_domain order="1" write_weight="true" mode="read_or_compute" />
    </domain>

This will automatically generate a NetCDF file containing the weights. Default file name will be something like xios_interpolation_weights_CONTEXT_INPUTDOMAIN_OUTPUTDOMAIN.nc , where CONTEXT, INPUTDOMAIN and OUTPUTDOMAIN are inherited from the context (i.e. definitions of these in the xml files).

One can specify the name of the file with the key "weight_filename", e.g.

    <domain id="dom_256_192" type="rectilinear" ni_glo="256" nj_glo="192" >
      <generate_rectilinear_domain/>
      <interpolate_domain order="1" write_weight="true" mode="read_or_compute" weight_filename="xios_weights" />
    </domain>

It can also happen that for a given variable we want the interpolation not to be conservative. For example, a variable like the area of a mesh grid should not be interpolated between different domains. Since the interpolation is specific to a domain (and defined in the "domain id"), we have to create a new domain for all the variable that should be interpolated in another way. For the variable "Area" for example, the syntax is as follow :

• Create the new domain:

<domain id="dom_64_48_quantity_T" type="rectilinear" ni_glo="64" nj_glo="48"   >
   <generate_rectilinear_domain/>
   <interpolate_domain quantity="true"/>
</domain>

• Assign the variable to this domain:

Later in the context file, the variable should be outputted using this new domain (note that it still can be outputed in the same file as the other variables) :

<field_group domain_ref="dom_64_48_quantity_T">
  <field_group operation="instant"
     freq_op="1ts">
    <field field_ref="area" operation="once" />
  </field_group>
</field_group>

Using the XIOS library in client-server mode

To run with XIOS in client-server mode requires the following:

• The client-server mode should be activated (in file iodef.xml):

                        <variable id="using_server" type="bool">true</variable>

• The xios_server.exe executable should be present alongside the GCM executable gcm_***.e and they should be run together in MPMD (Multiple Programs, Multiple Data) mode : some of the MPI processes being allocated to the GCM and the others to XIOS ; in practice much less are needed by XIOS than the GCM, this however also depends on the amount of outputs and postprocessing computations, e.g. temporal averaging and grid interpolations, that XIOS will have to do. For example if the MPI execution wrapper is mpirun and that 26 processes are to be used by the GCM gcm_64x52x20_phystd_para.e and 2 by XIOS (i.e. using overall 28 processes):

mpirun -np 26 gcm_64x52x20_phystd_para.e > gcm.out 2>&1 : -np 2 xios_server.exe