.. |br| raw:: html

   <br />


Runtime Model
*************

Using FleCSI requires proper initialization and configuration of the
FleCSI runtime. These examples illustrate some of the basic steps and
options that are available.

----

Example 1: Minimal
++++++++++++++++++

To use FleCSI, a ``runtime`` object must be configured and a `control policy` must be specified that describes the computation to be performed.
In the simplest case, both parts can be accomplished in one line; later examples will illustrate more advanced options.

FleCSI executes the control policy after setting up the task execution backend.
A simple control policy is supplied that calls a single function (object) given to it with a `scheduler` (used later for launching tasks) and uses its return value as an ``int`` exit status.

This example demonstrates a minimal use of FleCSI that just executes an
action to print out *Hello World*. Code for this example can be found in
*tutorial/1-runtime/1-minimal.cc*.

.. literalinclude:: ../../../tutorial/1-runtime/1-minimal.cc
  :language: cpp

----

Example 2: Program Options
++++++++++++++++++++++++++

FleCSI supports a program options capability based on `Boost Program
Options`__ to simplify the creation and management of user-defined
command-line options. The basic syntax for adding and accessing program
options is semantically similar to the Boost interface, but there are some notable
differences:

* FleCSI internally manages the ``boost::program_options::value`` variables
  for you, using ``boost::optional``.

* Positional options are the mechanism that should be used for
  *required* options.

* Default, implicit, zero, and multi value attributes are specified in
  the ``flecsi::program_option`` constructor as a ``std::initializer_list``.

This section of the tutorial provides examples of how to use FleCSI's
program option capability.

__ https://www.boost.org/doc/libs/release/doc/html/program_options.html

Example Program
^^^^^^^^^^^^^^^

In this example, imagine that you have a program that takes information about a taxi service (the options are silly and synthetic, but they demonstrate the basic usage of ``flecsi::program_option``).
The
command-line options and arguments for the program allow specification
of the following: trim level, transmission, child seat, purpose
(personal or business), light speed, and a passenger list. The first two
options will be in a *Car Options* section, while the purpose will be
under the *Ride Options* section. The passenger list is a positional
argument to the program.
Calling ``getopt::usage`` for this program produces the following:

.. code-block:: console

   Usage: ./runtime-program_options <passenger-list> 

   Positional Options:
     passenger-list The list of passengers for this trip [.txt].

   Basic Options:

   Car Options:
     -l [ --level ] arg (= 1)              Specify the trim level [1-10].
     -t [ --transmission ] arg (= manual)  Specify the transmission type 
                                           ["automatic", "manual"].
     -c [ --child-seat ] [=arg(= 1)] (= 0) Request a child seat.

   Ride Options:
     -p [ --purpose ] arg (= 1)            Specify the purpose of the trip 
                                           (personal=0, business=1).
     --lightspeed                          Travel at the speed of light.

Declaring Options
^^^^^^^^^^^^^^^^^

.. note::

  FleCSI program options must be created before calling ``flecsi::initialize`` (and must survive through all uses of their value).
  It is often convenient to declare them in a namespace in a header file (in which case, they must also be declared ``inline``).

Let's consider the first *Car Options* option, ``--level``.
To declare
this option, we use the following declaration:

.. literalinclude:: ../../../tutorial/1-runtime/2-program_options.cc
  :language: cpp
  :start-at: // Add an integer-valued command-line option with a default value
  :end-before: // Add a string-valued command-line option with a default value

First, notice that the flecsi::program_option type is templated on the
underlying option type ``int``.

This constructor to ``flecsi::program_option`` takes the following
parameters:

* ``section`` (``"Car Options"``): |br|
  Identifies the section. Sections are generated automatically, simply
  by referencing them in a program option.

* ``flag`` (``"level,l"``): |br|
  The long and short forms of the option. If the string contains a
  comma, it is split into *long name,short name*. If there is no comma,
  the string is used as the long name with no short name.

* ``help`` (``"Specify..."``) |br|
  The help description that will be displayed when the usage message
  is printed.

* ``values`` (``{{flecsi::option_default, ...}}``) |br|
  This is a
  ``std::initializer_list<flecsi::program_option::initializer_value>``.
  The possible values are ``flecsi::option_default``,
  ``flecsi::option_implicit``, ``flecsi::option_zero``, and
  ``flecsi::option_multi``. The default value is used if the option is not
  passed at invocation. The implicit value is used if the option is
  passed without a value. If zero is specified, the option does not take
  an argument, and an implicit value must be provided. If multi is
  specified, the option takes multiple values.

* ``check`` (``[](int, std::stringstream & ss) {...}``) |br|
  An optional, user-defined predicate to validate the value passed by
  the user.
  The first argument is of the option's type.

The next option ``--transmission`` is similar but uses a ``std::string``
value type:

.. literalinclude:: ../../../tutorial/1-runtime/2-program_options.cc
  :language: cpp
  :start-at: // Add a string-valued command-line option with a default value
  :end-before: // Add an option that defines an implicit value.

The only real difference is that (because the underlying type is
``std::string``) the default value is also a string.

The last option in the "Car Options" section ``--child-seat``
demonstrates the use of ``flecsi::option_implicit``:

.. literalinclude:: ../../../tutorial/1-runtime/2-program_options.cc
  :language: cpp
  :start-at: // Add an option that defines an implicit value.
  :end-before: // Add a an option to a different section,

Providing an implicit value defines the behavior for the case that the user invokes the program with the given flag but does not assign a
value, e.g., ``--child-seat`` vs. ``--child-seat=1``. The value is
*implied* by the flag itself.

.. caution::

  This style of option should not be used with positional arguments
  because Boost appears to have a bug when such options are invoked
  directly before a positional option (gets confused about separation).
  We break that convention here for the sake of completeness. If you
  need an option that simply acts as a switch (i.e., it is either *on*
  or *off*), consider using the ``--lightspeed`` style option below, as
  this type of option is safe to use with positional options.

The first option in the *Ride Options* section ``--purpose`` takes an
integer value *0* or *1*. This option is declared with the following
code:

.. literalinclude:: ../../../tutorial/1-runtime/2-program_options.cc
  :language: cpp
  :start-at: // Add a an option to a different section,
  :end-before: // Add an option with no default.

This option demonstrates how an enumeration can be used to define
possible values. Although FleCSI does not enforce correctness, the
enumeration can be used to check that the user-provided value is valid.

The next option in the *Ride Options* section ``--lightspeed`` defines
an implicit value and zero values (meaning that it takes no values). The
``--lightspeed`` option acts as a switch, taking the implicit value if
the flag is passed.  This will be useful to demonstrate how we can check
whether or not an option was passed in the next section:

.. literalinclude:: ../../../tutorial/1-runtime/2-program_options.cc
  :language: cpp
  :start-at: // Add an option with no default.
  :end-before: // Add a positional option. 

The final option in this example is a positional option: i.e., it is an
argument to the program itself.

.. literalinclude:: ../../../tutorial/1-runtime/2-program_options.cc
  :language: cpp
  :start-at: // Add a positional option. 
  :end-before: // User-defined program options are available after

Positional options are required: i.e., the program will error and print
the usage message if a value is not passed.

Checking & Using Options
^^^^^^^^^^^^^^^^^^^^^^^^

FleCSI option variables are implemented using an *optional* C++ type.
That type captures
the behavior that we want from an option (i.e., it either has a value or does not).
If the option has a value, the specific value depends on
whether or not the user explicitly passed the option on the command
line and on its default and implicit values.

Options that have a default value defined do not need to be tested:

.. literalinclude:: ../../../tutorial/1-runtime/2-program_options.cc
  :language: cpp
  :start-at: // Add cost for trim level.
  :end-before: // Add cost for lightspeed.

Here, we simply need to access the value of the option using ``value()``.

For options with no default value, we can check whether or not the
option has a value using ``has_value()``:

.. literalinclude:: ../../../tutorial/1-runtime/2-program_options.cc
  :language: cpp
  :start-at: // Add cost for lightspeed.
  :end-before: // Do something with the positional argument.

Our one positional option works like the defaulted options (because it
is required) and can be accessed using ``value()``:

.. literalinclude:: ../../../tutorial/1-runtime/2-program_options.cc
  :language: cpp
  :start-at: // Do something with the positional argument.
  :end-at: price *= passengers * 1.10 * price;

Here is the full source for this tutorial example:

.. literalinclude:: ../../../tutorial/1-runtime/2-program_options.cc
  :language: cpp

----

Example 3: Flog (FleCSI Logging Utility)
++++++++++++++++++++++++++++++++++++++++

Flog provides users with a mechanism to print logging information to
various stream buffers, similar to the C++ objects ``std::cout``, ``std::cerr``, and ``std::clog``.
Multiple streams can be used simultaneously, so that information about
the running state of a program can be captured and displayed at the same
time.
In this example, we show how Flog can be configured to stream output to
a file buffer and the ``std::clog`` stream buffer.

Before attempting this example, you should make sure that you have
configured and built FleCSI with ``ENABLE_FLOG=ON``.

.. important::

  One of the challenges of using distributed-memory and tasking runtimes
  is that output written to the console often collide because multiple
  threads of execution are all writing to the same descriptor
  concurrently.
  Flog fixes this by collecting output from different threads and
  serializing it.
  This is an important and useful feature of Flog.

Buffer Configuration
^^^^^^^^^^^^^^^^^^^^

By default, Flog does not produce any output (even when enabled).
In order to see or capture output, your application must add at least
one output stream.
This should be done after a ``flecsi::runtime`` has been created and before calling ``control`` on it.
Consider the main function for this example:

.. literalinclude:: ../../../tutorial/1-runtime/3-flog.cc
  :language: cpp
  :start-after: // simulation
  :end-at: } // main

The first output stream added is `std::clog`__.

__ https://en.cppreference.com/w/cpp/io/clog

.. literalinclude:: ../../../tutorial/1-runtime/3-flog.cc
  :language: cpp
  :start-at: // Add the standard log descriptor to Flog's buffers.
  :end-at: log::add_output_stream("clog", std::clog, true);

The arguments to ``add_output_stream`` are:

* ``label`` (``"clog"``): |br|
  This is an arbitrary label that may be used in future versions to
  enable or disable output. The label should be unique.

* ``stream buffer`` (``std::clog``): |br|
  A ``std::ostream`` object.

* ``colorize`` (``true``): |br|
  A boolean indicating whether or not output to this stream buffer
  should be colorized. It is useful to turn off colorization for
  non-interactive output.
  The default is ``false``.

To add an output stream to a file, we can do the following:

.. literalinclude:: ../../../tutorial/1-runtime/3-flog.cc
  :language: cpp
  :start-at: // Add an output file to Flog's buffers.
  :end-at: log::add_output_stream("log file", log_file);

.. important::

  Note that the ``std::ofstream`` is created (though not opened) before the ``flecsi::runtime`` object so that it is destroyed only after all logging is completed.

That's it! For this example, Flog is now configured to write output to
``std::clog``, and to *output.txt*.
Next, we will see how to actually write
output to these stream buffers.

Writing to Buffers
^^^^^^^^^^^^^^^^^^

Output with Flog is similar to ``std::cout``. Consider the Flog *info*
object:

.. code-block:: cpp

  flog(info) << "The value is " << value << std::endl;

This works just like any of the C++ output objects. Flog provides four
basic output objects: *trace*, *info*, *warn*, and *error*. These
provide different color decorations for easy identification in terminal
output and can be controlled using `strip levels` (discussed in the next
section).

The following code from this example shows some trivial usage of each of
the basic output objects:

.. literalinclude:: ../../../tutorial/1-runtime/3-flog.cc
  :language: cpp
  :start-at: // This output will always be generated because
  :end-at: flog(error) << "Error level output" << std::endl;


Controlling Output - Strip Levels
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

.. important::

  If FleCSI is configured with ``ENABLE_FLOG=OFF``, all Flog calls are
  compiled out: i.e., there is no runtime overhead.

The strip level is a runtime configuration option set via
``flog::config::strip_level``.

Valid strip levels are *[0-4]*. The default strip level is *0* (most
verbose). Depending on the strip level, Flog limits the type of messages
that are output.

* *trace* |br|
  Output written to the trace object is enabled for strip levels less
  than 1.

* *info* |br|
  Output written to the info object is enabled for strip levels less
  than 2.

* *warn* |br|
  Output written to the warn object is enabled for strip levels less
  than 3.

* *error* |br|
  Output written to the error object is enabled for strip levels less
  than 4.

Controlling Output - Tag Groups
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Tag groups provide a mechanism to control the runtime output generated
by Flog. The main idea here is that developers can use Flog to output
information that is useful in developing or debugging a program and
leave it in the code. Then, specific groups of messages can be enabled
or disabled to only output useful information for the current
development focus.

To create a new tag, we use the ``flog::tag`` type:

.. literalinclude:: ../../../tutorial/1-runtime/3-flog.cc
  :language: cpp
  :start-at: // Create some tags to control output.
  :end-at: log::tag tag2("tag2");

.. important::

  Flog tags must be declared at namespace scope.

Once you have declared a tag, it can be used to limit output to one or
more scoped regions.
The following code defines a guarded section of
output that will only be generated if *tag1* is enabled:

.. literalinclude:: ../../../tutorial/1-runtime/3-flog.cc
  :language: cpp
  :start-at: 'tag1'
  :end-at: } // scope

Here is another code example that defines a guarded section for *tag2*:

.. literalinclude:: ../../../tutorial/1-runtime/3-flog.cc
  :language: cpp
  :start-at: 'tag2'
  :end-at: } // scope

This example defines a command-line option to select a tag to enable:

.. literalinclude:: ../../../tutorial/1-runtime/3-flog.cc
  :language: cpp
  :start-at: program_option
  :end-at: ;

The selected tag is included in the configuration for the ``runtime`` object, discussed further below.

.. code-block:: console

  $ ./flog --flog=tag1
  [trace all p0] Trace level output
  [info all p0] Info level output
  [Warn all p0] Warn level output
  [ERROR all p0] Error level output
  [trace tag1 p0] Trace level output (in tag1 guard)
  [info tag1 p0] Info level output (in tag1 guard)
  [Warn tag1 p0] Warn level output (in tag1 guard)
  [ERROR tag1 p0] Error level output (in tag1 guard)

You can use ``flog::tags`` to discover all declared tags (as for displaying help).

Flog Options
^^^^^^^^^^^^

Defaults for the Flog options have been chosen in an attempt to most
closely model the behavior one would expect from the execution and
output of a standard MPI program.
However, because of the asynchronous nature of FleCSI's execution model,
it is important to understand the options that control Flog's behavior,
as it can sometimes be counter-intuitive.

As stated in the preceding sections, Flog buffers and serializes output
to avoid collisions from different threads.
FleCSI's default settings flush these buffers periodically for interactivity and case of process failure.
The Flog runtime configuration option ``serialization_interval``  defines
this behavior:

* ``flog::config::serialization_interval`` |br|
  Number of task executions between (bulk-synchronous) collections of buffered output. |br|
  *(default: 100)*

.. caution::

  It is important to understand and tune Flog serialization to your
  application.
  Serialization inhibits task asynchrony.
  When balanced, the performance effects should be very minimal.
  However, overly aggressive settings, e.g.,
  ``serialization_interval=1`` could force complete serialization
  of your application.
  This can be beneficial for debugging, but should not be used for
  actual simulation runs.

For many applications, there is a natural serialization interval that
implicitly starts at the beginning of the simulation time evolution.
FleCSI provides a function ``flecsi::flog::flush`` that can be used to
force FleCSI to serialize and flush output.

.. tip::

  Best practice for Flog serialization is to leave the default settings
  for ``serialization_interval`` and to use ``flecsi::flog::flush()``
  at an appropriate point in your application to force output.

Other parts of ``flog::config`` filter Flog output:

* ``tags`` |br|
  The tags for which to produce output.
  The example above specified just one tag (possibly "all"), but several may be supplied.

* ``verbose`` |br|
  How much additional information is output with
  your ``flog(severity)`` message.
  A value of ``-1`` will turn off any additional decorations, while a
  value of ``1`` will add additional information.
  By default, the severity level and process are output. |br|
  *(default: 0)*

* ``process`` |br|
  Which process should produce output.
  If ``-1``, enable output from all processes. |br|
  *(default: 0)*

.. caution::

  By default, Flog only writes output from process ``0``.
  Set ``process=-1`` to enable output from all processes.

.. tip::

  Logging output can sometimes have unexpected behavior.
  Consider the case where you are viewing output only from process
  ``0`` and the runtime maps a task to process ``1``.
  You will not see the messages from that task in the logging output.
  This is not an error.
  In general, some experimentation is necessary to achieve the desired
  level of output with Flog and FleCSI.

Finally, the ``flog::config::color`` runtime configuration option controls
whether coloring is enabled for Flog messages.

Example 4: Caliper Annotations
++++++++++++++++++++++++++++++

The `Caliper <https://software.llnl.gov/Caliper/>`_ Annotation interface in
FleCSI is used internally to inject Caliper instrumentation throughout the code.
This enables users to investigate runtime overhead and application performance
with Caliper.  Users can also use this interface to add additional annotations
to performance sensitive regions of their applications.

The CMake variable ``CALIPER_DETAIL`` is used to disable or control the
level of detail in included Caliper annotations.  The currently available
options are:

* ``CALIPER_DETAIL=none`` |br|
  Caliper annotations are disabled
* ``CALIPER_DETAIL=low`` |br|
  Annotations marked with low severity detail are included
* ``CALIPER_DETAIL=medium`` |br|
  Annotations marked with low and medium severity detail are included
* ``CALIPER_DETAIL=high`` |br|
  All annotations are included

.. caution::

   To use Caliper annotations with the Legion backend, the Legion option
   ``-ll:force_kthreads`` must be used.  Caliper is not aware of Legion
   user-level threads, so additional care must be practiced when using
   annotations with this runtime.


Adding Annotations
^^^^^^^^^^^^^^^^^^

In addition to instrumenting FleCSI runtime overhead, the annotation interface
can be used to add annotations to applications.  This allows users to instrument
their code and use Caliper to collect timing data.  An annotation for a code
region must specify a detail level, context, and name.  The detail level is used
to selectively control the inclusion of an annotation using the cmake variable
``CALIPER_DETAIL``.
The context for an annotation is used as a named
grouping for annotations.  In caliper, this can be used to filter and aggregate
annotations using the `caliper query language <http://software.llnl.gov/Caliper/calql.html>`_.

Scope guards are used to annotate a code region.
Consider the main function for this example:

.. literalinclude:: ../../../tutorial/1-runtime/4-caliper.cc
  :language: cpp
  :start-after: // main
  :end-at: } // main

A scope guard is used to annotate the ``control`` call:

.. literalinclude:: ../../../tutorial/1-runtime/4-caliper.cc
  :language: cpp
  :start-at: (annotation
  :end-at: run.control

For this region, the FleCSI execution context ``annotation::execution`` is
specified along with a detail level of ``annnotation::detail::low``.
To avoid hard-coding strings throughout an application, annotation regions can be
specified using structs that inherit from ``annotation::region``:

.. literalinclude:: ../../../tutorial/1-runtime/4-caliper.cc
  :language: cpp
  :start-at: struct user_execution : annotation::context<user_execution> {
  :end-before: void

This first defines a new annotation context ``user_execution`` by inheriting
from ``annotation::context`` and specifying a name for the context.  Three code
regions are then defined using this context.  The first two regions use the
default detail level of ``annotation::detail::medium``.
The main and sleeper functions are then annotated using region-based scope guards:

.. literalinclude:: ../../../tutorial/1-runtime/4-caliper.cc
  :language: cpp
  :start-at: annotation::rguard<main_region> main_guard;
  :end-at: annotation::rguard<main_region> main_guard;

.. literalinclude:: ../../../tutorial/1-runtime/4-caliper.cc
  :language: cpp
  :start-at: annotation::rguard<sleeper_subtask>(),
  :end-at: std::this_thread::sleep_for(std::chrono::milliseconds(400));

Generating Reports
^^^^^^^^^^^^^^^^^^
`Caliper configuration files
<http://software.llnl.gov/Caliper/configuration.html>`_ can be used to configure Caliper to generate reports for annotated regions of the code.
For
example, consider the following caliper configuration file:

.. literalinclude:: ../../../tutorial/1-runtime/caliper.config
  :language: bash

This file defines three caliper configuration profiles that can be used to
generate reports using the ``mpireport`` service (see
http://software.llnl.gov/Caliper/services.html).  This service aggregates
timings across all processes using ``CALI_MPI_REPORT_CONFIG`` query statements.
For
example, to run with the second configuration profile in this file (named
user), ensure ``caliper.config`` is in your working directory and run with::

  CALI_CONFIG_PROFILE=user ./runtime-caliper

When the program completes, caliper flushes the aggregated timings to a report
file named ``report.cali``::

  User-Execution  count min-time max-time total-time-%
  sleeper/subtask     1 0.400095 0.400095    57.141409
  sleeper             2 0.300089 0.300089    42.858591

The output represents collected timings for annotations in the
``User-Execution`` annotation context.

.. caution::
  For Caliper versions below v2.10, you need to replace all the ``duration.ns`` statements by ``duration``
  in the configuration file.

.. vim: set tabstop=2 shiftwidth=2 expandtab fo=cqt tw=72 :