Runtime Model

Using FleCSI requires proper initialization and configuration of the FleCSI runtime. These examples illustrate some of the basic steps and options that are available.

Example 1: Minimal

To use FleCSI, a runtime object must be configured and a control policy must be specified that describes the computation to be performed. In the simplest case, both parts can be accomplished in one line; later examples will illustrate more advanced options.

FleCSI executes the control policy after setting up the task execution backend. A simple control policy is supplied that calls a single function (object) given to it with no arguments and uses its return value as an int exit status.

This example demonstrates a minimal use of FleCSI that just executes an action to print out Hello World. Code for this example can be found in tutorial/1-runtime/1-minimal.cc.

#include <flecsi/execution.hh>
#include <flecsi/runtime.hh>

/*
  The top-level action can be any C/C++ function that takes no arguments and
  returns an int.

  In this simple example, we only print a message to indicate that the
  top-level action was actually executed by FleCSI. However, in a real
  application, the top-level action would execute FleCSI tasks and other
  functions to implement the simulation.
 */

int
top_level_action() {
  std::cout << "Hello World" << std::endl;
  return 0;
} // top_level_action

/*
  The main function must create a FleCSI \c runtime object.  Otherwise, the
  implementation of main is left to the user.
 */

int
main() {
  const flecsi::run::dependencies_guard dg;
  /*
    flecsi::run::call means to call the single function given as an argument to
    run.main.  (Multiple functions will be covered in later examples.)
    It will be skipped if, say, --help was passed as an argument; FleCSI's
    command-line support is documented in the next example.
   */
  return flecsi::runtime().control<flecsi::run::call>(top_level_action);
} // main

Example 2: Program Options

FleCSI supports a program options capability based on Boost Program Options to simplify the creation and management of user-defined command-line options. The basic syntax for adding and accessing program options is semantically similar to the Boost interface (You can find documentation using the above link.) However, there are some notable differences:

  • FleCSI internally manages the boost::program_options::value variables for you, using boost::optional.

  • Positional options are the mechanism that should be used for required options.

  • Default, implicit, zero, and multi value attributes are specified in the flecsi::program_option constructor as an std::initializer_list.

This section of the tutorial provides examples of how to use FleCSI’s program option capability.

Example Program

In this example, imagine that you have a program that takes information about a taxi service (The options are silly and synthetic. However they demonstrate the basic usage of the flecsi::program_option type.) The command-line options and arguments for the program allow specification of the following: trim level, transmission, child seat, purpose (personal or business), light speed, and a passenger list. The first two options will be in a Car Options section, while the purpose will be under the Ride Options section. The passenger list is a positional argument to the program. Calling getopt::usage for this program produces the following:

Usage: ./runtime-program_options <passenger-list>

Positional Options:
  passenger-list The list of passengers for this trip [.txt].

Basic Options:

Car Options:
  -l [ --level ] arg (= 1)              Specify the trim level [1-10].
  -t [ --transmission ] arg (= manual)  Specify the transmission type
                                        ["automatic", "manual"].
  -c [ --child-seat ] [=arg(= 1)] (= 0) Request a child seat.

Ride Options:
  -p [ --purpose ] arg (= 1)            Specify the purpose of the trip
                                        (personal=0, business=1).
  --lightspeed                          Travel at the speed of light.

Declaring Options

Note

FleCSI program options must be created before calling flecsi::initialize (and must survive through all uses of their value). It is often convenient to declare them in a namespace in a header file (in which case, they must also be declared inline).

Let’s consider the first Car Options option: --level. To declare this option, we use the following declaration:

// Add an integer-valued command-line option with a default value of '1'. The
// option will have a long form --level, and a short form -l. The option will be
// added under the "Car Options" section.

flecsi::program_option<int> trim("Car Options",
  "level,l",
  "Specify the trim level [1-10].",
  {{flecsi::option_default, 1}},
  [](int value, std::stringstream & ss) {
    return (value > 0 && value < 11) ||
           (ss << "value(" << value << ") out-of-range", false);
  });

First, notice that the flecsi::program_option type is templated on the underlying option type int. In general, this can be any valid C++ type.

This constructor to flecsi::program_option takes the following parameters:

  • section (“Car Options”):
    Identifies the section. Sections are generated automatically, simply by referencing them in a program option.

  • flag (“level,l”):
    The long and short forms of the option. If the string contains a comma, it is split into long name,short name. If there is no comma, the string is used as the long name with no short name.

  • help (“Specify…”)
    The help description that will be displayed when the usage message is printed.

  • values ({{flecsi::option_default, …}})
    This is a std::initializer_list<flecsi::program_option::initializer_value<int>>. The possible values are flecsi::option_default, flecsi::option_implicit, flecsi::option_zero, and flecsi::option_multi. The default value is used if the option is not passed at invocation. The implicit value is used if the option is passed without a value. If zero is specified, the option does not take an argument, and an implicit value must be provided. If multi is specified, the option takes multiple values.

  • check ([](int, std::stringstream & ss) {…})
    An optional, user-defined predicate to validate the value passed by the user. The first argument is of the option’s type.

The next option --transmission is similar but uses a std::string value type:

// Add a string-valued command-line option with a default value of "manual". The
// option will have a long form --transmission, and a short form -t. The option
// will be added under the "Car Options" section.

flecsi::program_option<std::string> transmission("Car Options",
  "transmission,t",
  "Specify the transmission type [\"automatic\", \"manual\"].",
  {{flecsi::option_default, "manual"}},
  [](const std::string & value, std::stringstream & ss) {
    return value == "manual" || value == "automatic" ||
           (ss << "option(" << value << ") is invalid", false);
  });

The only real difference is that (because the underlying type is std::string) the default value is also a string.

The last option in the “Car Options” section --child-seat demonstrates the use of flecsi::option_implicit:

// Add an option that defines an implicit value. If the program is invoked with
// --child-seat, the value will be true. If it is invoked without --child-seat,
// the value will be false. This style of option should not be used with
// positional arguments because Boost appears to have a bug when such options
// are invoked directly before a positional option (gets confused about
// separation). We break that convention here for the sake of completeness.

flecsi::program_option<bool> child_seat("Car Options",
  "child-seat,c",
  "Request a child seat.",
  {{flecsi::option_default, false}, {flecsi::option_implicit, true}});

Providing an implicit value defines the behavior for the case that the user invokes the program with the given flag but does not assign a value, e.g., --child-seat vs. --child-seat=1. The value is implied by the flag itself.

Caution

This style of option should not be used with positional arguments because Boost appears to have a bug when such options are invoked directly before a positional option (gets confused about separation). We break that convention here for the sake of completeness. If you need an option that simply acts as a switch (i.e., it is either on or off), consider using the --lightspeed style option below, as this type of option is safe to use with positional options.

The first option in the Ride Options section --purpose takes an integer value 0 or 1. This option is declared with the following code:

// Add a an option to a different section, i.e., "Ride Options". The enumeration
// type is not enforced by the FleCSI runtime, and is mostly for convenience.

enum purpose_option : size_t { personal, business };

flecsi::program_option<size_t> purpose("Ride Options",
  "purpose,p",
  "Specify the purpose of the trip (personal=0, business=1).",
  {{flecsi::option_default, purpose_option::business}},
  [](std::size_t value, std::stringstream & ss) {
    return value == personal || value == business ||
           (ss << "value(" << value << ") is invalid", false);
  });

This option demonstrates how an enumeration can be used to define possible values. Although FleCSI does not enforce correctness, the enumeration can be used to check that the user-provided value is valid.

The next option in the Ride Options section --lightspeed defines an implicit value and zero values (meaning that it takes no values). The --lightspeed option acts as a switch, taking the implicit value if the flag is passed. This will be useful to demonstrate how we can check whether or not an option was passed in the next section:

// Add an option with no default. This will allow us to demonstrate testing an
// option with has_value().

flecsi::program_option<bool> lightspeed("Ride Options",
  "lightspeed",
  "Travel at the speed of light.",
  {{flecsi::option_implicit, true}, {flecsi::option_zero}});

The final option in this example is a positional option: i.e., it is an argument to the program itself.

// Add a positional option. This uses a different constructor from the previous
// option declarations. Positional options are a replacement for required
// options (in the normal boost::program_options interface).

flecsi::program_option<std::string> passenger_list("passenger-list",
  "The list of passengers for this trip [.txt].",
  1,
  [](const std::string & value, std::stringstream & ss) {
    return value.find(".txt") != std::string::npos ||
           (ss << "file(" << value << ") has invalid suffix", false);
  });

Positional options are required: i.e., the program will error and print the usage message if a value is not passed.

Checking & Using Options

FleCSI option variables are implemented using an optional C++ type. The utility of this implementation is that optional already captures the behavior that we want from an option (i.e., it either has a value or does not). If the option has a value, the specific value depends on whether or not the user explicitly passed the option on the command line and on its default and implicit values.

Options that have a default value defined do not need to be tested:

  // Add cost for trim level. This option does not have to be checked with
  // has_value() because it is defaulted. It is also unnecessary to check the
  // value because it was declared with a validator function.

  price += trim.value() * 100.0;

  // Add cost for automatic transmission.

  price += transmission.value() == "automatic" ? 200.0 : 0.0;

  // Add cost for child seat.

  if(child_seat.value()) {
    price += 50.0;
  }

  // Deduction for business.

  if(purpose.value() == business) {
    price *= 0.8;
  }

Here, we simply need to access the value of the option using the value() method.

For options with no default value, we can check whether or not the option has a value using the has_value() method:

  // Add cost for lightspeed. Since this option does not have a default, we need
  // to check whether or not the flag was passed.

  if(lightspeed.has_value()) {
    price += 1000000.0;
  }

Our one positional option works like the defaulted options (because it is required) and can be accessed using the value() method:

  // Do something with the positional argument.

  auto read_file = [](std::string const &) {
    // Read passengers...
    return 5;
  };

  size_t passengers = read_file(passenger_list.value());

  price *= passengers * 1.10 * price;

Here is the full source for this tutorial example:

#include <flecsi/execution.hh>
#include <flecsi/runtime.hh>

// Add an integer-valued command-line option with a default value of '1'. The
// option will have a long form --level, and a short form -l. The option will be
// added under the "Car Options" section.

flecsi::program_option<int> trim("Car Options",
  "level,l",
  "Specify the trim level [1-10].",
  {{flecsi::option_default, 1}},
  [](int value, std::stringstream & ss) {
    return (value > 0 && value < 11) ||
           (ss << "value(" << value << ") out-of-range", false);
  });

// Add a string-valued command-line option with a default value of "manual". The
// option will have a long form --transmission, and a short form -t. The option
// will be added under the "Car Options" section.

flecsi::program_option<std::string> transmission("Car Options",
  "transmission,t",
  "Specify the transmission type [\"automatic\", \"manual\"].",
  {{flecsi::option_default, "manual"}},
  [](const std::string & value, std::stringstream & ss) {
    return value == "manual" || value == "automatic" ||
           (ss << "option(" << value << ") is invalid", false);
  });

// Add an option that defines an implicit value. If the program is invoked with
// --child-seat, the value will be true. If it is invoked without --child-seat,
// the value will be false. This style of option should not be used with
// positional arguments because Boost appears to have a bug when such options
// are invoked directly before a positional option (gets confused about
// separation). We break that convention here for the sake of completeness.

flecsi::program_option<bool> child_seat("Car Options",
  "child-seat,c",
  "Request a child seat.",
  {{flecsi::option_default, false}, {flecsi::option_implicit, true}});

// Add a an option to a different section, i.e., "Ride Options". The enumeration
// type is not enforced by the FleCSI runtime, and is mostly for convenience.

enum purpose_option : size_t { personal, business };

flecsi::program_option<size_t> purpose("Ride Options",
  "purpose,p",
  "Specify the purpose of the trip (personal=0, business=1).",
  {{flecsi::option_default, purpose_option::business}},
  [](std::size_t value, std::stringstream & ss) {
    return value == personal || value == business ||
           (ss << "value(" << value << ") is invalid", false);
  });

// Add an option with no default. This will allow us to demonstrate testing an
// option with has_value().

flecsi::program_option<bool> lightspeed("Ride Options",
  "lightspeed",
  "Travel at the speed of light.",
  {{flecsi::option_implicit, true}, {flecsi::option_zero}});

// Add a positional option. This uses a different constructor from the previous
// option declarations. Positional options are a replacement for required
// options (in the normal boost::program_options interface).

flecsi::program_option<std::string> passenger_list("passenger-list",
  "The list of passengers for this trip [.txt].",
  1,
  [](const std::string & value, std::stringstream & ss) {
    return value.find(".txt") != std::string::npos ||
           (ss << "file(" << value << ") has invalid suffix", false);
  });

// User-defined program options are available after getopt has been invoked.

int
top_level_action() {
  double price{0.0};

  // Add cost for trim level. This option does not have to be checked with
  // has_value() because it is defaulted. It is also unnecessary to check the
  // value because it was declared with a validator function.

  price += trim.value() * 100.0;

  // Add cost for automatic transmission.

  price += transmission.value() == "automatic" ? 200.0 : 0.0;

  // Add cost for child seat.

  if(child_seat.value()) {
    price += 50.0;
  }

  // Deduction for business.

  if(purpose.value() == business) {
    price *= 0.8;
  }

  // Add cost for lightspeed. Since this option does not have a default, we need
  // to check whether or not the flag was passed.

  if(lightspeed.has_value()) {
    price += 1000000.0;
  }

  // Do something with the positional argument.

  auto read_file = [](std::string const &) {
    // Read passengers...
    return 5;
  };

  size_t passengers = read_file(passenger_list.value());

  price *= passengers * 1.10 * price;

  std::cout << "Price: $" << price << std::endl;

  return 0;
} // top_level_action

int
main(int argc, char ** argv) {
  const flecsi::getopt g;
  try {
    g(argc, argv);
  }
  catch(const std::logic_error & e) {
    std::cerr << e.what() << '\n' << g.usage(argc ? argv[0] : "");
    return 1;
  }
  const flecsi::run::dependencies_guard dg;
  return flecsi::runtime().control<flecsi::run::call>(top_level_action);
} // main

Example 3: FLOG (FleCSI Logging Utility)

FLOG provides users with a mechanism to print logging information to various stream buffers, similar to the C++ objects std::cout, std::cerr, and std::clog. Multiple streams can be used simultaneously, so that information about the running state of a program can be captured and displayed at the same time. In this example, we show how FLOG can be configured to stream output to a file buffer and the std::clog stream buffer.

Before attempting this example, you should make sure that you have configured and built FleCSI with ENABLE_FLOG=ON.

Important

One of the challenges of using distributed-memory and tasking runtimes is that output written to the console often collide because multiple threads of execution are all writing to the same descriptor concurrently. FLOG fixes this by collecting output from different threads and serializing it. This is an important and useful feature of FLOG.

Buffer Configuration

By default, FLOG does not produce any output (even when enabled). In order to see or capture output, your application must add at least one output stream. This should be done after a flecsi::runtime has been created and before calling control on it. Consider the main function for this example:

int
main(int argc, char ** argv) {
  std::ofstream log_file; // to outlive runtime
  flecsi::getopt()(argc, argv);
  const run::dependencies_guard dg;
  // If FLECSI_ENABLE_FLOG is enabled, FLOG will automatically be initialized
  // when the runtime is created.
  run::config cfg;
  cfg.flog.tags = {tag};
  const runtime run(cfg);

  // In order to see or capture any output from FLOG, the user must add at least
  // one output stream. The function flog::add_output_stream provides an
  // interface for adding output streams to FLOG. The FleCSI runtime must have
  // been initialized before this function can be invoked.

  // Add the standard log descriptor to FLOG's buffers.

  flog::add_output_stream("clog", std::clog, true);

  // Add an output file to FLOG's buffers.

  log_file.open("output.txt");
  flog::add_output_stream("log file", log_file);

  return run.control<run::call>(top_level_action);
} // main

The first output stream added is std::clog.

  // Add the standard log descriptor to FLOG's buffers.

  flog::add_output_stream("clog", std::clog, true);

The arguments to add_output_stream are:

  • label (“clog”):
    This is an arbitrary label that may be used in future versions to enable or disable output. The label should be unique.

  • stream buffer (std::clog):
    A std::ostream object.

  • colorize (true):
    A boolean indicating whether or not output to this stream buffer should be colorized. It is useful to turn off colorization for non-interactive output. The default is false.

To add an output stream to a file, we can do the following:

  // Add an output file to FLOG's buffers.

  log_file.open("output.txt");
  flog::add_output_stream("log file", log_file);

Important

Note that the std::ofstream is created (though not opened) before the flecsi::runtime object so that it is destroyed only after all logging is completed.

That’s it! For this example, FLOG is now configured to write output to std::clog, and to output.txt. Next, we will see how to actually write output to these stream buffers.

Writing to Buffers

Output with FLOG is similar to std::cout. Consider the FLOG info object:

flog(info) << "The value is " << value << std::endl;

This works just like any of the C++ output objects. FLOG provides four basic output objects: trace, info, warn, and error. These provide different color decorations for easy identification in terminal output and can be controlled using strip levels (discussed in the next section).

The following code from this example shows some trivial usage of each of the basic output objects:

  // This output will always be generated because it is not scoped within a tag
  // guard.

  flog(trace) << "Trace level output" << std::endl;
  flog(info) << "Info level output" << std::endl;
  flog(warn) << "Warn level output" << std::endl;
  flog(error) << "Error level output" << std::endl;

Controlling Output - Strip Levels

Important

If FleCSI is configured with ENABLE_FLOG=OFF, all FLOG calls are compiled out: i.e., there is no runtime overhead.

The strip level is a runtime configuration option set via flog::config::strip_level.

Valid strip levels are [0-4]. The default strip level is 0 (most verbose). Depending on the strip level, FLOG limits the type of messages that are output.

  • trace
    Output written to the trace object is enabled for strip levels less than 1.

  • info
    Output written to the info object is enabled for strip levels less than 2.

  • warn
    Output written to the warn object is enabled for strip levels less than 3.

  • error
    Output written to the error object is enabled for strip levels less than 4.

Controlling Output - Tag Groups

Tag groups provide a mechanism to control the runtime output generated by FLOG. The main idea here is that developers can use FLOG to output information that is useful in developing or debugging a program and leave it in the code. Then, specific groups of messages can be enabled or disabled to only output useful information for the current development focus.

To create a new tag, we use the flog::tag type:

// Create some tags to control output.

flog::tag tag1("tag1");
flog::tag tag2("tag2");

Tags take a single std::string argument that is used in the help message to identify available tags.

Important

FLOG tags must be declared at namespace scope.

Once you have declared a tag, it can be used to limit output to one or more scoped regions. The following code defines a guarded section of output that will only be generated if tag1 is enabled:

  // This output will appear only if 'tag1' is enabled.

  {
    flog::guard guard(tag1);
    flog(trace) << "Trace level output (in tag1 guard)" << std::endl;
    flog(info) << "Info level output (in tag1 guard)" << std::endl;
    flog(warn) << "Warn level output (in tag1 guard)" << std::endl;
    flog(error) << "Error level output (in tag1 guard)" << std::endl;
  } // scope

Here is another code example that defines a guarded section for tag2:

  // This output will be generated only if 'tag2' is enabled.

  {
    flog::guard guard(tag2);
    flog(trace) << "Trace level output (in tag2 guard)" << std::endl;
    flog(info) << "Info level output (in tag2 guard)" << std::endl;
    flog(warn) << "Warn level output (in tag2 guard)" << std::endl;
    flog(error) << "Error level output (in tag2 guard)" << std::endl;
  } // scope

This example defines a command-line option to select a tag to enable:

flecsi::program_option<std::string> tag("Logging",
  "flog",
  "Specify the flog tag to enable.",
  {{flecsi::option_default, "all"}});

The selected tag is included in the configuration for the runtime object, discussed further below.

$ ./flog --flog=tag1
[trace all p0] Trace level output
[info all p0] Info level output
[Warn all p0] Warn level output
[ERROR all p0] Error level output
[trace tag1 p0] Trace level output (in tag1 guard)
[info tag1 p0] Info level output (in tag1 guard)
[Warn tag1 p0] Warn level output (in tag1 guard)
[ERROR tag1 p0] Error level output (in tag1 guard)

You can use flog::tags to discover all declared tags (as for displaying help).

FLOG Options

Defaults for the FLOG options have been chosen in an attempt to most closely model the behavior one would expect from the execution and output of a standard MPI program. However, because of the asynchronous nature of FleCSI’s execution model, it is important to understand the options that control FLOG’s behavior, as it can sometimes be counter-intuitive.

As stated in the preceding sections, FLOG buffers and serializes output to avoid collisions from different threads. As a safeguard, FleCSI’s default settings flush these buffers periodically, so as to avoid memory capacity issues. The FLOG runtime configuration option serialization_interval defines this behavior:

  • flog::config::serialization_interval
    The serialization interval specifies how often FleCSI should check for buffered output (requires reduction) as a number of tasks executed: i.e., if the serialization interval is set to 300, FleCSI will check how many messages have been injected into the stream of each process every multiple of 300 task executions.
    (default: 100)

Caution

It is important to understand and tune FLOG serialization to your application. Serialization inhibits task asynchrony. When balanced, the performance effects should be very minimal. However, overly aggressive settings, e.g., serialization_interval=1 could force complete serialization of your application. This can be beneficial for debugging, but should not be used for actual simulation runs.

For many applications, there is a natural serialization interval that implicitly starts at the beginning of the simulation time evolution. FleCSI provides a function flecsi::flog::flush() that can be used to force FleCSI to serialize and flush output.

Tip

Best practice for FLOG serialization is to leave the default settings for serialization_interval and to use flecsi::flog::flush() at an appropriate point in your application to force output.

Other parts of flog::config filter Flog output:

  • tags
    The tags for which to produce output. The example above specified just one tag (possibly “all”), but several may be supplied.

  • verbose
    How much additional information is output with your flog(severity) message. A value of -1 will turn off any additional decorations, while a value of 1 will add additional information. By default, the severity level and process are output.
    (default: 0)

  • process
    Which process should produce output. If -1, enable output from all processes.
    (default: 0)

Caution

By default, FLOG only writes output from process 0. Set process=-1 to enable output from all processes.

Tip

Logging output can sometimes have unexpected behavior. Consider the case where you are viewing output only from process 0 and the runtime maps a task to process 1. You will not see the messages from that task in the logging output. This is not an error. In general, some experimentation is necessary to achieve the desired level of output with FLOG and FleCSI.

Finally, the flog::config::color runtime configuration option controls whether coloring is enabled for FLOG messages.

Example 4: Caliper Annotations

The Caliper Annotation interface in FleCSI is used internally to inject Caliper instrumentation throughout the code. This enables users to investigate runtime overhead and application performance with Caliper. Users can also use this interface to add additional annotations to performance sensitive regions of their applications.

To CMake variable CALIPER_DETAIL is used to disable or control the level of detail in included Caliper annotations. The currently available options are:

  • CALIPER_DETAIL=none
    Caliper annotations are disabled

  • CALIPER_DETAIL=low
    Annotations marked with low severity detail are included

  • CALIPER_DETAIL=medium
    Annotations marked with low and medium severity detail are included

  • CALIPER_DETAIL=high
    All annotations are included

Caution

To use Caliper annotations with the Legion backend, the Legion option -ll:force_kthreads must be used. Caliper is not aware of Legion user-level threads, so additional care must be practiced when using annotations with this runtime.

Adding Annotations

In addition to instrumenting FleCSI runtime overhead, the annotation interface can be used to add annotations to applications. This allows users to instrument their code and use Caliper to collect timing data. An annotation for a code region must specify a detail level, context, and name. The detail level is used to selectively control the inclusion of an annotation using the cmake variable CALIPER_DETAIL. The context for an annotation is used as a named grouping for annotations. In caliper, this can be used to filter and aggregate annotations using the caliper query language.

Scope guards are used to annotate a code region. Consider the main function for this example:

int
main() {
  annotation::rguard<main_region> main_guard;

  const run::dependencies_guard dg;
  const runtime run;
  return (annotation::guard<annotation::execution, annotation::detail::low>(
            "top-level-task"),
    run.control<run::call>(top_level_action));
} // main

A scope guard is used to annotate the top level task:

  return (annotation::guard<annotation::execution, annotation::detail::low>(
            "top-level-task"),
    run.control<run::call>(top_level_action));

For this region, the FleCSI execution context annotation::execution is specified along with a detail level of annnotation::detail::low. To avoid hard coding strings throughout an application, annotation regions can be specified using structs that inherit from annotation::region:

struct user_execution : annotation::context<user_execution> {
  static constexpr char name[] = "User-Execution";
};

struct main_region : annotation::region<annotation::execution> {
  inline static const std::string name{"main"};
};

struct sleeper_region : annotation::region<user_execution> {
  inline static const std::string name{"sleeper"};
};

struct sleeper_subtask : annotation::region<user_execution> {
  inline static const std::string name{"subtask"};
  static constexpr annotation::detail detail_level = annotation::detail::high;
};

This first defines a new annotation context user_execution by inheriting from annotation::context and specifying a name for the context. Three code regions are then defined using this context. The first two regions use the default detail level of annotation::detail::medium. The main and sleeper functions are then annotated using region-based scope guards:

  annotation::rguard<main_region> main_guard;

  annotation::rguard<sleeper_subtask>(),
    std::this_thread::sleep_for(std::chrono::milliseconds(400));

Generating Reports

Caliper configuration files can be used to generate configure caliper to generate reports for annotated regions of the code. For example, consider the following caliper configuration file:

# [flecsi]
CALI_SERVICES_ENABLE=aggregate,event,mpi,mpireport,timestamp
CALI_MPIREPORT_FILENAME=report.cali
CALI_EVENT_ENABLE_SNAPSHOT_INFO=false
CALI_TIMER_SNAPSHOT_DURATION=true
CALI_MPIREPORT_CONFIG="select FleCSI-Execution, count(),min(sum#time.duration.ns) as \"min-time\", max(sum#time.duration.ns) as \"max-time\", percent_total(sum#time.duration.ns) as \"total-time-%\" where FleCSI-Execution format table order by percent_total#sum#time.duration.ns desc"

# [user]
CALI_SERVICES_ENABLE=aggregate,event,mpi,mpireport,timestamp
CALI_MPIREPORT_FILENAME=report.cali
CALI_EVENT_ENABLE_SNAPSHOT_INFO=false
CALI_TIMER_SNAPSHOT_DURATION=true
CALI_MPIREPORT_CONFIG="select User-Execution, count(),min(sum#time.duration.ns) as \"min-time\", max(sum#time.duration.ns) as \"max-time\", percent_total(sum#time.duration.ns) as \"total-time-%\" where User-Execution format table order by percent_total#sum#time.duration.ns desc"

# [all]
CALI_SERVICES_ENABLE=aggregate,event,mpi,mpireport,timestamp
CALI_MPIREPORT_FILENAME=report.cali
CALI_EVENT_ENABLE_SNAPSHOT_INFO=false
CALI_TIMER_SNAPSHOT_DURATION=true
CALI_MPIREPORT_CONFIG="select FleCSI-Execution,User-Execution,count(),min(sum#time.duration.ns) as \"min-time\", max(sum#time.duration.ns) as \"max-time\", percent_total(sum#time.duration.ns) as \"total-time-%\" format table order by percent_total#sum#time.duration.ns desc"

This file defines three caliper configuration profiles that can be used to generate reports using the mpireport service (see http://software.llnl.gov/Caliper/services.html). This service aggregates timings across all ranks using CALI_MPI_REPORT_CONFIG query statements. For example, to run with the second configuration profile in this file (named user), ensure caliper.config is in your working directory and run with:

CALI_CONFIG_PROFILE=user ./runtime-caliper

When the program completes, caliper flushes the aggregated timings to a report file named report.cali:

User-Execution  count min-time max-time total-time-%
sleeper/subtask     1 0.400095 0.400095    57.141409
sleeper             2 0.300089 0.300089    42.858591

The output represents collected timings for annotations in the User-Execution annotation context.

Caution

For Caliper versions below v2.10, you need to replace all the duration.ns statements by duration in the configuration file.