Release Notes

flecsi:: qualification is omitted throughout.

Changes in v2.4.0 (June 5 2025)

Possible incompatibilities

• Build

  • Kokkos is now a required build dependency.

  • OpenMP support depends on Kokkos configuration.

• Data

  • The automatic resizing for ragged fields no longer occurs after a manual update of the sizes. (This was changed without a release note in 2.3.2.)

Deprecated

• Build

  • ENABLE_KOKKOS — is ignored.

  • ENABLE_OPENMP — is ignored.

• Runtime

  • runtime::control on a const object — use a non-const object

  • existing control::invoke — provide a scheduler

  • initializing run::call with a nullary function — accept a scheduler

  • process and processes — use runtime versions

  • color and colors — use the space_base members

• Data

  • topology_slot — use topology or (for movability) ptr

  • partition_privilege_t — use privilege

• Execution

  • future<R, index>::get — pass to a task or use all

  • make_partial — use a lambda or std::bind, which can now be task arguments

  • default_accelerator — use accelerator

  • idempotent task attribute — has never had any effect

• Topologies

  • unstructured_base::coloring::index_space::entities — omit from initialization

• On-node parallelism

  • parallel_for, parallel_reduce — use accelerator

  • forall, reduceall without an execution space — use accelerator

• Utilities

  • nullary unit-test functions — accept a scheduler

  • util::annotation::context::ann — use guard or rguard

New features

• Build

  • FLECSI_BACKEND identifies the backend in use.

  • FLECSI_ENABLE_FLOG is defined as a macro if appropriate.

  • FLECSI_CALIPER_DETAIL reports the level of annotation detail.

  • flecsi_enable_mpi no longer requires a second argument (to request the unavailable C++ bindings).

• Runtime

  • control::invoke accepts a scheduler (typically provided by runtime::control).

  • control_base::scheduler provides access to the scheduler with which actions can launch tasks.

  • run::call supports functions that accept a scheduler.

  • control::write_graph and control::write_actions write Dot graphs of a control model. (This was added without a release note in 2.3.0.)

• Data

  • topology is the type of topology instances; unlike topology_slot, it is immovable.

    • specialization::topology is a convenience alias for it.

    • specialization::ptr is the appropriate std::unique_ptr type to hold a topology instance; like the deprecated topology_slot, it is movable and can defer initialization.

    • scheduler::allocate fills in a ptr and calls specialization::initialize (which is not otherwise automatic) with the scheduler and new topology instance.

    • get and operator-> in topology_slot access the topology.

  • Mutators support tracing.

  • privilege is the new name of partition_privilege_t.

  • Launch map rvalues can be used to create field references.

• Execution

  • Task parameters can be of a broader set of types; in particular, they need not be serializable.

  • A task parameter can be a std::vector of a FleCSI type such as a field accessor or a std::tuple that includes such a type (recursively).

  • exec::cpu, exec::gpu, and exec::omp are task parameter types that provide task-launch information and access to a Kokkos execution space. exec::accelerator is the one of these preferred for parallel kernels. exec::on is a token argument to pass for them.

    • Kokkos executors can have a number of threads specified to optimize GPU execution.

  • A scheduler launches tasks without attributes by reference to execution-space (template) parameters.

  • Index futures provide all to get all results.

• Topologies

  • narray specializations need not define dimension.

  • unstructured_base::peer_entities is a new type alias for convenience.

  • unstructured_base::coloring::index_space can be initialized without entities, which is unused.

  • user is a simple topology category without ghosts. (This was added without a release note in 2.2.0.)

• HPX backend

  • FleCSI now supports HPX as a runtime backend. However, GPU tasks are not supported with current versions of HPX.

• On-node parallelism

  • forall and reduceall may be used without a name.

• Utilities

  • mdspan and mdcolex provide element_type and value_type type aliases.

  • UNIT can be used in GPU kernels, with restrictions.

  • Unit-test functions may accept a scheduler.

Fixed

• Data

  • field<bool, data::ragged>::mutator works.

  • The particle layout works on the global topology.

• Legion backend

  • std::vector<bool> can be used as a task parameter.

  • Unit tests work without Kokkos but with OpenMP.

• Utilities

  • MPI collective helpers make use of MPI-4 large count support when available. Overflows due to MPI-3 restrictions are trapped.

  • mpi::all_gatherv works with bool.

Changes in v2.3.2 (March 11 2025)

Deprecated

• Build

  • Removed ParMETIS dependency. CMake support files will be removed.

• Data

  • That data.hh includes topology.hh — include it directly (this was deprecated without a release note in 2.3.0)

• Execution

  • That execution.hh includes runtime.hh — include it directly

Fixed

• MPI backend

  • Move-assignment of topology_slot works.

• Utilities

  • CTAD for mdcolex works with any integer type.

  • utilities.hh works with FLOG disabled.

Changes in v2.3.1 (February 3 2025)

Possible incompatibilities

• Legion backend

  • The error of destroying a topology after runtime::control returns (perhaps when main returns) is more likely to produce a crash.

Deprecated

• Runtime

  • threads and threads_per_process — thread-based parallelism must be configured outside the process

• Utilities

  • EXPECT_STRCASEEQ, EXPECT_STRCASENE, ASSERT_STRCASEEQ, and ASSERT_STRCASENE — use *_STREQ or *_STRNE for case-sensitive comparison, or write a case-insensitive comparison function and use *_TRUE or *_FALSE to check its output.

Fixed

• Data

  • Write-only multi-color particle accessors/mutators support non-trivial destructors.

  • Data synchronization now utilizes the correct memory space and logic for openmp.

• Legion backend

  • The speed of ghost copies has been significantly improved (without losing the decreased memory usage).

  • ragged or sparse fields do not generate spurious warnings about using uninitialized data.

  • Tasks with a single parameter whose type is a specialization of std::tuple work.

• Topologies

  • Arithmetic on topo::id objects checks types reliably.

• Utilities

  • dimensioned_array construction works reliably in constant expressions.

  • serial::get_tuple<std::tuple<...>> works (and produces a singleton tuple).

  • UNIT_CAPTURE works properly in non-MPI tasks.

• Topology

  • ntree initializes properly with a number of colors different from the number of processes.

Changes in v2.3.0 (June 20 2024)

Possible incompatibilities

• Topologies

  • narray auxiliaries between two colors are owned by the upper color.

Deprecated

• Build

  • ENABLE_DOXYGEN_WARN — ignored since 2.2.0

  • FLOG_ENABLE_COLOR_OUTPUT, FLOG_SERIALIZATION_INTERVAL, and FLOG_STRIP_LEVEL — use flog::config

• Runtime

  • initialize, start, finalize, run::status, and control::check_status — use runtime and, optionally, getopt

  • program — use argv directly

  • Use of a control policy’s node_policy type — remove it

  • control::execute — use control::invoke or runtime::control with run::control_base and not node_policy

  • option_value and needing one in a program_option validation function — accept the option type instead

  • flecsi/run/control.hh — use flecsi/runtime.hh

• Data

  • coloring_slot — use specialization::mpi_coloring

• Execution

  • Constructing a trace with an id — use the default constructor

• Topologies

  • specialization::cslot — use mpi_coloring

  • flecsi/topo/narray/interface.hh — use flecsi/topology.hh

  • narray_impl::index_definition::create_plan — always ignored

  • narray_base::domain and is_low, is_high, is_interior, is_degenerate, global_id, size, range, and offset in narray::access — use axis

• Utilities

  • util::dag, util::reorder, util::reorder_destructive, util::intersects, util::set_intersection, util::set_union, util::set_difference — superfluous

  • Passing binary functors to mpi::one_to_allv, mpi::one_to_alli, and mpi::all_to_allv — remove second parameter or use ranges

  • flecsi/util/annotation.hh, flecsi/util/array_ref.hh, flecsi/util/color_map.hh, flecsi/util/common.hh, flecsi/util/dag.hh, flecsi/util/demangle.hh, flecsi/util/dimensioned_array.hh, flecsi/util/mpi.hh, flecsi/util/reorder.hh, flecsi/util/serialize.hh, flecsi/util/set_intersection.hh, flecsi/util/set_utils.hh, flecsi/util/unit.hh — use flecsi/utilities.hh

New features

• Build

  • flecsi/config.hh provides information about the build of FleCSI in use.

  • A CMake utility library is provided. (This was added without a release note in 2.2.0.)

  • flecsi_add_target_test is a CMake function to define tests using existing targets.

  • flecsi_add_test can use a launcher command to wrap the test execution.

• Runtime

  • runtime.hh provides access specifically to the runtime model.

  • runtime represents FleCSI initialization as an object.

  • getopt parses user-defined command-line options.

  • run::dependencies_guard allows for application control over initialization of FleCSI dependencies.

  • control::invoke executes a control model with a policy object constructed from arguments provided.

  • run::call is a trivial predefined control model.

  • program_option validation functions can accept the option value directly.

  • task_names returns a mapping of shortened registered FleCSI task names to their full signature.

  • initialize accepts const char * const * for better compatibility.

• Data

  • topology_slot is now movable; empty slots may be detected with topology_slot::is_allocated.

• Execution

  • test convenience function launches unit test tasks.

  • trace objects can be move-assigned.

• Topologies

  • specialization::mpi_coloring creates a coloring eagerly.

  • topo::make_ids<S>(r) is a convenience function to convert a range r of ordinary integers into a range of id<S> objects.

  • unstructured_base::bounding_box computes the axis-aligned bounding box of a mesh.

  • unstructured::special_field is the field definition for special-entity lists.

  • unstructured::get_special_entities allows access to individual special-entity lists.

  • narray_base::distribute and narray_base::make_axes help construct coloring objects.

  • The names axis_definition and index_definition are available in narray_base.

  • axis, axis_color, axis_layout, and axis_info in narray_base describe axes in topology accessors.

  • ntree is a topology that tracks interacting particles. It only supports Legion as a backend and does not support ragged or sparse fields.

• Legion backend

  • Task names are now shortened for better usability in Legion profiling tools. See Profiling with Legion for details.

• Utilities

  • transform applies a transformation functor to a range.

  • partition_point and binary_index find values in sorted ranges.

  • permutation_view accesses a subset of a range.

  • mpi::one_to_allv, mpi::one_to_alli, and mpi::all_to_allv additionally accept ranges and unary functors.

  • KDTree efficiently finds intersections between shapes.

  • Values may be included in expectation/assertion failure messages.

  • FLECSI_DEVICE_CODE is a macro defined during the device compilation for either CUDA or HIP.

• Logging

  • flog::config is the collection of FLOG options that can be changed at runtime.

  • flog::tags returns the names of all defined tags.

Fixed

• Build

  • flecsi_add_sphinx_target doesn’t use -n (which can break for references into other libraries).

• Data

  • The size of one color of an index space can exceed \(2^{32}\) (if FLECSI_ID_TYPE is configured approprately).

• Execution

  • Tasks may be declared noexcept. (This was fixed but not documented in 2.2.1.)

• Topologies

  • narray

    • index_definition::diagonals being false works. (This was fixed but not documented in 2.2.1.)

    • Periodic axes require equal boundary and halo depths (differing values never worked reliably).

    • Auxiliary axes support full_ghosts and periodic properly.

  • unstructured supports num_intervals of 0.

• Legion backend

  • omp tasks now work in builds with GPU support.

  • MPI tasks properly wait on previously launched tasks.

  • Memory usage for implementing ghost copies has been significantly reduced.

  • Processes with multiple CPUs schedule tasks more efficiently.

  • Launches of zero point tasks succeed (vacuously).

  • The --Xbackend -dm:memoize option is no longer required to enable tracing.

• On-node parallelism

  • Custom reductions and reduced types work with Kokkos.

• Utilities

  • one_to_alli no longer triggers a race condition.

• Logging

  • Special options like --control-model and --help work reliably. (This was documented but not actually implemented in 2.2.1.)

  • FLOG messages are now variable size and are no longer truncated.

Changes in v2.2.1 (July 12 2023)

Fixed

• Runtime

  • Control policy objects are value-initialized by run::control::execute.

  • Unrecognized options are properly rejected along with unrecognized positional arguments.

  • The same exit status is used for all erroneous command lines.

  • Control-model graphs are labeled with the program name.

  • Control model output strips parameters, return values, wrappers, and common namespaces from actions for better readability.

• Data

  • ragged accessors with wo privileges work for GPU tasks. (The field type must still be trivially default-constructible.)

• Utilities

  • transform_view supports pointers to members (though not during constant evaluation).

Changes in v2.2.0 (April 14 2023)

Possible incompatibilities

• Build

  • FLECSI_RUNTIME_MODEL is renamed to FLECSI_BACKEND (it never referred to Runtime Model).

• Utilities

  • util::mdspan::operator() is removed (it had an inconsistent argument order).

Deprecated

• Runtime

  • control::state — derive the control policy type from control_base and define actions accepting a reference to it

  • run::control_point — use control_base::point

  • run::cycle — use control_base::cycle

• Data

  • global_topology and process_topology — create instances of topo::global and topo::index as needed

• Utilities

  • in util::annotation, begin and end — use guard or rguard

• Logging

  • The namespace flecsi::log — use flecsi::flog instead

  • The CMAKE option FLOG_SERIALIZATION_THRESHOLD is not used anymore

New features

• Build

  • The HIP compiler is supported.

• Runtime

  • Control policies may inherit from control_base to be provided to actions, to allow those actions to throw control_base::exception, and to limit their lifetime to that of control::execute.

  • control_base::meta defines special control points for a specialization’s use via control::meta.

  • New option --Xbackend to pass single backend arguments. Can be used multiple times.

• Data

  • field_reference::get_elements and associated types allow control of the storage for elements of ragged/sparse fields.

  • launch::mapping allows field data to be accessed from other colors (using multi task parameters) with the Legion backend.

  • topology_slot::colors reports on the number of colors for an allocated topology.

  • reduction_accessor allows dense fields on a global topology to be updated collectively.

  • The particle layout supports efficient creation and destruction of unordered field elements.

  • Field definitions are copyable and may be destroyed after use.

  • Fields may be of non-portable types so long as they are used only by MPI tasks.

  • single accessors support -> to access the members of a field value of class type.

• Execution

  • Tasks may be executed as omp, allowing efficient use of OpenMP.

  • exec::trace improves performance of loops when used with the Legion backend.

  • task_local objects define global variables with task-local values.

• Topologies

  • unstructured represents unstructured meshes.

  • narray represents hyperrectangular arrays, including structured meshes.

  • global topology instances may be created with sizes other than 1. (All elements are shared together among all point tasks in an index launch.)

  • util::id and util::gid store topology-entity IDs.

• MPI backend

  • The toc processor type is supported.

• On-node parallelism

  • exec::parallel_for, forall, exec::parallel_reduce, and reduceall may be used without Kokkos enabled, in which case they run serially.

  • exec::mdiota_view provides support for multi dimensional ranges, with exec::full_range, exec::prefix_range and exec::sub_range options.

• Utilities

  • unit.hh provides macros, based on Google Test, for writing unit tests in or outside of tasks.

  • serial provides a general-purpose serialization interface. It may be extended to allow additional types to be used as task parameters or with MPI communication.

  • mdcolex accesses a multi-dimensional array using Fortran-like syntax.

  • mpi::one_to_alli scatters generated and/or serialized data with bounded memory usage.

  • MPI communication functions compute values to send in rank order and support mutable functors.

  • substring_view represents part of another range.

  • equal_map and offsets represent partitions of integer ranges.

• Logging

  • flog_fatal suppresses backtraces (already unavailable if NDEBUG is defined) unless FLECSI_BACKTRACE is set in the environment.

  • flog is a new name that avoids collisions with ::log in code lacking proper namespace qualifications.

Fixed

• Build

  • HDF5 is supported by the CMake configuration.

  • Caliper is supported by the MPI backend.

  • Building with the MPI backend properly links to the threading library.

  • Added FLECSI_VERSION define to record version

• Runtime

  • Certain control-flow graphs compile with Graphviz support and are drawn correctly.

  • --backend-args can be specified multiple times. The collected arguments are passed to the backend.

  • MPI and Kokkos are initialized with no arguments (so that they cannot misinterpret arguments not meant for them).

• Data

  • Mutators support write-only privileges (which, as with accessors, are necessary to properly initialize fields).

  • Mutators work on fields over zero index points.

• Topologies

  • Topology accessors that provide metadata for direct access in a toc task work properly when writable (although without such direct access in that case).

• Legion backend

  • Certain user-level fields are allocated properly.

  • MPI tasks with reference parameters support argument conversions correctly.

  • Errors at process termination in certain configurations are avoided.

  • Accessors with both wo and na privileges are processed correctly.

• MPI backend

  • exec::fold::min and exec::fold::max work with bool (but may be slower than using another type like int).

  • Index futures work properly when declared const.

  • Index futures work with bool return type.

• On-node parallelism

  • More accessor and topology accessor functions are available on a device.

  • Several uses of “iterator” in documentation have been corrected to use “range”.

• Logging

  • Tags work correctly in tasks executed in parallel.

  • Messages are sorted by timestamp correctly.

  • Exiting the process before flecsi::finalize does not crash.

Changes in v2.1.0 (April 16 2021)

New features

• Topologies

  • topo::help is a convenient base class for specialization class templates that defines the non-dependent names in specialization.

Changes in v2.0.0 (March 30 2021)

The changes between versions 1 and 2 are extensive and so are merely summarized here. The broadest change is that macros are used only for on-node parallelism constructs and for logging. Some macro arguments become template arguments.

Removed

• Data

  • flecsi_get_client_handle — use the topology slot directly

  • flecsi_get_handles, flecsi_get_handles_all, flecsi_is_at, flecsi_has_attribute_at, flecsi_has_attribute — will be provided differently by io

• Execution

  • flecsi_register_task, flecsi_register_task_simple, flecsi_register_mpi_task, flecsi_register_mpi_task_simple — unneeded

  • specialization_tlt_init, specialization_spmd_init, execution::driver — the application provides main

  • flecsi_execute_task_simple — namespaces require no special treatment

  • flecsi_register_program — unneeded

  • flecsi_register_global_object, flecsi_set_global_object, flecsi_initialize_global_object, flecsi_get_global_object — use non-local variables directly, with the same restrictions

  • flecsi_register_reduction_operation — unneeded

• Topologies

  • Topology element id() — iteration produces indices

  • Unstructured mesh — will be reimplemented

Other incompatibilities

• Data

  • flecsi_register_data_client — now topology slots

  • flecsi_register_field, flecsi_register_global, flecsi_register_color — now field::definition, or a container of same to implement multiple versions

  • flecsi_get_handle, flecsi_get_global, flecsi_get_color, flecsi_get_mutator — now definition::operator()

  • data_client_handle_u<T,P> — now T::accessor<P>

  • dense_accessor, ragged_accessor, ragged_mutator, sparse_accessor, sparse_mutator — now definition::accessor or definition::mutator

    • ragged fields have a std::vector-like interface at each index point

    • sparse fields have a std::map-like interface at each index point

• Execution

  • flecsi_execute_task — now execute

  • flecsi_execute_mpi_task, flecsi_execute_mpi_task_simple — pass mpi to execute

  • flecsi_execute_reduction_task — now reduce

  • flecsi_color(), flecsi_colors() — now color() and colors()

  • flecsi_register_function, flecsi_execute_function, flecsi_function_handle, flecsi_define_function_type — now exec::make_partial

  • execution::flecsi_future — now future, with a simpler interface

• Topologies

  • Specializations inherit from topo::specialization rather than from a core topology type directly.

New features

• Runtime

  • Application command-line options can be specified using the program_option class template and associated helper functions.

  • The control class template is used to define an extensible control model for an application.

• Data

  • The raw and single layouts support special cases of field usage.

• Execution

  • Tasks can be (specializations of) function templates.

  • Tasks can accept parameters of dynamic size (e.g., std::vector<double>), although passing large objects is inefficient.

  • Index launches can be of any number of point tasks (determined by the task’s arguments, including exec::launch_domain).

  • test conveniently executes tasks for unit testing.

• Topologies

  • Multiple user-defined topology instances may exist sequentially or simultaneously.