Developer Guide

This documentation is intended for developers of the core FleCSI library.

Style Guide

Unless otherwise specified, follow the Boost “Design and Programming” guidelines and header guidelines. They do not address purely stylistic matters of indentation, spacing, and brace placement; those are addressed for FleCSI by clang-format.


Starting from the 2019 version:

  • Optimization does matter for FleCSI, but only where the overhead it introduces is expected to be a significant fraction of the application’s runtime.

  • Obviously FleCSI uses certain additional libraries (like Legion). Do not add further dependencies without discussion. As a single component, there is no need to limit use of one part of FleCSI by another.

  • The ABI concerns, especially for Windows, are irrelevant.

  • noexcept has replaced exception-specifications and should be used as appropriate, especially for move operations.

  • FleCSI uses its own unit-testing framework, not Boost’s.

  • The preferable line-length limit is 78 characters, so that even a diff of such a file avoids the line continuations used in some displays.

  • Individual documentation files do not get their own copyright/license.

  • The source-file header is of course different.

  • FleCSI uses flog_assert, not BOOST_ASSERT.

  • min and max do not require special treatment.

  • Header and source files are suffixed with .hh and .cc.

  • There are no “primary directories”.

  • Code documentation is generated by Doxygen, prose by Sphinx.

Additional Rules

  • Changes purely for stylistic reasons should be rare and must appear in a separate commit (preferably a separate merge request).

  • Class and enumeration names do not get a suffix _t.

  • Header names are “absolute” (i.e., begin with flecsi/).

  • Function documentation uses the imperative mood.

Directory Structure

The source code for the core FleCSI infrastructure is located in the top-level/flecsi directory. For the most part, the subdirectories of this directory correspond to the different namespaces in the core infrastructure. Each of these subdirectories must contain a valid CMakeLists.txt file. However, none of their children should have a CMakeLists.txt file, i.e., the build system will not recurse beyond the first level of subdirectories. Developers should use relative paths within a CMakeLists.txt file to identify source in subdirectories.

Unit test files should be placed in the test subdirectory of each namespace subdirectory. By convention, developers should not create subdirectories within the test subdirectory.


FleCSI uses the Semantic Versioning system. Note that it defines the three kinds of releases in terms of restrictions on what changes can appear in each, and that those restrictions are phrased in terms of the documented interface. We interpret compatibility strictly in terms of source (with the inevitable judgment calls for things like SFINAE). Even altogether new code can appear in a “patch” release if it serves to fix a bug or improve performance rather than as a new documented feature.

There is one active, shared branch for each of the three types of release, as described below. Tags are used to identify releases as well as certain internal reference points for development.

Branch Types

  • incompatible
    The develop branch is where work on the next major release takes place, potentially with interface and feature changes that are incompatible with previous versions.

  • feature
    Feature branches (named for their major version number, e.g., 1, 2, 3) are for feature development on the current major version.

  • release
    Release branches (named for their major.minor version number, e.g., 1.1, 1.2) are interface-stable versions of the code base. At appropriate points, tags (named for their major.minor.patch version number, e.g., 1.1.2) are used to identify patched versions.


At the time of writing, FleCSI has the following branches:

develop (incompatible)
2 (feature)
2.1 (release)

In general, each change should be made on the most restrictive permissible relevant branch so as to minimize divergence between them (after merging) and the associated potential for future merge conflicts. The condition of relevance pertains to an internal feature might be added only on the feature branch if it is not expected to accrue any clients on the release branch. A sometimes countervailing consideration is stability: users expect that patch releases are less likely to cause problems when upgrading even though it is simply a bug if even a feature release does so. It is also unfortunate to need to consider reverting a change because an official release is needed in the interval between introducing it and becoming confident in it.


The form of a tag is determined by the underlying branch type and commit it is intended to reference:

  • devel branch
    In general, a commit on the devel branch should only be tagged if it is the first commit after a new feature branch has been created, and should be named with d and the major version of the feature branch, e.g., when feature branch 2 is created, a new devel tag d2 should be created on the next devel commit.

  • feature branch
    Feature tags should be created on the next commit after a new release branch is created, with f and the major.minor version of the release branch, e.g., when release branch 1.5 is created, a new feature tag f1.5 should be created on the next feature commit.

  • release branch
    Release tags should be created for each new release, with v and the release version, e.g., v1.4.1.


Tags mark a whence or origin for development rather than a whither or completion. The first letter of the tag is important, i.e., d, f, or v, as it is used by FleCSI to determine the branch type of the tag during CMake configuration.


FleCSI development uses a devel -> feature -> release forking workflow that can be visualized as in Fig. 10. Bugfixes and features can be back-merged into feature or devel, as appropriate.


Fig. 10 Workflow diagram illustrating basic workflow using the three branch types described above. (Figure due to Angela Herring.)

Container Images


These sections assumes that you are familiar with container management using a container engine like Docker. If you are not, there is excellent documentation starting here. Additionally, we include instructions and links for installation and basic usage below.

Installing a Container Engine

Some FleCSI developer tasks require a container engine, e.g., Docker. This section provides links to installation and usage documentation for Docker, and Podman.


The Docker daemon is available for a variety of platforms from the docker website.

For linux, the community engine is available for several distributions:


An alternative, drop-in replacement for Docker is Podman. Podman has the advantage that it is daemonless, and does not require root privileges to run.

Tutorial Image

This section details the steps to modify and update the docker image for the FleCSI tutorials.


These instructions are only relevant if you want to modify or rename the automatically generated tutorial images. The docker files in this directory are automatically built on Docker Hub, whenever they get modified on GitHub. These instructions are primarily intended for developers who are modifying the image scripts and need to test them.

Building and Testing Locally

The tutorial docker files are located in flecsi/tutorial/docker. These instructions assume that you are in that directory.

To build a new docker image, you can execute the following:

$ docker build -t laristra/flecsi-tutorial:TAG -f BACKEND .

where TAG and BACKEND are replaced with a tag and a specific backend, e.g., legion or mpi. The TAG argument may be any name that you would like to give the image. The default is latest.

Docker will try to cache build steps and information that it doesn’t think has changed. Sometimes it is useful to force docker to rebuild everything from scratch. To do this, you can pass the –no-cache flag:

$ docker build --no-cache -t laristra/flecsi-tutorial:TAG -f BACKEND .

This is particularly useful if a repository has changed.

The default repository and branch for the container are and devel, respectively. If you are in the process of updating the container, and would like to use a different repository and branch, you can specifiy them like:

$ docker build --build-arg REPO= \
  --build-arg BRANCH=tutorial-update --no-cache \
  -t laristra/flecsi-tutorial:TAG -f BACKEND .

where REPO is the git repo to use, and BRANCH is the branch name to use.

Once you have successfully built the imager, you can test it using the run command:

$docker run -it --shm-size=512m -u flecsi laristra/flecsi-tutorial:TAG /bin/bash

This will open an interactive shell (/bin/bash) on the running image.

Pushing to

If you want to push images to the docker hub repository, you will need to log in:

$ docker login

This will prompt you for a username and password. You will need to be a member of the laristra organization to successfully push images to the docker hub. Once you are logged in, you can push an image to Docker Hub with:

$ docker push laristra/flecsi-tutorial:TAG

where TAG is the tag that you specified during the build.

In general, pushing the image manually is unnecessary because it will automatically be rebuilt when the docker file is updated on GitHub.

Gitlab CI

FleCSI has a special branch gitlab-ci that provides images and configuration files that are used for Gitlab’s continuous integration (CI) services. The current set of utilties include:

  • Images
    Operating system, build environment, and FleCSI images. These use dockerfile syntax, and can be built and run with docker, and podman engines, or potentially, with any engine that is compatible with docker-generated images.

    The images are modular, and are intended to be used to build up capabilities:

    • OS Images
      Located in the os directory, these are designed to provide a standard base of functionality, i.e., a standard set of system packages, which can be used interchangeably with downstream environment images.

    • Environment
      Located in a directory named for the corresponding branch, e.g., the environment dockerfile file for devel is located in the devel directory, these are designed to provide a standard spack environment for building FleCSI.

      It is intended that developers add new directories when necessary to support custom configuration of a branch that they are testing.

    • Build
      Also located in the branch directory, the build image provides a pre-built FleCSI installation that can be used for debugging, or as a building block for other projects.

  • Spack Configuration Files
    Located in the spack subdirectory of an environment directory, these allow customization of the spack install for a particular environment. In particular, the spack/ file should specify explicit versions for as many packages as is feasible. Package versions should only be elevated when necessary for new feature support. Following this guideline will greatly increase stability.

Build System

The gitlab-ci branch uses a standard CMake build system. There are several important configuration options that you need to be aware of:

    This is the branch name of the branch or fork of gitlab-ci that you wish to use. The default is gitlab-ci. However, it is useful to be able to set this to a personal branch name, so that you can test local changes without doing a merge request to the central FleCSI Gitlab project. Use this in conjunction with the REPOSITORY option.

    This is the url for the repository where the build system will look for the CI_BRANCH option that you specified. The default is the main FleCSI Gitlab repository However, it is useful to set this to your personal fork for testing new images.

    This is the container registry that will be used for push-, and pull- targets. In general, this needs to be set to the registry that is being used by the CI to pull images, e.g., laristra/flecsi-ci (this is on, or (internal container registry).

    This should be the name of the container engine that is installed on your system. Currently, only docker, and podman are supported. Additional engines may be added in the future.

As an example configuration, consider the following:

$ mkdir build
$ cd build
$ cmake .. -DCI_BRANCH=ci-environment -DREPOSITORY= -DENGINE=podman

This configuration will use the author’s ci-environment branch on Github with the podman engine.

To build an image, try:

$ make centos-8

This will build the OS image defined in os/centos-8.


A note of caution on building the images under the gitlab-ci branch. Simply typing make after configuration will attempt to build all of the images in the project! This can take several hours to complete! If this is not your intention, you can list the build targets by typing make help. To build only a specific target, type make TARGET, where TARGET is the target that you would like to build. Note also that building targets with dependencies will build all of the dependencies of the target (This is unaviodable.)


The build system also supports passing runtime options to the container engine. These are of the form STAGE_EXTRA, e.g., BUILD_EXTRA, PUSH_EXTRA, and CLEAN_EXTRA. For example, BUILD_EXTRA can be used to force complete rebuild of an image like:

$ make BUILD_EXTRA="--pull --no-cache" centos-8

This will force the engine to start from scratch, i.e., pulling the base image from the registry, and ignoring cached stages.

Published Documentation

The project homepage is a GitHub Pages site generated from the current GitHub repository. The deploy-documentation Make target sets up a repository to update it by pushing to the appropriate branch.

Spack Cheat Sheet

To remove all cached sources:

$ spack clean -d

To remove cached sources for a particular spec:

$ spack clean -d spec

To uninstall all spack packages:

$ spack uninstall -fay

To keep temporary staging files in /tmp/$USER:

$ spack install --keep-stage ...

Git Cheat Sheet

To lookup the hash referenced by a tag:

$ git rev-list -n 1 $TAG

To get the message for an annotated tag:

$ git tag -nX (X specifies lines of annotation)

To sync tags:

$ git fetch --prune --prune-tags

Building for Darwin

This is a recipe for building the current development version of FleCSI on the LANL Darwin cluster.

Checkout spack and flecsi:

$ git clone --branch devel
$ git clone --single-branch --branch develop

Source the spack environment script:

$ source spack/share/spack/

Add the flecsi spack repo:

$ spack repo add flecsi/spack-repo

Load compiler modules (both work if loaded in this order) and update spack compilers:

$ module load clang/8.0.1 gcc/9.3.0
$ spack compilers

Create a spack environment and install the flecsi dependencies.

$ spack env create legion
$ spack env activate legion
$ spack install --only dependencies flecsi@devel backend=legion +hdf5 +graphviz +flog ^mpich
$ spack install cmake

Load doxygen, install sphinx, and update your path:

$ module load doxygen
$ pip3 install --user Sphinx
$ pip3 install --user recommonmark
$ pip3 install --user sphinx_rtd_theme
$ export PATH=$HOME/.local/bin:$PATH

Configure and build flecsi:

$ cd flecsi && mkdir -p build/legion && cd build/legion
$ ../../tools/configure clang legion
$ make -j 16

Graphviz Notes

FleCSI uses the libcgraph interface to Graphviz to create control model visualizations. The libcgraph interface is fairly counterintuitive. One particular gotcha is that graph, node, and edge attributes can only be set on attributes that have been defined for the graph. If an attribute type has not been defined, the graph will ignore it. There is not easy to remedy to this problem: attributes that are added after initialization will reset all previously added elements to whatever the default of the new attribute is. Therefore, if you need to add an attribute, the best thing to do is to look at the graphviz.hh file in ‘flecsi/util’, and add it there with a reasonable default.


The API reference is organized exclusively using the groups feature; none of the files and namespaces are documented, since they have little relevance to the user. See the manual for details, but note that members of namespaces enclosed by the \{ and \} of a grouping command are not included in the group.

The developers’ version of the API reference includes a selection of internal interfaces for core developers (who of course must nonetheless consult the source in general). Use \cond core and \endcond to mark material that should appear only there. Developers must also check for such markers to identify internal interfaces; the output does not distinguish them, since users should consult only the users’ version anyway.


Sphinx documentation is here. The following are some examples of frequently-used elements. However, a good practice is to just look at the existing documentation to figure out how something was done.


.. _undersea:
.. figure:: images/undersea.png
  :align: center
  :width: 70%

  A colorful image resembling a cosmic version of an undersea world.

This will be rendered like:


Fig. 11 A colorful image resembling a cosmic version of an undersea world.

You can reference the figure using its label undersea like:

As can be seen in :numref:`undersea`...

This will be rendered like:

As can be seen in Fig. 11

Code Blocks

Syntax highlighting for codes blocks uses pygments, which supports many programming and markup languages.

Here are two examples:


.. code-block:: console

  $ xterm -hold -fs 10 -bg black -fg white -geometry 128x40 -e curl

This will be rendered like:

$ xterm -hold -fs 10 -bg black -fg white -geometry 128x40 -e curl


.. code-block:: cpp

  template<typename Bar>
  using Baz = Foo<Bar>;

This will be rendered like:

template<typename Bar>
using Baz = Foo<Bar>;

Literal Includes

Literal includes allow you to directly include source code or other inputs from the actual file you are referencing. This is useful because any changes to the file will automatically be captured in the documentation.

.. literalinclude:: ../../../tutorial/standalone/
  :language: cpp
  :lines: 23-45

This will be rendered like: