Developer Guide

This documentation is intended for developers of the core FleCSI library.


Style Guide

Unless otherwise specified, follow the Boost “Design and Programming” guidelines and header guidelines. They do not address purely stylistic matters of indentation, spacing, and brace placement; those are addressed for FleCSI by clang-format.

Exceptions

Starting from the 2019 version:

  • Optimization does matter for FleCSI, but only where the overhead it introduces is expected to be a significant fraction of the application’s runtime.

  • Obviously FleCSI uses certain additional libraries (like Legion). Do not add further dependencies without discussion. As a single component, there is no need to limit use of one part of FleCSI by another.

  • The ABI concerns, especially for Windows, are irrelevant.

  • noexcept has replaced exception-specifications and should be used as appropriate, especially for move operations.

  • FleCSI uses its own unit-testing framework, not Boost’s.

  • The preferable line-length limit is 78 characters, so that even a diff of such a file avoids the line continuations used in some displays.

  • Individual documentation files do not get their own copyright/license.

  • The source-file header is of course different.

  • FleCSI uses flog_assert, not BOOST_ASSERT.

  • min and max do not require special treatment.

  • Header and source files are suffixed with .hh and .cc.

  • There are no “primary directories”.

  • Code documentation is generated by Doxygen (where \ is used instead of @), prose by Sphinx.

Additional Rules

  • Class and enumeration names do not get a suffix _t.

  • Header names are “absolute” (i.e., begin with flecsi/).

  • Function documentation uses the imperative mood.

Directory Structure

The source code for the core FleCSI infrastructure is located in the top-level/flecsi directory. For the most part, the subdirectories of this directory correspond to the different namespaces in the core infrastructure. Each of these subdirectories must contain a valid CMakeLists.txt file. However, none of their children should have a CMakeLists.txt file; the build system will not recurse beyond the first level of subdirectories. Developers should use relative paths within a CMakeLists.txt file to identify source in subdirectories.

Unit test files should be placed in the test subdirectory of each namespace subdirectory. By convention, developers should not create subdirectories within the test subdirectory.

Commits

While exceptions must occasionally be made to each of the following, ideally each commit should

  • build and pass tests (partly to support git bisect)

  • not add or change material changed later in the same merge request (partly to help git blame)

  • include the smallest subset of an overall set of changes that follows the above rules

  • add or update any relevant documentation, including the Release Notes

  • not be a meaningless merge from the destination branch (as introduced by git pull)

  • have a meaningful message that follows the 50/72 rule

  • be properly formatted per clang-format

  • either make only purely stylistic changes or make no such changes

Stylistic changes should be rare and are best put in their own merge request (so that they do not complicate any substantive review). Improvements to the style of code already being modified are encouraged, so long as they do not materially interfere with review.

Expending a reasonable effort to rewrite the history of a merge request to obtain the above properties is encouraged. The usual tool is git rebase -i, although retroactive formatting is much easier with (a script that runs) git filter-branch.

The name of the source branch of a merge request appears in the machine-generated commit that merges it. Those names should therefore be meaningful; in particular, to avoid confusion they shouldn’t just be that of the destination branch (as from commands like git checkout 2.1).


Versioning

FleCSI uses the Semantic Versioning system. Note that it defines the three kinds of releases in terms of restrictions on what changes can appear in each, and that those restrictions are phrased in terms of the documented interface. We interpret compatibility strictly in terms of source (with the inevitable judgment calls for things like SFINAE). Even altogether new code can appear in a “patch” release if it serves to fix a bug or improve performance rather than as a new documented feature. As an exception to the specification, deprecations can be made in such a release as well, since they are merely documentation for the careful user who wants to upgrade early for forward compatibility. However, since features are typically deprecated in favor of some new feature, most will nonetheless appear in feature releases.

There is one active, shared branch for each of the three types of release, as described below. Tags are used to identify releases as well as certain internal reference points for development.

Branch Types

incompatible

The develop branch is where work on the next major release takes place, potentially with interface and feature changes that are incompatible with previous versions.

feature

Feature branches (named for their major version number, e.g., 1, 2, 3) are for feature development on the current major version.

release

Release branches (named for their major.minor version number, e.g., 1.1, 1.2) are interface-stable versions of the code base. At appropriate points, tags (named for their major.minor.patch version number, e.g., 1.1.2) are used to identify patched versions.

In general, each change should be made on the most restrictive permissible relevant branch so as to minimize divergence between them (after merging) and the associated potential for future merge conflicts. The condition of relevance pertains to an internal feature that might be added only on the feature branch if it is not expected to accrue any clients on the release branch. A sometimes countervailing consideration is stability: users expect that patch releases are less likely to cause problems when upgrading even though it is simply a bug if even a feature release does so. It is also unfortunate to need to consider reverting a change because an official release is needed in the interval between introducing it and becoming confident in it.

Tags

Release tags should be created for each new release, with v and the release version, e.g., v1.4.1.

FleCSI Version File (.version) and FLECSI_VERSION

The contents of the .version file in the root of a FleCSI source checkout is used to identify the branch type and version of FleCSI. Given the three different branch types and tagged versioned releases, its content will be one of the following four schemes:

develop branch

develop

feature branch

f<major>

release branch

r<major>.<minor>

tagged release

v<major>.<minor>.<patch>

FleCSI uses the information in .version to define FLECSI_VERSION in its flecsi/config.hh header. This constant encodes the major, minor and patch version of FleCSI in a single integer value.

For tagged releases (v<major>.<minor>.<patch>) it is defined as (major << 16) | (minor << 8) | patch.

On release branches both major and minor version components are set to the release version, however, the patch version component is set to its maximal value 255.

On feature branches only the major version component is set to the version value, while the minor version component is set to its maximum 255 and the patch version component is set to 0.

Finally, on the develop branch, the major version component is set to its maximum 255, while the other components are set to 0.

The following table summarizes these rules with some examples:

Table 1 Examples of .version and FLECSI_VERSION

Source

Contents of .version

FLECSI_VERSION

develop branch

develop

0xff0000

2 feature branch

f2

0x02ff00

2.1 release branch

r2.1

0x0201ff

v2.2.0 tagged release

v2.2.0

0x020200

Workflow

FleCSI development uses a devel -> feature -> release forking workflow that can be visualized as in Fig. 16. Bugfixes and features can be back-merged into feature or devel, as appropriate.

../_images/branch.png

Fig. 16 Workflow diagram illustrating basic workflow using the three branch types described above. (Figure due to Angela Herring.)


Published Documentation

The project homepage is a GitHub Pages site generated from the current GitHub repository. The deploy-docs Make target sets up a repository to update it by pushing to the appropriate branch.


Spack Cheat Sheet

To remove all cached sources:

$ spack clean -d

To remove cached sources for a particular spec:

$ spack clean -d spec

To uninstall all spack packages:

$ spack uninstall -fay

To keep temporary staging files in /tmp/$USER:

$ spack install --keep-stage ...

Git Cheat Sheet

To lookup the hash referenced by a tag:

$ git rev-list -n 1 $TAG

To get the message for an annotated tag:

$ git tag -nX (X specifies lines of annotation)

To sync tags:

$ git fetch --prune --prune-tags

Building for Darwin

Darwin is a testbed cluster at LANL that provides a wide variety of node hardware configurations. The FleCSI distribution provides a script, tools/darwin.sh, that automates downloading, building, and installing FleCSI and all of its dependencies. The script can be run either from a checked-out version of the FleCSI repository, in which case it will not re-download FleCSI, or as a standalone script, in which case it will clone the flecsi repository and build from there.

The former is the preferred approach. From a back-end node, run

$ git clone ssh://git@re-git.lanl.gov:10022/flecsi/flecsi.git
$ flecsi/tools/darwin.sh

The script performs the following operations:

  1. Clone flecsi if the script was not run as above, from a cloned repository.

  2. Install a version of the Spack package manager known to work with FleCSI into $HOME/spack.

  3. Load Darwin’s environment modules for known-to-work-with-FleCSI versions of various tools.

  4. Create and activate a flecsi-mpich Spack environment. Download, build, and install FleCSI’s dependencies into this environment. (This is by far the more time-consuming part of the script. Plan on about 45 minutes.)

  5. Configure, build, test, and install FleCSI into $HOME/flecsi-inst, including documentation.

  6. Configure and build the FleCSI tutorial files. This ensures that it is possible to compile and link against the headers and libraries in $HOME/flecsi-inst.

The script expects a fairly virgin environment. It currently fails if Spack is already installed, conflicting modules are already loaded, or other aspects of the installation already have been run.

Once the script completes, you can activate the FleCSI environment with

$ source ~/spack/share/spack/setup-env.sh
$ spack env activate flecsi-mpich

The complete tools/darwin.sh script is reproduced below. Although the script is intended to be run on the Darwin cluster, it should not be too hard to adapt it to other systems or even simply use the script as a reference for the commands needed to get FleCSI up and running.

#! /bin/bash

######################################
# Build FleCSI entirely from scratch #
# (tested on LANL's Darwin cluster)  #
######################################

# Trace commands and abort on the first error.
set -e

# Clone flecsi into the current directory unless the script was run
# from what looks like a flecsi clone.
script_dir=$(dirname $(readlink -f "$0"))
cd "$script_dir"
if [ -d ../flecsi ] ; then
    cd ..
else
    cd -
    git clone ssh://git@re-git.lanl.gov:10022/flecsi/flecsi.git
    cd flecsi
fi

# Log where we came from in Git in case this is needed for troubleshooting.
(
    set +e
    git config --get remote.origin.url 2>/dev/null || \
      echo "${0}: Directory $(pwd) does not appear to have come from Git" 1>&2
    git rev-parse HEAD 2>/dev/null
)

# Define an installation directory.
FLECSI_INSTALL="$HOME/flecsi-inst"

# Set GCC version we're going to use
GCC_VERSION=$(echo $(grep GCC_VERSION: .gitlab-ci.yml | cut -d: -f2))

# Download a version of Spack known to work with FleCSI and activate it.
SPACK_VERSION=$(echo $(grep USE_SPACK_UPSTREAM: .gitlab-ci.yml | cut -d: -f2))

if [[ "${SPACK_VERSION}" =~ "develop-" ]]; then
  SPACK_COMMIT=$(echo $SPACK_VERSION | cut -d- -f2)
fi

pushd "$HOME"
if [ ! -d spack ]; then
  git clone https://github.com/spack/spack.git
  cd spack
  if [[ "${SPACK_VERSION}" =~ "develop-" ]]; then
    git checkout $SPACK_COMMIT
  else
    git switch releases/$SPACK_VERSION
  fi
  git rev-parse HEAD
else
  cd spack
  echo "Found existing Spack install in ~/spack"
  git fetch origin

  if [[ "${SPACK_VERSION}" =~ "develop-" ]]; then
    if [ "$(_TMP=$(git rev-parse HEAD); echo ${_TMP:0:8})" != "${SPACK_COMMIT}" ]; then
      echo "ERROR: The current checkout does not match ${SPACK_COMMIT}!"
      echo
      echo "Please update manually with:"
      echo " git -C ~/spack checkout ${SPACK_COMMIT}"
      echo
      echo "WARNING: This may invalidate other Spack environments that rely on" \
           "this Spack instance!"
      exit 1
    fi
  else
    if [ "$(git rev-parse HEAD)" != "$(git rev-parse origin/releases/$SPACK_VERSION)" ]; then
      echo "ERROR: The current checkout does not match origin/releases/$SPACK_VERSION!"
      echo
      echo "Please update manually with:"
      echo " git -C ~/spack fetch +releases/$SPACK_VERSION:refs/remotes/origin/releases/$SPACK_VERSION"
      echo " git -C ~/spack switch origin/releases/$SPACK_VERSION"
      echo
      echo "WARNING: This may invalidate other Spack environments that rely on" \
           "this Spack instance!"
      exit 1
    fi
  fi
fi
source "$HOME/spack/share/spack/setup-env.sh"
popd


# Create new Spack environment and activate it
spack env remove -y flecsi-mpich || true
spack env create flecsi-mpich
spack env activate flecsi-mpich

# Load GCC version known to work with FleCSI
# and make it visible to Spack
module load gcc/${GCC_VERSION}

# ignore configuration in ~/.spack to avoid conflicts
export SPACK_DISABLE_LOCAL_CONFIG=true

# allow spack to dynamically concretize packages together instead of separately.
# this avoids some installation errors due to multiple python packages depending
# on different versions of a dependency
spack config add concretizer:reuse:false
spack config add packages:all:compiler:["gcc@${GCC_VERSION}"]

# add FleCSI spack package repository
spack repo add spack-repo

# On Darwin we have a Spack upstream that already has prebuilt dependencies
DARWIN_SPACK_UPSTREAM=/projects/flecsi-devel/gitlab/spack-upstream/$SPACK_VERSION

if [ -d "$DARWIN_SPACK_UPSTREAM" ] && [ -x "${DARWIN_SPACK_UPSTREAM}" ]; then
  # add spack upstream if accessible
  spack config add upstreams:default:install_tree:${DARWIN_SPACK_UPSTREAM}/opt/spack/
  cp ${DARWIN_SPACK_UPSTREAM}/etc/spack/{compilers.yaml,packages.yaml} $HOME/spack/etc/spack/
else
  # Otherwise, load a compatible cmake and expose whatever else happens to be
  # sitting around as Spack externals
  module load cmake/3.26.3
  spack compiler find
  spack external find
fi

# Install FleCSI's dependencies with Spack.
# Note: cxxstd is only needed for legion@master. Remove once no longer needed.
spack add flecsi%gcc@${GCC_VERSION} backend=legion +doc +graphviz +hdf5 +kokkos +flog \
          ^legion network=gasnet cxxstd=17 conduit=mpi build_type=Debug \
          ^mpich
spack install -j $(nproc) --only dependencies

# Build, test, and install FleCSI.

# configure FleCSI, inheriting (-i) the its configuration from the Spack environment
test -d build && rm -rf build
tools/spack_cmake -i -t flecsi -- -DCMAKE_INSTALL_PREFIX="$FLECSI_INSTALL"
cd build
cmake --build . -j $(nproc)
ctest
cmake --build . --target install
cd ..

# Ensure the tutorial examples build properly.

# This makes the installed FleCSI visible to CMake's find_package
export CMAKE_PREFIX_PATH=$FLECSI_INSTALL:$CMAKE_PREFIX_PATH
cd tutorial
test -d build && rm -rf build

# configure tutorial examples with the same compiler and compiler flags as flecsi
../tools/spack_cmake -c -i flecsi
cmake --build build -j $(nproc)

# Build complete
echo "BUILD COMPLETE"
echo
echo "To continue working in the same Spack environment, execute the following commands:"
echo
echo " source \$HOME/spack/share/spack/setup-env.sh"
echo " spack env activate flecsi-mpich"
echo " module load gcc/${GCC_VERSION}"
echo

Graphviz Notes

FleCSI uses the libcgraph interface to Graphviz to create control model visualizations. The libcgraph interface is fairly counterintuitive. One particular gotcha is that graph, node, and edge attributes can only be set on attributes that have been defined for the graph. If an attribute type has not been defined, the graph will ignore it. There is not easy to remedy to this problem: attributes that are added after initialization will reset all previously added elements to whatever the default of the new attribute is. Therefore, if you need to add an attribute, the best thing to do is to look at the graphviz.hh file in ‘flecsi/util’ and add it there with a reasonable default.


Doxygen

The API reference is organized exclusively using the groups feature; none of the files and namespaces are documented, since they have little relevance to the user. See the manual for details, but note that members of namespaces enclosed by the \{ and \} of a grouping command are not included in the group.

The developers’ version of the API reference includes a selection of internal interfaces for core developers (who of course must nonetheless consult the source in general). Use \cond core and \endcond to mark material that should appear only there. Developers must also check for such markers to identify internal interfaces; the output does not distinguish them, since users should consult only the users’ version anyway.


Sphinx

Sphinx documentation is here. The following are some examples of frequently-used elements. However, a good practice is to just look at the existing documentation to figure out how something was done.

Headings

By convention, headings are underlined with characters in the order *+^=.

Figures

.. _undersea:
.. figure:: images/undersea.png
  :align: center
  :width: 70%

  A colorful image resembling a cosmic version of an undersea world.

This will be rendered like:

../_images/undersea.png

Fig. 17 A colorful image resembling a cosmic version of an undersea world.

You can reference the figure using its label undersea like:

As can be seen in :numref:`undersea`...

This will be rendered like:

As can be seen in Fig. 17

Code Blocks

Syntax highlighting for codes blocks uses pygments, which supports many programming and markup languages.

Here are two examples:

Console

.. code-block:: console

  $ xterm -hold -fs 10 -bg black -fg white -geometry 128x40 -e curl wttr.in

This will be rendered like:

$ xterm -hold -fs 10 -bg black -fg white -geometry 128x40 -e curl wttr.in

C++

.. code-block:: cpp

  template<typename Bar>
  using Baz = Foo<Bar>;

This will be rendered like:

template<typename Bar>
using Baz = Foo<Bar>;

Literal Includes

Literal includes allow you to directly include source code or other inputs from the actual file you are referencing. This is useful because any changes to the file will automatically be captured in the documentation.

.. literalinclude:: ../../../tutorial/2-control/1-simple.cc
  :language: cpp
  :start-at: // Function definition of an advance action.
  :end-before: // Register the finalize action under the 'finalize' control point.

This will be rendered like:

// Function definition of an advance action.

void
advance(control_policy &) {
  flog(info) << "advance" << std::endl;
}

// Register the advance action under the 'advance' control point.

control::action<advance, cp::advance> advance_action;

// Function definition of a finalize action.

void
finalize(control_policy &) {
  flog(info) << "finalize" << std::endl;
}

The included parts of the file begin with a :start-at: or :start-after: input and end with an :end-at: or :end-before: input. Each of these performs a literal string match for the string that follows the colon. That is, :start-at: some text will match against the string some text and will start including the entire line it contains. Similarly, :start-after: will match the same line, but only start including after the line containing the string. :end-at: and :end-before: work in a similar fashion. Note that for the above we included the block of comments that matched // Function definition of an advance action. and we stopped including just before a line that begins another comment block section for another part of the code. Since an end-before was used, the line containing the matched string was not included.