How to run ERSEM

Build ERSEM

1. Make a “build” folder, and go into the build folder.

2. Create “build_archer2.edit.sh”

#!/usr/bin/env bash# Script for compiling FVCOM-FABM-ERSEM for ARCHER2
#
# The build is split into three phases:
#
# 1) Build the FABM-ERSEM library. Here, ERSEM is a marine
# biogeochemical model while FABM is a piece of software
# which allows the ocean model (FVCOM) and ERSEM to
# communicate. ERSEM's main job is to compute d/dt sink/source
# terms for biogeochemical tracers. FABM communicates these
# to FVCOM, which in turn factors them in during the
# forward integration of the model.
#
# 2) Build FVCOM ancillary libraries. These include the
# sub-packages julian, metis, fproj and proj which are
# distributed with the FVCOM source code to facilitate
# the build process.
#
# 3) Build FVCOM.
#
# ---------------------------------------------------------------------set -euo pipefailusage() {
cat <<USAGEBuild FVCOM-FABM-ERSEM.Usage: build.sh [-ch]Options:
c    Clean up after a previous build and exit
h    Display helpUSAGE
}clean_past_build() {printf "Cleaning build directory ... \n\n"# Remove FVCOM build and install directoriesif [ -d ${BUILD_DIR} ]; thenread -r -p "Remove directory ${BUILD_DIR} (y/n): " removeif [ ${remove} = "y" ]; thenecho "Removing ${BUILD_DIR} ..."rm -r ${BUILD_DIR}fifiif [ -d ${INSTALL_DIR} ]; thenread -r -p "Remove directory ${INSTALL_DIR} (y/n): " removeif [ ${remove} = "y" ]; thenecho "Removing ${INSTALL_DIR} ..."rm -r ${INSTALL_DIR}fifi# Clean up FVCOM libs dircd ${FVCOM_LIBS_DIR}printf "\nCleaning ${FVCOM_LIBS_DIR} ...\n"make clean TOPDIR=${FVCOM_TOP_DIR} BIODIR=${FABM_INSTALL_DIR}  PETSC_DIRN=${PETSC_DIRN}# Clean up FVCOM code dircd ${FVCOM_TOP_DIR}printf "\nCleaning ${FVCOM_TOP_DIR} ...\n"make clean TOPDIR=${FVCOM_TOP_DIR} BIODIR=${FABM_INSTALL_DIR}  PETSC_DIRN=${PETSC_DIRN}
}clean=false
while getopts "hc" options; docase $options in(c)clean=true;;(h)usageexit 0;;(?)usageexit 1;;esac
done# <<TO CHANGE - Load required modules (tested on ARCHER2)>>#module use /sw/com/modulefiles/
#module swap PrgEnv-cray/8.3.3 PrgEnv-intel/8.3.3
#module swap intel/2021.4.0 intel-classic/2021.4.0
#module load cray-hdf5-parallel
#module load cray-netcdf-hdf5parallel
#module list# Set up paths
# ------------# Root directory - assumes the package has been unzipped into an
# appropriate location on the HPC (e.g. somewhere in /work).
ROOT_DIR=$(pwd)# All other paths are set relative to ROOTDIR
CODE_DIR="${ROOT_DIR}/code"
BUILD_DIR="${ROOT_DIR}/build"
INSTALL_DIR="${ROOT_DIR}/install"# FABM/ERSEM paths
FABM_SRC_DIR="${CODE_DIR}/FABM/src"
#ERSEM_SRC_DIR="${CODE_DIR}/ERSEM-erie/src"
ERSEM_SRC_DIR="${CODE_DIR}/ersem_lake_erie-main/src"
FABM_BUILD_DIR="${BUILD_DIR}/fabm"
FABM_INSTALL_DIR="${INSTALL_DIR}/fabm"# FVCOM ancillary library paths
FVCOM_TOP_DIR="${CODE_DIR}/FVCOM_source"
FVCOM_LIBS_DIR="${FVCOM_TOP_DIR}/libs"
PETSC_DIRN="/pexue6/chenfuh/PETSc_intel2022a/petsc-2.3.3-p16_HYPRE"# Copy over main.inc for HPC dependent compiler flags
# ---------------------------------------------------# <<TO_CHANGE - copy in FVCOM make.inc for compiler flags etc (tested on ARCHER2)>>
#ln -sf ${FVCOM_TOP_DIR}/build_utils/make_PML_ARCHER2.edit.inc ${FVCOM_TOP_DIR}/make.inc
#ln -sf ${ROOT_DIR}/make_PML_ARCHER2.edit.inc ${FVCOM_TOP_DIR}/make.inc
ln -sf ${ROOT_DIR}/make_lakeErie.inc ${FVCOM_TOP_DIR}/make.incif [ ${clean} = true ]; thenclean_past_buildexit 0
fi# Step 1 - Build the FABM-ERSEM library
# -------------------------------------echo "STEP 1: Building the FABM-ERSEM library"if [ ! -d ${FABM_BUILD_DIR} ]; thenmkdir -p ${FABM_BUILD_DIR}
fi
if [ ! -d ${FABM_INSTALL_DIR} ]; thenmkdir -p ${FABM_INSTALL_DIR}
fi# Number of jobs to use when building with cmake
NJOBS=4cd $FABM_BUILD_DIR# <<TO_CHANGE - the specified fortran compiler (tested on ARCHER2)>>
#cmake ${FABM_SRC_DIR} -DCMAKE_INSTALL_PREFIX=${FABM_INSTALL_DIR} \
#      -DFABM_ERSEM_BASE=${ERSEM_SRC_DIR} -DFABM_HOST=fvcom \
#      -DCMAKE_Fortran_COMPILER=mpiifx
#/pexue2/pexue/local2/cmake/bin/cmake ${FABM_SRC_DIR} -DCMAKE_INSTALL_PREFIX=${FABM_INSTALL_DIR} \
#      -DFABM_ERSEM_BASE=${ERSEM_SRC_DIR} -DFABM_HOST=fvcom \
#      -DCMAKE_Fortran_COMPILER=mpiifort
cmake ${FABM_SRC_DIR} -DCMAKE_INSTALL_PREFIX=${FABM_INSTALL_DIR} \-DFABM_ERSEM_BASE=${ERSEM_SRC_DIR} -DFABM_HOST=fvcom \-DCMAKE_Fortran_COMPILER=mpiifort
make install -j ${NJOBS}cd ${ROOT_DIR}# Step 2 - Build FVCOM ancillary libraries
# ----------------------------------------echo "STEP 2: Building FVCOM ancillary libraries"cd ${FVCOM_LIBS_DIR}#make TOPDIR=${FVCOM_TOP_DIR} BIODIR=${FABM_INSTALL_DIR} PETSC_DIRN=${PETSC_DIRN}cd ${ROOT_DIR}# Step 3 - Build FVCOM
# ----------------------------------------echo "STEP 3: Build FVCOM"cd ${FVCOM_TOP_DIR}make -j 20 TOPDIR=${FVCOM_TOP_DIR} BIODIR=${FABM_INSTALL_DIR}  PETSC_DIRN=${PETSC_DIRN}cd ${ROOT_DIR}# Check if we have an executable and finish
FVCOM_EXE="${FVCOM_TOP_DIR}/fvcom"
if [ -x ${FVCOM_EXE} ]; thenFVCOM_RUN_DIR="${ROOT_DIR}/run"mkdir -p ${FVCOM_RUN_DIR}rsync -avP ${FVCOM_EXE} ${FVCOM_RUN_DIR}/bin/ && echo "Build was successful"
elseecho "Build failed"
fi

3. Make “code” folder according to the “build_archer2.edit.sh” above.

# All other paths are set relative to ROOTDIR
CODE_DIR="${ROOT_DIR}/code"

[czhao4@compute-0-200 build]$ ls
build_archer2.edit.sh  code

4. FVCOM ancillary library paths: Make “FVCOM_source” under “code” folder according to the “build_archer2.edit.sh” above.

# FVCOM ancillary library paths
FVCOM_TOP_DIR="${CODE_DIR}/FVCOM_source"

Now code folder shows like below:

[czhao4@compute-0-200 build]$ ls code/
ersem_lake_erie-main  FABM  FVCOM_source

The source code of “ersem_lake_erie-main FABM FVCOM_source” can be found on HPC /pexue7/czhao4/ERSEM/example/build/code

5. In root_dir “build” folder, create “make_lakeErie.inc”. It is the namelist file for FVCOM, but modified for ERSEM. It can be found at /pexue7/czhao4/ERSEM/example/build/make_lakeErie.inc.
   Modify the following directory accordingly:

#========== TOPDIR ========================================================
# TOPDIR is the directory in which this make file and the fvcom source reside#           TOPDIR  = /users/modellers/rito/src/FVCOMv4.3-FABMv1/FVCOM_source
#           TOPDIR  = /pexue6/chenfuh/GLE_ERSEM_PML2/Erie_ERSEM_PML2/build_ch_new20250224/code/FVCOM_sourceTOPDIR  = /pexue7/czhao4/ERSEM/example/build/code/FVCOM_source
#                   PETSC_DIRN     = /mnt/projects/hpc/code/fvcom/PETSc_new/petsc-2.3.3-p16
#           PETSC_DIRN=/gpfs1/users/modellers/pica/Software/apps/petsc/intelmpi/2.3.3PETSC_DIRN=/pexue6/chenfuh/PETSc_intel2022a/petsc-2.3.3-p16_HYPRE

Then, it will be soft linked to build/code/FVCOM_source/make.inc by “build_archer2.edit.sh”.
6. Create "run_build.sh "

#!/bin/bash#module load oneapi/{tbb,oclfpga,compiler-rt,compiler,mkl,mpi}
#module load toolchain/intel/2022a
module load devel/CMake/3.23.1-GCCcore-11.3.0
module list#export maindir=/users/modellers/rito/Code/lakeerie#export NETCDF_DIR=/scyllapfs/hpe/fthomas/css/.local/netcdf-4.9.0/oneapi
#export HDF5_DIR=/scyllapfs/hpe/fthomas/css/.local/hdf5-1.14.0/oneapi/
export PETSC_DIR=/pexue6/chenfuh/PETSc_intel2022a/petsc-2.3.3-p16_HYPRE
export PETSC_ARCH=arch-linux2-c-debug
export NETCDF_DIR=/pexue6/local_px/netcdf4.6.1
export HDF5_DIR=/pexue6/local_px/hdf5-1.14.2
#export LD_LIBRARY_PATH=/pexue6/chenfuh/test_xy_gle_DA/NOAA_FVCOM/fvcom3.2.2_xinyu_xy_proj_DA_weight_control_all_layers/libs/install/lib:$LD_LIBRARY_PATH
#export LD_LIBRARY_PATH=/pexue6/chenfuh/PETSc_intel2022a/petsc-2.3.3-p16/lib/arch-linux2-c-debug:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=/pexue6/chenfuh/PETSc_intel2022a/petsc-2.3.3-p16_HYPRE/lib/arch-linux2-c-debug:$LD_LIBRARY_PATH#cd $maindir
#cp -r $maindir/IMP_files_Intel/build_archer2.edit.sh PML_FVCOM_Benchmark/
#cp -r $maindir/make_PML_ARCHER2.edit.inc code/FVCOM_source/build_utils/
#cd PML_FVCOM_Benchmark./build_archer2.edit.sh -c
./build_archer2.edit.sh 2>&1 | tee build.log

7. source run_build.sh
   And if build success, it shows:

.....
-- Installing: /pexue7/czhao4/ERSEM/example/build/install/fabm/include/fabm_coupling.mod
-- Installing: /pexue7/czhao4/ERSEM/example/build/install/fabm/include/fabm_config_types.mod
-- Installing: /pexue7/czhao4/ERSEM/example/build/install/fabm/include/fabm_builtin_reduction.mod
-- Installing: /pexue7/czhao4/ERSEM/example/build/install/fabm/include/fabm.h
-- Installing: /pexue7/czhao4/ERSEM/example/build/install/fabm/include/fabm_version.h
-- Installing: /pexue7/czhao4/ERSEM/example/build/install/fabm/include/fabm_driver.h
-- Installing: /pexue7/czhao4/ERSEM/example/build/install/fabm/cmake/fabmConfig.cmake
-- Installing: /pexue7/czhao4/ERSEM/example/build/install/fabm/cmake/fabmConfig-release.cmake
STEP 2: Building FVCOM ancillary libraries
STEP 3: Build FVCOM
make: Circular mod_prec.o <- mod_prec.o dependency dropped.
make: `fvcom' is up to date.
sending incremental file list
created directory /pexue7/czhao4/ERSEM/example/build/run/bin
fvcom33,496,320 100%  239.95MB/s    0:00:00 (xfr#1, to-chk=0/1)sent 33,504,606 bytes  received 100 bytes  67,009,412.00 bytes/sec
total size is 33,496,320  speedup is 1.00
Build was successful

And it will generate an excutable file “fvcom” at:

[czhao4@compute-0-200 build]$ ls
build  build_archer2.edit.sh  build.log  code  install  make_lakeErie.inc  run  run_build.sh
[czhao4@compute-0-200 build]$ ls run/bin/
fvcom

Run ERSEM

1. Make a run folder and generally, its structure is same with FVCOM (because ERSEM is actually a bio module in FVCOM).

[czhao4@compute-0-200 test]$ ll
total 892K
lrwxrwxrwx 1 czhao4 pexue-users   45 Apr 15 07:54 erie_run.nml
lrwxrwxrwx 1 czhao4 pexue-users   51 Apr 15 08:18 fabm_output.yaml # Define the output var list. If this file does not exist, all var will be output.
lrwxrwxrwx 1 czhao4 pexue-users   27 Apr 15 08:15 fabm.yaml # Input for ERSEM
lrwxrwxrwx 1 czhao4 pexue-users   48 Apr 15 06:50 fvcom -> /pexue7/czhao4/ERSEM/example/build/run/bin/fvcom
-rwxr-xr-x 1 czhao4 pexue-users 1.5K Apr 15 08:10 fvcom.sh
drwxr-xr-x 2 czhao4 pexue-users 4.0K Apr 15 08:13 input
drwxr-xr-x 2 czhao4 pexue-users 4.0K Apr 15 08:13 output
-rw-r--r-- 1 czhao4 pexue-users  646 Apr 15 08:22 run.sh

[czhao4@compute-0-200 test]$ ll input/
total 19G
-rwxr-xr-x 1 czhao4 pexue-users 532K Feb 24 17:54 erie_brf.nc
-rwxr-xr-x 1 czhao4 pexue-users 257K Feb 24 17:57 erie_cor.dat
-rwxr-xr-x 1 czhao4 pexue-users 8.1G Feb 19 07:04 erie_forcing_interp_2018-2019.nc
-rw-r--r-- 1 czhao4 pexue-users 4.1G Apr  8 14:33 erie_forcing_interp_2019.nc
-rwxr-xr-x 1 czhao4 pexue-users  48M Feb 19 06:59 erie_restart_0013.nc
-rwxr-xr-x 1 czhao4 pexue-users  48M Feb 19 07:02 erie_restart_2018_cycle7.nc
-rwxr-xr-x 1 czhao4 pexue-users  91M Feb 19 07:02 erie_restart_2018.nc
lrwxrwxrwx 1 czhao4 pexue-users  106 Apr  8 14:01 erie_restart_20190101.nc -> /pexue6/chenfuh/GLE_ERSEM_PML2/Erie_ERSEM_PML2/test_run_ch_new20250224/output2018/erie_restart_20190101.nc
-rwxr-xr-x 1 czhao4 pexue-users  48M Feb 19 07:01 erie_restart_2019.nc
-rwxr-xr-x 1 czhao4 pexue-users 892K Feb 19 07:05 julian_obc_2018-2019.nc
-rwxr-xr-x 1 czhao4 pexue-users  52M Feb 19 07:05 River_data_2018-2019.nc
-rwxr-xr-x 1 czhao4 pexue-users 9.9K Feb 19 07:16 RIVERS_NAMELIST.nml
-rwxr-xr-x 1 czhao4 pexue-users  13K Feb 24 17:56 sediment.inp
-rwxr-xr-x 1 czhao4 pexue-users 263K Feb 19 07:05 tsobc_2018-2019.nc
-rwxr-xr-x 1 czhao4 pexue-users 6.0G Feb 19 07:07 ww3_mod.2018-2019.nc

run.sh shows like:

export NETCDF_DIR=/pexue6/local_px/netcdf4.6.1
export HDF5_DIR=/pexue6/local_px/hdf5-1.14.2
export LD_LIBRARY_PATH=/pexue6/chenfuh/PETSc_intel2022a/petsc-2.3.3-p16_HYPRE/lib/arch-linux2-c-debug:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=/pexue6/local_px/netcdf4.6.1/lib:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=/pexue6/local_px/hdf5-1.14.2/lib:$LD_LIBRARY_PATH
module load toolchain/intel/2022a
mpirun -n 1 ./fvcom --casename=erie

Some notes about ERSEM var

  N1p: N1/p                                            # phosphate (mmol P/m^3)N3n: N3/n                                            # nitrate (mmol N/m^3)N4n: N4/n                                            # ammonium (mmol N/m^3)N5s: N5/s                                            # silicate (mmol Si/m^3)R1c: R1/c                                            # dissolved organic carbon (mg C/m^3)R1p: R1/p                                            # dissolved organic phosphorus (mmol P/m^3)R1n: R1/n                                            # dissolved organic nitrogen (mmol N/m^3)R2c: R2/c                                            # semi-labile dissolved organic carbon (mg C/m^3)O2o: O2/o                                            # oxygen (mmol O_2/m^3)O3c: O3/c                                            # carbon dioxide (mmol C/m^3)