User Manual

Here are key functionalities offered in ClimateModels.jl.

Climate Model Interface

The interface ties the ModelConfig data structure with methods like setup, build, and launch. In return, it provides standard methods to deal with inputs and outputs, as well as capabilities described below.

The ModelRun method, or just run, streamlines the process. It executes all three steps at once (setup, build, and launch). For example, let's use RandomWalker as the model.

fun=ClimateModels.RandomWalker

With the simplified ModelConfig constructors, we can just write any of the following:

ModelRun(ModelConfig(model=fun))

or

MC=run(ModelConfig(fun))
log(MC)
5-element Vector{String}:
 "87971cc initial setup"
 "47b00f5 add Project.toml to log"
 "103ace1 add Manifest.toml to log"
 "e763c89 task started [6383c1f8-2a2b-404c-acbb-2a7af395e4f9]"
 "575e6fd (HEAD -> main) task ended   [6383c1f8-2a2b-404c-acbb-2a7af395e4f9]"

or

@ModelRun ClimateModels.RandomWalker
  ID            = 96599d08-c6ca-4601-8244-40c271219a04
  model         = RandomWalker
  configuration = anonymous
  run folder    = /tmp/96599d08-c6ca-4601-8244-40c271219a04
  log subfolder = /tmp/96599d08-c6ca-4601-8244-40c271219a04/log

By design of the ClimateModels interface, it is required that fun receives a ModelConfig as its sole input argument. This requirement is easily satisfied in practice.

Input parameters can be specified via the inputs keyword argument, or via files. See Parameters.

Breaking Things Down

Let's start with defining the model:

MC=ModelConfig(model=fun)
  ID            = a357a982-eed1-4190-8cb3-77d75d88c139
  model         = RandomWalker
  configuration = anonymous
  run folder    = /tmp/a357a982-eed1-4190-8cb3-77d75d88c139
  log subfolder = /tmp/a357a982-eed1-4190-8cb3-77d75d88c139/log

The sequence of calls within ModelRun is expanded below. In practice, setup typically handles files and software, build gets the model ready, and launch starts the model computation.

setup(MC)
build(MC)
launch(MC)

The model's top level function gets called via launch. In our example, it generates a CSV file found in the run folder as shown below.

Note

It is not required that compilation takes place during build. It can also be done beforehand or within launch.

Sometimes it is convenient to further break down the computational workflow into several tasks. These can be added to the ModelConfig via put! and then executed via launch, as demonstrated in Parameters.

The run folder name and its content can be viewed using pathof and readdir, respectively.

pathof(MC)
"/tmp/a357a982-eed1-4190-8cb3-77d75d88c139"
readdir(MC)
2-element Vector{String}:
 "RandomWalker.csv"
 "log"

The log subfolder was created earlier by setup. The log function can then retrieve the workflow log.

log(MC)
5-element Vector{String}:
 "1a93d0c initial setup"
 "5a4905a add Project.toml to log"
 "0d75755 add Manifest.toml to log"
 "6c5b43a task started [b082c618-cc29-4e3f-b448-980ab6c861d5]"
 "ecef099 (HEAD -> main) task ended   [b082c618-cc29-4e3f-b448-980ab6c861d5]"

This highlights that Project.toml and Manifest.toml for the environment being used have been archived. This happens during setup to document all dependencies and make the workflow reproducible.

Customization

A key point is that everything can be customized to, e.g., use popular models previously written in Fortran or C just as simply.

Here are simple ways to start usinf the ClimateModels.jl interface with your favorite model.

  • specify model directly as a function, and use defaults for everything else, as illustrated in random walk
  • specify model name as a String and the main Function as the configuration, as in CMIP6
  • put model in a Pluto notebook and ingest it via PlutoConfig as shown below

Sometimes, one may also want to define custom setup, build, or launch methods. To do this, one can define a concrete type of AbstractModelConfig using ModelConfig as a blueprint. This is the recommended approach when other languanges like Fortran or Python are involved (e.g., Hector.

Note

Defining a concrete type of AbstractModelConfig can also be practical with pure Julia model, e.g. to speed up launch, generate ensembles, facilitate checkpointing, etc. That's the case in the Oceananigans.jl example.

For popular models the customized interface elements can be provided via a dedicated package. This may allow them to be maintained independently by developers and users most familiar with each model. MITgcmTools.jl does this for MITgcm. It provides its own suite of examples that use the ClimateModels.jl interface.

Tracked Worklow Support

When creating a ModelConfig, it receives a unique identifier (UUIDs.uuid4()). By default, this identifier is used in the name of the run folder attached to the ModelConfig.

The run folder normally gets created by setup. During this phase, log is used to create a git enabled subfolder called log. This will allow us to record steps in our workflow – again via log.

As shown in the Parameters example:

  • Parameters specified via inputs are automatically recorded into tracked_parameters.toml during setup.
  • Modified parameters are automatically recorded in tracked_parameters.toml during launch.
  • log called on a ModelConfig with no other argument shows the workflow record.

Parameters

Let's now mofdify model parameters, then rerun a model, and keep track of these workflow steps.

After an initial model run of 100 steps, duration NS is extended to 200 time steps. The put! and launch sequence then reruns the model.

Note

The same method can be used to break down a workflow in several steps. Each call to launch sequentially takes the next task from the stack (i.e., channel). Once the task channel is empty then launch does nothing.

mc=ModelConfig(fun,(NS=100,filename="run01.csv"))
run(mc)

mc.inputs[:NS]=200
mc.inputs[:filename]="run02.csv"
put!(mc)
launch(mc)

log(mc)
9-element Vector{String}:
 "5d1c3c2 initial setup"
 "f6b2e3e initial tracked_parameters.toml"
 "4a71420 add Project.toml to log"
 "8bc4e63 add Manifest.toml to log"
 "7d47a3f task started [39e6048a-2a3a-4ce0-bdda-243e9027491c]"
 "44ed750 task ended   [39e6048a-2a3a-4ce0-bdda-243e9027491c]"
 "93389ce task started [b09e693d-d833-493c-94fb-6fde8b9b3162]"
 "72e0a63 modify tracked_parameters.toml"
 "ba7bc35 (HEAD -> main) task ended   [b09e693d-d833-493c-94fb-6fde8b9b3162]"

The call sequence is readily reflected in the workflow log, and the run folder now has two output files.

readdir(mc)
3-element Vector{String}:
 "log"
 "run01.csv"
 "run02.csv"

In more complex models, there generally is a large number of parameters that are often organized in a collection of text files.

The ClimateModels.jl interface is easily customized to turn those into a tracked_parameters.toml file as demonstrated in the Hector and in the MITgcm.

ClimateModels.jl thus readily enables interacting with parameters and tracking their values even with complex models as highlighted in the JuliaCon 2021 Presentation.

Pluto Notebook Integration

Any Pluto notebook is easily integrated to the ClimateModels.jl framework via PlutoConfig.

filename=joinpath(tempdir(),"notebook.jl")
PC=PlutoConfig(filename,(linked_model="MC",))
run(PC)
readdir(PC)
7-element Vector{String}:
 "CellOrder.txt"
 "MC.0cefc74a-0252-451e-a18e-84c623b41ce6"
 "Manifest.toml"
 "Project.toml"
 "log"
 "main.jl"
 "stdout.txt"

The Pluto notebook gets split up into main code (1) and environment (2). This approach provides a simple way to go from model documentation, in notebook format, to large simulations run, done in batch mode.

Files get copied into pathof(PC) as before. If notebook.jl contains a ModelConfig, let's call it MC, then the pathof(MC) folder can be linked into pathof(PC) at the end. This feature is controlled by linked_model as illustrated just before. A data input folder can be specified via the data_folder key. This will result in the specified folder getting linked into pathof(PC) before running the notebook.

update provides a simple method for updating notebook dependencies. Such routine maintanance is often followed by rerunning the notebook to detect potential updating issues.

update(PlutoConfig(filename))
run(PlutoConfig(filename))
  Activating project at `/tmp/60c59267-9686-42ea-a840-7823a65d69a5`
    Updating registry at `~/.julia/registries/General.toml`
   Installed MPIPreferences ──── v0.1.11
   Installed MPICH_jll ───────── v4.2.1+1
   Installed MPItrampoline_jll ─ v5.3.3+1
   Installed NetCDF_jll ──────── v400.902.211+0
   Installed HDF5_jll ────────── v1.14.3+3
   Installed PlutoUI ─────────── v0.7.59
   Installed OpenMPI_jll ─────── v4.1.6+0
   Installed ClimateModels ───── v0.3.2
    Updating `/tmp/60c59267-9686-42ea-a840-7823a65d69a5/Project.toml`
  [13f3f980] ↑ CairoMakie v0.11.9 ⇒ v0.11.10
  [f6adb021] ↑ ClimateModels v0.3.1 ⇒ v0.3.2
  [7f904dfe] ↑ PlutoUI v0.7.58 ⇒ v0.7.59
    Updating `/tmp/60c59267-9686-42ea-a840-7823a65d69a5/Manifest.toml`
  [6e696c72] ↑ AbstractPlutoDingetjes v1.3.0 ⇒ v1.3.2
  [66dad0bd] + AliasTables v1.1.1
  [4fba245c] ↑ ArrayInterface v7.9.0 ⇒ v7.10.0
  [336ed68f] ↑ CSV v0.10.13 ⇒ v0.10.14
  [13f3f980] ↑ CairoMakie v0.11.9 ⇒ v0.11.10
  [f6adb021] ↑ ClimateModels v0.3.1 ⇒ v0.3.2
  [3da002f7] ↑ ColorTypes v0.11.4 ⇒ v0.11.5
  [34da2185] ↑ Compat v4.14.0 ⇒ v4.15.0
  [d38c429a] ↑ Contour v0.6.2 ⇒ v0.6.3
  [864edb3b] ↑ DataStructures v0.18.18 ⇒ v0.18.20
  [31c24e10] ↑ Distributions v0.25.107 ⇒ v0.25.108
  [1a297f60] ↑ FillArrays v1.9.3 ⇒ v1.11.0
  [6a86dc24] ↑ FiniteDiff v2.23.0 ⇒ v2.23.1
  [663a7486] ↑ FreeTypeAbstraction v0.10.1 ⇒ v0.10.3
  [cf35fbd7] ↑ GeoInterface v1.3.3 ⇒ v1.3.4
  [5c1252a2] ↑ GeometryBasics v0.4.10 ⇒ v0.4.11
  [d1acc4aa] ↑ IntervalArithmetic v0.22.9 ⇒ v0.22.11
  [5ab0869b] ↑ KernelDensity v0.6.8 ⇒ v0.6.9
  [3da0fdf6] ↑ MPIPreferences v0.1.10 ⇒ v0.1.11
  [ee78f7c6] ↑ Makie v0.20.8 ⇒ v0.20.9
  [e1d29d7a] ↑ Missings v1.1.0 ⇒ v1.2.0
  [6fe1bfb0] ↑ OffsetArrays v1.13.0 ⇒ v1.14.0
  [429524aa] ↑ Optim v1.9.3 ⇒ v1.9.4
  [65ce6f38] - PackageExtensionCompat v1.0.2
  [2ae35dd2] ↑ Permutations v0.4.20 ⇒ v0.4.21
  [3bbf5609] - PikaParser v0.6.1
  [7f904dfe] ↑ PlutoUI v0.7.58 ⇒ v0.7.59
  [f27b6e38] ↑ Polynomials v4.0.6 ⇒ v4.0.8
  [c5dd0088] - StableHashTraits v1.1.8
  [2913bbd2] ↑ StatsBase v0.34.2 ⇒ v0.34.3
  [9d95972d] - TupleTools v1.5.0
  [2e619515] ↑ Expat_jll v2.5.0+0 ⇒ v2.6.2+0
  [a3f928ae] ↑ Fontconfig_jll v2.13.93+0 ⇒ v2.13.96+0
  [559328eb] ↑ FriBidi_jll v1.0.10+0 ⇒ v1.0.14+0
  [f8c6e375] ↑ Git_jll v2.44.0+1 ⇒ v2.44.0+2
  [0234f1f7] ↑ HDF5_jll v1.14.2+1 ⇒ v1.14.3+3
  [905a6f67] ↑ Imath_jll v3.1.7+0 ⇒ v3.1.11+0
  [1d5cc7b8] ↑ IntelOpenMP_jll v2024.0.2+0 ⇒ v2024.1.0+0
  [c1c5ebd0] ↑ LAME_jll v3.100.1+0 ⇒ v3.100.2+0
  [dd4b983a] ↑ LZO_jll v2.10.1+0 ⇒ v2.10.2+0
  [d4300ac3] ↑ Libgcrypt_jll v1.8.7+0 ⇒ v1.8.11+0
  [7add5ba3] ↑ Libgpg_error_jll v1.42.0+0 ⇒ v1.49.0+0
  [4b2f31a3] ↑ Libmount_jll v2.39.3+0 ⇒ v2.40.0+0
  [38a345b3] ↑ Libuuid_jll v2.39.3+1 ⇒ v2.40.0+0
  [856f044c] ↑ MKL_jll v2024.0.0+0 ⇒ v2024.1.0+0
  [7cb0a576] ↑ MPICH_jll v4.2.0+0 ⇒ v4.2.1+1
  [f1f71cc9] ↑ MPItrampoline_jll v5.3.2+0 ⇒ v5.3.3+1
  [7243133f] ↑ NetCDF_jll v400.902.209+0 ⇒ v400.902.211+0
  [18a262bb] ↑ OpenEXR_jll v3.1.4+0 ⇒ v3.2.4+0
⌅ [fe0851c0] ↓ OpenMPI_jll v5.0.2+0 ⇒ v4.1.6+0
  [32165bc3] - PMIx_jll v4.2.7+0
  [36c8627f] ↑ Pango_jll v1.52.1+0 ⇒ v1.52.2+0
  [1082639a] ↑ Xorg_libXext_jll v1.3.4+4 ⇒ v1.3.6+0
  [ea2f1a96] ↑ Xorg_libXrender_jll v0.9.10+4 ⇒ v0.9.11+0
  [1080aeaf] - libevent_jll v2.1.13+1
  [1317d2d5] + oneTBB_jll v2021.12.0+0
  [eb928a42] - prrte_jll v3.0.2+0
  [e66e0078] ↑ CompilerSupportLibraries_jll v1.1.0+0 ⇒ v1.1.1+0
        Info Packages marked with ⌅ have new versions available but compatibility constraints restrict them from upgrading. To see why use `status --outdated -m`
Precompiling project...
  ✓ MPIPreferences
  ✓ MPItrampoline_jll
  ✓ DiskArrays
  ✓ OpenMPI_jll
  ✓ MPICH_jll
  ✓ HDF5_jll
  ✓ NetCDF_jll
  ✓ PlutoUI
  ✓ NetCDF
  ✓ ClimateModels
  10 dependencies successfully precompiled in 5 seconds. 291 already precompiled.
  1 dependency precompiled but a different version is currently loaded. Restart julia to access the new version
        Info We haven't cleaned this depot up for a bit, running Pkg.gc()...
      Active manifest files: 3 found
      Active artifact files: 101 found
      Active scratchspaces: 4 found
     Deleted no artifacts, repos, packages or scratchspaces
  Activating project at `~/work/ClimateModels.jl/ClimateModels.jl/docs`
  Activating project at `/tmp/152d37b6-4c83-4ea2-9b33-5579d3ccff80`
  Activating project at `~/work/ClimateModels.jl/ClimateModels.jl/docs`

Files and Cloud Support

Numerical model output often gets archived, distributed, and retrieved over the web. Some times, downloading data is most convenient. In other cases, it is preferable to compute in the cloud and just download final results.

ClimateModels.jl has examples for most common file formats. These are handled via Downloads.jl, NetCDF.jl, DataFrames.jl, CSV.jl, and TOML.jl.

fil=joinpath(pathof(mc),"run02.csv")

CSV=ClimateModels.CSV
DataFrame=ClimateModels.DataFrame

CSV.File(fil) |> DataFrame
"200×2 DataFrame"
Note

For more examples with NetCDF.jl and Zarr.jl, please look at IPCC notebook and CMIP6 notebok.