User Manual
Here are key functionalities offered in ClimateModels.jl
.
Climate Model Interface
The interface ties the ModelConfig
data structure with methods like setup
, build
, and launch
. In return, it provides standard methods to deal with inputs and outputs, as well as capabilities described below.
The ModelRun
method, or just run
, streamlines the process. It executes all three steps at once (setup
, build
, and launch
). For example, let's use RandomWalker
as the model.
fun=ClimateModels.RandomWalker
With the simplified ModelConfig
constructors, we can just write any of the following:
ModelRun(ModelConfig(model=fun))
or
MC=run(ModelConfig(fun))
log(MC)
5-element Vector{String}:
"da7bcde initial setup"
"e043448 add Project.toml to log"
"363123f add Manifest.toml to log"
"f0d2959 task started [ebfab54f-3790-480c-b6c3-a2b8fce9e89e]"
"9ecc694 (HEAD -> main) task ended [ebfab54f-3790-480c-b6c3-a2b8fce9e89e]"
or
@ModelRun ClimateModels.RandomWalker
ID = 8ff0fa0e-b0d3-4402-b0df-7fccb55daf2e
model = RandomWalker
configuration = anonymous
run folder = /tmp/8ff0fa0e-b0d3-4402-b0df-7fccb55daf2e
log subfolder = /tmp/8ff0fa0e-b0d3-4402-b0df-7fccb55daf2e/log
By design of the ClimateModels
interface, it is required that fun
receives a ModelConfig
as its sole input argument. This requirement is easily satisfied in practice.
Input parameters can be specified via the inputs
keyword argument, or via files. See Parameters.
Breaking Things Down
Let's start with defining the model:
MC=ModelConfig(model=fun)
ID = 81bbd4e5-6b26-45c3-86a9-364c83d47fec
model = RandomWalker
configuration = anonymous
run folder = /tmp/81bbd4e5-6b26-45c3-86a9-364c83d47fec
log subfolder = /tmp/81bbd4e5-6b26-45c3-86a9-364c83d47fec/log
The sequence of calls within ModelRun
is expanded below. In practice, setup
typically handles files and software, build
gets the model ready, and launch
starts the model computation.
setup(MC)
build(MC)
launch(MC)
The model
's top level function gets called via launch
. In our example, it generates a CSV file found in the run folder as shown below.
It is not required that compilation takes place during build
. It can also be done beforehand or within launch
.
Sometimes it is convenient to further break down the computational workflow into several tasks. These can be added to the ModelConfig
via put!
and then executed via launch
, as demonstrated in Parameters.
The run folder name and its content can be viewed using pathof
and readdir
, respectively.
pathof(MC)
"/tmp/81bbd4e5-6b26-45c3-86a9-364c83d47fec"
readdir(MC)
2-element Vector{String}:
"RandomWalker.csv"
"log"
The log
subfolder was created earlier by setup
. The log
function can then retrieve the workflow log.
log(MC)
5-element Vector{String}:
"57da7b3 initial setup"
"d140168 add Project.toml to log"
"f97a02a add Manifest.toml to log"
"dca648d task started [553d383b-c541-4fe1-9757-12b0d8b2dd0c]"
"72aa358 (HEAD -> main) task ended [553d383b-c541-4fe1-9757-12b0d8b2dd0c]"
This highlights that Project.toml
and Manifest.toml
for the environment being used have been archived. This happens during setup
to document all dependencies and make the workflow reproducible.
Customization
A key point is that everything can be customized to, e.g., use popular models previously written in Fortran or C just as simply.
Here are simple ways to start usinf the ClimateModels.jl
interface with your favorite model
.
- specify
model
directly as a function, and use defaults for everything else, as illustrated in random walk - specify
model
name as aString
and the mainFunction
as theconfiguration
, as in CMIP6 - put
model
in a Pluto notebook and ingest it viaPlutoConfig
as shown below
Sometimes, one may also want to define custom setup
, build
, or launch
methods. To do this, one can define a concrete type of AbstractModelConfig
using ModelConfig
as a blueprint. This is the recommended approach when other languanges like Fortran or Python are involved (e.g., Hector.
Defining a concrete type of AbstractModelConfig
can also be practical with pure Julia model, e.g. to speed up launch
, generate ensembles, facilitate checkpointing, etc. That's the case in the Oceananigans.jl example.
For popular models the customized interface elements can be provided via a dedicated package. This may allow them to be maintained independently by developers and users most familiar with each model. MITgcmTools.jl does this for MITgcm. It provides its own suite of examples that use the ClimateModels.jl
interface.
Tracked Worklow Support
When creating a ModelConfig
, it receives a unique identifier (UUIDs.uuid4()
). By default, this identifier is used in the name of the run folder attached to the ModelConfig
.
The run folder normally gets created by setup
. During this phase, log
is used to create a git
enabled subfolder called log
. This will allow us to record steps in our workflow – again via log
.
As shown in the Parameters example:
- Parameters specified via
inputs
are automatically recorded intotracked_parameters.toml
duringsetup
. - Modified parameters are automatically recorded in
tracked_parameters.toml
duringlaunch
. log
called on aModelConfig
with no other argument shows the workflow record.
Parameters
Let's now mofdify model parameters, then rerun a model, and keep track of these workflow steps.
After an initial model run of 100 steps, duration NS
is extended to 200 time steps. The put!
and launch
sequence then reruns the model.
The same method can be used to break down a workflow in several steps. Each call to launch
sequentially takes the next task from the stack (i.e., channel
). Once the task channel
is empty then launch
does nothing.
mc=ModelConfig(fun,(NS=100,filename="run01.csv"))
run(mc)
mc.inputs[:NS]=200
mc.inputs[:filename]="run02.csv"
put!(mc)
launch(mc)
log(mc)
9-element Vector{String}:
"2ff9e8d initial setup"
"58a041f initial tracked_parameters.toml"
"6629d29 add Project.toml to log"
"40e31dc add Manifest.toml to log"
"fbcc30a task started [f3dee56a-439f-4f73-a598-55df16d8ceba]"
"a26ad20 task ended [f3dee56a-439f-4f73-a598-55df16d8ceba]"
"a742e63 task started [5460ff7c-6ec7-41cf-9800-e55559f9ff43]"
"fc84b59 modify tracked_parameters.toml"
"7e5436c (HEAD -> main) task ended [5460ff7c-6ec7-41cf-9800-e55559f9ff43]"
The call sequence is readily reflected in the workflow log, and the run folder now has two output files.
readdir(mc)
3-element Vector{String}:
"log"
"run01.csv"
"run02.csv"
In more complex models, there generally is a large number of parameters that are often organized in a collection of text files.
The ClimateModels.jl
interface is easily customized to turn those into a tracked_parameters.toml
file as demonstrated in the Hector and in the MITgcm.
ClimateModels.jl
thus readily enables interacting with parameters and tracking their values even with complex models as highlighted in the JuliaCon 2021 Presentation.
Pluto Notebook Integration
Any Pluto notebook is easily integrated to the ClimateModels.jl
framework via PlutoConfig
.
filename=joinpath(tempdir(),"notebook.jl")
PC=PlutoConfig(filename,(linked_model="MC",))
run(PC)
readdir(PC)
7-element Vector{String}:
"CellOrder.txt"
"MC.1a280e23-3703-4d2d-9148-d5082cd8a80f"
"Manifest.toml"
"Project.toml"
"log"
"main.jl"
"stdout.txt"
The Pluto notebook gets split up into main code (1) and environment (2). This approach provides a simple way to go from model documentation, in notebook format, to large simulations run, done in batch mode.
Files get copied into pathof(PC)
as before. If notebook.jl
contains a ModelConfig
, let's call it MC
, then the pathof(MC)
folder can be linked into pathof(PC)
at the end. This feature is controlled by linked_model
as illustrated just before. A data input folder can be specified via the data_folder
key. This will result in the specified folder getting linked into pathof(PC)
before running the notebook.
update
provides a simple method for updating notebook dependencies. Such routine maintanance is often followed by rerunning the notebook to detect potential updating issues.
update(PlutoConfig(filename))
run(PlutoConfig(filename))
Activating project at `/tmp/fcd353f0-c552-48a5-b73b-b1a584711d29`
Updating registry at `~/.julia/registries/General.toml`
Installed PlutoUI ─────── v0.7.61
Installed ClimateModels ─ v0.3.8
Updating `/tmp/fcd353f0-c552-48a5-b73b-b1a584711d29/Project.toml`
[13f3f980] ↑ CairoMakie v0.12.16 ⇒ v0.13.1
[f6adb021] ↑ ClimateModels v0.3.6 ⇒ v0.3.8
[7f904dfe] ↑ PlutoUI v0.7.60 ⇒ v0.7.61
Updating `/tmp/fcd353f0-c552-48a5-b73b-b1a584711d29/Manifest.toml`
[179af706] ↑ CFTime v0.1.3 ⇒ v0.1.4
[13f3f980] ↑ CairoMakie v0.12.16 ⇒ v0.13.1
[d360d2e6] ↑ ChainRulesCore v1.25.0 ⇒ v1.25.1
[f6adb021] ↑ ClimateModels v0.3.6 ⇒ v0.3.8
[a2cac450] ↑ ColorBrewer v0.4.0 ⇒ v0.4.1
[35d6a980] ↑ ColorSchemes v3.27.1 ⇒ v3.28.0
[5ae59095] ↑ Colors v0.12.11 ⇒ v0.13.0
[f0e56b4a] ↑ ConcurrentUtilities v2.4.2 ⇒ v2.4.3
[927a84f5] ↑ DelaunayTriangulation v1.6.3 ⇒ v1.6.4
[31c24e10] ↑ Distributions v0.25.113 ⇒ v0.25.117
[411431e0] ↑ Extents v0.1.4 ⇒ v0.1.5
[7a1cc6ca] ↑ FFTW v1.8.0 ⇒ v1.8.1
[68eda718] ↑ GeoFormatTypes v0.4.2 ⇒ v0.4.4
[cf35fbd7] ↑ GeoInterface v1.3.8 ⇒ v1.4.1
[5c1252a2] ↑ GeometryBasics v0.4.11 ⇒ v0.5.2
[cd3eb016] ↑ HTTP v1.10.11 ⇒ v1.10.15
[34004b35] ↑ HypergeometricFunctions v0.3.25 ⇒ v0.3.27
[d1acc4aa] ↑ IntervalArithmetic v0.22.19 ⇒ v0.22.22
[41ab1584] ↑ InvertedIndices v1.3.0 ⇒ v1.3.1
[033835bb] ↑ JLD2 v0.5.9 ⇒ v0.5.11
[692b3bcd] ↑ JLLWrappers v1.6.1 ⇒ v1.7.0
[2ab3a3ac] ↑ LogExpFunctions v0.3.28 ⇒ v0.3.29
[6c6e2e6c] ↑ MIMEs v0.1.4 ⇒ v1.0.0
[1914dd2f] ↑ MacroTools v0.5.13 ⇒ v0.5.15
[ee78f7c6] ↑ Makie v0.21.16 ⇒ v0.22.1
[20f20a25] ↑ MakieCore v0.8.10 ⇒ v0.9.0
[77ba4419] ↑ NaNMath v1.0.2 ⇒ v1.1.1
[6fe1bfb0] ↑ OffsetArrays v1.14.1 ⇒ v1.15.0
[bac558e1] ↑ OrderedCollections v1.6.3 ⇒ v1.7.0
[90014a1f] ↑ PDMats v0.11.31 ⇒ v0.11.32
[7f904dfe] ↑ PlutoUI v0.7.60 ⇒ v0.7.61
[43287f4e] ↑ PtrArrays v1.2.1 ⇒ v1.3.0
[fdea26ae] ↑ SIMD v3.7.0 ⇒ v3.7.1
[91c51154] ↑ SentinelArrays v1.4.7 ⇒ v1.4.8
[65257c39] ↑ ShaderAbstractions v0.4.1 ⇒ v0.5.0
[276daf66] ↑ SpecialFunctions v2.4.0 ⇒ v2.5.0
[90137ffa] ↑ StaticArrays v1.9.8 ⇒ v1.9.11
[2913bbd2] ↑ StatsBase v0.34.3 ⇒ v0.34.4
[09ab397b] ↑ StructArrays v0.6.21 ⇒ v0.7.0
[731e570b] ↑ TiffImages v0.11.1 ⇒ v0.11.3
[410a4b4d] ↑ Tricks v0.1.9 ⇒ v0.1.10
[1986cc42] ↑ Unitful v1.21.0 ⇒ v1.22.0
[6e34b625] ↑ Bzip2_jll v1.0.8+2 ⇒ v1.0.8+4
[2e619515] ↑ Expat_jll v2.6.4+0 ⇒ v2.6.4+3
[f5851436] ↑ FFTW_jll v3.3.10+1 ⇒ v3.3.10+3
[a3f928ae] ↑ Fontconfig_jll v2.13.96+0 ⇒ v2.15.0+0
[d7e528f0] ↑ FreeType2_jll v2.13.3+0 ⇒ v2.13.3+1
[559328eb] ↑ FriBidi_jll v1.0.14+0 ⇒ v1.0.16+0
[59f7168a] ↑ Giflib_jll v5.2.2+0 ⇒ v5.2.3+0
[f8c6e375] ↑ Git_jll v2.46.2+0 ⇒ v2.47.1+0
[7746bdde] ↑ Glib_jll v2.82.2+0 ⇒ v2.82.4+0
[2e76f6c2] ↑ HarfBuzz_jll v8.3.1+0 ⇒ v8.5.0+0
[1d5cc7b8] ↑ IntelOpenMP_jll v2024.2.1+0 ⇒ v2025.0.4+0
[aacddb02] ↑ JpegTurbo_jll v3.0.4+0 ⇒ v3.1.1+0
[88015f11] ↑ LERC_jll v4.0.0+0 ⇒ v4.0.1+0
[dd4b983a] ↑ LZO_jll v2.10.2+1 ⇒ v2.10.3+0
⌅ [e9f186c6] ↑ Libffi_jll v3.2.2+1 ⇒ v3.2.2+2
[7e76a0d4] ↑ Libglvnd_jll v1.6.0+0 ⇒ v1.7.0+0
[7add5ba3] ↑ Libgpg_error_jll v1.50.0+0 ⇒ v1.51.1+0
[94ce4f54] ↑ Libiconv_jll v1.17.0+1 ⇒ v1.18.0+0
[4b2f31a3] ↑ Libmount_jll v2.40.2+0 ⇒ v2.40.3+0
[89763e89] ↑ Libtiff_jll v4.7.0+0 ⇒ v4.7.1+0
[38a345b3] ↑ Libuuid_jll v2.40.2+0 ⇒ v2.40.3+0
[856f044c] ↑ MKL_jll v2024.2.0+0 ⇒ v2025.0.1+1
[458c3c95] ↑ OpenSSL_jll v3.0.15+1 ⇒ v3.0.15+3
[efe28fd5] ↑ OpenSpecFun_jll v0.5.5+0 ⇒ v0.5.6+0
[36c8627f] ↑ Pango_jll v1.54.1+0 ⇒ v1.55.5+0
[aed1982a] ↑ XSLT_jll v1.1.41+0 ⇒ v1.1.42+0
[ffd25f8a] ↑ XZ_jll v5.6.3+0 ⇒ v5.6.4+1
[4f6342f7] ↑ Xorg_libX11_jll v1.8.6+0 ⇒ v1.8.6+3
[0c0b7dd1] ↑ Xorg_libXau_jll v1.0.11+1 ⇒ v1.0.12+0
[a3789734] ↑ Xorg_libXdmcp_jll v1.1.4+1 ⇒ v1.1.5+0
[1082639a] ↑ Xorg_libXext_jll v1.3.6+0 ⇒ v1.3.6+3
[ea2f1a96] ↑ Xorg_libXrender_jll v0.9.11+0 ⇒ v0.9.11+1
[14d82f49] ↑ Xorg_libpthread_stubs_jll v0.1.1+1 ⇒ v0.1.2+0
[c7cfdc94] ↑ Xorg_libxcb_jll v1.17.0+0 ⇒ v1.17.0+3
[c5fb5394] ↑ Xorg_xtrans_jll v1.5.0+1 ⇒ v1.5.1+0
[3161d3a3] ↑ Zstd_jll v1.5.6+1 ⇒ v1.5.7+0
[a4ae2306] ↑ libaom_jll v3.9.0+0 ⇒ v3.11.0+0
[b53b4c65] ↑ libpng_jll v1.6.44+0 ⇒ v1.6.45+1
[075b6546] ↑ libsixel_jll v1.10.3+1 ⇒ v1.10.5+0
[c5f90fcd] ↑ libwebp_jll v1.4.0+0 ⇒ v1.5.0+0
[1270edf5] ↑ x264_jll v10164.0.0+0 ⇒ v10164.0.1+0
Info Packages marked with ⌅ have new versions available but compatibility constraints restrict them from upgrading. To see why use `status --outdated -m`
Precompiling project...
769.7 ms ✓ ColorVectorSpace → SpecialFunctionsExt
3118.1 ms ✓ PlutoUI
3693.2 ms ✓ ClimateModels
191511.7 ms ✓ ClimateModels → ClimateModelsMakieExt
4 dependencies successfully precompiled in 196 seconds. 308 already precompiled.
3 dependencies precompiled but different versions are currently loaded. Restart julia to access the new versions. Otherwise, loading dependents of these packages may trigger further precompilation to work with the unexpected versions.
Info We haven't cleaned this depot up for a bit, running Pkg.gc()...
Active manifest files: 3 found
Active artifact files: 105 found
Active scratchspaces: 2 found
Deleted no artifacts, repos, packages or scratchspaces
Activating project at `~/work/ClimateModels.jl/ClimateModels.jl/docs`
Activating project at `/tmp/bf5c2795-d3ae-4c37-8d95-e2890d8145c3`
Precompiling CairoMakie...
973.1 ms ✓ Graphics
2387.9 ms ✓ QOI
2398.4 ms ✓ OpenEXR
4511.6 ms ✓ Sixel
5437.1 ms ✓ JpegTurbo
3165.6 ms ✓ ImageAxes
1930.1 ms ✓ Cairo
3483.3 ms ✓ WebP
1429.9 ms ✓ ImageMetadata
1564.2 ms ✓ Netpbm
60610.9 ms ✓ TiffImages
42268.0 ms ✓ CairoMakie
12 dependencies successfully precompiled in 104 seconds. 258 already precompiled.
Activating project at `~/work/ClimateModels.jl/ClimateModels.jl/docs`
Files and Cloud Support
Numerical model output often gets archived, distributed, and retrieved over the web. Some times, downloading data is most convenient. In other cases, it is preferable to compute in the cloud and just download final results.
ClimateModels.jl
has examples for most common file formats. These are handled via Downloads.jl, NetCDF.jl, DataFrames.jl, CSV.jl, and TOML.jl.
fil=joinpath(pathof(mc),"run02.csv")
CSV=ClimateModels.CSV
DataFrame=ClimateModels.DataFrame
CSV.File(fil) |> DataFrame
"200×2 DataFrame"
For more examples with NetCDF.jl and Zarr.jl, please look at IPCC notebook and CMIP6 notebok.