CbiomesProcessing.jl documentation
CbiomesProcessing.jl provides a simple, distributed workflow to e.g. post-process binary model output. Inter-operability with MeshArrays.jl and NCTiles.jl is an important prospect of this package.
Contents
Main Features
Currently the reference implementation consists of
TaskDriver()(insideutilities.jl) is the driver function- It calls e.g.
loop_task1.(i)
- It calls e.g.
CbiomesProcessing.loop_task1(indx::Int)(insidefileloops.jl) loadsMfromMTRX.jld- It calls
loop_task1(indx,M)
- It calls
CbiomesProcessing.loop_task1(indx,M)(insidefileloops.jl) loops over files and variables- It calls
MatrixInterp(in,MTRX,siz)and writes result to aFortranFile
- It calls
API / Functions further documents the CbiomesProcessing module functions that are included in:
examples.jlincludes simple examples withoutTaskDriver.fileloops.jlincludes task loops such asCbiomesProcessing.loop_task1()](@ref). A real-life example is provided in theTaskDriverdocumentation reported below.utilities.jlprovides low-level functions such asTaskDriver.src/CMAP2019/contains a workflow that was used to process model output for ingestion in CMAP
Index
CbiomesProcessing.StartWorkersCbiomesProcessing.TaskDriverCbiomesProcessing.cbioproc_example1CbiomesProcessing.cbioproc_example2CbiomesProcessing.loop_exampleACbiomesProcessing.loop_exampleBCbiomesProcessing.loop_task1CbiomesProcessing.loop_task1CbiomesProcessing.loop_task2CbiomesProcessing.loop_task3CbiomesProcessing.loop_task4
API / Functions
CbiomesProcessing.StartWorkers — MethodStartWorkers(nwrkrs::Int)Start workers if needed.
CbiomesProcessing.TaskDriver — MethodTaskDriver(indx,fn)Broacast / distribute task (fn; e.g. loop_task1) over indices (indx; e.g. file indices)
Examples:
using CbiomesProcessing, Distributed, SparseArrays
TaskDriver(1,CbiomesProcessing.loop_task1)
StartWorkers(4)
@everywhere using CbiomesProcessing, SparseArrays
TaskDriver(1:4,CbiomesProcessing.loop_task1)Visualize results:
using FortranFiles, Plots
k=1
recl=720*360*4
fil="diags_interp/ETAN/ETAN.0000000732.data"
f = FortranFile(fil,"r",access="direct",recl=recl,convert="big-endian")
tmp=read(f,rec=k,(Float32,(720,360))); close(f)
heatmap(tmp)CbiomesProcessing.cbioproc_example1 — Methodcbioproc_example1()A 3D example without land mask or MeshArrays.
CbiomesProcessing.cbioproc_example2 — Methodcbioproc_example2()A 2D example with MeshArrays + mask or loop.
CbiomesProcessing.loop_exampleA — Methodloop_exampleA(indx,SPM,siz)Applies MatrixInterp in a loop over a subset of model output files (filList[indx])
CbiomesProcessing.loop_exampleB — Methodloop_exampleB(indx,SPM,siz)Applies MatrixInterp in a loop over a subset of model output files (filList[indx])
CbiomesProcessing.loop_task1 — Methodloop_task1(indx,M)Loop over a subset of model output files (filList[indx]), apply MatrixInterp (M) as a postprocessing step, and write the result to file (one subfolder for each variable)
CbiomesProcessing.loop_task1 — Methodloop_task1(indx::Int)Interpolate all variables for one record
CbiomesProcessing.loop_task2 — Methodloop_task2(indx)Loop over a subset of model output files (filList[indx]), convert Rirr* to Satellite Chl as a postprocessing step, and write the result to file (one subfolder for each variable)
CbiomesProcessing.loop_task3 — Methodloop_task3(indx)Loop over a subset of model output files (filList[indx]), compute export as a postprocessing step, and write the result to file (one subfolder for each variable)
CbiomesProcessing.loop_task4 — Methodloop_task4(indx)Loop over model output files and time average them as a postprocessing step, and write the result to file (one subfolder for each variable)
/