How do I do it?

The purpose of this section is to do a collection of small convinient pieces of code on how to do simple things.

Extract the axes names from a Cube

using YAXArrays
using DimensionalData

julia> c = YAXArray(rand(10, 10, 5))

╭─────────────────────────────╮
│ 10×10×5 YAXArray{Float64,3} │
├─────────────────────────────┴────────────────────────────────── dims ┐
  ↓ Dim_1 Sampled{Int64} Base.OneTo(10) ForwardOrdered Regular Points,
  → Dim_2 Sampled{Int64} Base.OneTo(10) ForwardOrdered Regular Points,
  ↗ Dim_3 Sampled{Int64} Base.OneTo(5) ForwardOrdered Regular Points
├──────────────────────────────────────────────────────────── metadata ┤
  Dict{String, Any}()
├─────────────────────────────────────────────────────────── file size ┤
  file size: 3.91 KB
└──────────────────────────────────────────────────────────────────────┘

julia> caxes(c) # former way of doing it

↓ Dim_1 Sampled{Int64} Base.OneTo(10) ForwardOrdered Regular Points,
→ Dim_2 Sampled{Int64} Base.OneTo(10) ForwardOrdered Regular Points,
↗ Dim_3 Sampled{Int64} Base.OneTo(5) ForwardOrdered Regular Points

WARNING

To get the axes of a YAXArray use the dims function instead of the caxes function

julia> dims(c)

↓ Dim_1 Sampled{Int64} Base.OneTo(10) ForwardOrdered Regular Points,
→ Dim_2 Sampled{Int64} Base.OneTo(10) ForwardOrdered Regular Points,
↗ Dim_3 Sampled{Int64} Base.OneTo(5) ForwardOrdered Regular Points

INFO

Also, use DD.rebuild(ax, values) instead of axcopy(ax, values) to copy an axes with the same name but different values.

Obtain values from axes and data from the cube

There are two options to collect values from axes. In this examples the axis ranges from 1 to 10.

These two examples bring the same result

collect(getAxis("Dim_1", c).val)
collect(c.axes[1].val)

10-element Vector{Int64}:
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10

to collect data from a cube works exactly the same as doing it from an array

julia> c[:, :, 1]

╭───────────────────────────╮
│ 10×10 YAXArray{Float64,2} │
├───────────────────────────┴────────────────────────── dims ┐
  ↓ Dim_1 Sampled{Int64} 1:10 ForwardOrdered Regular Points,
  → Dim_2 Sampled{Int64} 1:10 ForwardOrdered Regular Points
├────────────────────────────────────────────────── metadata ┤
  Dict{String, Any}()
├───────────────────────────────────────────────── file size ┤
  file size: 800.0 bytes
└────────────────────────────────────────────────────────────┘

How do I concatenate cubes

It is possible to concatenate several cubes that shared the same dimensions using the [concatenatecubes]@ref function.

Let's create two dummy cubes

using YAXArrays
axlist = (
    Dim{:time}(range(1, 20, length=20)),
    Dim{:lon}(range(1, 10, length=10)),
    Dim{:lat}(range(1, 5, length=15))
    )

data1 = rand(20, 10, 15)
ds1 = YAXArray(axlist, data1)

data2 = rand(20, 10, 15)
ds2 = YAXArray(axlist, data2)

Now we can concatenate ds1 and ds2:

julia> dsfinal = concatenatecubes([ds1, ds2], Dim{:Variables}(["var1", "var2"]))

╭────────────────────────────────╮
│ 20×10×15×2 YAXArray{Float64,4} │
├────────────────────────────────┴─────────────────────────────────────── dims ┐
  ↓ time      Sampled{Float64} 1.0:1.0:20.0 ForwardOrdered Regular Points,
  → lon       Sampled{Float64} 1.0:1.0:10.0 ForwardOrdered Regular Points,
  ↗ lat       Sampled{Float64} 1.0:0.2857142857142857:5.0 ForwardOrdered Regular Points,
  ⬔ Variables Categorical{String} ["var1", "var2"] ForwardOrdered
├──────────────────────────────────────────────────────────────────── metadata ┤
  Dict{String, Any}()
├─────────────────────────────────────────────────────────────────── file size ┤
  file size: 46.88 KB
└──────────────────────────────────────────────────────────────────────────────┘

How do I subset a Cube?

Let's start by creating a dummy cube. Define the time span of the cube

using Dates
t = Date("2020-01-01"):Month(1):Date("2022-12-31")

Date("2020-01-01"):Dates.Month(1):Date("2022-12-01")

create cube axes

axes = (Dim{:Lon}(1:10), Dim{:Lat}(1:10), Dim{:Time}(t))

↓ Lon  1:10,
→ Lat  1:10,
↗ Time Date("2020-01-01"):Dates.Month(1):Date("2022-12-01")

assign values to a cube

julia> c = YAXArray(axes, reshape(1:3600, (10, 10, 36)))

╭────────────────────────────╮
│ 10×10×36 YAXArray{Int64,3} │
├────────────────────────────┴─────────────────────────────────────────── dims ┐
  ↓ Lon  Sampled{Int64} 1:10 ForwardOrdered Regular Points,
  → Lat  Sampled{Int64} 1:10 ForwardOrdered Regular Points,
  ↗ Time Sampled{Date} Date("2020-01-01"):Dates.Month(1):Date("2022-12-01") ForwardOrdered Regular Points
├──────────────────────────────────────────────────────────────────── metadata ┤
  Dict{String, Any}()
├─────────────────────────────────────────────────────────────────── file size ┤
  file size: 28.12 KB
└──────────────────────────────────────────────────────────────────────────────┘

Now we subset the cube by any dimension.

Subset cube by years

julia> ctime = c[Time=Between(Date(2021,1,1), Date(2021,12,31))]

╭────────────────────────────╮
│ 10×10×12 YAXArray{Int64,3} │
├────────────────────────────┴─────────────────────────────────────────── dims ┐
  ↓ Lon  Sampled{Int64} 1:10 ForwardOrdered Regular Points,
  → Lat  Sampled{Int64} 1:10 ForwardOrdered Regular Points,
  ↗ Time Sampled{Date} Date("2021-01-01"):Dates.Month(1):Date("2021-12-01") ForwardOrdered Regular Points
├──────────────────────────────────────────────────────────────────── metadata ┤
  Dict{String, Any}()
├─────────────────────────────────────────────────────────────────── file size ┤
  file size: 9.38 KB
└──────────────────────────────────────────────────────────────────────────────┘

Subset cube by a specific date and date range

julia> ctime2 = c[Time=At(Date("2021-05-01"))]

╭─────────────────────────╮
│ 10×10 YAXArray{Int64,2} │
├─────────────────────────┴────────────────────────── dims ┐
  ↓ Lon Sampled{Int64} 1:10 ForwardOrdered Regular Points,
  → Lat Sampled{Int64} 1:10 ForwardOrdered Regular Points
├──────────────────────────────────────────────── metadata ┤
  Dict{String, Any}()
├─────────────────────────────────────────────── file size ┤
  file size: 800.0 bytes
└──────────────────────────────────────────────────────────┘

julia> ctime3 = c[Time=Date("2021-05-01") .. Date("2021-12-01")]

╭───────────────────────────╮
│ 10×10×8 YAXArray{Int64,3} │
├───────────────────────────┴──────────────────────────────────────────── dims ┐
  ↓ Lon  Sampled{Int64} 1:10 ForwardOrdered Regular Points,
  → Lat  Sampled{Int64} 1:10 ForwardOrdered Regular Points,
  ↗ Time Sampled{Date} Date("2021-05-01"):Dates.Month(1):Date("2021-12-01") ForwardOrdered Regular Points
├──────────────────────────────────────────────────────────────────── metadata ┤
  Dict{String, Any}()
├─────────────────────────────────────────────────────────────────── file size ┤
  file size: 6.25 KB
└──────────────────────────────────────────────────────────────────────────────┘

Subset cube by longitude and latitude

julia> clonlat = c[Lon=1 .. 5, Lat=5 .. 10] # check even numbers range, it is ommiting them

╭──────────────────────────╮
│ 5×6×36 YAXArray{Int64,3} │
├──────────────────────────┴───────────────────────────────────────────── dims ┐
  ↓ Lon  Sampled{Int64} 1:5 ForwardOrdered Regular Points,
  → Lat  Sampled{Int64} 5:10 ForwardOrdered Regular Points,
  ↗ Time Sampled{Date} Date("2020-01-01"):Dates.Month(1):Date("2022-12-01") ForwardOrdered Regular Points
├──────────────────────────────────────────────────────────────────── metadata ┤
  Dict{String, Any}()
├─────────────────────────────────────────────────────────────────── file size ┤
  file size: 8.44 KB
└──────────────────────────────────────────────────────────────────────────────┘

How do I apply map algebra?

Our next step is map algebra computations. This can be done effectively using the 'map' function. For example:

Multiplying cubes with only spatio-temporal dimensions

julia> map((x, y) -> x * y, ds1, ds2)

╭──────────────────────────────╮
│ 20×10×15 YAXArray{Float64,3} │
├──────────────────────────────┴───────────────────────────────────────── dims ┐
  ↓ time Sampled{Float64} 1.0:1.0:20.0 ForwardOrdered Regular Points,
  → lon  Sampled{Float64} 1.0:1.0:10.0 ForwardOrdered Regular Points,
  ↗ lat  Sampled{Float64} 1.0:0.2857142857142857:5.0 ForwardOrdered Regular Points
├──────────────────────────────────────────────────────────────────── metadata ┤
  Dict{String, Any}()
├─────────────────────────────────────────────────────────────────── file size ┤
  file size: 23.44 KB
└──────────────────────────────────────────────────────────────────────────────┘

Cubes with more than 3 dimensions

julia> map((x, y) -> x * y, dsfinal[Variables=At("var1")], dsfinal[Variables=At("var2")])

╭──────────────────────────────╮
│ 20×10×15 YAXArray{Float64,3} │
├──────────────────────────────┴───────────────────────────────────────── dims ┐
  ↓ time Sampled{Float64} 1.0:1.0:20.0 ForwardOrdered Regular Points,
  → lon  Sampled{Float64} 1.0:1.0:10.0 ForwardOrdered Regular Points,
  ↗ lat  Sampled{Float64} 1.0:0.2857142857142857:5.0 ForwardOrdered Regular Points
├──────────────────────────────────────────────────────────────────── metadata ┤
  Dict{String, Any}()
├─────────────────────────────────────────────────────────────────── file size ┤
  file size: 23.44 KB
└──────────────────────────────────────────────────────────────────────────────┘

To add some complexity, we will multiply each value for π and then divided for the sum of each time step. We will use the ds1 cube for this purpose.

julia> mapslices(ds1, dims=("Lon", "Lat")) do xin
           (xin * π) ./ maximum(skipmissing(xin))
       end

"Running nonthreaded" = "Running nonthreaded"
╭──────────────────────────────────────────────╮
│ 10×15×20 YAXArray{Union{Missing, Float64},3} │
├──────────────────────────────────────────────┴───────────────────────── dims ┐
  ↓ lon  Sampled{Float64} 1.0:1.0:10.0 ForwardOrdered Regular Points,
  → lat  Sampled{Float64} 1.0:0.2857142857142857:5.0 ForwardOrdered Regular Points,
  ↗ time Sampled{Float64} 1.0:1.0:20.0 ForwardOrdered Regular Points
├──────────────────────────────────────────────────────────────────── metadata ┤
  Dict{String, Any}()
├─────────────────────────────────────────────────────────────────── file size ┤
  file size: 23.44 KB
└──────────────────────────────────────────────────────────────────────────────┘

How do I use the CubeTable function?

The function "CubeTable" creates an iterable table and the result is a DataCube. It is therefore very handy for grouping data and computing statistics by class. It uses OnlineStats.jl to calculate statistics, and weighted statistics can be calculated as well.

Here we will use the ds1 Cube defined previously and we create a mask for data classification.

Cube containing a mask with classes 1, 2 and 3.

julia> classes = YAXArray((getAxis("lon", dsfinal), getAxis("lat", dsfinal)), rand(1:3, 10, 15))

╭─────────────────────────╮
│ 10×15 YAXArray{Int64,2} │
├─────────────────────────┴────────────────────────────────────────────── dims ┐
  ↓ lon Sampled{Float64} 1.0:1.0:10.0 ForwardOrdered Regular Points,
  → lat Sampled{Float64} 1.0:0.2857142857142857:5.0 ForwardOrdered Regular Points
├──────────────────────────────────────────────────────────────────── metadata ┤
  Dict{String, Any}()
├─────────────────────────────────────────────────────────────────── file size ┤
  file size: 1.17 KB
└──────────────────────────────────────────────────────────────────────────────┘

using GLMakie
GLMakie.activate!()
# This is how our classification map looks like
fig, ax, obj = heatmap(classes;
    colormap=Makie.Categorical(cgrad([:grey15, :orangered, :snow3])))
cbar = Colorbar(fig[1,2], obj)
fig

Now we define the input cubes that will be considered for the iterable table

t = CubeTable(values=ds1, classes=classes)

Datacube iterator with 1 subtables with fields: (:values, :classes, :time, :lon, :lat)

using DataFrames
using OnlineStats
## visualization of the CubeTable
c_tbl = DataFrame(t[1])
first(c_tbl, 5)

In this line we calculate the Mean for each class

julia> fitcube = cubefittable(t, Mean, :values, by=(:classes))

╭───────────────────────────────────────────────╮
│ 3-element YAXArray{Union{Missing, Float64},1} │
├───────────────────────────────────────────────┴────────────── dims ┐
  ↓ classes Sampled{Int64} [1, 2, 3] ForwardOrdered Irregular Points
├────────────────────────────────────────────────────────── metadata ┤
  Dict{String, Any}()
├───────────────────────────────────────────────────────── file size ┤
  file size: 24.0 bytes
└────────────────────────────────────────────────────────────────────┘

We can also use more than one criteria for grouping the values. In the next example, the mean is calculated for each class and timestep.

julia> fitcube = cubefittable(t, Mean, :values, by=(:classes, :time))

╭──────────────────────────────────────────╮
│ 3×20 YAXArray{Union{Missing, Float64},2} │
├──────────────────────────────────────────┴────────────────────── dims ┐
  ↓ classes Sampled{Int64} [1, 2, 3] ForwardOrdered Irregular Points,
  → time    Sampled{Float64} 1.0:1.0:20.0 ForwardOrdered Regular Points
├───────────────────────────────────────────────────────────── metadata ┤
  Dict{String, Any}()
├──────────────────────────────────────────────────────────── file size ┤
  file size: 480.0 bytes
└───────────────────────────────────────────────────────────────────────┘

convertions types DimArray & YAXArray

using YAXArrays, YAXArrayBase
using DimensionalData

DimArray to YAXArray

julia> dim_arr = rand(X(1:5), Y(10.0:15.0), metadata = Dict{String, Any}())

╭─────────────────────────╮
│ 5×6 DimArray{Float64,2} │
├─────────────────────────┴────────────────────────────────── dims ┐
  ↓ X Sampled{Int64} 1:5 ForwardOrdered Regular Points,
  → Y Sampled{Float64} 10.0:1.0:15.0 ForwardOrdered Regular Points
├──────────────────────────────────────────────────────── metadata ┤
  Dict{String, Any}()
└──────────────────────────────────────────────────────────────────┘
 ↓ →  10.0       11.0       12.0        13.0        14.0       15.0
 1     0.103148   0.144857   0.875862    0.937395    0.984028   0.104553
 2     0.915685   0.177268   0.0361709   0.495258    0.614792   0.11859
 3     0.747971   0.776771   0.861844    0.40649     0.464619   0.561726
 4     0.252875   0.319771   0.117366    0.640748    0.757698   0.938727
 5     0.317355   0.540248   0.857564    0.0380851   0.452824   0.943581

metadata

Note the metadata argument. Needed by yaxconvert.

julia> yax_arr = yaxconvert(YAXArray, dim_arr)

╭─────────────────────────╮
│ 5×6 YAXArray{Float64,2} │
├─────────────────────────┴────────────────────────────────── dims ┐
  ↓ X Sampled{Int64} 1:5 ForwardOrdered Regular Points,
  → Y Sampled{Float64} 10.0:1.0:15.0 ForwardOrdered Regular Points
├──────────────────────────────────────────────────────── metadata ┤
  Dict{String, Any}()
├─────────────────────────────────────────────────────── file size ┤
  file size: 240.0 bytes
└──────────────────────────────────────────────────────────────────┘

And saving it:

using Zarr, NetCDF
savecube(yax_arr, "yax_arr.nc", driver=:netcdf, overwrite=true);

or as a zarr file

savecube(yax_arr, "yax_arr.zarr", driver=:zarr);

And going back to the DimArray type is also possible.

YAXArray into a DimArray

julia> dim_arr = yaxconvert(DimArray, yax_arr)

╭─────────────────────────╮
│ 5×6 DimArray{Float64,2} │
├─────────────────────────┴────────────────────────────────── dims ┐
  ↓ X Sampled{Int64} 1:5 ForwardOrdered Regular Points,
  → Y Sampled{Float64} 10.0:1.0:15.0 ForwardOrdered Regular Points
├──────────────────────────────────────────────────────── metadata ┤
  Dict{String, Any}()
└──────────────────────────────────────────────────────────────────┘
 ↓ →  10.0       11.0       12.0        13.0        14.0       15.0
 1     0.103148   0.144857   0.875862    0.937395    0.984028   0.104553
 2     0.915685   0.177268   0.0361709   0.495258    0.614792   0.11859
 3     0.747971   0.776771   0.861844    0.40649     0.464619   0.561726
 4     0.252875   0.319771   0.117366    0.640748    0.757698   0.938727
 5     0.317355   0.540248   0.857564    0.0380851   0.452824   0.943581

at the moment there is no support to save a DimArray directly into disk as a NetCDF or a Zarr file.

How do I assing variable names to YAXArrays in a Dataset

One variable name

julia> ds = YAXArrays.Dataset(; (:a => YAXArray(rand(10)),)...)

YAXArray Dataset
Shared Axes:
↓ Dim_1 Sampled{Int64} Base.OneTo(10) ForwardOrdered Regular Points
Variables:
a,

Multiple variable names

keylist = (:a, :b, :c)
varlist = (YAXArray(rand(10)), YAXArray(rand(10,5)), YAXArray(rand(2,5)))

julia> ds = YAXArrays.Dataset(; (keylist .=> varlist)...)

YAXArray Dataset
Shared Axes:
()
Variables:

a
  ↓ Dim_1 Sampled{Int64} Base.OneTo(10) ForwardOrdered Regular Points
b
  ↓ Dim_1 Sampled{Int64} Base.OneTo(10) ForwardOrdered Regular Points,
  → Dim_2 Sampled{Int64} Base.OneTo(5) ForwardOrdered Regular Points
c
  ↓ Dim_1 Sampled{Int64} Base.OneTo(2) ForwardOrdered Regular Points,
  → Dim_2 Sampled{Int64} Base.OneTo(5) ForwardOrdered Regular Points