Understanding intake-ESM keys and how to use them

Intake-ESM helps with aggregating your datasets using some keys. Here, we dig into what exactly these keys are, how they are constructed, and how you can change them. Understanding how this work will help you control how your datasets are merged together, and remove the mystery behind these strings of text.

Import packages and spin up a Dask cluster

We start first with importing intake and a Client from dask.distributed

import intake
from distributed import Client

client = Client()

Investigate a CMIP6 catalog

Let’s start with a sample CMIP6 catalog! This is a fairly large dataset.

url ="https://raw.githubusercontent.com/intake/intake-esm/main/tutorial-catalogs/GOOGLE-CMIP6.json"
catalog = intake.open_esm_datastore(url)
catalog.df.head()

	activity_id	institution_id	source_id	experiment_id	member_id	table_id	variable_id	grid_label	zstore	dcpp_init_year	version
0	CMIP	IPSL	IPSL-CM6A-LR	historical	r2i1p1f1	Amon	va	gr	gs://cmip6/CMIP6/CMIP/IPSL/IPSL-CM6A-LR/histor...	NaN	20180803
1	CMIP	IPSL	IPSL-CM6A-LR	historical	r2i1p1f1	Amon	ua	gr	gs://cmip6/CMIP6/CMIP/IPSL/IPSL-CM6A-LR/histor...	NaN	20180803
2	CMIP	IPSL	IPSL-CM6A-LR	historical	r8i1p1f1	Oyr	o2	gn	gs://cmip6/CMIP6/CMIP/IPSL/IPSL-CM6A-LR/histor...	NaN	20180803
3	CMIP	IPSL	IPSL-CM6A-LR	historical	r30i1p1f1	Amon	ua	gr	gs://cmip6/CMIP6/CMIP/IPSL/IPSL-CM6A-LR/histor...	NaN	20180803
4	CMIP	IPSL	IPSL-CM6A-LR	historical	r30i1p1f1	Amon	va	gr	gs://cmip6/CMIP6/CMIP/IPSL/IPSL-CM6A-LR/histor...	NaN	20180803

Typically, the next step would be to search and load your datasets using to_dataset_dict() or to_datatree()

catalog_subset = catalog.search(variable_id='ua')
dsets = catalog_subset.to_dataset_dict()
print(dsets)

--> The keys in the returned dictionary of datasets are constructed as follows:
	'activity_id.institution_id.source_id.experiment_id.table_id.grid_label'

100.00% [2/2 01:49<00:00]

/home/docs/checkouts/readthedocs.org/user_builds/intake-esm/checkouts/latest/intake_esm/source.py:109: UserWarning: The specified chunks separate the stored chunks along dimension "time" starting at index 1. This could degrade performance. Instead, consider rechunking after loading.
  ds = xr.open_dataset(url, **xarray_open_kwargs)
/home/docs/checkouts/readthedocs.org/user_builds/intake-esm/checkouts/latest/intake_esm/source.py:109: UserWarning: The specified chunks separate the stored chunks along dimension "plev" starting at index 1. This could degrade performance. Instead, consider rechunking after loading.
  ds = xr.open_dataset(url, **xarray_open_kwargs)

{'CMIP.CCCma.CanESM5.historical.Amon.gn': <xarray.Dataset> Size: 80GB
Dimensions:         (member_id: 65, dcpp_init_year: 1, time: 1980, plev: 19,
                     lat: 64, lon: 128, bnds: 2)
Coordinates:
  * member_id       (member_id) object 520B 'r10i1p1f1' ... 'r9i1p2f1'
  * dcpp_init_year  (dcpp_init_year) float64 8B nan
  * time            (time) object 16kB 1850-01-16 12:00:00 ... 2014-12-16 12:...
  * plev            (plev) float64 152B 1e+05 9.25e+04 8.5e+04 ... 500.0 100.0
  * lat             (lat) float64 512B -87.86 -85.1 -82.31 ... 82.31 85.1 87.86
  * lon             (lon) float64 1kB 0.0 2.812 5.625 ... 351.6 354.4 357.2
    lat_bnds        (lat, bnds) float64 1kB dask.array<chunksize=(64, 2), meta=np.ndarray>
    lon_bnds        (lon, bnds) float64 2kB dask.array<chunksize=(128, 2), meta=np.ndarray>
    time_bnds       (time, bnds) object 32kB dask.array<chunksize=(1980, 2), meta=np.ndarray>
Dimensions without coordinates: bnds
Data variables:
    ua              (member_id, dcpp_init_year, time, plev, lat, lon) float32 80GB dask.array<chunksize=(1, 1, 174, 19, 64, 128), meta=np.ndarray>
Attributes: (12/46)
    Conventions:                      CF-1.7 CMIP-6.2
    YMDH_branch_time_in_child:        1850:01:01:00
    activity_id:                      CMIP
    branch_method:                    Spin-up documentation
    branch_time_in_child:             0.0
    contact:                          ec.cccma.info-info.ccmac.ec@canada.ca
    ...                               ...
    intake_esm_attrs:table_id:        Amon
    intake_esm_attrs:variable_id:     ua
    intake_esm_attrs:grid_label:      gn
    intake_esm_attrs:version:         20190429
    intake_esm_attrs:_data_format_:   zarr
    intake_esm_dataset_key:           CMIP.CCCma.CanESM5.historical.Amon.gn, 'CMIP.IPSL.IPSL-CM6A-LR.historical.Amon.gr': <xarray.Dataset> Size: 99GB
Dimensions:         (member_id: 32, dcpp_init_year: 1, time: 1980, plev: 19,
                     lat: 143, lon: 144, axis_nbounds: 2)
Coordinates:
  * member_id       (member_id) object 256B 'r10i1p1f1' ... 'r9i1p1f1'
  * dcpp_init_year  (dcpp_init_year) float64 8B nan
  * time            (time) datetime64[ns] 16kB 1850-01-16T12:00:00 ... 2014-1...
  * plev            (plev) float32 76B 1e+05 9.25e+04 8.5e+04 ... 500.0 100.0
  * lat             (lat) float32 572B -90.0 -88.73 -87.46 ... 87.46 88.73 90.0
  * lon             (lon) float32 576B 0.0 2.5 5.0 7.5 ... 352.5 355.0 357.5
    time_bounds     (time, axis_nbounds) datetime64[ns] 32kB dask.array<chunksize=(1980, 2), meta=np.ndarray>
Dimensions without coordinates: axis_nbounds
Data variables:
    ua              (member_id, dcpp_init_year, time, plev, lat, lon) float32 99GB dask.array<chunksize=(1, 1, 1, 1, 143, 144), meta=np.ndarray>
Attributes: (12/48)
    Conventions:                      CF-1.7 CMIP-6.2
    EXPID:                            historical
    activity_id:                      CMIP
    branch_method:                    standard
    branch_time_in_child:             0.0
    contact:                          ipsl-cmip6@listes.ipsl.fr
    ...                               ...
    intake_esm_attrs:variable_id:     ua
    intake_esm_attrs:grid_label:      gr
    intake_esm_attrs:_data_format_:   zarr
    NCO:                              "4.6.0"
    variant_info:                     Restart from another point in piControl...
    intake_esm_dataset_key:           CMIP.IPSL.IPSL-CM6A-LR.historical.Amon.gr}

Investigating the keys

The keys for these datasets include some helpful information - but you might be wondering what this all means and where this text comes from…

print(list(dsets))

['CMIP.CCCma.CanESM5.historical.Amon.gn', 'CMIP.IPSL.IPSL-CM6A-LR.historical.Amon.gr']

When intake-esm aggregates these datasets, it uses some pre-determined metadata, defined in the catalog file. We can look at which fields are used for aggregation, or merging of the datasets, using the following

print(catalog.esmcat.aggregation_control.groupby_attrs)

['activity_id', 'institution_id', 'source_id', 'experiment_id', 'table_id', 'grid_label']

Let’s go back to our data catalog… and find these fields. You’ll notice they are all column labels! These are key components of the metadata.

catalog_subset.df

	activity_id	institution_id	source_id	experiment_id	member_id	table_id	variable_id	grid_label	zstore	dcpp_init_year	version
0	CMIP	IPSL	IPSL-CM6A-LR	historical	r2i1p1f1	Amon	ua	gr	gs://cmip6/CMIP6/CMIP/IPSL/IPSL-CM6A-LR/histor...	NaN	20180803
1	CMIP	IPSL	IPSL-CM6A-LR	historical	r30i1p1f1	Amon	ua	gr	gs://cmip6/CMIP6/CMIP/IPSL/IPSL-CM6A-LR/histor...	NaN	20180803
2	CMIP	IPSL	IPSL-CM6A-LR	historical	r8i1p1f1	Amon	ua	gr	gs://cmip6/CMIP6/CMIP/IPSL/IPSL-CM6A-LR/histor...	NaN	20180803
3	CMIP	IPSL	IPSL-CM6A-LR	historical	r29i1p1f1	Amon	ua	gr	gs://cmip6/CMIP6/CMIP/IPSL/IPSL-CM6A-LR/histor...	NaN	20180803
4	CMIP	IPSL	IPSL-CM6A-LR	historical	r3i1p1f1	Amon	ua	gr	gs://cmip6/CMIP6/CMIP/IPSL/IPSL-CM6A-LR/histor...	NaN	20180803
...	...	...	...	...	...	...	...	...	...	...	...
92	CMIP	CCCma	CanESM5	historical	r31i1p2f1	Amon	ua	gn	gs://cmip6/CMIP6/CMIP/CCCma/CanESM5/historical...	NaN	20190429
93	CMIP	CCCma	CanESM5	historical	r33i1p2f1	Amon	ua	gn	gs://cmip6/CMIP6/CMIP/CCCma/CanESM5/historical...	NaN	20190429
94	CMIP	CCCma	CanESM5	historical	r6i1p2f1	Amon	ua	gn	gs://cmip6/CMIP6/CMIP/CCCma/CanESM5/historical...	NaN	20190429
95	CMIP	CCCma	CanESM5	historical	r6i1p1f1	Amon	ua	gn	gs://cmip6/CMIP6/CMIP/CCCma/CanESM5/historical...	NaN	20190429
96	CMIP	IPSL	IPSL-CM6A-LR	historical	r32i1p1f1	Amon	ua	gr	gs://cmip6/CMIP6/CMIP/IPSL/IPSL-CM6A-LR/histor...	NaN	20190802

97 rows × 11 columns

Using keys_info()

These groupby attributes are columns in our catalog! This means that the datasets which will be aggregated using the hierarchy:

activity_id --> institution_id --> source_id --> experiment_id --> table_id --> grid_label

A more clear of taking a look at these aggregation variables using the .keys_info() method for the catalog:

catalog_subset.keys_info()

	activity_id	institution_id	source_id	experiment_id	table_id	grid_label
key
CMIP.CCCma.CanESM5.historical.Amon.gn	CMIP	CCCma	CanESM5	historical	Amon	gn
CMIP.IPSL.IPSL-CM6A-LR.historical.Amon.gr	CMIP	IPSL	IPSL-CM6A-LR	historical	Amon	gr

Change our groupby/aggregation controls

If we wanted to instead aggregate our datasets at the member_id level, we can change that using the following method:

original_groupby_attributes = catalog.esmcat.aggregation_control.groupby_attrs
new_groupby_attributes = original_groupby_attributes + ["member_id"]
print(new_groupby_attributes)

['activity_id', 'institution_id', 'source_id', 'experiment_id', 'table_id', 'grid_label', 'member_id']

Now that we have our new groupby attributes, we can assign these to our catalog subset.

catalog_subset.esmcat.aggregation_control.groupby_attrs = new_groupby_attributes

Let’s check our new keys! You’ll notice we now have 97 keys, aggregated on

activity_id --> institution_id --> source_id --> experiment_id --> table_id --> grid_label --> member_id

catalog_subset.keys_info()

	activity_id	institution_id	source_id	experiment_id	table_id	grid_label	member_id
key
CMIP.CCCma.CanESM5.historical.Amon.gn.r10i1p1f1	CMIP	CCCma	CanESM5	historical	Amon	gn	r10i1p1f1
CMIP.CCCma.CanESM5.historical.Amon.gn.r10i1p2f1	CMIP	CCCma	CanESM5	historical	Amon	gn	r10i1p2f1
CMIP.CCCma.CanESM5.historical.Amon.gn.r11i1p1f1	CMIP	CCCma	CanESM5	historical	Amon	gn	r11i1p1f1
CMIP.CCCma.CanESM5.historical.Amon.gn.r11i1p2f1	CMIP	CCCma	CanESM5	historical	Amon	gn	r11i1p2f1
CMIP.CCCma.CanESM5.historical.Amon.gn.r12i1p1f1	CMIP	CCCma	CanESM5	historical	Amon	gn	r12i1p1f1
...	...	...	...	...	...	...	...
CMIP.IPSL.IPSL-CM6A-LR.historical.Amon.gr.r5i1p1f1	CMIP	IPSL	IPSL-CM6A-LR	historical	Amon	gr	r5i1p1f1
CMIP.IPSL.IPSL-CM6A-LR.historical.Amon.gr.r6i1p1f1	CMIP	IPSL	IPSL-CM6A-LR	historical	Amon	gr	r6i1p1f1
CMIP.IPSL.IPSL-CM6A-LR.historical.Amon.gr.r7i1p1f1	CMIP	IPSL	IPSL-CM6A-LR	historical	Amon	gr	r7i1p1f1
CMIP.IPSL.IPSL-CM6A-LR.historical.Amon.gr.r8i1p1f1	CMIP	IPSL	IPSL-CM6A-LR	historical	Amon	gr	r8i1p1f1
CMIP.IPSL.IPSL-CM6A-LR.historical.Amon.gr.r9i1p1f1	CMIP	IPSL	IPSL-CM6A-LR	historical	Amon	gr	r9i1p1f1

97 rows × 7 columns

Load our datasets with the new keys

We can now load our new datasets to our dictionary of datasets using:

dsets = catalog_subset.to_dataset_dict()

--> The keys in the returned dictionary of datasets are constructed as follows:
	'activity_id.institution_id.source_id.experiment_id.table_id.grid_label.member_id'

100.00% [99/99 00:49<00:00]

/home/docs/checkouts/readthedocs.org/user_builds/intake-esm/checkouts/latest/intake_esm/source.py:109: UserWarning: The specified chunks separate the stored chunks along dimension "time" starting at index 1. This could degrade performance. Instead, consider rechunking after loading.
  ds = xr.open_dataset(url, **xarray_open_kwargs)
/home/docs/checkouts/readthedocs.org/user_builds/intake-esm/checkouts/latest/intake_esm/source.py:109: UserWarning: The specified chunks separate the stored chunks along dimension "plev" starting at index 1. This could degrade performance. Instead, consider rechunking after loading.
  ds = xr.open_dataset(url, **xarray_open_kwargs)

/home/docs/checkouts/readthedocs.org/user_builds/intake-esm/checkouts/latest/intake_esm/source.py:109: UserWarning: The specified chunks separate the stored chunks along dimension "time" starting at index 1. This could degrade performance. Instead, consider rechunking after loading.
  ds = xr.open_dataset(url, **xarray_open_kwargs)
/home/docs/checkouts/readthedocs.org/user_builds/intake-esm/checkouts/latest/intake_esm/source.py:109: UserWarning: The specified chunks separate the stored chunks along dimension "plev" starting at index 1. This could degrade performance. Instead, consider rechunking after loading.
  ds = xr.open_dataset(url, **xarray_open_kwargs)

And if we only wanted the first key, we could use the following to grab the first key in the list. Notice how we now have our member_id at the end!

first_key = catalog_subset.keys()[0]
first_key

'CMIP.CCCma.CanESM5.historical.Amon.gn.r10i1p1f1'

And the .to_dask() method to load our dataset into our notebook.

ds = catalog_subset[first_key].to_dask()
ds

<xarray.Dataset> Size: 1GB
Dimensions:         (member_id: 1, dcpp_init_year: 1, time: 1980, plev: 19,
                     lat: 64, lon: 128, bnds: 2)
Coordinates:
  * member_id       (member_id) object 8B 'r10i1p1f1'
  * dcpp_init_year  (dcpp_init_year) float64 8B nan
  * time            (time) object 16kB 1850-01-16 12:00:00 ... 2014-12-16 12:...
  * plev            (plev) float64 152B 1e+05 9.25e+04 8.5e+04 ... 500.0 100.0
  * lat             (lat) float64 512B -87.86 -85.1 -82.31 ... 82.31 85.1 87.86
  * lon             (lon) float64 1kB 0.0 2.812 5.625 ... 351.6 354.4 357.2
    lat_bnds        (lat, bnds) float64 1kB dask.array<chunksize=(64, 2), meta=np.ndarray>
    lon_bnds        (lon, bnds) float64 2kB dask.array<chunksize=(128, 2), meta=np.ndarray>
    time_bnds       (time, bnds) object 32kB dask.array<chunksize=(1980, 2), meta=np.ndarray>
Dimensions without coordinates: bnds
Data variables:
    ua              (member_id, dcpp_init_year, time, plev, lat, lon) float32 1GB dask.array<chunksize=(1, 1, 180, 19, 64, 128), meta=np.ndarray>
Attributes: (12/67)
    CCCma_model_hash:                 55f484f90aff0e32c5a8e92a42c6b9ae7ffe6224
    CCCma_parent_runid:               rc3.1-pictrl
    CCCma_pycmor_hash:                33c30511acc319a98240633965a04ca99c26427e
    CCCma_runid:                      rc3.1-his10
    Conventions:                      CF-1.7 CMIP-6.2
    YMDH_branch_time_in_child:        1850:01:01:00
    ...                               ...
    intake_esm_attrs:variable_id:     ua
    intake_esm_attrs:grid_label:      gn
    intake_esm_attrs:zstore:          gs://cmip6/CMIP6/CMIP/CCCma/CanESM5/his...
    intake_esm_attrs:version:         20190429
    intake_esm_attrs:_data_format_:   zarr
    intake_esm_dataset_key:           CMIP.CCCma.CanESM5.historical.Amon.gn.r...

xarray.Dataset

• Dimensions:
  • member_id: 1
  • dcpp_init_year: 1
  • time: 1980
  • plev: 19
  • lat: 64
  • lon: 128
  • bnds: 2
• Coordinates: (9)
  • member_id
    (member_id)
    object
    'r10i1p1f1'

    array(['r10i1p1f1'], dtype=object)

  • dcpp_init_year
    (dcpp_init_year)
    float64
    nan

    array([nan])

  • time
    (time)
    object
    1850-01-16 12:00:00 ... 2014-12-...

    axis :

    T

    bounds :

    time_bnds

    long_name :

    time

    standard_name :

    time

    array([cftime.DatetimeNoLeap(1850, 1, 16, 12, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1850, 2, 15, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1850, 3, 16, 12, 0, 0, 0, has_year_zero=True),
           ...,
           cftime.DatetimeNoLeap(2014, 10, 16, 12, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(2014, 11, 16, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(2014, 12, 16, 12, 0, 0, 0, has_year_zero=True)],
          shape=(1980,), dtype=object)

  • plev
    (plev)
    float64
    1e+05 9.25e+04 ... 500.0 100.0

    axis :

    Z

    long_name :

    pressure

    positive :

    down

    standard_name :

    air_pressure

    units :

    Pa

    array([100000.,  92500.,  85000.,  70000.,  60000.,  50000.,  40000.,  30000.,
            25000.,  20000.,  15000.,  10000.,   7000.,   5000.,   3000.,   2000.,
             1000.,    500.,    100.])

  • lat
    (lat)
    float64
    -87.86 -85.1 -82.31 ... 85.1 87.86

    axis :

    Y

    bounds :

    lat_bnds

    long_name :

    Latitude

    standard_name :

    latitude

    units :

    degrees_north

    array([-87.863799, -85.096527, -82.312913, -79.525607, -76.7369  , -73.947515,
           -71.157752, -68.367756, -65.577607, -62.787352, -59.99702 , -57.206632,
           -54.4162  , -51.625734, -48.835241, -46.044727, -43.254195, -40.463648,
           -37.67309 , -34.882521, -32.091944, -29.30136 , -26.510769, -23.720174,
           -20.929574, -18.138971, -15.348365, -12.557756,  -9.767146,  -6.976534,
            -4.185921,  -1.395307,   1.395307,   4.185921,   6.976534,   9.767146,
            12.557756,  15.348365,  18.138971,  20.929574,  23.720174,  26.510769,
            29.30136 ,  32.091944,  34.882521,  37.67309 ,  40.463648,  43.254195,
            46.044727,  48.835241,  51.625734,  54.4162  ,  57.206632,  59.99702 ,
            62.787352,  65.577607,  68.367756,  71.157752,  73.947515,  76.7369  ,
            79.525607,  82.312913,  85.096527,  87.863799])

  • lon
    (lon)
    float64
    0.0 2.812 5.625 ... 354.4 357.2

    axis :

    X

    bounds :

    lon_bnds

    long_name :

    Longitude

    standard_name :

    longitude

    units :

    degrees_east

    array([  0.    ,   2.8125,   5.625 ,   8.4375,  11.25  ,  14.0625,  16.875 ,
            19.6875,  22.5   ,  25.3125,  28.125 ,  30.9375,  33.75  ,  36.5625,
            39.375 ,  42.1875,  45.    ,  47.8125,  50.625 ,  53.4375,  56.25  ,
            59.0625,  61.875 ,  64.6875,  67.5   ,  70.3125,  73.125 ,  75.9375,
            78.75  ,  81.5625,  84.375 ,  87.1875,  90.    ,  92.8125,  95.625 ,
            98.4375, 101.25  , 104.0625, 106.875 , 109.6875, 112.5   , 115.3125,
           118.125 , 120.9375, 123.75  , 126.5625, 129.375 , 132.1875, 135.    ,
           137.8125, 140.625 , 143.4375, 146.25  , 149.0625, 151.875 , 154.6875,
           157.5   , 160.3125, 163.125 , 165.9375, 168.75  , 171.5625, 174.375 ,
           177.1875, 180.    , 182.8125, 185.625 , 188.4375, 191.25  , 194.0625,
           196.875 , 199.6875, 202.5   , 205.3125, 208.125 , 210.9375, 213.75  ,
           216.5625, 219.375 , 222.1875, 225.    , 227.8125, 230.625 , 233.4375,
           236.25  , 239.0625, 241.875 , 244.6875, 247.5   , 250.3125, 253.125 ,
           255.9375, 258.75  , 261.5625, 264.375 , 267.1875, 270.    , 272.8125,
           275.625 , 278.4375, 281.25  , 284.0625, 286.875 , 289.6875, 292.5   ,
           295.3125, 298.125 , 300.9375, 303.75  , 306.5625, 309.375 , 312.1875,
           315.    , 317.8125, 320.625 , 323.4375, 326.25  , 329.0625, 331.875 ,
           334.6875, 337.5   , 340.3125, 343.125 , 345.9375, 348.75  , 351.5625,
           354.375 , 357.1875])

  • lat_bnds
    (lat, bnds)
    float64
    dask.array<chunksize=(64, 2), meta=np.ndarray>

    Array Chunk Bytes 1.00 kiB 1.00 kiB Shape (64, 2) (64, 2) Dask graph 1 chunks in 2 graph layers Data type float64 numpy.ndarray

  • lon_bnds
    (lon, bnds)
    float64
    dask.array<chunksize=(128, 2), meta=np.ndarray>

    Array Chunk Bytes 2.00 kiB 2.00 kiB Shape (128, 2) (128, 2) Dask graph 1 chunks in 2 graph layers Data type float64 numpy.ndarray

  • time_bnds
    (time, bnds)
    object
    dask.array<chunksize=(1980, 2), meta=np.ndarray>

    Array Chunk Bytes 30.94 kiB 30.94 kiB Shape (1980, 2) (1980, 2) Dask graph 1 chunks in 2 graph layers Data type object numpy.ndarray

• Data variables: (1)
  • ua
    (member_id, dcpp_init_year, time, plev, lat, lon)
    float32
    dask.array<chunksize=(1, 1, 180, 19, 64, 128), meta=np.ndarray>

    cell_measures :

    area: areacella

    cell_methods :

    time: mean

    comment :

    Zonal wind (positive in a eastward direction).

    history :

    2019-04-30T17:40:15Z altered by CMOR: Reordered dimensions, original order: lat lon plev time. 2019-04-30T17:40:15Z altered by CMOR: replaced missing value flag (1e+38) with standard missing value (1e+20).

    long_name :

    Eastward Wind

    original_name :

    U

    standard_name :

    eastward_wind

    units :

    m s-1

    Array Chunk Bytes 1.15 GiB 106.88 MiB Shape (1, 1, 1980, 19, 64, 128) (1, 1, 180, 19, 64, 128) Dask graph 11 chunks in 3 graph layers Data type float32 numpy.ndarray

• Attributes: (67)

  CCCma_model_hash :

  55f484f90aff0e32c5a8e92a42c6b9ae7ffe6224

  CCCma_parent_runid :

  rc3.1-pictrl

  CCCma_pycmor_hash :

  33c30511acc319a98240633965a04ca99c26427e

  CCCma_runid :

  rc3.1-his10

  Conventions :

  CF-1.7 CMIP-6.2

  YMDH_branch_time_in_child :

  1850:01:01:00

  YMDH_branch_time_in_parent :

  5651:01:01:00

  activity_id :

  CMIP

  branch_method :

  Spin-up documentation

  branch_time_in_child :

  0.0

  branch_time_in_parent :

  1387365.0

  cmor_version :

  3.4.0

  contact :

  ec.cccma.info-info.ccmac.ec@canada.ca

  creation_date :

  2019-04-30T17:40:15Z

  data_specs_version :

  01.00.29

  experiment :

  all-forcing simulation of the recent past

  experiment_id :

  historical

  external_variables :

  areacella

  forcing_index :

  1

  frequency :

  mon

  further_info_url :

  https://furtherinfo.es-doc.org/CMIP6.CCCma.CanESM5.historical.none.r10i1p1f1

  grid :

  T63L49 native atmosphere, T63 Linear Gaussian Grid; 128 x 64 longitude/latitude; 49 levels; top level 1 hPa

  grid_label :

  gn

  history :

  2019-04-30T17:40:15Z ;rewrote data to be consistent with CMIP for variable ua found in table Amon.; Output from $runid

  initialization_index :

  1

  institution :

  Canadian Centre for Climate Modelling and Analysis, Environment and Climate Change Canada, Victoria, BC V8P 5C2, Canada

  institution_id :

  CCCma

  license :

  CMIP6 model data produced by The Government of Canada (Canadian Centre for Climate Modelling and Analysis, Environment and Climate Change Canada) is licensed under a Creative Commons Attribution ShareAlike 4.0 International License (https://creativecommons.org/licenses). Consult https://pcmdi.llnl.gov/CMIP6/TermsOfUse for terms of use governing CMIP6 output, including citation requirements and proper acknowledgment. Further information about this data, including some limitations, can be found via the further_info_url (recorded as a global attribute in this file) and at https:///pcmdi.llnl.gov/. The data producers and data providers make no warranty, either express or implied, including, but not limited to, warranties of merchantability and fitness for a particular purpose. All liabilities arising from the supply of the information (including any liability arising in negligence) are excluded to the fullest extent permitted by law.

  mip_era :

  CMIP6

  nominal_resolution :

  500 km

  parent_activity_id :

  CMIP

  parent_experiment_id :

  piControl

  parent_mip_era :

  CMIP6

  parent_source_id :

  CanESM5

  parent_time_units :

  days since 1850-01-01 0:0:0.0

  parent_variant_label :

  r1i1p1f1

  physics_index :

  1

  product :

  model-output

  realization_index :

  10

  realm :

  atmos

  references :

  Geophysical Model Development Special issue on CanESM5 (https://www.geosci-model-dev.net/special_issues.html)

  source :

  CanESM5 (2019): aerosol: interactive atmos: CanAM5 (T63L49 native atmosphere, T63 Linear Gaussian Grid; 128 x 64 longitude/latitude; 49 levels; top level 1 hPa) atmosChem: specified oxidants for aerosols land: CLASS3.6/CTEM1.2 landIce: specified ice sheets ocean: NEMO3.4.1 (ORCA1 tripolar grid, 1 deg with refinement to 1/3 deg within 20 degrees of the equator; 361 x 290 longitude/latitude; 45 vertical levels; top grid cell 0-6.19 m) ocnBgchem: Canadian Model of Ocean Carbon (CMOC); NPZD ecosystem with OMIP prescribed carbonate chemistry seaIce: LIM2

  source_id :

  CanESM5

  source_type :

  AOGCM

  sub_experiment :

  none

  sub_experiment_id :

  none

  table_id :

  Amon

  table_info :

  Creation Date:(20 February 2019) MD5:374fbe5a2bcca535c40f7f23da271e49

  title :

  CanESM5 output prepared for CMIP6

  tracking_id :

  hdl:21.14100/3aa8545a-6f58-4fde-8b2b-d3a00c1e3ea9

  variable_id :

  ua

  variant_label :

  r10i1p1f1

  version :

  v20190429

  status :

  2019-10-25;created;by nhn2@columbia.edu

  intake_esm_vars :

  ['ua']

  intake_esm_attrs:activity_id :

  CMIP

  intake_esm_attrs:institution_id :

  CCCma

  intake_esm_attrs:source_id :

  CanESM5

  intake_esm_attrs:experiment_id :

  historical

  intake_esm_attrs:member_id :

  r10i1p1f1

  intake_esm_attrs:table_id :

  Amon

  intake_esm_attrs:variable_id :

  ua

  intake_esm_attrs:grid_label :

  gn

  intake_esm_attrs:zstore :

  gs://cmip6/CMIP6/CMIP/CCCma/CanESM5/historical/r10i1p1f1/Amon/ua/gn/v20190429/

  intake_esm_attrs:version :

  20190429

  intake_esm_attrs:_data_format_ :

  zarr

  intake_esm_dataset_key :

  CMIP.CCCma.CanESM5.historical.Amon.gn.r10i1p1f1

Compare this dataset with the original catalog configuration

Compare this to our original catalog, which aggregated one level higher, placing all of the member_ids into the same dataset.

Note

Notice how our metadata now mentions there are 65 member_ids in this dataset, compared to 1 in the previous dataset

original_ds = catalog[catalog.keys()[0]].to_dask()
original_ds

/home/docs/checkouts/readthedocs.org/user_builds/intake-esm/checkouts/latest/intake_esm/source.py:308: FutureWarning: In a future version of xarray the default value for compat will change from compat='no_conflicts' to compat='override'. This is likely to lead to different results when combining overlapping variables with the same name. To opt in to new defaults and get rid of these warnings now use `set_options(use_new_combine_kwarg_defaults=True) or set compat explicitly.
  self._ds = xr.combine_by_coords(

/home/docs/checkouts/readthedocs.org/user_builds/intake-esm/checkouts/latest/intake_esm/source.py:308: FutureWarning: In a future version of xarray the default value for compat will change from compat='no_conflicts' to compat='override'. This is likely to lead to different results when combining overlapping variables with the same name. To opt in to new defaults and get rid of these warnings now use `set_options(use_new_combine_kwarg_defaults=True) or set compat explicitly.
  self._ds = xr.combine_by_coords(

/home/docs/checkouts/readthedocs.org/user_builds/intake-esm/checkouts/latest/intake_esm/source.py:308: FutureWarning: In a future version of xarray the default value for compat will change from compat='no_conflicts' to compat='override'. This is likely to lead to different results when combining overlapping variables with the same name. To opt in to new defaults and get rid of these warnings now use `set_options(use_new_combine_kwarg_defaults=True) or set compat explicitly.
  self._ds = xr.combine_by_coords(

<xarray.Dataset> Size: 160GB
Dimensions:         (member_id: 65, dcpp_init_year: 1, time: 1980, plev: 19,
                     lat: 64, lon: 128, bnds: 2)
Coordinates:
  * member_id       (member_id) object 520B 'r10i1p1f1' ... 'r9i1p2f1'
  * dcpp_init_year  (dcpp_init_year) float64 8B nan
  * time            (time) object 16kB 1850-01-16 12:00:00 ... 2014-12-16 12:...
  * plev            (plev) float64 152B 1e+05 9.25e+04 8.5e+04 ... 500.0 100.0
  * lat             (lat) float64 512B -87.86 -85.1 -82.31 ... 82.31 85.1 87.86
  * lon             (lon) float64 1kB 0.0 2.812 5.625 ... 351.6 354.4 357.2
    lat_bnds        (lat, bnds) float64 1kB dask.array<chunksize=(64, 2), meta=np.ndarray>
    lon_bnds        (lon, bnds) float64 2kB dask.array<chunksize=(128, 2), meta=np.ndarray>
    time_bnds       (time, bnds) object 32kB dask.array<chunksize=(1980, 2), meta=np.ndarray>
Dimensions without coordinates: bnds
Data variables:
    ua              (member_id, dcpp_init_year, time, plev, lat, lon) float32 80GB dask.array<chunksize=(1, 1, 174, 19, 64, 128), meta=np.ndarray>
    va              (member_id, dcpp_init_year, time, plev, lat, lon) float32 80GB dask.array<chunksize=(1, 1, 170, 19, 64, 128), meta=np.ndarray>
Attributes: (12/43)
    Conventions:                      CF-1.7 CMIP-6.2
    YMDH_branch_time_in_child:        1850:01:01:00
    activity_id:                      CMIP
    branch_method:                    Spin-up documentation
    branch_time_in_child:             0.0
    contact:                          ec.cccma.info-info.ccmac.ec@canada.ca
    ...                               ...
    intake_esm_attrs:experiment_id:   historical
    intake_esm_attrs:table_id:        Amon
    intake_esm_attrs:grid_label:      gn
    intake_esm_attrs:version:         20190429
    intake_esm_attrs:_data_format_:   zarr
    intake_esm_dataset_key:           CMIP.CCCma.CanESM5.historical.Amon.gn

xarray.Dataset

• Dimensions:
  • member_id: 65
  • dcpp_init_year: 1
  • time: 1980
  • plev: 19
  • lat: 64
  • lon: 128
  • bnds: 2
• Coordinates: (9)
  • member_id
    (member_id)
    object
    'r10i1p1f1' ... 'r9i1p2f1'

    array(['r10i1p1f1', 'r10i1p2f1', 'r11i1p1f1', 'r11i1p2f1', 'r12i1p1f1',
           'r12i1p2f1', 'r13i1p1f1', 'r13i1p2f1', 'r14i1p1f1', 'r14i1p2f1',
           'r15i1p1f1', 'r15i1p2f1', 'r16i1p1f1', 'r16i1p2f1', 'r17i1p1f1',
           'r17i1p2f1', 'r18i1p1f1', 'r18i1p2f1', 'r19i1p1f1', 'r19i1p2f1',
           'r1i1p1f1', 'r1i1p2f1', 'r20i1p1f1', 'r20i1p2f1', 'r21i1p1f1',
           'r21i1p2f1', 'r22i1p1f1', 'r22i1p2f1', 'r23i1p1f1', 'r23i1p2f1',
           'r24i1p1f1', 'r24i1p2f1', 'r25i1p1f1', 'r25i1p2f1', 'r26i1p2f1',
           'r27i1p2f1', 'r28i1p2f1', 'r29i1p2f1', 'r2i1p1f1', 'r2i1p2f1',
           'r30i1p2f1', 'r31i1p2f1', 'r32i1p2f1', 'r33i1p2f1', 'r34i1p2f1',
           'r35i1p2f1', 'r36i1p2f1', 'r37i1p2f1', 'r38i1p2f1', 'r39i1p2f1',
           'r3i1p1f1', 'r3i1p2f1', 'r40i1p2f1', 'r4i1p1f1', 'r4i1p2f1', 'r5i1p1f1',
           'r5i1p2f1', 'r6i1p1f1', 'r6i1p2f1', 'r7i1p1f1', 'r7i1p2f1', 'r8i1p1f1',
           'r8i1p2f1', 'r9i1p1f1', 'r9i1p2f1'], dtype=object)

  • dcpp_init_year
    (dcpp_init_year)
    float64
    nan

    array([nan])

  • time
    (time)
    object
    1850-01-16 12:00:00 ... 2014-12-...

    axis :

    T

    bounds :

    time_bnds

    long_name :

    time

    standard_name :

    time

    array([cftime.DatetimeNoLeap(1850, 1, 16, 12, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1850, 2, 15, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(1850, 3, 16, 12, 0, 0, 0, has_year_zero=True),
           ...,
           cftime.DatetimeNoLeap(2014, 10, 16, 12, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(2014, 11, 16, 0, 0, 0, 0, has_year_zero=True),
           cftime.DatetimeNoLeap(2014, 12, 16, 12, 0, 0, 0, has_year_zero=True)],
          shape=(1980,), dtype=object)

  • plev
    (plev)
    float64
    1e+05 9.25e+04 ... 500.0 100.0

    axis :

    Z

    long_name :

    pressure

    positive :

    down

    standard_name :

    air_pressure

    units :

    Pa

    array([100000.,  92500.,  85000.,  70000.,  60000.,  50000.,  40000.,  30000.,
            25000.,  20000.,  15000.,  10000.,   7000.,   5000.,   3000.,   2000.,
             1000.,    500.,    100.])

  • lat
    (lat)
    float64
    -87.86 -85.1 -82.31 ... 85.1 87.86

    axis :

    Y

    bounds :

    lat_bnds

    long_name :

    Latitude

    standard_name :

    latitude

    units :

    degrees_north

    array([-87.863799, -85.096527, -82.312913, -79.525607, -76.7369  , -73.947515,
           -71.157752, -68.367756, -65.577607, -62.787352, -59.99702 , -57.206632,
           -54.4162  , -51.625734, -48.835241, -46.044727, -43.254195, -40.463648,
           -37.67309 , -34.882521, -32.091944, -29.30136 , -26.510769, -23.720174,
           -20.929574, -18.138971, -15.348365, -12.557756,  -9.767146,  -6.976534,
            -4.185921,  -1.395307,   1.395307,   4.185921,   6.976534,   9.767146,
            12.557756,  15.348365,  18.138971,  20.929574,  23.720174,  26.510769,
            29.30136 ,  32.091944,  34.882521,  37.67309 ,  40.463648,  43.254195,
            46.044727,  48.835241,  51.625734,  54.4162  ,  57.206632,  59.99702 ,
            62.787352,  65.577607,  68.367756,  71.157752,  73.947515,  76.7369  ,
            79.525607,  82.312913,  85.096527,  87.863799])

  • lon
    (lon)
    float64
    0.0 2.812 5.625 ... 354.4 357.2

    axis :

    X

    bounds :

    lon_bnds

    long_name :

    Longitude

    standard_name :

    longitude

    units :

    degrees_east

    array([  0.    ,   2.8125,   5.625 ,   8.4375,  11.25  ,  14.0625,  16.875 ,
            19.6875,  22.5   ,  25.3125,  28.125 ,  30.9375,  33.75  ,  36.5625,
            39.375 ,  42.1875,  45.    ,  47.8125,  50.625 ,  53.4375,  56.25  ,
            59.0625,  61.875 ,  64.6875,  67.5   ,  70.3125,  73.125 ,  75.9375,
            78.75  ,  81.5625,  84.375 ,  87.1875,  90.    ,  92.8125,  95.625 ,
            98.4375, 101.25  , 104.0625, 106.875 , 109.6875, 112.5   , 115.3125,
           118.125 , 120.9375, 123.75  , 126.5625, 129.375 , 132.1875, 135.    ,
           137.8125, 140.625 , 143.4375, 146.25  , 149.0625, 151.875 , 154.6875,
           157.5   , 160.3125, 163.125 , 165.9375, 168.75  , 171.5625, 174.375 ,
           177.1875, 180.    , 182.8125, 185.625 , 188.4375, 191.25  , 194.0625,
           196.875 , 199.6875, 202.5   , 205.3125, 208.125 , 210.9375, 213.75  ,
           216.5625, 219.375 , 222.1875, 225.    , 227.8125, 230.625 , 233.4375,
           236.25  , 239.0625, 241.875 , 244.6875, 247.5   , 250.3125, 253.125 ,
           255.9375, 258.75  , 261.5625, 264.375 , 267.1875, 270.    , 272.8125,
           275.625 , 278.4375, 281.25  , 284.0625, 286.875 , 289.6875, 292.5   ,
           295.3125, 298.125 , 300.9375, 303.75  , 306.5625, 309.375 , 312.1875,
           315.    , 317.8125, 320.625 , 323.4375, 326.25  , 329.0625, 331.875 ,
           334.6875, 337.5   , 340.3125, 343.125 , 345.9375, 348.75  , 351.5625,
           354.375 , 357.1875])

  • lat_bnds
    (lat, bnds)
    float64
    dask.array<chunksize=(64, 2), meta=np.ndarray>

    Array Chunk Bytes 1.00 kiB 1.00 kiB Shape (64, 2) (64, 2) Dask graph 1 chunks in 321 graph layers Data type float64 numpy.ndarray

  • lon_bnds
    (lon, bnds)
    float64
    dask.array<chunksize=(128, 2), meta=np.ndarray>

    Array Chunk Bytes 2.00 kiB 2.00 kiB Shape (128, 2) (128, 2) Dask graph 1 chunks in 321 graph layers Data type float64 numpy.ndarray

  • time_bnds
    (time, bnds)
    object
    dask.array<chunksize=(1980, 2), meta=np.ndarray>

    Array Chunk Bytes 30.94 kiB 30.94 kiB Shape (1980, 2) (1980, 2) Dask graph 1 chunks in 321 graph layers Data type object numpy.ndarray

• Data variables: (2)
  • ua
    (member_id, dcpp_init_year, time, plev, lat, lon)
    float32
    dask.array<chunksize=(1, 1, 174, 19, 64, 128), meta=np.ndarray>

    cell_measures :

    area: areacella

    cell_methods :

    time: mean

    comment :

    Zonal wind (positive in a eastward direction).

    long_name :

    Eastward Wind

    original_name :

    U

    standard_name :

    eastward_wind

    units :

    m s-1

    Array Chunk Bytes 74.62 GiB 103.31 MiB Shape (65, 1, 1980, 19, 64, 128) (1, 1, 174, 19, 64, 128) Dask graph 1430 chunks in 261 graph layers Data type float32 numpy.ndarray

  • va
    (member_id, dcpp_init_year, time, plev, lat, lon)
    float32
    dask.array<chunksize=(1, 1, 170, 19, 64, 128), meta=np.ndarray>

    cell_measures :

    area: areacella

    cell_methods :

    time: mean

    comment :

    Meridional wind (positive in a northward direction).

    long_name :

    Northward Wind

    original_name :

    V

    standard_name :

    northward_wind

    units :

    m s-1

    Array Chunk Bytes 74.62 GiB 100.94 MiB Shape (65, 1, 1980, 19, 64, 128) (1, 1, 170, 19, 64, 128) Dask graph 2145 chunks in 261 graph layers Data type float32 numpy.ndarray

• Attributes: (43)

  Conventions :

  CF-1.7 CMIP-6.2

  YMDH_branch_time_in_child :

  1850:01:01:00

  activity_id :

  CMIP

  branch_method :

  Spin-up documentation

  branch_time_in_child :

  0.0

  contact :

  ec.cccma.info-info.ccmac.ec@canada.ca

  experiment :

  all-forcing simulation of the recent past

  experiment_id :

  historical

  external_variables :

  areacella

  forcing_index :

  1

  frequency :

  mon

  grid :

  T63L49 native atmosphere, T63 Linear Gaussian Grid; 128 x 64 longitude/latitude; 49 levels; top level 1 hPa

  grid_label :

  gn

  initialization_index :

  1

  institution :

  Canadian Centre for Climate Modelling and Analysis, Environment and Climate Change Canada, Victoria, BC V8P 5C2, Canada

  institution_id :

  CCCma

  license :

  CMIP6 model data produced by The Government of Canada (Canadian Centre for Climate Modelling and Analysis, Environment and Climate Change Canada) is licensed under a Creative Commons Attribution ShareAlike 4.0 International License (https://creativecommons.org/licenses). Consult https://pcmdi.llnl.gov/CMIP6/TermsOfUse for terms of use governing CMIP6 output, including citation requirements and proper acknowledgment. Further information about this data, including some limitations, can be found via the further_info_url (recorded as a global attribute in this file) and at https:///pcmdi.llnl.gov/. The data producers and data providers make no warranty, either express or implied, including, but not limited to, warranties of merchantability and fitness for a particular purpose. All liabilities arising from the supply of the information (including any liability arising in negligence) are excluded to the fullest extent permitted by law.

  mip_era :

  CMIP6

  nominal_resolution :

  500 km

  parent_activity_id :

  CMIP

  parent_experiment_id :

  piControl

  parent_mip_era :

  CMIP6

  parent_source_id :

  CanESM5

  parent_time_units :

  days since 1850-01-01 0:0:0.0

  product :

  model-output

  realm :

  atmos

  source :

  CanESM5 (2019): aerosol: interactive atmos: CanAM5 (T63L49 native atmosphere, T63 Linear Gaussian Grid; 128 x 64 longitude/latitude; 49 levels; top level 1 hPa) atmosChem: specified oxidants for aerosols land: CLASS3.6/CTEM1.2 landIce: specified ice sheets ocean: NEMO3.4.1 (ORCA1 tripolar grid, 1 deg with refinement to 1/3 deg within 20 degrees of the equator; 361 x 290 longitude/latitude; 45 vertical levels; top grid cell 0-6.19 m) ocnBgchem: Canadian Model of Ocean Carbon (CMOC); NPZD ecosystem with OMIP prescribed carbonate chemistry seaIce: LIM2

  source_id :

  CanESM5

  source_type :

  AOGCM

  sub_experiment :

  none

  sub_experiment_id :

  none

  table_id :

  Amon

  title :

  CanESM5 output prepared for CMIP6

  version :

  v20190429

  intake_esm_attrs:activity_id :

  CMIP

  intake_esm_attrs:institution_id :

  CCCma

  intake_esm_attrs:source_id :

  CanESM5

  intake_esm_attrs:experiment_id :

  historical

  intake_esm_attrs:table_id :

  Amon

  intake_esm_attrs:grid_label :

  gn

  intake_esm_attrs:version :

  20190429

  intake_esm_attrs:_data_format_ :

  zarr

  intake_esm_dataset_key :

  CMIP.CCCma.CanESM5.historical.Amon.gn

Conclusion

These intake-esm keys can be a bit abstract when first accessing your data, but understanding them is essential to understand how intake-esm aggregates your data, and how you can change these aggregation controls for your desired analysis! We hope this helped demystify these keys.

Show code cell source

Hide code cell source

import intake_esm  # just to display version information
intake_esm.show_versions()

Show code cell output

Hide code cell output

INSTALLED VERSIONS
------------------

cftime: 1.6.5
dask: 2026.3.0
fastprogress: 1.1.3
fsspec: 2026.2.0
gcsfs: 2026.2.0
intake: 2.0.9
intake_esm: 2025.12.12.post7+g414c4cfc1.d20260406
netCDF4: 1.7.4
pandas: 3.0.2
requests: 2.33.1
s3fs: 2026.2.0
xarray: 2026.2.0
zarr: 3.1.5