Modify a catalog

import intake

The in-memory representation of an Earth System Model (ESM) catalog is a Pandas DataFrame, and is accessible via the .df property:

url ="https://raw.githubusercontent.com/intake/intake-esm/main/tutorial-catalogs/GOOGLE-CMIP6.json"
cat = intake.open_esm_datastore(url)
cat.df.head()

	activity_id	institution_id	source_id	experiment_id	member_id	table_id	variable_id	grid_label	zstore	dcpp_init_year	version
0	CMIP	IPSL	IPSL-CM6A-LR	historical	r2i1p1f1	Amon	va	gr	gs://cmip6/CMIP6/CMIP/IPSL/IPSL-CM6A-LR/histor...	NaN	20180803
1	CMIP	IPSL	IPSL-CM6A-LR	historical	r2i1p1f1	Amon	ua	gr	gs://cmip6/CMIP6/CMIP/IPSL/IPSL-CM6A-LR/histor...	NaN	20180803
2	CMIP	IPSL	IPSL-CM6A-LR	historical	r8i1p1f1	Oyr	o2	gn	gs://cmip6/CMIP6/CMIP/IPSL/IPSL-CM6A-LR/histor...	NaN	20180803
3	CMIP	IPSL	IPSL-CM6A-LR	historical	r30i1p1f1	Amon	ua	gr	gs://cmip6/CMIP6/CMIP/IPSL/IPSL-CM6A-LR/histor...	NaN	20180803
4	CMIP	IPSL	IPSL-CM6A-LR	historical	r30i1p1f1	Amon	va	gr	gs://cmip6/CMIP6/CMIP/IPSL/IPSL-CM6A-LR/histor...	NaN	20180803

In this notebook we will go through some examples showing how to modify this dataframe and some of its behavior during data loading steps.

Note

Pandas is a powerful tool for data manipulation. If you are not familiar with it, we recommend you to read the Pandas documentation.

Use case 1: complex search queries

Let’s say we are interested in datasets with the following attributes:

• experiment_id=["historical"]

• table_id="Amon"

• variable_id="ua"

In addition to these attributes, we are interested in the first ensemble member (member_id) of each model (source_id) only.

This can be achieved in two steps:

Step 1: run a query against the catalog

We can run a query against the catalog:

cat_subset = cat.search(
    experiment_id=["historical"],
    table_id="Amon",
    variable_id="ua",
)
cat_subset

GOOGLE-CMIP6 catalog with 2 dataset(s) from 97 asset(s):

	unique
activity_id	1
institution_id	2
source_id	2
experiment_id	1
member_id	72
table_id	1
variable_id	1
grid_label	2
zstore	97
dcpp_init_year	0
version	3
derived_variable_id	0

Step 2: select the first member_id for each source_id

The subsetted catalog contains source_id with the following number of member_id per source_id:

cat_subset.df.groupby("source_id")["member_id"].nunique()

source_id
CanESM5         65
IPSL-CM6A-LR    32
Name: member_id, dtype: int64

To get the first member_id for each source_id, we group the dataframe by source_id and use the .first() method to retrieve the first member_id:

grouped = cat_subset.df.groupby(["source_id"])
df = grouped.first().reset_index()

# Confirm that we have one ensemble member per source_id

df.groupby("source_id")["member_id"].nunique()

source_id
CanESM5         1
IPSL-CM6A-LR    1
Name: member_id, dtype: int64

df

	source_id	activity_id	institution_id	experiment_id	member_id	table_id	variable_id	grid_label	zstore	dcpp_init_year	version
0	CanESM5	CMIP	CCCma	historical	r11i1p1f1	Amon	ua	gn	gs://cmip6/CMIP6/CMIP/CCCma/CanESM5/historical...	NaN	20190429
1	IPSL-CM6A-LR	CMIP	IPSL	historical	r2i1p1f1	Amon	ua	gr	gs://cmip6/CMIP6/CMIP/IPSL/IPSL-CM6A-LR/histor...	NaN	20180803

Step 3: attach the new dataframe to our catalog object

cat_subset.esmcat._df = df
cat_subset

GOOGLE-CMIP6 catalog with 2 dataset(s) from 2 asset(s):

	unique
source_id	2
activity_id	1
institution_id	2
experiment_id	1
member_id	2
table_id	1
variable_id	1
grid_label	2
zstore	2
dcpp_init_year	0
version	2
derived_variable_id	0

Let’s load the subsetted catalog into a dictionary of datasets:

dsets = cat_subset.to_dataset_dict()
[key for key in dsets]

--> The keys in the returned dictionary of datasets are constructed as follows:
	'activity_id.institution_id.source_id.experiment_id.table_id.grid_label'

100.00% [2/2 00:16<00:00]

['CMIP.IPSL.IPSL-CM6A-LR.historical.Amon.gr',
 'CMIP.CCCma.CanESM5.historical.Amon.gn']

dsets["CMIP.IPSL.IPSL-CM6A-LR.historical.Amon.gr"]

<xarray.Dataset> Size: 3GB
Dimensions:         (lat: 143, lon: 144, plev: 19, time: 1980, axis_nbounds: 2,
                     member_id: 1, dcpp_init_year: 1)
Coordinates:
  * lat             (lat) float32 572B -90.0 -88.73 -87.46 ... 87.46 88.73 90.0
  * lon             (lon) float32 576B 0.0 2.5 5.0 7.5 ... 352.5 355.0 357.5
  * plev            (plev) float32 76B 1e+05 9.25e+04 8.5e+04 ... 500.0 100.0
  * time            (time) datetime64[ns] 16kB 1850-01-16T12:00:00 ... 2014-1...
    time_bounds     (time, axis_nbounds) datetime64[ns] 32kB dask.array<chunksize=(1980, 2), meta=np.ndarray>
  * member_id       (member_id) object 8B 'r2i1p1f1'
  * dcpp_init_year  (dcpp_init_year) float64 8B nan
Dimensions without coordinates: axis_nbounds
Data variables:
    ua              (member_id, dcpp_init_year, time, plev, lat, lon) float32 3GB dask.array<chunksize=(1, 1, 60, 19, 143, 144), meta=np.ndarray>
Attributes: (12/67)
    CMIP6_CV_version:                 cv=6.2.3.5-2-g63b123e
    Conventions:                      CF-1.7 CMIP-6.2
    EXPID:                            historical
    activity_id:                      CMIP
    branch_method:                    standard
    branch_time_in_child:             0.0
    ...                               ...
    intake_esm_attrs:variable_id:     ua
    intake_esm_attrs:grid_label:      gr
    intake_esm_attrs:zstore:          gs://cmip6/CMIP6/CMIP/IPSL/IPSL-CM6A-LR...
    intake_esm_attrs:version:         20180803
    intake_esm_attrs:_data_format_:   zarr
    intake_esm_dataset_key:           CMIP.IPSL.IPSL-CM6A-LR.historical.Amon.gr

xarray.Dataset

• Dimensions:
  • lat: 143
  • lon: 144
  • plev: 19
  • time: 1980
  • axis_nbounds: 2
  • member_id: 1
  • dcpp_init_year: 1
• Coordinates: (7)
  • lat
    (lat)
    float32
    -90.0 -88.73 -87.46 ... 88.73 90.0

    axis :

    Y

    long_name :

    Latitude

    standard_name :

    latitude

    units :

    degrees_north

    array([-90.      , -88.73239 , -87.46479 , -86.19718 , -84.92958 , -83.66197 ,
           -82.39436 , -81.12676 , -79.85915 , -78.59155 , -77.323944, -76.056335,
           -74.788734, -73.521126, -72.253525, -70.985916, -69.71831 , -68.45071 ,
           -67.1831  , -65.91549 , -64.64789 , -63.380283, -62.112675, -60.84507 ,
           -59.577465, -58.30986 , -57.042255, -55.774647, -54.507042, -53.239437,
           -51.971832, -50.704224, -49.43662 , -48.169014, -46.90141 , -45.633804,
           -44.366196, -43.09859 , -41.830986, -40.56338 , -39.295776, -38.028168,
           -36.760563, -35.492958, -34.225353, -32.957745, -31.690142, -30.422535,
           -29.15493 , -27.887323, -26.619719, -25.352112, -24.084507, -22.816902,
           -21.549295, -20.28169 , -19.014084, -17.746479, -16.478872, -15.211267,
           -13.943662, -12.676056, -11.408451, -10.140845,  -8.87324 ,  -7.605634,
            -6.338028,  -5.070423,  -3.802817,  -2.535211,  -1.267606,   0.      ,
             1.267606,   2.535211,   3.802817,   5.070423,   6.338028,   7.605634,
             8.87324 ,  10.140845,  11.408451,  12.676056,  13.943662,  15.211267,
            16.478872,  17.746479,  19.014084,  20.28169 ,  21.549295,  22.816902,
            24.084507,  25.352112,  26.619719,  27.887323,  29.15493 ,  30.422535,
            31.690142,  32.957745,  34.225353,  35.492958,  36.760563,  38.028168,
            39.295776,  40.56338 ,  41.830986,  43.09859 ,  44.366196,  45.633804,
            46.90141 ,  48.169014,  49.43662 ,  50.704224,  51.971832,  53.239437,
            54.507042,  55.774647,  57.042255,  58.30986 ,  59.577465,  60.84507 ,
            62.112675,  63.380283,  64.64789 ,  65.91549 ,  67.1831  ,  68.45071 ,
            69.71831 ,  70.985916,  72.253525,  73.521126,  74.788734,  76.056335,
            77.323944,  78.59155 ,  79.85915 ,  81.12676 ,  82.39436 ,  83.66197 ,
            84.92958 ,  86.19718 ,  87.46479 ,  88.73239 ,  90.      ],
          dtype=float32)

  • lon
    (lon)
    float32
    0.0 2.5 5.0 ... 352.5 355.0 357.5

    axis :

    X

    long_name :

    Longitude

    standard_name :

    longitude

    units :

    degrees_east

    array([  0. ,   2.5,   5. ,   7.5,  10. ,  12.5,  15. ,  17.5,  20. ,  22.5,
            25. ,  27.5,  30. ,  32.5,  35. ,  37.5,  40. ,  42.5,  45. ,  47.5,
            50. ,  52.5,  55. ,  57.5,  60. ,  62.5,  65. ,  67.5,  70. ,  72.5,
            75. ,  77.5,  80. ,  82.5,  85. ,  87.5,  90. ,  92.5,  95. ,  97.5,
           100. , 102.5, 105. , 107.5, 110. , 112.5, 115. , 117.5, 120. , 122.5,
           125. , 127.5, 130. , 132.5, 135. , 137.5, 140. , 142.5, 145. , 147.5,
           150. , 152.5, 155. , 157.5, 160. , 162.5, 165. , 167.5, 170. , 172.5,
           175. , 177.5, 180. , 182.5, 185. , 187.5, 190. , 192.5, 195. , 197.5,
           200. , 202.5, 205. , 207.5, 210. , 212.5, 215. , 217.5, 220. , 222.5,
           225. , 227.5, 230. , 232.5, 235. , 237.5, 240. , 242.5, 245. , 247.5,
           250. , 252.5, 255. , 257.5, 260. , 262.5, 265. , 267.5, 270. , 272.5,
           275. , 277.5, 280. , 282.5, 285. , 287.5, 290. , 292.5, 295. , 297.5,
           300. , 302.5, 305. , 307.5, 310. , 312.5, 315. , 317.5, 320. , 322.5,
           325. , 327.5, 330. , 332.5, 335. , 337.5, 340. , 342.5, 345. , 347.5,
           350. , 352.5, 355. , 357.5], dtype=float32)

  • plev
    (plev)
    float32
    1e+05 9.25e+04 ... 500.0 100.0

    long_name :

    pressure

    name :

    plev

    positive :

    down

    standard_name :

    air_pressure

    units :

    Pa

    array([100000.,  92500.,  85000.,  70000.,  60000.,  50000.,  40000.,  30000.,
            25000.,  20000.,  15000.,  10000.,   7000.,   5000.,   3000.,   2000.,
             1000.,    500.,    100.], dtype=float32)

  • time
    (time)
    datetime64[ns]
    1850-01-16T12:00:00 ... 2014-12-...

    axis :

    T

    bounds :

    time_bounds

    long_name :

    Time axis

    standard_name :

    time

    time_origin :

    1850-01-01 00:00:00

    array(['1850-01-16T12:00:00.000000000', '1850-02-15T00:00:00.000000000',
           '1850-03-16T12:00:00.000000000', ..., '2014-10-16T12:00:00.000000000',
           '2014-11-16T00:00:00.000000000', '2014-12-16T12:00:00.000000000'],
          dtype='datetime64[ns]')

  • time_bounds
    (time, axis_nbounds)
    datetime64[ns]
    dask.array<chunksize=(1980, 2), meta=np.ndarray>

    Array Chunk Bytes 30.94 kiB 30.94 kiB Shape (1980, 2) (1980, 2) Dask graph 1 chunks in 2 graph layers Data type datetime64[ns] numpy.ndarray

  • member_id
    (member_id)
    object
    'r2i1p1f1'

    array(['r2i1p1f1'], dtype=object)

  • dcpp_init_year
    (dcpp_init_year)
    float64
    nan

    array([nan])

• Data variables: (1)
  • ua
    (member_id, dcpp_init_year, time, plev, lat, lon)
    float32
    dask.array<chunksize=(1, 1, 60, 19, 143, 144), meta=np.ndarray>

    cell_measures :

    area: areacella

    cell_methods :

    time: mean

    description :

    Eastward Wind

    history :

    none

    interval_operation :

    900 s

    interval_write :

    1 month

    long_name :

    Eastward Wind

    online_operation :

    average

    standard_name :

    eastward_wind

    units :

    m s-1

    Array Chunk Bytes 2.89 GiB 89.55 MiB Shape (1, 1, 1980, 19, 143, 144) (1, 1, 60, 19, 143, 144) Dask graph 33 chunks in 3 graph layers Data type float32 numpy.ndarray

• Indexes: (6)
  • lat
    PandasIndex

    PandasIndex(Index([             -90.0, -88.73239135742188, -87.46479034423828,
           -86.19718170166016, -84.92958068847656, -83.66197204589844,
           -82.39436340332031, -81.12676239013672,  -79.8591537475586,
             -78.591552734375,
           ...
              78.591552734375,   79.8591537475586,  81.12676239013672,
            82.39436340332031,  83.66197204589844,  84.92958068847656,
            86.19718170166016,  87.46479034423828,  88.73239135742188,
                         90.0],
          dtype='float32', name='lat', length=143))

  • lon
    PandasIndex

    PandasIndex(Index([  0.0,   2.5,   5.0,   7.5,  10.0,  12.5,  15.0,  17.5,  20.0,  22.5,
           ...
           335.0, 337.5, 340.0, 342.5, 345.0, 347.5, 350.0, 352.5, 355.0, 357.5],
          dtype='float32', name='lon', length=144))

  • plev
    PandasIndex

    PandasIndex(Index([100000.0,  92500.0,  85000.0,  70000.0,  60000.0,  50000.0,  40000.0,
            30000.0,  25000.0,  20000.0,  15000.0,  10000.0,   7000.0,   5000.0,
             3000.0,   2000.0,   1000.0,    500.0,    100.0],
          dtype='float32', name='plev'))

  • time
    PandasIndex

    PandasIndex(DatetimeIndex(['1850-01-16 12:00:00', '1850-02-15 00:00:00',
                   '1850-03-16 12:00:00', '1850-04-16 00:00:00',
                   '1850-05-16 12:00:00', '1850-06-16 00:00:00',
                   '1850-07-16 12:00:00', '1850-08-16 12:00:00',
                   '1850-09-16 00:00:00', '1850-10-16 12:00:00',
                   ...
                   '2014-03-16 12:00:00', '2014-04-16 00:00:00',
                   '2014-05-16 12:00:00', '2014-06-16 00:00:00',
                   '2014-07-16 12:00:00', '2014-08-16 12:00:00',
                   '2014-09-16 00:00:00', '2014-10-16 12:00:00',
                   '2014-11-16 00:00:00', '2014-12-16 12:00:00'],
                  dtype='datetime64[ns]', name='time', length=1980, freq=None))

  • member_id
    PandasIndex

    PandasIndex(Index(['r2i1p1f1'], dtype='object', name='member_id'))

  • dcpp_init_year
    PandasIndex

    PandasIndex(Index([nan], dtype='float64', name='dcpp_init_year'))

• Attributes: (67)

  CMIP6_CV_version :

  cv=6.2.3.5-2-g63b123e

  Conventions :

  CF-1.7 CMIP-6.2

  EXPID :

  historical

  activity_id :

  CMIP

  branch_method :

  standard

  branch_time_in_child :

  0.0

  branch_time_in_parent :

  7305.0

  contact :

  ipsl-cmip6@listes.ipsl.fr

  creation_date :

  2018-05-11T14:34:10Z

  data_specs_version :

  01.00.21

  description :

  CMIP6 historical

  dr2xml_md5sum :

  00e1a4f623b35a33620b9828c66bd1c8

  dr2xml_version :

  1.3

  experiment :

  all-forcing simulation of the recent past

  experiment_id :

  historical

  external_variables :

  areacella

  forcing_index :

  1

  frequency :

  mon

  further_info_url :

  https://furtherinfo.es-doc.org/CMIP6.IPSL.IPSL-CM6A-LR.historical.none.r2i1p1f1

  grid :

  LMDZ grid

  grid_label :

  gr

  history :

  Mon Sep 3 14:33:33 2018: ncatted -O -a parent_variant_label,global,m,c,r1i1p1f1 ua_Amon_IPSL-CM6A-LR_historical_r2i1p1f1_gr_185001-201412.nc none

  initialization_index :

  1

  institution :

  Institut Pierre Simon Laplace, Paris 75252, France

  institution_id :

  IPSL

  license :

  CMIP6 model data produced by IPSL is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License (https://creativecommons.org/licenses). Consult https://pcmdi.llnl.gov/CMIP6/TermsOfUse for terms of use governing CMIP6 output, including citation requirements and proper acknowledgment. Further information about this data, including some limitations, can be found via the further_info_url (recorded as a global attribute in this file) and at https://cmc.ipsl.fr/. The data producers and data providers make no warranty, either express or implied, including, but not limited to, warranties of merchantability and fitness for a particular purpose. All liabilities arising from the supply of the information (including any liability arising in negligence) are excluded to the fullest extent permitted by law.

  mip_era :

  CMIP6

  model_version :

  6.1.2

  name :

  /ccc/work/cont003/gencmip6/lurtont/IGCM_OUT/IPSLCM6/PROD/historical/CM61-LR-hist-03-10/CMIP6/ATM/ua_Amon_IPSL-CM6A-LR_historical_r2i1p1f1_gr_%start_date%-%end_date%

  nominal_resolution :

  250 km

  parent_activity_id :

  CMIP

  parent_experiment_id :

  piControl

  parent_mip_era :

  CMIP6

  parent_source_id :

  IPSL-CM6A-LR

  parent_time_units :

  days since 1850-01-01 00:00:00

  parent_variant_label :

  r1i1p1f1

  physics_index :

  1

  product :

  model-output

  realization_index :

  2

  realm :

  atmos

  source :

  IPSL-CM6A-LR (2017): atmos: LMDZ (NPv6, N96; 144 x 143 longitude/latitude; 79 levels; top level 40000 m) land: ORCHIDEE (v2.0, Water/Carbon/Energy mode) ocean: NEMO-OPA (eORCA1.3, tripolar primarily 1deg; 362 x 332 longitude/latitude; 75 levels; top grid cell 0-2 m) ocnBgchem: NEMO-PISCES seaIce: NEMO-LIM3

  source_id :

  IPSL-CM6A-LR

  source_type :

  AOGCM

  sub_experiment :

  none

  sub_experiment_id :

  none

  table_id :

  Amon

  title :

  IPSL-CM6A-LR model output prepared for CMIP6 / CMIP historical

  tracking_id :

  hdl:21.14100/270c7de1-9463-478c-a0a4-050195cf96a4

  variable_id :

  ua

  variant_info :

  Restart from another point in piControl.. Information provided by this attribute may in some cases be flawed. Users can find more comprehensive and up-to-date documentation via the further_info_url global attribute.

  variant_label :

  r2i1p1f1

  status :

  2019-10-25;created;by nhn2@columbia.edu

  netcdf_tracking_ids :

  hdl:21.14100/270c7de1-9463-478c-a0a4-050195cf96a4

  version_id :

  v20180803

  intake_esm_vars :

  ['ua']

  intake_esm_attrs:source_id :

  IPSL-CM6A-LR

  intake_esm_attrs:activity_id :

  CMIP

  intake_esm_attrs:institution_id :

  IPSL

  intake_esm_attrs:experiment_id :

  historical

  intake_esm_attrs:member_id :

  r2i1p1f1

  intake_esm_attrs:table_id :

  Amon

  intake_esm_attrs:variable_id :

  ua

  intake_esm_attrs:grid_label :

  gr

  intake_esm_attrs:zstore :

  gs://cmip6/CMIP6/CMIP/IPSL/IPSL-CM6A-LR/historical/r2i1p1f1/Amon/ua/gr/v20180803/

  intake_esm_attrs:version :

  20180803

  intake_esm_attrs:_data_format_ :

  zarr

  intake_esm_dataset_key :

  CMIP.IPSL.IPSL-CM6A-LR.historical.Amon.gr

Use case 2: save a catalog subset as a new catalog

Another use case is to save a subset of the catalog as a new catalog. This is highly useful when you want to share a subset of the catalog or preserve a copy of the catalog for future use.

Tip

We highly recommend that you save the subset of the catalog which you use in your analysis. Remote catalogs can change over time, and you may want to preserve a copy of the original catalog to ensure reproducibility of your analysis.

To save a subset of the catalog as a new catalog, we can use the serialize() method:

import tempfile
directory = tempfile.gettempdir()
cat_subset.serialize(directory=directory, name="my_catalog_subset")

Successfully wrote ESM catalog json file to: file:///tmp/my_catalog_subset.json

By default, the serialize() method will write a single JSON file containing the catalog subset.

!cat {directory}/my_catalog_subset.json

{
  "esmcat_version": "0.1.0",
  "attributes": [
    {
      "column_name": "activity_id",
      "vocabulary": "https://raw.githubusercontent.com/WCRP-CMIP/CMIP6_CVs/master/CMIP6_activity_id.json"
    },
    {
      "column_name": "source_id",
      "vocabulary": "https://raw.githubusercontent.com/WCRP-CMIP/CMIP6_CVs/master/CMIP6_source_id.json"
    },
    {
      "column_name": "institution_id",
      "vocabulary": "https://raw.githubusercontent.com/WCRP-CMIP/CMIP6_CVs/master/CMIP6_institution_id.json"
    },
    {
      "column_name": "experiment_id",
      "vocabulary": "https://raw.githubusercontent.com/WCRP-CMIP/CMIP6_CVs/master/CMIP6_experiment_id.json"
    },
    {
      "column_name": "member_id",
      "vocabulary": ""
    },
    {
      "column_name": "table_id",
      "vocabulary": "https://raw.githubusercontent.com/WCRP-CMIP/CMIP6_CVs/master/CMIP6_table_id.json"
    },
    {
      "column_name": "variable_id",
      "vocabulary": ""
    },
    {
      "column_name": "grid_label",
      "vocabulary": "https://raw.githubusercontent.com/WCRP-CMIP/CMIP6_CVs/master/CMIP6_grid_label.json"
    },
    {
      "column_name": "version",
      "vocabulary": ""
    },
    {
      "column_name": "dcpp_start_year",
      "vocabulary": ""
    }
  ],
  "assets": {
    "column_name": "zstore",
    "format": "zarr",
    "format_column_name": null
  },
  "aggregation_control": {
    "variable_column_name": "variable_id",
    "groupby_attrs": [
      "activity_id",
      "institution_id",
      "source_id",
      "experiment_id",
      "table_id",
      "grid_label"
    ],
    "aggregations": [
      {
        "type": "union",
        "attribute_name": "variable_id",
        "options": {}
      },
      {
        "type": "join_new",
        "attribute_name": "member_id",
        "options": {
          "coords": "minimal",
          "compat": "override"
        }
      },
      {
        "type": "join_new",
        "attribute_name": "dcpp_init_year",
        "options": {
          "coords": "minimal",
          "compat": "override"
        }
      }
    ]
  },
  "id": "my_catalog_subset",
  "description": "This is an ESM catalog for CMIP6 Zarr data residing in Pangeo's Google Storage.",
  "title": null,
  "last_updated": "2024-10-07T15:59:41Z",
  "catalog_dict": [
    {
      "source_id": "CanESM5",
      "activity_id": "CMIP",
      "institution_id": "CCCma",
      "experiment_id": "historical",
      "member_id": "r11i1p1f1",
      "table_id": "Amon",
      "variable_id": "ua",
      "grid_label": "gn",
      "zstore": "gs://cmip6/CMIP6/CMIP/CCCma/CanESM5/historical/r11i1p1f1/Amon/ua/gn/v20190429/",
      "dcpp_init_year": NaN,
      "version": 20190429
    },
    {
      "source_id": "IPSL-CM6A-LR",
      "activity_id": "CMIP",
      "institution_id": "IPSL",
      "experiment_id": "historical",
      "member_id": "r2i1p1f1",
      "table_id": "Amon",
      "variable_id": "ua",
      "grid_label": "gr",
      "zstore": "gs://cmip6/CMIP6/CMIP/IPSL/IPSL-CM6A-LR/historical/r2i1p1f1/Amon/ua/gr/v20180803/",
      "dcpp_init_year": NaN,
      "version": 20180803
    }
  ]
}

For large catalogs, we recommend that you write the catalog subset to its own CSV file. This can be achieved by setting catalog_type to file:

cat_subset.serialize(directory=directory, name="my_catalog_subset", catalog_type="file")

Successfully wrote ESM catalog json file to: file:///tmp/my_catalog_subset.json

!cat {directory}/my_catalog_subset.json
!cat {directory}/my_catalog_subset.csv

{
  "esmcat_version": "0.1.0",
  "attributes": [
    {
      "column_name": "activity_id",
      "vocabulary": "https://raw.githubusercontent.com/WCRP-CMIP/CMIP6_CVs/master/CMIP6_activity_id.json"
    },
    {
      "column_name": "source_id",
      "vocabulary": "https://raw.githubusercontent.com/WCRP-CMIP/CMIP6_CVs/master/CMIP6_source_id.json"
    },
    {
      "column_name": "institution_id",
      "vocabulary": "https://raw.githubusercontent.com/WCRP-CMIP/CMIP6_CVs/master/CMIP6_institution_id.json"
    },
    {
      "column_name": "experiment_id",
      "vocabulary": "https://raw.githubusercontent.com/WCRP-CMIP/CMIP6_CVs/master/CMIP6_experiment_id.json"
    },
    {
      "column_name": "member_id",
      "vocabulary": ""
    },
    {
      "column_name": "table_id",
      "vocabulary": "https://raw.githubusercontent.com/WCRP-CMIP/CMIP6_CVs/master/CMIP6_table_id.json"
    },
    {
      "column_name": "variable_id",
      "vocabulary": ""
    },
    {
      "column_name": "grid_label",
      "vocabulary": "https://raw.githubusercontent.com/WCRP-CMIP/CMIP6_CVs/master/CMIP6_grid_label.json"
    },
    {
      "column_name": "version",
      "vocabulary": ""
    },
    {
      "column_name": "dcpp_start_year",
      "vocabulary": ""
    }
  ],
  "assets": {
    "column_name": "zstore",
    "format": "zarr",
    "format_column_name": null
  },
  "aggregation_control": {
    "variable_column_name": "variable_id",
    "groupby_attrs": [
      "activity_id",
      "institution_id",
      "source_id",
      "experiment_id",
      "table_id",
      "grid_label"
    ],
    "aggregations": [
      {
        "type": "union",
        "attribute_name": "variable_id",
        "options": {}
      },
      {
        "type": "join_new",
        "attribute_name": "member_id",
        "options": {
          "coords": "minimal",
          "compat": "override"
        }
      },
      {
        "type": "join_new",
        "attribute_name": "dcpp_init_year",
        "options": {
          "coords": "minimal",
          "compat": "override"
        }
      }
    ]
  },
  "id": "my_catalog_subset",
  "description": "This is an ESM catalog for CMIP6 Zarr data residing in Pangeo's Google Storage.",
  "title": null,
  "last_updated": "2024-10-07T15:59:42Z",
  "catalog_file": "file:///tmp/my_catalog_subset.csv"
}

source_id,activity_id,institution_id,experiment_id,member_id,table_id,variable_id,grid_label,zstore,dcpp_init_year,version
CanESM5,CMIP,CCCma,historical,r11i1p1f1,Amon,ua,gn,gs://cmip6/CMIP6/CMIP/CCCma/CanESM5/historical/r11i1p1f1/Amon/ua/gn/v20190429/,,20190429
IPSL-CM6A-LR,CMIP,IPSL,historical,r2i1p1f1,Amon,ua,gr,gs://cmip6/CMIP6/CMIP/IPSL/IPSL-CM6A-LR/historical/r2i1p1f1/Amon/ua/gr/v20180803/,,20180803

Conclusion

Intake-ESM provides a powerful search API, however, there are cases where you may want to modify the catalog by using pandas directly. In this notebook we showed how to do that and how to attach the modified dataframe to the catalog object and/or save the modified catalog as a new catalog.

Show code cell source Hide code cell source

import intake_esm
intake_esm.show_versions()

Show code cell output Hide code cell output

INSTALLED VERSIONS
------------------

cftime: 1.6.4
dask: 2024.9.1
fastprogress: 1.0.3
fsspec: 2024.9.0
gcsfs: 2024.9.0post1
intake: 0.7.0
intake_esm: 2024.2.6.post17+gecd3833.d20241007
netCDF4: 1.7.1
pandas: 2.2.3
requests: 2.32.3
s3fs: 2024.9.0
xarray: 2024.9.0
zarr: 2.18.3