Intake-esm catalogs include two pieces:
An ESM-Collection file: an ESM-Collection file is a simple json file that provides metadata about the catalog. The specification for this json file is found in the esm-collection-spec repository.
A catalog file: the catalog file is a CSV file that lists the catalog contents. This file includes one row per dataset granule (e.g. a NetCDF file or Zarr dataset). The columns in this CSV must match the attributes and assets listed in the ESM-Collection file. A short example of a catalog file is shown below::
activity_id,institution_id,source_id,experiment_id,member_id,table_id,variable_id,grid_label,zstore,dcpp_init_year AerChemMIP,BCC,BCC-ESM1,piClim-CH4,r1i1p1f1,Amon,ch4,gn,gs://cmip6/AerChemMIP/BCC/BCC-ESM1/piClim-CH4/r1i1p1f1/Amon/ch4/gn/, AerChemMIP,BCC,BCC-ESM1,piClim-CH4,r1i1p1f1,Amon,clt,gn,gs://cmip6/AerChemMIP/BCC/BCC-ESM1/piClim-CH4/r1i1p1f1/Amon/clt/gn/, AerChemMIP,BCC,BCC-ESM1,piClim-CH4,r1i1p1f1,Amon,co2,gn,gs://cmip6/AerChemMIP/BCC/BCC-ESM1/piClim-CH4/r1i1p1f1/Amon/co2/gn/, AerChemMIP,BCC,BCC-ESM1,piClim-CH4,r1i1p1f1,Amon,evspsbl,gn,gs://cmip6/AerChemMIP/BCC/BCC-ESM1/piClim-CH4/r1i1p1f1/Amon/evspsbl/gn/, ...
The table below is an incomplete list of existing catalogs. Please feel free to add to this list or raise an issue on GitHub.
CMIP6-GLADE
Description: CMIP6 data accessible on the NCAR’s GLADE disk storage system
Platform: NCAR-GLADE
Catalog path or url: /glade/collections/cmip/catalog/intake-esm-datastore/catalogs/glade-cmip6.json
Data Format: netCDF
Documentation Page: https://pcmdi.llnl.gov/CMIP6/Guide/dataUsers.html
CMIP6-CESM2-Timeseries
Description: CESM2 raw output (non-cmorized) that went into CMIP6 data
Platform: NCAR-CAMPAIGN
Catalog path or url: /glade/collections/cmip/catalog/intake-esm-datastore/catalogs/campaign-cesm2-cmip6-timeseries.json
Documentation Page: http://www.cesm.ucar.edu/models/cesm2/
CMIP5-GLADE
Description: CMIP5 data accessible on the NCAR’s GLADE disk storage system
Catalog path or url: /glade/collections/cmip/catalog/intake-esm-datastore/catalogs/glade-cmip5.json
Documentation Page: https://pcmdi.llnl.gov/mips/cmip5/guide.html
CESM1-LENS-AWS
Description: CESM1 Large Ensemble data publicly available on Amazon S3
Platform: AWS S3 (us-west-2 region)
Catalog path or url: https://raw.githubusercontent.com/NCAR/cesm-lens-aws/master/intake-catalogs/aws-cesm1-le.json
Data Format: Zarr
Documentation Page: https://doi.org/10.26024/wt24-5j82
CESM1-LENS-GLADE
Description: CESM1 Large Ensemble data stored on NCAR’s GLADE disk storage system
Catalog path or url: /glade/collections/cmip/catalog/intake-esm-datastore/catalogs/glade-cesm1-le.json
Documentation Page: https://doi.org/10.5065/d6j101d1
CMIP6-GCP
Description: CMIP6 Zarr data residing in Pangeo’s Google Storage
Platform: Google Cloud Platform
Catalog path or url: https://raw.githubusercontent.com/NCAR/intake-esm-datastore/master/catalogs/pangeo-cmip6.json
CMIP6-MISTRAL
Description: CMIP6 data accessible on the DKRZ’s MISTRAL disk storage system
Platform: DKRZ (German Climate Computing Centre)-MISTRAL
Catalog path or url: /home/mpim/m300524/intake-esm-datastore/catalogs/mistral-cmip6.json
CMIP5-MISTRAL
Description: CMIP5 data accessible on the DKRZ’s MISTRAL disk storage system
Catalog path or url: /home/mpim/m300524/intake-esm-datastore/catalogs/mistral-cmip5.json
MiKlip-MISTRAL
Description: Data from MiKlip projects at the Max Planck Institute for Meteorology (MPI-M)
Catalog path or url: /home/mpim/m300524/intake-esm-datastore/catalogs/mistral-miklip.json
Documentation Page: https://www.fona-miklip.de/
MPI-GE-MISTRAL
Description: Max Planck Institute Grand Ensemble cmorized by CMIP5-standards
Catalog path or url: /home/mpim/m300524/intake-esm-datastore/catalogs/mistral-MPI-GE.json
Documentation Page: https://doi.org/10/gf3kgt
CMIP6-LDEO-OpenDAP
Description: CMIP6 data accessible via Hyrax OpenDAP Server at Lamont-Doherty Earth Observatory
Platform: LDEO-OpenDAP
Catalog path or url: http://haden.ldeo.columbia.edu/catalogs/hyrax_cmip6.json
Note
Some of these catalogs are also stored in intake-esm-datastore GitHub repository at https://github.com/NCAR/intake-esm-datastore/tree/master/catalogs