Intake-esm provides functionality to execute queries against the catalog. This notebook provided a more in-depth treatment of the search API in intake-esm, with detailed information that you can refer to when needed.
import warnings warnings.filterwarnings("ignore") import intake
catalog_url = "https://ncar-cesm-lens.s3-us-west-2.amazonaws.com/catalogs/aws-cesm1-le.json" col = intake.open_esm_datastore(catalog_url) col
aws-cesm1-le catalog with 56 dataset(s) from 429 asset(s):
col.df.head()
The search() method allows the user to perform a query on a catalog using keyword arguments. The keyword argument names must be the names of the columns in the catalog. By default, the search() method looks for exact matches, and is case sensitive:
search()
col.search(experiment="20C", variable_long_name="wind")
aws-cesm1-le catalog with 0 dataset(s) from 0 asset(s):
As you can see, the example above returns an empty catalog.
In some cases, you may not know the exact term to look for. For such cases, inkake-esm supports searching for substring matches. With use of wildcards and/or regular expressions, we can find all items with a particular substring in a given column. Let’s search for:
entries from experiment = ‘20C’
experiment
all entries whose variable long name contains wind
wind
col.search(experiment="20C", variable_long_name="wind*").df
Now, let’s search for:
all entries whose variable long name starts with wind
col.search(experiment="20C", variable_long_name="^wind").df
require_all_on argument
By default intake-esm’s search() method returns entries that fulfill any of the criteria specified in the query. Intake-esm can return entries that fulfill all query criteria when the user supplies the require_all_on argument. The require_all_on parameter can be a dataframe column or a list of dataframe columns across which all elements must satisfy the query criteria. The require_all_on argument is best explained with the following example.
require_all_on
Let’s define a query for our collection that requests multiple variable_ids and multiple experiment_ids from the Omon table_id, all from 3 different source_ids:
catalog_url = ( "https://raw.githubusercontent.com/NCAR/intake-esm-datastore/master/catalogs/pangeo-cmip6.json" ) col = intake.open_esm_datastore(catalog_url) col
pangeo-cmip6 catalog with 6539 dataset(s) from 402033 asset(s):
# Define our query query = dict( variable_id=["thetao", "o2"], experiment_id=["historical", "ssp245", "ssp585"], table_id=["Omon"], source_id=["ACCESS-ESM1-5", "AWI-CM-1-1-MR", "FGOALS-f3-L"], )
Now, let’s use this query to search for all assets in the collection that satisfy any combination of these requests (i.e., with require_all_on=None, which is the default):
require_all_on=None
col_subset = col.search(**query) col_subset
pangeo-cmip6 catalog with 8 dataset(s) from 76 asset(s):
# Group by `source_id` and count unique values for a few columns col_subset.df.groupby("source_id")[["experiment_id", "variable_id", "table_id"]].nunique()
As you can see, the search results above include source_ids for which we only have one of the two variables, and one or two of the three experiments.
We can tell intake-esm to discard any source_id that doesn’t have both variables ["thetao", "o2"] and all three experiments ["historical", "ssp245", "ssp585"] by passing require_all_on=["source_id"] to the search method:
["thetao", "o2"]
["historical", "ssp245", "ssp585"]
require_all_on=["source_id"]
col_subset = col.search(require_all_on=["source_id"], **query) col_subset
pangeo-cmip6 catalog with 3 dataset(s) from 63 asset(s):
col_subset.df.groupby("source_id")[["experiment_id", "variable_id", "table_id"]].nunique()
Notice that with the require_all_on=["source_id"] option, the only source_id that was returned by our query was the source_id for which all of the variables and experiments were found.