Zarr tutorial (#323)

* adding the links to all indexing materials

* typo fix + remove a redundant example.

* initial commits and contents

* data storage

* updates and cloud storage

* updates and cloud storage

* additional resources

* updates

* more organized

* sharding

* sharding added

* zarr added

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* minor updates to zarr tutorial

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* ensure zarr notebook is rendered, add cell metadata for large outputs

* remove accidentaly advaced-indexing edit

* use data subfolder for io notebook as in zarr notebook

* fix cat and tree paths, toggle output

* link to cmip6 notebook at end

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Joseph Hamman <joe@earthmover.io>
Co-authored-by: Scott Henderson <3924836+scottyhq@users.noreply.github.com>

Author: 4 people

_toc.yml

@@ -44,6 +44,7 @@ parts:
           - file: intermediate/indexing/boolean-masking-indexing.ipynb
       - file: intermediate/hierarchical_computation.ipynb
       - file: intermediate/xarray_and_dask
+      - file: intermediate/intro-to-zarr.ipynb
       - file: intermediate/xarray_ecosystem
       - file: intermediate/hvplot
       - file: intermediate/remote_data/index

fundamentals/01.1_io.ipynb

@@ -51,9 +51,27 @@
    ]
   },
   {
-   "cell_type": "markdown",
+   "cell_type": "code",
+   "execution_count": null,
    "id": "2",
    "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Ensure we start with a clean directory for the tutorial\n",
+    "import pathlib\n",
+    "import shutil\n",
+    "\n",
+    "datadir = pathlib.Path('../data/io-tutorial')\n",
+    "if datadir.exists():\n",
+    "    shutil.rmtree(datadir)\n",
+    "else:\n",
+    "    datadir.mkdir()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "3",
+   "metadata": {},
    "source": [
     "The constructor of `Dataset` takes three parameters:\n",
     "\n",
@@ -66,7 +84,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "3",
+   "id": "4",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -94,16 +112,16 @@
     ")\n",
     "\n",
     "# write datasets\n",
-    "ds1.to_netcdf(\"ds1.nc\")\n",
-    "ds2.to_netcdf(\"ds2.nc\")\n",
+    "ds1.to_netcdf(datadir / \"ds1.nc\")\n",
+    "ds2.to_netcdf(datadir / \"ds2.nc\")\n",
     "\n",
     "# write dataarray\n",
-    "ds1.a.to_netcdf(\"da1.nc\")"
+    "ds1.a.to_netcdf(datadir / \"da1.nc\")"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "4",
+   "id": "5",
    "metadata": {},
    "source": [
     "Reading those files is just as simple:\n"
@@ -112,26 +130,26 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "5",
+   "id": "6",
    "metadata": {},
    "outputs": [],
    "source": [
-    "xr.open_dataset(\"ds1.nc\")"
+    "xr.open_dataset(datadir / \"ds1.nc\")"
    ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "6",
+   "id": "7",
    "metadata": {},
    "outputs": [],
    "source": [
-    "xr.open_dataarray(\"da1.nc\")"
+    "xr.open_dataarray(datadir / \"da1.nc\")"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "7",
+   "id": "8",
    "metadata": {},
    "source": [
     "<img src=\"https://zarr.readthedocs.io/en/stable/_static/logo1.png\" align=\"right\" width=\"20%\">\n",
@@ -151,16 +169,16 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "8",
+   "id": "9",
    "metadata": {},
    "outputs": [],
    "source": [
-    "ds1.to_zarr(\"ds1.zarr\", mode=\"w\")"
+    "ds1.to_zarr(datadir / \"ds1.zarr\", mode=\"w\")"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "9",
+   "id": "10",
    "metadata": {},
    "source": [
     "We can then read the created file with:\n"
@@ -169,16 +187,16 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "10",
+   "id": "11",
    "metadata": {},
    "outputs": [],
    "source": [
-    "xr.open_zarr(\"ds1.zarr\", chunks=None)"
+    "xr.open_zarr(datadir / \"ds1.zarr\", chunks=None)"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "11",
+   "id": "12",
    "metadata": {},
    "source": [
     "setting the `chunks` parameter to `None` avoids `dask` (more on that in a later\n",
@@ -187,7 +205,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "12",
+   "id": "13",
    "metadata": {},
    "source": [
     "**tip:** You can write to any dictionary-like (`MutableMapping`) interface:"
@@ -196,7 +214,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "13",
+   "id": "14",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -207,7 +225,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "14",
+   "id": "15",
    "metadata": {},
    "source": [
     "## Raster files using rioxarray\n",
@@ -220,7 +238,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "15",
+   "id": "16",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -241,16 +259,16 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "16",
+   "id": "17",
    "metadata": {},
    "outputs": [],
    "source": [
-    "da.rio.to_raster('ds1_a.tiff')"
+    "da.rio.to_raster(datadir / 'ds1_a.tiff')"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "17",
+   "id": "18",
    "metadata": {},
    "source": [
     "NOTE: you can now load this file into GIS tools like [QGIS](https://www.qgis.org)! Or open back into Xarray:"
@@ -259,11 +277,11 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "18",
+   "id": "19",
    "metadata": {},
    "outputs": [],
    "source": [
-    "DA = xr.open_dataarray('ds1_a.tiff', engine='rasterio')\n",
+    "DA = xr.open_dataarray(datadir / 'ds1_a.tiff', engine='rasterio')\n",
     "DA.rio.crs"
    ]
   }
@@ -279,11 +297,6 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3"
-  },
-  "vscode": {
-   "interpreter": {
-    "hash": "31f2aee4e71d21fbe5cf8b01ff0e069b9275f58929596ceb00d14d90e3e16cd6"
-   }
   }
  },
  "nbformat": 4,