{ "cells": [ { "cell_type": "markdown", "id": "205a15d4-e3b1-420d-aa95-0b64e829e297", "metadata": {}, "source": [ "# Effect of rounding on coordinates with Xarray\n", "**Claire Carouge, CLEX CMS**\n", "\n", "This notebook demonstrates how to work with two datasets that should have the same dimensional coordinates, but for whatever reason don’t.\n", "\n", "When doing operations involving the 2 DataArrays, Xarray will either return the results on the common locations only or return an indexing error. The solution is then to replace one set of coordinates with the other set.\n", "\n", "## The problem" ] }, { "cell_type": "markdown", "id": "4e53cc1f-2500-4576-8713-8368151a21cc", "metadata": {}, "source": [ "To demonstrate, we create two DataArrays that have a slightly different coordinate array. Note how the difference is so small that Jupyter’s default presentation obscures it by rounding." ] }, { "cell_type": "code", "execution_count": 1, "id": "4cec3123-ef3f-424f-80a4-f1eb7d9335fc", "metadata": {}, "outputs": [], "source": [ "import xarray as xr\n", "import numpy as np" ] }, { "cell_type": "code", "execution_count": 2, "id": "8451bb26-afd0-4914-8508-284b68b136b5", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
<xarray.DataArray (x: 3)>\n",
       "array([1, 2, 3])\n",
       "Coordinates:\n",
       "  * x        (x) float64 1.1 2.1 3.1
" ], "text/plain": [ "\n", "array([1, 2, 3])\n", "Coordinates:\n", " * x (x) float64 1.1 2.1 3.1" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
<xarray.DataArray (x: 3)>\n",
       "array([1, 2, 3])\n",
       "Coordinates:\n",
       "  * x        (x) float64 1.1 2.1 3.1
" ], "text/plain": [ "\n", "array([1, 2, 3])\n", "Coordinates:\n", " * x (x) float64 1.1 2.1 3.1" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "arr1 = xr.DataArray([1,2,3],dims=\"x\",coords={\"x\":[1.1,2.1,3.1]})\n", "arr2 = xr.DataArray([1,2,3],dims=\"x\",coords={\"x\":[1.1,2.099999999,3.1]})\n", "display(arr1, arr2)" ] }, { "cell_type": "markdown", "id": "276b722c-1e50-4ea2-a002-1d81b1538b79", "metadata": {}, "source": [ "You can specify a precision for numpy to suppress the rounding. We are using `np.printoptions()` here to easily swap between the default rounding and a specific precision. You can use `np.set_printoptions()` to set a specific precision for your whole notebook. Note that Xarray may override the precision set, make sure you print the `.data` array if you need to double-check the precision in the print:" ] }, { "cell_type": "code", "execution_count": 3, "id": "4dcaa536-2c4c-4337-a2d3-700a4b2c8ece", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[1.1 2.099999999 3.1 ]\n" ] } ], "source": [ "with np.printoptions(precision=9):\n", " print(arr2.x.data)" ] }, { "cell_type": "markdown", "id": "60d63db5-e54f-4ff8-82ca-b3f7eddfccf4", "metadata": {}, "source": [ "When we make element-wise calculations with these two DataArrays, we would expect that the resulting array would have the same length as the two input arrays, i.e. arr1 - arr2 == [0, 0, 0]. \n", "\n", "But instead we only get a length 2 array, containing only the points where the coordinates were truly identical." ] }, { "cell_type": "code", "execution_count": 4, "id": "0c148c79-280f-4303-8232-c383c13f068b", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
<xarray.DataArray (x: 2)>\n",
       "array([0, 0])\n",
       "Coordinates:\n",
       "  * x        (x) float64 1.1 3.1
" ], "text/plain": [ "\n", "array([0, 0])\n", "Coordinates:\n", " * x (x) float64 1.1 3.1" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "arr1-arr2" ] }, { "cell_type": "markdown", "id": "0a8b7c84", "metadata": {}, "source": [ "There was no error, but xarray automatic broadcasting has silently ignored the second value where the coordinates are different for the 2 arrays." ] }, { "cell_type": "markdown", "id": "ed07a75b-f56e-482c-8abc-6fce321404c1", "metadata": {}, "source": [ "## Solutions\n", "### Re-assign coordinates\n", "\n", "Usually, these issues arise when the coordinates **should** be identical, but aren't. \n", "In that case, the easiest, straight-forward method is simply assign the coordinate array from one to the other:" ] }, { "cell_type": "code", "execution_count": 5, "id": "a4546537-60e1-47e8-afbc-b8fabeee023f", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[1.1 2.1 3.1]\n" ] } ], "source": [ "arr2=arr2.assign_coords({\"x\":arr1.x})\n", "with np.printoptions(precision=9):\n", " print(arr2.x.data)" ] }, { "cell_type": "code", "execution_count": 6, "id": "b95f66c6-6f5a-4223-8644-e51878960fc0", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
<xarray.DataArray (x: 3)>\n",
       "array([0, 0, 0])\n",
       "Coordinates:\n",
       "  * x        (x) float64 1.1 2.1 3.1
" ], "text/plain": [ "\n", "array([0, 0, 0])\n", "Coordinates:\n", " * x (x) float64 1.1 2.1 3.1" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "arr1-arr2" ] }, { "cell_type": "markdown", "id": "413f2757", "metadata": {}, "source": [ "### Interpolate the data\n", "\n", "If the coordinates are so different that this would lead to unacceptable errors, we can also use `xarray`'s interpolation routine to force the arrays on the same grid.\n", "\n", "Let's first re-assign the coordinates from the beginning." ] }, { "cell_type": "code", "execution_count": 7, "id": "81343434-02f6-40ba-92aa-ec5c8a4e3624", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[1.1 2.099999999 3.1 ]\n" ] } ], "source": [ "arr2=arr2.assign_coords({\"x\":[1.1,2.099999999,3.1]})\n", "with np.printoptions(precision=9):\n", " print(arr2.x.data)" ] }, { "cell_type": "markdown", "id": "e668baf4", "metadata": {}, "source": [ "Now we interpolate using the x-coordinate from `arr1` as the target for the interpolation." ] }, { "cell_type": "code", "execution_count": 8, "id": "6daf8e03", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
<xarray.DataArray (x: 3)>\n",
       "array([1., 2., 3.])\n",
       "Coordinates:\n",
       "  * x        (x) float64 1.1 2.1 3.1
" ], "text/plain": [ "\n", "array([1., 2., 3.])\n", "Coordinates:\n", " * x (x) float64 1.1 2.1 3.1" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "arr2 = arr2.interp(x=arr1.x)\n", "display(arr2)" ] }, { "cell_type": "markdown", "id": "b5e5a547", "metadata": {}, "source": [ "The two arrays now have the same coordinate array, but the values have ever so slightly changed. In this case, the rounding obscures this fact again, but the difference is no longer all-zeros:" ] }, { "cell_type": "code", "execution_count": 9, "id": "32664138", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
<xarray.DataArray (x: 3)>\n",
       "array([ 0.00000000e+00, -1.00000008e-09,  0.00000000e+00])\n",
       "Coordinates:\n",
       "  * x        (x) float64 1.1 2.1 3.1
" ], "text/plain": [ "\n", "array([ 0.00000000e+00, -1.00000008e-09, 0.00000000e+00])\n", "Coordinates:\n", " * x (x) float64 1.1 2.1 3.1" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "arr1 - arr2" ] }, { "cell_type": "markdown", "id": "b4315746-ff3b-499d-963f-6f0951af603c", "metadata": { "tags": [] }, "source": [ "## Why is Xarray doing this\n", "Although this behaviour may seem annoying in this example, it is something that is very useful for Xarray. For example, if you have a field defined on a subset of a grid (e.g surface field) and one on the full grid (e.g on all model levels), you can get the difference, sum, etc. of the two fields directly. You don't need to create a subset of the full grid field. Or if you have fields covering different but overlapping time periods, to subset the fields to the common period, you can do:\n", "```\n", " array1 = array1.sel(time=array2.time)\n", " array2 = array2.sel(time=array1.time)\n", "```\n", "\n", "That is true as long as the subset grid has a coordinate along the dimensions that are subsetted even if the dimensions do not appear in the subsetted field." ] }, { "cell_type": "code", "execution_count": null, "id": "e0382dec-a380-4530-a88e-3cf038710ae8", "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python [conda env:analysis3-21.07]", "language": "python", "name": "conda-env-analysis3-21.07-py" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.7" } }, "nbformat": 4, "nbformat_minor": 5 }