diff --git a/README.md b/README.md index c4fa5f5..b67999e 100644 --- a/README.md +++ b/README.md @@ -1,84 +1,60 @@ - - -[![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/CosmoStat/Tutorials/master) - -# CosmoStat Tutorials ---- - -CosmoStat is committed to the philosophy of reproducible research, endeavouring -to provide source code and data for all publications. In this spirit, we have -additionally put significant effort into providing useful educational -materials. The aim being to provide other researchers with an in-depth -understanding of the various tools we use in our work. - - -## Contents ---- - -1. [Set Up](#Set-Up) - * [Requirements](#Requirements) - * [Running Remotely](#Running-Remotely) - * [Running Locally](#Running-Locally) -1. [Introduction to Python](./python/README.md) - * [Requirements](./python/README.md#Requirements) - * [Beginner Topics](./python/README.md#tutorial-1-beginner-topics) - * [Intermediate and Advanced Topics](./python/README.md#tutorial-2-intermediate-and-advanced-topics) -1. [Sparsity](./ada/README.md) - * [Requirements](./ada/README.md#Requirements) - * [Notebooks](./ada/README.md#Notebooks) - * [Acknowledgements](./ada/README.md#Acknowledgements) -1. [Low-Rank](./low-rank/README.md) - * [Requirements](./low-rank/README.md#Requirements) - * [Notebooks](./low-rank/README.md#Notebooks) - * [Acknowledgements](./low-rank/README.md#Acknowledgements) - -## Set Up ---- +# ADA Sparsity Tutorial -### Requirements +> Author: [Samuel Farrens](http://www.cosmostat.org/people/sfarrens) +> Year: 2018 (updated in 2023) +> Email: [samuel.farrens@cea.fr](mailto:samuel.farrens@cea.fr) + +[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/CosmoStat/Tutorials/blob/ada/])[![Open In Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/CosmoStat/Tutorials/ada) -Each tutorial has its own set of requirements, but all require Python 3.5 or higher. +This tutorial was originally presented at the ninth edition of the +[Astronomical Data Analysis (ADAIX)](http://ada.cosmostat.org/) summer school held in Valencia +in 2018. The objective is to provide a beginner level introduction to the concept of sparsity, in particular as a regularisation method for solving linear inverse problems. The tutorial does not provide an in-depth mathematical background nor detailed explanations for every topic. Tutees should supplement this tutorial with further reading of the various references provided in the notebooks for a more comprehensive understanding of the subject. More information about how cutting-edge sparsity techniques are being applied to astrophysical data is available on the CosmoStat website. -* Python (require >=3.5) +This tutorial is comprised of a series of Jupyter notebooks that demonstrate how the tools implemented in sparsity work as well as showing the applicability of these tools to various simple problems. +All code blocks are provided in Python and the number of external packages required to run the examples has be kept to a minimum. With the exception of plotting routines, all of the code has been clearly presented inside the notebooks at least once to avoid the use of any "black boxes" for solving the problems presented. -### Running Remotely +## Running the notebooks remotely -All of the tutorials can be run remotely by launching the repository -[Binder](https://mybinder.org/v2/gh/CosmoStat/Tutorials/master). No further set -up is required. +All of the Jupyter notebooks for the tutorial can be run remotely (i.e. without needing to install anything locally) using either [Google Colab](https://colab.research.google.com/) or [Binder](https://mybinder.org/). Click on either of the badges above to launch one of these services. Note that running the notebooks in this way will require a stable internet connection. Also note that Binder make take several minutes to launch. -### Running Locally +## Running the notebooks locally -In order to run the tutorials offline, please follow these steps: +Alternatively, you can [clone](https://docs.github.com/en/repositories/creating-and-managing-repositories/cloning-a-repository) or simply [download](https://github.com/CosmoStat/Tutorials/archive/refs/heads/ada.zip) the repository contents to your computer. Once the content is downloaded and all of the requirements have been installed, you will be able to run the notebooks locally without the need of an internet connection. + +### Requirements -1. Download or clone the GitHub repository. +In order to run the tutorial notebooks locally, you will need to have the following packages installed: - +* Python +* NumPy (recommend v1.24+) +* SciPy (recommend v1.10+) +* Matplotlib (recommend v3.6+) +* Jupyter (recommend v1.0+) - *e.g.* +All of the packages listed above can easily be installed using either `pip` or Conda. - ```bash - git clone https://github.com/CosmoStat/Tutorials.git - ``` +*e.g.* -2. Install the tutorial dependencies. +```bash +$ pip install numpy scipy matplotlib jupyter +``` - This can be done using `pip` as follows: +or - ```bash - pip install -r requirements.txt - ``` +```bash +$ conda install numpy scipy matplotlib jupyter +``` - or using `conda` as follows: +## List of notebooks - ```bash - conda env create -f environment.yml - conda activate cstutorials - ``` +1. [Introduction to Inverse Problems](inverse_problems_1.ipynb) +1. [Introduction to Sparsity I](sparsity_1.ipynb) +1. [Introduction to Sparsity II: Compressed Sensing](sparsity_2_compressed_sensing.ipynb) +1. [Introduction to Sparsity III: Deconvolution](sparsity_3_deconvolution.ipynb) +1. [Introduction to Wavelets](wavelets_1.ipynb) +1. [Exercise Solutions](exercise_solutions.ipynb) -3. Finally, the notebooks can be launched by running: +## Acknowledgements - ```bash - jupyter notebook - ``` +S. Farrens would like to acknowledge the following people for helpful feedback and ideas for producing this tutorial: Fred Ngolè-Mboula and Jean-Luc Starck. diff --git a/ada/README.md b/ada/README.md deleted file mode 100644 index 7210041..0000000 --- a/ada/README.md +++ /dev/null @@ -1,55 +0,0 @@ -## Sparsity ---- - -> Author: [Samuel Farrens](http://www.cosmostat.org/people/sfarrens) -> Year: 2018 -> Email: [samuel.farrens@cea.fr](mailto:samuel.farrens@cea.fr) - -This tutorial was originally presented at the ninth edition of the -[Astronomical Data Analysis (ADAIX)](http://ada.cosmostat.org/) summer school held in Valencia -in 2018. The objective is to provide a beginner level introduction to the concept of sparsity, in particular as a regularisation method for solving linear inverse problems. The tutorial does not provide an in-depth mathematical background nor detailed explanations for every topic. Tutees should supplement this tutorial with further reading of the various references provided in the notebooks for a more comprehensive understanding of the subject. More information about how cutting-edge sparsity techniques are being applied to astrophysical data is available on the CosmoStat website. - -This tutorial is comprised of a series of Jupyter notebooks that demonstrate how the tools implemented in sparsity work as well as showing the applicability of these tools to various simple problems. - -All code blocks are provided in Python (support for both 2.7 and 3.6+) and the number of external packages required to run the examples has be kept to a minimum. With the exception of plotting routines, all of the code has been clearly presented inside the notebooks at least once to avoid the use of any "black boxes" for solving the problems presented. - -### Requirements - -In order to run the tutorial notebooks tutees will need to have the following installed: - -* Python (2.7 or 3.6+, users are recommended to use Python 3.6) -* NumPy (recommend v1.14.1+) -* SciPy (recommend v1.0.0+) -* Matplotlib (recommend v2.1.0+) -* Jupyter (recommend v1.0.0+) - -Python 2.7 users should additonally install: - -* Future (recommend v0.16.0+) - -All of the packages listed above can easily be installed using either `pip` or Anaconda. - -*e.g.* - -```bash -$ pip install numpy scipy matplotlib jupyter -``` - -or - -```bash -$ conda install numpy scipy matplotlib jupyter -``` - -### Notebooks - -1. [Introduction to Inverse Problems](https://mybinder.org/v2/gh/CosmoStat/Tutorials/master?filepath=%2Fada%2Finverse_problems_1.ipynb) -1. [Introduction to Sparsity I](https://mybinder.org/v2/gh/CosmoStat/Tutorials/master?filepath=%2Fada%2Fsparsity_1.ipynb) -1. [Introduction to Sparsity II: Compressed Sensing](https://mybinder.org/v2/gh/CosmoStat/Tutorials/master?filepath=%2Fada%2Fsparsity_2_compressed_sensing.ipynb) -1. [Introduction to Sparsity III: Deconvolution](https://mybinder.org/v2/gh/CosmoStat/Tutorials/master?filepath=%2Fada%2Fsparsity_3_deconvolution.ipynb) -1. [Introduction to Wavelets](https://mybinder.org/v2/gh/CosmoStat/Tutorials/master?filepath=%2Fada%2Fwavelets_1.ipynb) -1. [Exercise Solutions](https://mybinder.org/v2/gh/CosmoStat/Tutorials/master?filepath=%2Fada%2Fexercise_solutions.ipynb) - -### Acknowledgements - -S. Farrens would like to acknowledge the following people for helpful feedback and ideas for producing this tutorial: Fred Ngolè-Mboula and Jean-Luc Starck. diff --git a/ada/data/cs_mask.npy b/data/cs_mask.npy similarity index 100% rename from ada/data/cs_mask.npy rename to data/cs_mask.npy diff --git a/ada/data/cs_obs_data.npy b/data/cs_obs_data.npy similarity index 100% rename from ada/data/cs_obs_data.npy rename to data/cs_obs_data.npy diff --git a/ada/data/cs_true_data.npy b/data/cs_true_data.npy similarity index 100% rename from ada/data/cs_true_data.npy rename to data/cs_true_data.npy diff --git a/ada/data/deconv_obs_data.npy b/data/deconv_obs_data.npy similarity index 100% rename from ada/data/deconv_obs_data.npy rename to data/deconv_obs_data.npy diff --git a/ada/data/deconv_psf.npy b/data/deconv_psf.npy similarity index 100% rename from ada/data/deconv_psf.npy rename to data/deconv_psf.npy diff --git a/ada/data/venus.npy b/data/venus.npy similarity index 100% rename from ada/data/venus.npy rename to data/venus.npy diff --git a/environment.yml b/environment.yml index c3f3461..892142f 100644 --- a/environment.yml +++ b/environment.yml @@ -1,14 +1,12 @@ -name: cstutorials +name: sparsity channels: - conda-forge - defaults dependencies: - - python>=3.5 - - astropy>=3.0 + - python + - astropy - jupyter - matplotlib - numpy - - pandas - - scikit-learn - scipy - - seaborn + - pooch diff --git a/ada/exercise_solutions.ipynb b/exercise_solutions.ipynb similarity index 91% rename from ada/exercise_solutions.ipynb rename to exercise_solutions.ipynb index 2c96639..4c157f8 100644 --- a/ada/exercise_solutions.ipynb +++ b/exercise_solutions.ipynb @@ -4,6 +4,8 @@ "cell_type": "markdown", "metadata": {}, "source": [ + "
\n", + "\n", "# Exercise Solutions\n", "\n", "---\n", @@ -22,12 +24,14 @@ " * [Linear Regression Exercise](#Linear-Regression-Exercise)\n", " * [Gradient Descent Exercise](#Gradient-Descent-Exercise)\n", "1. [Introduction to Sparsity I](#Introduction-to-Sparsity-I)\n", - " * [Denoising Example](#Denoising Example)\n", + " * [Denoising Example](#Denoising-Example)\n", "1. [Introduction to Sparsity II: Compressed Sensing](#Introduction-to-Sparsity-II:-Compressed-Sensing)\n", " * [1D Example](#1D-Example)\n", " * [2D Exercise](#2D-Exercise)\n", "1. [Introduction to Sparsity III: Deconvolution](#Introduction-to-Sparsity-II:-Deconvolution)\n", - " * [Deconvolution Exercise](#Deconvolution-Exercise)" + " * [Deconvolution Exercise](#Deconvolution-Exercise)\n", + "1. [Introduction to Wavelets](#Introduction-to-Wavelets)\n", + " * [Starlet Transform](#Starlet-Transform)" ] }, { @@ -77,7 +81,7 @@ "lamdba_val = 3 * sigma_mad(y_sparse) \n", "\n", "# Perform thresholding.\n", - "x_rec = ifft(hard_thresh(y_sparse, lamda_val))\n", + "x_rec = ifft(hard_thresh(y_sparse, lambda_val))\n", "```" ] }, @@ -124,7 +128,7 @@ "The gradient and optimisation problem are the same as for the 1D example.\n", "\n", "```Python\n", - "sample_percentage = mask2d[mask2d == 1].size / float(mask2d.size) * 100)\n", + "sample_percentage = mask2d[mask2d == 1].size / float(mask2d.size) * 100\n", "```\n", "The percentage of samples used is: 5.0%\n", "\n", @@ -206,21 +210,21 @@ ], "metadata": { "kernelspec": { - "display_name": "Python 2", + "display_name": "Python 3 (ipykernel)", "language": "python", - "name": "python2" + "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", - "version": 2 + "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", - "pygments_lexer": "ipython2", - "version": "2.7.13" + "pygments_lexer": "ipython3", + "version": "3.10.8" } }, "nbformat": 4, diff --git a/ada/images/cea_logo.png b/images/cea_logo.png similarity index 100% rename from ada/images/cea_logo.png rename to images/cea_logo.png diff --git a/images/cosmostat_logo.png b/images/cosmostat_logo.png new file mode 100644 index 0000000..e9de5e2 Binary files /dev/null and b/images/cosmostat_logo.png differ diff --git a/ada/images/dwt.png b/images/dwt.png similarity index 100% rename from ada/images/dwt.png rename to images/dwt.png diff --git a/ada/images/galaxy_example.npy b/images/galaxy_example.npy similarity index 100% rename from ada/images/galaxy_example.npy rename to images/galaxy_example.npy diff --git a/ada/images/uwt.png b/images/uwt.png similarity index 100% rename from ada/images/uwt.png rename to images/uwt.png diff --git a/ada/inverse_problems_1.ipynb b/inverse_problems_1.ipynb similarity index 50% rename from ada/inverse_problems_1.ipynb rename to inverse_problems_1.ipynb index b3d261b..fb13166 100644 --- a/ada/inverse_problems_1.ipynb +++ b/inverse_problems_1.ipynb @@ -2,19 +2,32 @@ "cells": [ { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "slide" + } + }, "source": [ + "
\n", + "\n", "# Introduction to Inverse Problems\n", "\n", "---\n", "\n", "> Author: Samuel Farrens \n", - "> Year: 2018 \n", + "> Year: 2018 (updated in 2023) \n", "> Email: [samuel.farrens@cea.fr](mailto:samuel.farrens@cea.fr) \n", - "> Website: https://sfarrens.github.io\n", - "\n", - "---\n", - "\n", + "> Website: https://sfarrens.github.io" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "slideshow": { + "slide_type": "skip" + } + }, + "source": [ "## Contents\n", " \n", "1. [Set-Up](#Set-Up)\n", @@ -30,16 +43,17 @@ " * [L2 Norm](#L2-Norm)\n", " * [Linear Regression Application](#Linear-Regression-Application)\n", "1. [Gradient Descent Exercise](#Gradient-Descent-Exercise)\n", - " * [Solutions](#Solutions)\n", "1. [Hints](#Hints)" ] }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "slide" + } + }, "source": [ - "---\n", - "\n", "## Set-Up\n", "\n", "Here we will import a couple of packages that will be needed throughout the notebook. \n", @@ -53,106 +67,209 @@ "metadata": {}, "outputs": [], "source": [ - "# Tell Jupyter to display plots in this notebook.\n", - "%matplotlib inline\n", - "\n", - "# Import the numpy package with the alias np.\n", - "import numpy as np \n", - "\n", - "# Import the pyplot package from matplotlib with the alias plt.\n", - "from matplotlib import pylab\n", - "\n", - "# Import interaction module\n", - "from ipywidgets.widgets import *\n", - "\n", - "# Import tutorial functions.\n", - "from sparsity_tutorial import plot" + "# Uncomment the following lines if you are running this on Google Colab\n", + "# !git clone -b ada https://github.com/CosmoStat/Tutorials.git\n", + "# import sys\n", + "# sys.path.insert(0,'/content/Tutorials')" ] }, { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "fragment" + } + }, "outputs": [], "source": [ - "# For Python 2 users only\n", - "# Uncomment the following commands\n", + "# Import the numpy package with the alias np.\n", + "import numpy as np \n", "\n", - "# Import Python 3 behaviour\n", - "# from __future__ import print_function" + "# Import interactive widgets.\n", + "from ipywidgets.widgets import interact, IntSlider, FloatSlider\n", + "\n", + "# Import tutorial plotting functions.\n", + "from sparsity_tutorial import plot" ] }, { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], + "cell_type": "markdown", + "metadata": { + "slideshow": { + "slide_type": "skip" + } + }, "source": [ - "# Set size of figures.\n", - "pylab.rcParams['figure.figsize'] = (10.0, 8.0)" + "Note that the plotting routines have been abstracted to keep the code cells as simple as possible. You can run\n", + "\n", + "```python\n", + "help(plot)\n", + "```\n", + "\n", + "to see the expected inputs for these routines." ] }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "slide" + } + }, "source": [ - "---\n", - "\n", "## Introduction\n", "\n", - "### Objective\n", - "\n", - "The objective of this notebook is to introduce the concept of an inverse problem. The focus will be on linear inverse problems, which are those that can be modelled using linear algebra.\n", + "### Objective\n", "\n", - "### Inverse Problem Definition\n", + "The objective of this notebook is to introduce the concept of an inverse problem. The focus will be on linear inverse problems, which are those that can be modelled using linear algebra." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, + "source": [ + "### Inverse Problem Definition\n", "\n", "A standard \"forward\" problem is one in which data are obtained from model parameters, *i.e.*\n", "\n", - "$$\\text{Model}\\rightarrow\\text{Data,}$$\n", - "\n", + "$$\\text{Model}\\rightarrow\\text{Data,}$$" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "slideshow": { + "slide_type": "fragment" + } + }, + "source": [ "while an inverse problem is one in which the model parameters are determined from the data, *i.e.*\n", "\n", - "$$\\text{Data}\\rightarrow\\text{Model.}$$\n", - "\n", - "In other words, with an inverse problem one attempts to obtain information about a physical system from observed measurements. This can be very useful as some model parameters can not be measured directly.\n", - "\n", - "Inverse problems have been applied to various topics such as oceanography, weather prediction, astrophysics, medical imaging and geophysics.\n", - "\n", + "$$\\text{Data}\\rightarrow\\text{Model.}$$" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "slideshow": { + "slide_type": "fragment" + } + }, + "source": [ + "In other words, with an inverse problem one attempts to obtain information about a physical system from observed measurements. This can be very useful as some model parameters can not be measured directly." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, + "source": [ + "Inverse problems have been applied to various topics such as oceanography, weather prediction, astrophysics, medical imaging and geophysics." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "slideshow": { + "slide_type": "fragment" + } + }, + "source": [ "See the following links for more information about inverse problems.\n", "* [Wikipedia](https://en.wikipedia.org/wiki/Inverse_problem)\n", - "* [Scatting Ideas](https://cmontalto.wordpress.com/2013/03/08/what-are-inverse-problems/)\n" + "* [Scatting Ideas](https://cmontalto.wordpress.com/2013/03/08/what-are-inverse-problems/)" ] }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "slide" + } + }, "source": [ - "---\n", - "\n", "## Linear Regression\n", "\n", - "One of the simplest and most intuitive applications of an inverse problem is fitting a line to a set of data points. \n", - "\n", - "### Mathematical Representation\n", - "\n", - "We will start by defining some variables. We will use $y \\in \\mathbb{R}^{m}$ to represent data measurements/observations, $a \\in \\mathbb{R}^{n}$ to represent a given model and $H \\in \\mathbb{R}^{m \\times n}$ to represent a matrix of equations that relate the model parameters to the measured data. Now we can pose a linear inverse problem of the following form\n", - "\n", - "$$y = Ha$$\n", - "\n", - "When dealing with forward problems $a$ and $H$ are known and can be used to obtain $y$. For inverse problems $y$ and $H$ are known and can be used to obtain $a$.\n", - "\n", - "> Note: Here we use $a$ for our model vector to avoid confusion with data points on the $x$-axis. Commonly (and in the other tutorials) the model vector will be represented by the variable $x$." + "One of the simplest and most intuitive applications of an inverse problem is fitting a line to a set of data points. " ] }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "source": [ - "### Straight Line Example \n", + "### Mathematical Representation\n", "\n", - "The best place to start is with something that should be intimately familiar to anyone with even the most basic background in mathematics, a straight line.\n", + "We will start by defining some variables. We will use $y \\in \\mathbb{R}^{m}$ to represent data measurements/observations, $a \\in \\mathbb{R}^{n}$ to represent a given model, and $H \\in \\mathbb{R}^{m \\times n}$ to represent a matrix of equations that relate the model parameters to the measured data. " + ] + }, + { + "cell_type": "markdown", + "metadata": { + "slideshow": { + "slide_type": "fragment" + } + }, + "source": [ + "Now we can pose a linear inverse problem of the following form\n", "\n", + "$$y = Ha$$" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "slideshow": { + "slide_type": "fragment" + } + }, + "source": [ + "When dealing with forward problems $a$ and $H$ are known and can be used to obtain $y$. For inverse problems $y$ and $H$ are known and can be used to obtain $a$." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "slideshow": { + "slide_type": "skip" + } + }, + "source": [ + "Note that here we use $a$ for our model vector to avoid confusion with data points on the $x$-axis. Commonly (and in the other tutorials) the model vector will be represented by the variable $x$." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, + "source": [ + "### Straight Line Example \n", "\n", + "The best place to start is with something that should be intimately familiar to anyone with even the most basic background in mathematics, a straight line." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, + "source": [ "#### Equation of a Straight Line\n", "\n", "A simple straight line can be represented with the following well known equation\n", @@ -165,7 +282,11 @@ { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "fragment" + } + }, "outputs": [], "source": [ "# This defines a function called y_func with input variables x, m and b, and returns the values of mx + b.\n", @@ -176,31 +297,55 @@ }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "source": [ "#### Forward Problem\n", "\n", "A straightforward way to approach this problem is, when we have some data on the x-axis\n", "\n", - "$$x = \\begin{bmatrix} 8 & 2 & 11 & 6 & 5 & 4 & 12 & 9 & 6 & 11 \\end{bmatrix}$$\n", + "$$x = \\begin{bmatrix} 8 & 2 & 11 & 6 & 5 & 4 & 12 & 9 & 6 & 1 \\end{bmatrix}$$\n", "\n", "and we know the slope and intercept of the line,\n", "\n", "$$m = -1.1$$\n", "\n", - "$$b = 14.0,$$\n", - "\n", - "to simply plug this information into the equation above to get the corresponding data on the y-axis. To think of the of this in terms of a linear forward problem, however, we need to know that the slope and intercept correspond to our model parameters and the data on the x-axis needs to be converted into a matrix operator. So, a better way of representing our line equation would be:\n", + "$$b = 14,$$\n", "\n", - "$$y = a_0x^0 + a_1x^1$$\n", + "to simply plug this information into the equation above to get the corresponding data on the y-axis. " + ] + }, + { + "cell_type": "markdown", + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, + "source": [ + "To think of the of this in terms of a linear forward problem, however, we need to know that the slope and intercept correspond to our model parameters and the data on the x-axis need to be converted into a matrix operator. So, a better way of representing our line equation would be:\n", "\n", + "$$y = a_0x^0 + a_1x^1$$" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, + "source": [ "Then, in terms of our forward problem formulation ($y=Ha$), \n", "\n", - "$$a = \\begin{bmatrix} a_0 & a_1 \\end{bmatrix}$$\n", + "$$a = \\begin{bmatrix} a_0 & a_1 \\end{bmatrix} = \\begin{bmatrix} 14 & -1.1 \\end{bmatrix}$$\n", "\n", "and\n", "\n", - "$$H = \\begin{bmatrix} \n", + "$$H = \\begin{bmatrix} x^0 & x^1 \\end{bmatrix} = \\begin{bmatrix} \n", "1 & 8 \\\\\n", "1 & 2 \\\\\n", "1 & 11 \\\\\n", @@ -210,22 +355,35 @@ "1 & 12 \\\\\n", "1 & 9 \\\\\n", "1 & 6 \\\\\n", - "1 & 11 \n", - "\\end{bmatrix}$$\n", - "\n", - "We can implement this in python as follows. First we define $a$ and $H$ " + "1 & 1 \n", + "\\end{bmatrix}$$" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, + "source": [ + "We can implement this in Python as follows. First we define $a$ and $H$ " ] }, { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "fragment" + } + }, "outputs": [], "source": [ "# Set points in x-axis.\n", "x = np.array([8, 2, 11, 6, 5, 4, 12, 9, 6, 1])\n", "\n", - "# Define the matrix operator.\n", + "# Define the H matrix operator.\n", "H = np.array([np.ones(x.size), x]).T\n", "\n", "# Set the model values.\n", @@ -234,44 +392,80 @@ }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "fragment" + } + }, "source": [ - "then we can use these values to determine $y$" + "then we can use these values to determine $y$." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "slideshow": { + "slide_type": "skip" + } + }, + "source": [ + "> Note that the `@` operator in the following cell is equivalent to [`np.matmul`](https://numpy.org/doc/stable/reference/generated/numpy.matmul.html) or [`np.dot`](https://numpy.org/doc/stable/reference/generated/numpy.dot.html)." ] }, { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "outputs": [], "source": [ "# Calculate Y.\n", - "y = np.dot(H, a)\n", + "y = H @ a \n", "\n", "# Plot results. \n", - "plot.regression_plot(x, y, np.arange(14), y_func(np.arange(14), a[1], a[0]))" + "plot.regression_plot(data=(x, y), model=(np.arange(14), y_func(np.arange(14), a[1], a[0])))" ] }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "fragment" + } + }, "source": [ - "As our data has been produced using the model the fit is perfect." + "As our data has been produced using the model, the fit is perfect." ] }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "source": [ "#### Inverse Problem\n", "\n", - "For the inverse problem we will assume we already know $y$ and use this to work out $a$, so in other words we already have a set of data points and we want to work out the model parameters of the best fitting line.\n", - "\n", - "$$x = \\begin{bmatrix} 8 & 2 & 11 & 6 & 5 & 4 & 12 & 9 & 6 & 11 \\end{bmatrix}$$\n", + "For the inverse problem we will assume we already know $y$ and use this to work out $a$. So, in other words, we already have a set of data points, $(x, y)$, and we want to work out the model parameters, $a$, of the best fitting line.\n", "\n", - "$$y = \\begin{bmatrix} 3 & 10 & 3 & 6 & 8 & 12 & 1 & 4 & 9 & 14 \\end{bmatrix}$$\n", + "$$x = \\begin{bmatrix} 8 & 2 & 11 & 6 & 5 & 4 & 12 & 9 & 6 & 1 \\end{bmatrix}$$\n", "\n", - "and again\n", + "$$y = \\begin{bmatrix} 3 & 10 & 3 & 6 & 8 & 12 & 1 & 4 & 9 & 14 \\end{bmatrix}$$" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, + "source": [ + "The matrix $H$ remains unchanged\n", "\n", "$$H = \\begin{bmatrix} \n", "1 & 8 \\\\\n", @@ -283,39 +477,85 @@ "1 & 12 \\\\\n", "1 & 9 \\\\\n", "1 & 6 \\\\\n", - "1 & 11 \n", - "\\end{bmatrix}$$\n", - "\n", - "to solve this problem we need to invert $H$ *i.e.*\n", - "\n", - "$$a = H^{-1}y$$\n", - "\n", - "however, $H$ is not a square matrix which means it cannot be inverted directly. So, first we need to multiply by $H^{T}$.\n", + "1 & 1 \n", + "\\end{bmatrix}$$" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, + "source": [ + "but, to solve this problem, we need to invert $H$ *i.e.*\n", "\n", - "> Because $H \\in \\mathbb{R}^{m \\times n}$ and $H^{T} \\in \\mathbb{R}^{n \\times m}$ so $H^{T}H \\in \\mathbb{R}^{n \\times n}$\n", + "$$a = H^{-1}y$$" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "slideshow": { + "slide_type": "fragment" + } + }, + "source": [ + "however, $H$ is not a square matrix, which means it cannot be inverted directly. So, first we need to multiply by $H^{T}$.\n", "\n", + "> Because $H \\in \\mathbb{R}^{m \\times n}$ and $H^{T} \\in \\mathbb{R}^{n \\times m}$ so $H^{T}H \\in \\mathbb{R}^{n \\times n}$" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "slideshow": { + "slide_type": "fragment" + } + }, + "source": [ "So, the solution is\n", "\n", "$$a = (H^TH)^{-1}H^{T}y$$\n", "\n", - "which is often called the Normal Equation. We can implement this in python as follows." + "which is often called the Normal Equation." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, + "source": [ + "We can implement this in python as follows." ] }, { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "fragment" + } + }, "outputs": [], "source": [ "# This function implements the normal equation\n", "def normal_eq(H, y):\n", " \n", - " return np.dot(np.dot(np.linalg.inv(np.dot(H.T, H)), H.T), y)" + " return np.linalg.inv(H.T @ H) @ H.T @ y" ] }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "fragment" + } + }, "source": [ "With this function we can attempt to fit a line to our data." ] @@ -323,7 +563,11 @@ { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "outputs": [], "source": [ "# Set points in y-axis.\n", @@ -335,49 +579,86 @@ "print('a =', a)\n", "\n", "# Plot results.\n", - "plot.regression_plot(x, y, np.arange(14), y_func(np.arange(14), a[1], a[0]))" + "plot.regression_plot(data=(x, y), model=(np.arange(14), y_func(np.arange(14), a[1], a[0])))" ] }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "fragment" + } + }, "source": [ "As you can see we can easily recover a good fit to the data points with only a few lines of code." ] }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "slide" + } + }, "source": [ "## Linear Regression Exercise\n", "\n", - "For this exercise you should apply the techniques learned for fitting a straight line to a set of data points to a new set of data that requires a polynomial fit.\n", - "\n", - "### Equation of a Polynomial Line \n", + "For this exercise you should apply the techniques learned for fitting a straight line to a set of data points to a new set of data that requires a polynomial fit." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "slideshow": { + "slide_type": "fragment" + } + }, + "source": [ + "### Equation of a Polynomial Line \n", "\n", "The expression for a straight line (*i.e.* $k = 1$) generalises to the following form for a $k^{\\textrm{th}}$ degree polynomial:\n", "\n", "$$y = a_0 + a_1x + a_2x^2 + ... + a_kx^k$$\n", "\n", - "where $a_i$ are the polynomial coefficients. We can represent this in Python with the following function:" + "where $a_i$ are the polynomial coefficients." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, + "source": [ + "We can represent this in Python with the following function:" ] }, { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "fragment" + } + }, "outputs": [], "source": [ "# Define the new function y(x) for any kth degree polynomial.\n", - "# This defines a function called y_func2 with input variables x and a, and returns the values of a0 + a1x + a2x^2 + ...\n", + "# This defines a function called y_func2 with input variables x and a, and returns the values \n", + "# of a0 + a1x + a2x^2 + ...\n", "def y_func2(x, a):\n", " \n", - " return sum([(a_i * x ** n) for a_i, n in zip(a, range(a.size))])" + " return sum([(a_i * x ** i) for i, a_i in enumerate(a)])" ] }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "source": [ "The data for this exercise are the following:\n", "\n", @@ -403,37 +684,71 @@ "| 0.85 | 1.265 |\n", "| 0.90 | 1.380 |\n", "| 0.95 | 1.575 |\n", - "| 1.00 | 1.857 |\n", - "\n", + "| 1.00 | 1.857 |" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, + "source": [ "These values have already been defined for you." ] }, { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "fragment" + } + }, "outputs": [], "source": [ "# The values for x and y.\n", "x_k = np.linspace(0.0, 1.0, 21)\n", - "y_k = np.array([0.486, 0.866, 0.944, 1.144, 1.103, 1.202, 1.166, 1.191, 1.124, 1.095, 1.122, 1.102, 1.099, 1.017, \n", - " 1.111, 1.117, 1.152, 1.265, 1.380, 1.575, 1.857])" + "y_k = np.array([\n", + " 0.486, 0.866, 0.944, 1.144, 1.103, 1.202, 1.166, \n", + " 1.191, 1.124, 1.095, 1.122, 1.102, 1.099, 1.017, \n", + " 1.111, 1.117, 1.152, 1.265, 1.380, 1.575, 1.857\n", + "])" ] }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "fragment" + } + }, + "source": [ + "Your job is to define the matrix operator $H$ and find the model parameters $a$ by solving the inverse problem $y=Ha$." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "source": [ - "Your job is to define the matrix operator $H$ and find the model parameters $a$ by solving the inverse problem $y=Ha$.\n", + "### EXERCISE \n", "\n", - "
\n", - "EXERCISE CELL " + "Modify the cell below." ] }, { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "fragment" + } + }, "outputs": [], "source": [ "##############################\n", @@ -449,7 +764,11 @@ }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "source": [ "Now you can test how well your model line fits the data." ] @@ -458,89 +777,206 @@ "cell_type": "code", "execution_count": null, "metadata": { - "scrolled": false + "scrolled": false, + "slideshow": { + "slide_type": "fragment" + } }, "outputs": [], "source": [ "# Display the plot.\n", "if a_k is not None:\n", - " plot.regression_plot(x_k, y_k, x_k, y_func2(x_k, a_k))" + " plot.regression_plot(data=(x_k, y_k), model=(x_k, y_func2(x_k, a_k)))" ] }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "slide" + } + }, "source": [ "## Gradient Descent\n", "\n", - "In the previous examples we used an analytical approach to solve the inverse problem, however for many problems this is not possible (*e.g.* if the matrix $H$ is not invertable).\n", - "\n", - "An alternative is to try an iterative method such as gradient descent, which searches for the local optimum of a function from a given starting position.\n", - "\n", + "In the previous examples we used an analytical approach to solve the inverse problem, however for many problems this is not possible (*e.g.* if the matrix $H$ is not invertable)." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "slideshow": { + "slide_type": "fragment" + } + }, + "source": [ + "An alternative is to try an iterative method such as gradient descent, which searches for the local optimum of a function from a given starting position." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, + "source": [ "For this approach we want to define a convex function that measures the accuracy of a given reconstruction. One way to test the accuracy is to measure the residual, $y-Hx$, for a given estimate of $x$. With a convex function, like the l2-norm, we will be searching for the global minimum of the residual\n", "\n", - "$$F(x) = \\frac{1}{2}\\|Hx-y\\|_2^2$$\n", - "\n", + "$$F(x) = \\frac{1}{2}\\|Hx-y\\|_2^2$$" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "slideshow": { + "slide_type": "fragment" + } + }, + "source": [ "The corresponding gradient is\n", "\n", - "$$\\nabla F(x) = H^{T}(Hx-y)$$\n", - "\n", - "### L2 Norm \n", + "$$\\nabla F(x) = H^{T}(Hx-y)$$" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, + "source": [ + "### L2 Norm \n", "\n", "The l2 or Euclidian norm is calculated as follows:\n", "\n", "$$\\|x\\|_2 = \\Big(\\sum_{i=1}^n|x_i|^2\\Big)^{\\frac{1}{2}}$$\n", "\n", - "We can show that this function is convex as follows. \n", - "\n", - "
\n", - "INTERACTIVE CELL" + "We can show that this function is convex as follows. First we define a range of values, then we calculate $\\|x\\|_2$ and $\\|x\\|_2^2$." ] }, { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "fragment" + } + }, + "outputs": [], + "source": [ + "# Set a range of x values\n", + "values = np.around(np.linspace(-1, 1, 101), 2)\n", + "# Calculate the l2-norm of x\n", + "norm = np.array([np.linalg.norm(value) for value in values])\n", + "# Calculate the squared l2-norm of x\n", + "norm2 = norm ** 2" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "slideshow": { + "slide_type": "fragment" + } + }, + "source": [ + "Now let's plot these functions." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, + "source": [ + "#### INTERACTIVE CELL" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "slideshow": { + "slide_type": "fragment" + } + }, "outputs": [], "source": [ "######################################\n", "# YOU SHOULD INTERACT WITH THIS CELL #\n", "######################################\n", - "\n", - "@interact(point=(0, 99, 1))\n", - "def show_grad(point=20):\n", - " \n", - " x = np.linspace(-1, 1, 100)\n", - " y1 = np.array([np.linalg.norm(xi) for xi in x])\n", - " y2 = np.array([np.linalg.norm(xi) ** 2 for xi in x])\n", - " dy = x[point] ** 2 + 2 * x[point] * (x - x[point])\n", - "\n", - " print('Gradient =', 2 * x[point])\n", - "\n", + "@interact(x_val=FloatSlider(value=-0.8, min=-1.0, max=1.0, step=0.2))\n", + "def show_grad(x_val):\n", + " x_val = np.around(x_val, 2)\n", + " # Calculate the gradient of the squared l2-norm \n", + " grad = 2 * x_val\n", + " # Calculate the tangent line at this point\n", + " tangent = x_val ** 2 + 2 * x_val * (values - x_val)\n", " # Display\n", - " plot.grad_plot(x, y1, y2, dy, point)" + " plot.grad_plot(values, norm, norm2, grad, tangent, index=np.where(values == x_val)[0][0])" ] }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "skip" + } + }, "source": [ - "The plot clearly shows that both $\\|x\\|_2$ and $\\|x\\|_2^2$ have a clearly defined global minimum. The plot also displays the gradient of $\\|x\\|_2^2$ at the point $x_{20}$, try ajusting the position of this point and see what happens to the gradient.\n", - "\n", - "Unsurprisingly, as the point apporachs the minimum the gradient tends to zero. Also, depending on which side of the minimum the point is, the gradient will be either positive or negative. Gradient descent captialises on this by iteratively updating solutions in the following way\n", + "The plot shows that both $\\|x\\|_2$ and $\\|x\\|_2^2$ have a clearly defined global minimum. The plot also displays the gradient of $\\|x\\|_2^2$ at the point $x=-0.8$, try ajusting the position of this point and see what happens to the gradient." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, + "source": [ + "Unsurprisingly, as the point apporachs the minimum, the gradient tends to zero. Also, depending on which side of the minimum the point is, the gradient will be either positive or negative." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "slideshow": { + "slide_type": "fragment" + } + }, + "source": [ + "Gradient descent captialises on this by iteratively updating solutions in the following way\n", "\n", "$$x_{n+1} = x_n - \\gamma \\nabla F(x_n)$$\n", "\n", - "where $x_n$ corresponds to the solution at given iteration. So, as the gradient approaches zero and consequenly as $x$ approaches the global minimum the solution will converge. \n", - "\n", + "where $x_n$ corresponds to the solution at given iteration. So, as the gradient approaches zero and consequenly as $x$ approaches the global minimum, the solution will converge. " + ] + }, + { + "cell_type": "markdown", + "metadata": { + "slideshow": { + "slide_type": "fragment" + } + }, + "source": [ "Now, we can try this apporach to out linear regression problem. " ] }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "source": [ - "### Linear Regression Application \n", + "### Linear Regression Application \n", "\n", "The cost function for our problem is\n", "\n", @@ -552,18 +988,26 @@ { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "fragment" + } + }, "outputs": [], "source": [ "# Cost function for linear regression problem\n", "def cost_func(y, a, H):\n", " \n", - " return 0.5 * np.linalg.norm(y - np.dot(H, a)) ** 2" + " return 0.5 * np.linalg.norm(H @ a - y) ** 2" ] }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "source": [ "The gradient of this problem is given by\n", "\n", @@ -575,18 +1019,26 @@ { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "fragment" + } + }, "outputs": [], "source": [ "# Gradient of linear regression problem\n", "def grad(y, a, H):\n", " \n", - " return np.dot(H.T, np.dot(H, a) - y)" + " return H.T @ (H @ a - y)" ] }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "source": [ "The gradient descent algorithm is simply\n", "\n", @@ -598,7 +1050,11 @@ { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "fragment" + } + }, "outputs": [], "source": [ "# Function that performs gradient descent.\n", @@ -607,7 +1063,7 @@ " a_rec = a_guess\n", " cost = []\n", " \n", - " for i in range(n_iter):\n", + " for _ in range(n_iter):\n", " \n", " a_rec = a_rec - gamma * grad(y, a_rec, H)\n", " cost.append(cost_func(y, a_rec, H))\n", @@ -617,19 +1073,34 @@ }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "fragment" + } + }, "source": [ - "We can test this approach on our first straight line example.\n", - "\n", - "
\n", - "INTERACTIVE CELL" + "We can test this approach on our first straight line example." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, + "source": [ + "#### INTERACTIVE CELL" ] }, { "cell_type": "code", "execution_count": null, "metadata": { - "scrolled": false + "scrolled": false, + "slideshow": { + "slide_type": "fragment" + } }, "outputs": [], "source": [ @@ -637,26 +1108,36 @@ "# YOU SHOULD INTERACT WITH THIS CELL #\n", "######################################\n", "\n", + "@interact(n_iter=IntSlider(value=1, min=1, max=1001, step=100))\n", "def test_gradient_descent(n_iter):\n", "\n", " a_new, cost = grad_desc(y, [0, 0], H, grad, cost_func, gamma=0.003, n_iter=n_iter)\n", - " plot.regression_plot(x, y, np.arange(14), y_func(np.arange(14), a_new[1], a_new[0]))\n", - " plot.cost_plot(cost)\n", - "\n", - "iter_slider = IntSlider(value=1, min=1, max=601, step=50)\n", - "interact(test_gradient_descent, n_iter=iter_slider)" + " \n", + " plot.regression_plot_cost(\n", + " data=(x, y), \n", + " model=(np.arange(14), y_func(np.arange(14), a_new[1], a_new[0])), \n", + " cost=cost,\n", + " )" ] }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "fragment" + } + }, "source": [ "You can clearly see how the model parameters are improved after each iteration and as the cost function converges we are left with a good fit to the data points." ] }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "slide" + } + }, "source": [ "## Gradient Descent Exercise\n", "\n", @@ -672,34 +1153,47 @@ }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "source": [ - "### Solutions\n", + "### EXERCISE\n", "\n", - "EXERCISE: provide your solutions in the cell(s) below." + "Provide your solutions in the cell(s) below." ] }, { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "scrolled": false, + "slideshow": { + "slide_type": "fragment" + } + }, "outputs": [], "source": [] }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "skip" + } + }, "source": [ "---\n", "\n", - "## Hints\n", + "## Hints\n", "\n", - "** Linear Regression Exercise **\n", + "**Linear Regression Exercise**\n", "\n", "1. You do not need to redefine the function for the normal equation. \n", "1. The polynomial should have as many degrees as there are good Indiana Jones films. \n", "\n", - "** Gradient Descent Exercise **\n", + "**Gradient Descent Exercise**\n", "\n", "1. Make sure your first guess for $a$ has the right size.\n", "1. If your cost function is increasing you may need to decrease the value of $\\gamma$.\n", @@ -709,7 +1203,11 @@ }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "skip" + } + }, "source": [ "[Open next notebook ->](./sparsity_1.ipynb)" ] @@ -717,7 +1215,7 @@ ], "metadata": { "kernelspec": { - "display_name": "Python 3", + "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, @@ -731,7 +1229,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.7.2" + "version": "3.10.8" } }, "nbformat": 4, diff --git a/low-rank/README.md b/low-rank/README.md deleted file mode 100644 index edaf4af..0000000 --- a/low-rank/README.md +++ /dev/null @@ -1,45 +0,0 @@ -## Low-Rank Approximation ---- - -> Author: [Samuel Farrens](http://www.cosmostat.org/people/sfarrens) -> Year: 2018 -> Email: [samuel.farrens@cea.fr](mailto:samuel.farrens@cea.fr) - -The objective is to provide a beginner level introduction to the concept of low-rank approximation, in particular as a regularisation method for solving linear inverse problems. The tutorial does not provide an in-depth mathematical background nor detailed explanations for every topic. Tutees should supplement this tutorial with further reading of the various references provided in the notebooks for a more comprehensive understanding of the subject. More information about how cutting-edge sparsity techniques are being applied to astrophysical data is available on the CosmoStat website. - -This tutorial is comprised of a Jupyter notebook that demonstrate how the low-rank approximation works as well as showing the applicability of this tools to various simple problems. - -All code blocks are provided in Python (support for >=3.5) and the number of external packages required to run the examples has be kept to a minimum. All of the code has been clearly presented inside the notebook at least once to avoid the use of any "black boxes" for solving the problems presented. - -### Requirements - -In order to run the tutorial notebooks tutees will need to have the following installed: - -* Python (require >=3.5) -* NumPy (recommend >=1.16.2) -* SciPy (recommend >=1.2.1) -* Astropy (recommend >=3.1.2) -* Matplotlib (recommend >=3.0.3) -* Jupyter (recommend >=1.0.0) - -All of the packages listed above can easily be installed using either `pip` or Anaconda. - -*e.g.* - -```bash -$ pip install numpy scipy astropy matplotlib jupyter -``` - -or - -```bash -$ conda install numpy scipy astropy matplotlib jupyter -``` - -### Notebooks - -1. [Introduction to Low-Rank Approximation](https://mybinder.org/v2/gh/CosmoStat/Tutorials/master?filepath=%2Flow-rank%2Flow-rank.ipynb) - -### Acknowledgements - -S. Farrens would like to acknowledge the following people for helpful feedback and ideas for producing this tutorial: Fred Ngolè-Mboula. diff --git a/low-rank/low-rank.ipynb b/low-rank/low-rank.ipynb deleted file mode 100644 index 446b0e2..0000000 --- a/low-rank/low-rank.ipynb +++ /dev/null @@ -1,894 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# Introduction to Low-Rank Approximation\n", - "\n", - "---\n", - "\n", - "> Author: Samuel Farrens \n", - "> Year: 2017 \n", - "> Email: [samuel.farrens@gmail.com](mailto:samuel.farrens@gmail.com) \n", - "> Website: https://sfarrens.github.io\n", - "\n", - "---\n", - "\n", - "\n", - "## Contents\n", - "\n", - "---\n", - " \n", - "1. [Set-Up](#Set-Up)\n", - "1. [Introduction](#Introduction)\n", - "1. [Rank of a Matrix](#Rank-of-a-Matrix)\n", - " * [Full Rank Matrix](#Full-Rank-Matrix) \n", - " * [Reduced Rank Matrix](#Reduced-Rank-Matrix) \n", - " * [Uneven Matrix](#Uneven-Matrix)\n", - "1. [Singular Value Decomposition](#Singular-Value-Decomposition) \n", - " * [Reducing the Rank](#Reducing-the-Rank)\n", - " * [Hard Thresholding](#Hard-Thresholding)\n", - "1. [Low Rank Approximation](#Low-Rank-Approximation)\n", - " * [Nuclear Norm](#Nuclear-Norm)\n", - "1. [Deconvolution Exercise](#Deconvolution-Exercise)\n", - " * [Gradient](#Gradient)\n", - " * [Rank Reduction](#Rank-Reduction)\n", - " * [Optimisation](#Optimisation)\n", - " * [Deconvolution](#Deconvolution)\n", - "1. [Hints](#Hints)\n", - "1. [Exercise Solutions](#Exercise-Solutions)\n", - " * [Deconvolution Exercise Solution](#Deconvolution-Exercise-Solution)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "---\n", - "\n", - "## Set-Up\n", - "\n", - "Here we will import a couple of packages that we will need throughout the notebook." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Tell Jupyter to display plots in this notebook.\n", - "%matplotlib inline\n", - "\n", - "# Import the numpy package with the alias np.\n", - "import numpy as np \n", - "\n", - "# Import the FFT convolution function from Scipy.\n", - "from scipy.signal import fftconvolve\n", - "\n", - "# Import the Gaussian kernel function from Astropy.\n", - "from astropy.convolution import Gaussian2DKernel\n", - "\n", - "# Import the pyplot package from matplotlib with the alias plt.\n", - "import matplotlib.pyplot as plt \n", - "from matplotlib import pylab\n", - "\n", - "# Ignore warnings.\n", - "import warnings\n", - "warnings.filterwarnings(\"ignore\")" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "---\n", - "\n", - "## Introduction\n", - "\n", - "The knowledge that a given matrix contains a certain amount of redundancy can be used a powerful prior for solving inverse problems. The notebook provides a brief introduction to the concepts of matrix rank and the low-rank approximation for regularisation. This notebook does not provide a comprehensive review on these topics, rather it is intended to give the user some intuitive hands-on experience." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "---\n", - "\n", - "## Rank of a Matrix\n", - "\n", - "The rank of a matrix can be defined in the following ways:\n", - "\n", - "1. the maximum number of linearly independent column vectors in a given matrix\n", - "1. the maximum number of linearly independent row vectors in a given matrix\n", - "\n", - "Both of these definitions are equivalent.\n", - "\n", - "### Full Rank Matrix\n", - "\n", - "For the matrix\n", - "\n", - "$$ A = \\begin{bmatrix} 6 & 1 & 2 \\\\ 5 & 6 & 1 \\\\ 5 & 7 & 7 \\end{bmatrix} $$\n", - "\n", - "the rank is given by\n", - "\n", - "$$\\textrm{rank}(A) = 3$$\n", - "\n", - "since all three rows are linearly idependent. $A$ is then said to have **full rank**.\n", - "\n", - "This can be implemented in Python as follows" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "A = np.array([[6, 1, 2], [5, 6, 1], [5, 7, 7]])\n", - "\n", - "print('A =')\n", - "print(A)\n", - "print('')\n", - "print('rank(A) =', np.linalg.matrix_rank(A))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Reduced Rank Matrix\n", - "\n", - "For the matrix\n", - "\n", - "$$ A = \\begin{bmatrix} 1 & 0 & 1 \\\\ -2 & -3 & 1 \\\\ 3 & 3 & 0 \\end{bmatrix} $$\n", - "\n", - "the rank is given by\n", - "\n", - "$$\\textrm{rank}(A) = 2$$\n", - "\n", - "since the third row is simply the difference of the first two, which are linearly idependent. In this case $A$ is **rank deficient** or has **reduced rank**." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "A = np.array([[1, 0, 1], [-2, -3, 1], [3, 3, 0]])\n", - "\n", - "print('A =')\n", - "print(A)\n", - "print('')\n", - "print('rank(A) =', np.linalg.matrix_rank(A))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Uneven Matrix\n", - "\n", - "For a matrix $\\in\\mathbb{R}^{m\\times n}$, if $m>n$ then the maximum rank of the matrix is $n$ (*i.e.* if the matrix has more rows than columns then the maximum value of the rank is the number of columns). Conversely, if $mReducing the Rank\n", - "\n", - "By definition setting one or more singular values to zero reduces the rank of a given matrix.\n", - "\n", - "For example, for a matrix with rank 2" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Define the matrix A\n", - "A = np.array([[7, 3, -4], [-5, 1, 2], [2, 4, -2]])\n", - "\n", - "# Calculate the SVD of A\n", - "U, S, V = np.linalg.svd(A)\n", - "S = np.around(S, 2)\n", - "\n", - "print('A =')\n", - "print(A)\n", - "print('')\n", - "print('Sigma =')\n", - "print(np.diag(S))\n", - "print('')\n", - "print('rank(A) =', np.linalg.matrix_rank(A))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "the rank is reduced to 1 when the second singular value is set to zero." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "#Set the second singular value to 0\n", - "S[1] = 0\n", - "\n", - "# Create a new matrix with the updated singular values\n", - "B = np.dot(U, np.dot(np.diag(S), V))\n", - "\n", - "print('Sigma =')\n", - "print(np.diag(S))\n", - "print('')\n", - "print('B =')\n", - "print(B)\n", - "print('')\n", - "print('rank(B) =', np.linalg.matrix_rank(B))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Hard Thresholding\n", - "\n", - "Another way of looking at this would be apply a **Hard Threshold** to the singular values of the matrix.\n", - "\n", - "$$ \n", - "HT_{\\lambda}(\\mathbf{x}) = \n", - "\\begin{cases} \n", - "\\mathbf{x} & \\text{if}\\ |\\mathbf{x}| \\geq \\lambda \\\\ \n", - "0 & \\text{otherwise} \\\\\n", - "\\end{cases}\n", - "$$" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "def hard_thresh(value, threshold):\n", - " '''Hard Threshold\n", - " \n", - " Returns hard threshold of input\n", - " \n", - " '''\n", - " \n", - " return value * (np.abs(value) >= threshold)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "For example, for a full rank matrix $A$ ($\\textrm{rank}(A)=3$)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Define a matrix of random values with seed 1\n", - "np.random.seed(1)\n", - "A = np.random.randint(-9, 9, (3, 3))\n", - "\n", - "# Calculate the SVD of A\n", - "U, S, V = np.linalg.svd(A)\n", - "S = np.around(S, 2)\n", - "\n", - "print('A =')\n", - "print(A)\n", - "print('')\n", - "print('Sigma =')\n", - "print(np.diag(S))\n", - "print('')\n", - "print('rank(A) =', np.linalg.matrix_rank(A))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "a new matrix $B$ with a rank of 1 can be obtained by applying a hard threshold with $\\lambda=6$ to the singular values of $A$." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Hard threshold values of sigma\n", - "S_ht = hard_thresh(S, 6)\n", - "\n", - "# Create a new matrix with the updated singular values\n", - "B = np.dot(U, np.dot(np.diag(S_ht), V))\n", - "\n", - "print('HT(Sigma) =')\n", - "print(np.diag(S_ht))\n", - "print('')\n", - "print('B =')\n", - "print(B)\n", - "print('')\n", - "print('rank(B) =', np.linalg.matrix_rank(B))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Therefore, with an appropriate threshold, one can reduce the rank of a given matrix." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "---\n", - "\n", - "## Low Rank Approximation\n", - "\n", - "The low-rank approximation is a minimisation constraint that is subject to the condition that the matrix one wishes to obtain has reduced rank. In other words, we can use the prior knowledge that the matrix one wants to recover is of reduced rank when solving an inverse problem.\n", - "\n", - "### Nuclear Norm\n", - "\n", - "As with the $l_0$ norm in sparse regularisation, the rank of the matrix cannot be used directly for regularisation as it is a non-convex function. Instead the nuclear norm (or trace norm) is used to promote low-rank solutions. The nuclear norm is given by the sum of singular values,\n", - "\n", - "$$\\|X\\|_* = \\sum_{k=1} \\sigma_k(X)$$\n", - "\n", - "where $\\sigma_k(X)$ denotes the $k^{\\textrm{th}}$ largest singular value of $X$. This can be implemented in Python as follows" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "def nuclear_norm(data):\n", - " '''Nuclear Norm\n", - " \n", - " This function returns the nuclear norm of the input data.\n", - " \n", - " '''\n", - "\n", - " # Get SVD of the data.\n", - " u, s, v = np.linalg.svd(data)\n", - "\n", - " # Return nuclear norm.\n", - " return np.sum(s)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "For example, given a full rank matrix $A$ and a rank 2 matrix $B$ one can see that the nuclear norm of $B$ is smaller than that of $A$." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "A = np.array([[6, 1, 2], [5, 6, 1], [5, 7, 7]])\n", - "B = np.array([[1, 0, 1], [-2, -3, 1], [3, 3, 0]])\n", - "\n", - "print('rank(A)=', np.linalg.matrix_rank(A))\n", - "print('||A||*=', nuclear_norm(A))\n", - "print('')\n", - "print('rank(B)=', np.linalg.matrix_rank(B))\n", - "print('||B||*=', nuclear_norm(B))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "---\n", - "\n", - "## Deconvolution Exercise\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "This exercise deals with the concept of deconvolution.\n", - "\n", - "We begin by defining a matrix $X$ with rank 1." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "scrolled": false - }, - "outputs": [], - "source": [ - "#############################\n", - "# NO NEED TO EDIT THIS CELL #\n", - "# JUST EXECUTE IT #\n", - "#############################\n", - "\n", - "# Define a 11x11 matrix X\n", - "vals = np.array(list(range(6)) + list(range(5))[::-1]) \n", - "X = np.array([i * vals for i in range(6)] + [i * vals for i in range(5)[::-1]], dtype='float')\n", - "X /= X.sum()\n", - "\n", - "# Calculate the rank of X\n", - "print('rank(X)=', np.linalg.matrix_rank(X))\n", - "\n", - "# Display X\n", - "plt.imshow(X, interpolation='none', cmap='magma')\n", - "plt.colorbar()\n", - "plt.show()" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "A slightly blurred version of $X$ with noise is given by\n", - "\n", - "$$Y = MX + N$$\n", - "\n", - "where $M$ represents the convolution of $X$ with a blurring kernel and $N$ is additive Gaussian noise." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "#############################\n", - "# NO NEED TO EDIT THIS CELL #\n", - "# JUST EXECUTE IT #\n", - "#############################\n", - "\n", - "def blur(data, kernel):\n", - " '''Blurring operator\n", - "\n", - " This function convolves the input data with the blurring kernel.\n", - "\n", - " '''\n", - " \n", - " return fftconvolve(data, kernel, 'same')\n", - "\n", - "def blur_transpose(data, kernel):\n", - " '''Adjoint blurring operator\n", - " \n", - " This function convolves the input data with the rotated blurring kernel.\n", - " \n", - " '''\n", - " \n", - " return fftconvolve(data, np.rot90(kernel, 2), 'same')\n", - "\n", - "# Define a 3x3 Gaussian kernel\n", - "gauss_kernel = np.array(Gaussian2DKernel(0.5, x_size=3, y_size=3))\n", - "\n", - "# Blur X and add noise\n", - "Y = blur(X, gauss_kernel) + 0.003 * np.random.randn(*X.shape)\n", - "\n", - "# Calculate the rank of Y\n", - "print('rank(Y)=', np.linalg.matrix_rank(Y))\n", - "\n", - "# Display X and Y\n", - "plt.figure(figsize=(15, 12))\n", - "plt.subplot(121)\n", - "plt.imshow(X, interpolation='none', cmap='magma')\n", - "plt.title('X', fontsize=20)\n", - "plt.subplot(122)\n", - "plt.imshow(Y, interpolation='none', cmap='magma')\n", - "plt.title('Y', fontsize=20)\n", - "plt.show()" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "$Y$ is clearly distorted with respect to $X$. This sort of distortion is present in virtually all images obtained with optical instruments (*e.g.* telescopes, microscopes, *etc.*).\n", - "\n", - "In order to recover $X$ we will aim to solve the problem using the following minimsation:\n", - "\n", - "$$\\begin{aligned} & \\underset{X}{\\text{argmin}} & \\frac{1}{2}\\|Y-MX\\|_2^2 + \\lambda\\|X\\|_*\\end{aligned}$$\n", - "\n", - "which imposes a low-rank solution of $X$ via the nuclear norm term.\n", - "\n", - "### Gradient\n", - "\n", - "Your first task is to define a function that calculates the gradient of the minimisation problem, which is given by\n", - "\n", - "$$\\nabla F(X) = M^T(MX-Y)$$\n", - "\n", - "> See the [Hints](#Hints) if you get stuck.\n", - "\n", - "
\n", - "EDIT THE CELL BELOW" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "##############################\n", - "# YOU NEED TO EDIT THIS CELL #\n", - "##############################\n", - "\n", - "def grad(y, x_rec, kernel):\n", - " '''Gradient\n", - " \n", - " This function calculates the gradient given the observation and an approximation of the true signal.\n", - " \n", - " '''\n", - "\n", - " return None" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Rank Reduction\n", - "\n", - "Your next task is to define a function for imposing that the recovered matrix has a reduced rank.\n", - "\n", - "To complete this task you will need to:\n", - "1. calculate some measure of the rank of the input data\n", - "1. reduce the this rank using the input threshold\n", - "1. return the updated matrix\n", - "\n", - "> See the [Hints](#Hints) if you get stuck.\n", - "\n", - "
\n", - "EDIT THE CELL BELOW" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "##############################\n", - "# YOU NEED TO EDIT THIS CELL #\n", - "##############################\n", - "\n", - "def reduce_rank(x, threshold):\n", - " '''Reduce Rank\n", - " \n", - " This function reduces the rank of the input value x using the specified threshold.\n", - " \n", - " '''\n", - " \n", - " return None" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Optimisation\n", - "\n", - "For this exercise we will use the forward-backward splitting algorithm with a rank reduction operation to promote low-rank solutions. It has the following form for each iteration:\n", - "\n", - "$$X_{n+1} = X_n - \\gamma * \\nabla F(X_n)$$\n", - "$$X_{n+1} = \\text{RR}_\\lambda(X_{n+1})$$" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "#############################\n", - "# NO NEED TO EDIT THIS CELL #\n", - "# JUST EXECUTE IT #\n", - "#############################\n", - "\n", - "def cost_func(y, x_rec, threshold, kernel):\n", - " '''Cost Function\n", - " \n", - " Return the cost of the given estimate of x\n", - " \n", - " '''\n", - " \n", - " return 0.5 * np.linalg.norm(y - blur(x_rec, kernel)) ** 2 + threshold * nuclear_norm(x_rec)\n", - "\n", - "def forwardBackward(observation, first_guess, threshold, kernel):\n", - " '''Forward-Backward Algorithm\n", - " \n", - " Return the recovered signal using a rank reduction operator\n", - " \n", - " '''\n", - " \n", - " x_rec = first_guess\n", - " \n", - " n_iter = 15\n", - " gamma = 1.0\n", - " \n", - " for i in range(n_iter):\n", - " \n", - " x_temp = x_rec - gamma * grad(observation, x_rec, kernel)\n", - " x_rec = reduce_rank(x_temp, threshold)\n", - " cost = cost_func(observation, x_rec, threshold, kernel)\n", - " print('Iteration', i + 1, 'Cost =', cost)\n", - " \n", - " return x_rec" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Deconvolution\n", - "\n", - "Your final task is to perform the deconvolution.\n", - "\n", - "1. Set the low-rank threshold value ($\\lambda$).\n", - "1. Set an initial value for $X$.\n", - "1. Implement the forward-backward algorithm with the following inputs:\n", - " 1. The blurred noisy matrix.\n", - " 1. The initial value for $X$.\n", - " 1. The threshold.\n", - " 1. The blurring kernel.\n", - "\n", - "> See the [Hints](#Hints) if you get stuck.\n", - "\n", - "
\n", - "EDIT THE CELL BELOW" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "##############################\n", - "# YOU NEED TO EDIT THIS CELL #\n", - "##############################\n", - "\n", - "# Set the threshold value.\n", - "threshold = None\n", - "\n", - "# Set initial guess for X.\n", - "first_guess = None\n", - "\n", - "# Recover the matrix.\n", - "X_rec = None\n", - "\n", - "# Calculate the rank of recoverd matrix.\n", - "rank_x_rec = None\n", - "\n", - "if not isinstance(rank_x_rec, type(None)):\n", - " print('')\n", - " print('rank(X_rec)=', rank_x_rec)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "#############################\n", - "# NO NEED TO EDIT THIS CELL #\n", - "# JUST EXECUTE IT #\n", - "#############################\n", - "\n", - "# Display\n", - "if not isinstance(X_rec, type(None)):\n", - " plt.figure(figsize=(17, 12))\n", - " plt.subplot(131)\n", - " plt.imshow(X, interpolation='none', cmap='magma')\n", - " plt.title('$X$', fontsize=20)\n", - " plt.subplot(132)\n", - " plt.imshow(Y, interpolation='none', cmap='magma')\n", - " plt.title('$Y$', fontsize=20)\n", - " plt.subplot(133)\n", - " plt.imshow(X_rec, interpolation='none', cmap='magma')\n", - " plt.title('$\\hat{X}$', fontsize=20)\n", - " plt.show()" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Does your final $\\hat{X}$ look more like $X$ or $Y$? Try ajusting $\\lambda$ to see if that improves your results." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "---\n", - "\n", - "## Hints\n", - "\n", - "** Gradient **\n", - "\n", - "1. The blurring operators are already defined.\n", - "\n", - "** Rank Reduction **\n", - "\n", - "1. Make sure you are thresholding something *singular*.\n", - "1. Thresholding matrices can be quite *hard*.\n", - "\n", - "** Deconvolution **\n", - "\n", - "1. Make sure your first guess of $X$ is the right shape.\n", - "1. Try adjusting your value of $lambda$\n", - "\n", - "---\n", - "\n", - "## Exercise Solutions\n", - "\n", - "**Warning!** Try your best to solve all of the exercises on your own before checking the answers.\n", - "\n", - "
\n", - "\n", - "### Deconvolution Exercise Solution\n", - "\n", - "** Gradient **\n", - "\n", - "```Python\n", - "\n", - "def grad(y, x_rec, kernel):\n", - " '''Gradient\n", - " \n", - " This function calculates the gradient given the observation and an approximation of the true signal.\n", - " \n", - " '''\n", - "\n", - " return blur_transpose(blur(x_rec, kernel) - y, kernel)\n", - " \n", - "```\n", - "\n", - "** Rank Reduction **\n", - "\n", - "```Python\n", - "\n", - "def reduce_rank(x, threshold):\n", - " '''Reduce Rank\n", - " \n", - " This function reduces the rank of the input value x using the specified threshold.\n", - " \n", - " '''\n", - " \n", - " u, s, v = np.linalg.svd(x)\n", - " \n", - " s_new = hard_thresh(s, threshold)\n", - " \n", - " return np.dot(u, np.dot(np.diag(s_new), v))\n", - "\n", - "```\n", - "\n", - "** Deconvolution **\n", - "\n", - "```Python\n", - "\n", - "# Set the threshold value.\n", - "threshold = 0.1\n", - "\n", - "# Set initial guess for X.\n", - "first_guess = np.ones(Y.shape)\n", - "\n", - "# Recover the matrix.\n", - "X_rec = forwardBackward(Y, first_guess, threshold, gauss_kernel)\n", - "\n", - "# Calculate the rank of recoverd matrix.\n", - "rank_x_rec = np.linalg.matrix_rank(X_rec)\n", - "\n", - "```" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.7.2" - } - }, - "nbformat": 4, - "nbformat_minor": 1 -} diff --git a/python/Answers-to-exercises-2.ipynb b/python/Answers-to-exercises-2.ipynb deleted file mode 100644 index ccddda9..0000000 --- a/python/Answers-to-exercises-2.ipynb +++ /dev/null @@ -1,454 +0,0 @@ -{ - "cells": [ - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "import numpy as np" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "> **Puzzle 3:** Obtain the array `[4., 4., 4.]` from `extended_array` by using slicing and `log`." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "#Hidden answer Puzzle 3:\n", - "#np.log(extended_array[3::4])" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "> **Puzzle 4:** Change all the last two columns by zeros using slicing, np.shape and np.zeros (assuming you don't know the size of the array beforehand)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "#Hidden answer Puzzle 4:\n", - "#array2d[:,-2:]=np.zeros((array2d.shape[0],2))\n", - "#print(array2d)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "> **Exercise 1:** Using broadcasting and in max. 2 lines of code, construct a multiplication table of the numbers from 1 to 10, i.e., where for each column corresponding to 1,2,3,..., the rows correspond to their integer multiples. For 1 to 3, it looks like this:\n", - "\n", - "---\n", - "\n", - "|1|2|3|\n", - "|--|--|--|\n", - "|2|4|6|\n", - "|3|6|9|\n", - "\n", - "> Then, using masking, remove all multiples of 3, in one line of code. So that the result (in this case for 3x3) looks like this:\n", - "\n", - "|1|2|0|\n", - "|--|--|--|\n", - "|2|4|0|\n", - "|0|0|0|\n", - "\n", - "> Finally, compute the sum over each column and write it into a list:\n", - "\n", - " [3,6,0]\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "x1 = np.arange(1,11)\n", - "mult = x1[:,np.newaxis] * x1[None,:]\n", - "print(mult)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "mult[(mult %3 == 0)] = 0\n", - "mult" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "#With list comprehensions\n", - "[np.sum(mult[:,ii]) for ii in range(mult.shape[1])]" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "#With numpy functions\n", - "np.apply_along_axis(np.sum,1, mult)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### **Exercise 2:** \n", - " * Produce a random set of points, following a normal distribution with mean 0 and covariance matrix `cov=[[1, 3.0/5], [3/5.0, 2]]`.\n", - " * Plot the set of points using `plt.scatter(X, Y)`\n", - " * Comment: Write equation in LaTeX $$\\exp(x^2)$$\n", - " * Compute the marginal distributions for the x and y coordinates and using numpy functions, check that they follow a Gaussian distribution when compared to their histograms. (Hints: `plt.hist(points, bins, density=True)`, `np.exp`, `np.sqrt`, `np.std`, `np.mean`). Use 25 bins.\n", - " * Using masking and `np.where` remove all points whose y-coordinate is more than 2-sigma away from the mean.\n", - " * Plot the remaining points in a scatter plot with blue color, with `facecolor='none'` and size `s=200`, together with the old set of points in red color with a transparency of alpha=0.6." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "mean = [0, 0]\n", - "cov = [[1, 3.0/5],\n", - " [3/5.0, 2]]\n", - "X = np.random.multivariate_normal(mean, cov, 1000)\n", - "X.shape" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "%matplotlib inline\n", - "import matplotlib.pyplot as plt\n", - "#import seaborn; seaborn.set() # for plot styling\n", - "\n", - "plt.scatter(X[:, 0], X[:, 1]);" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "xx = np.linspace(-3,3,1000)\n", - "yy = np.empty((2,xx.shape[0]))\n", - "yy[0] = 1/np.sqrt(2*np.pi*(np.std(X[:,0]))**2) * np.exp(-xx**2 / (2*(np.std(X[:,0]))**2))\n", - "yy[1] = 1/np.sqrt(2*np.pi*(np.std(X[:,1]))**2) * np.exp(-xx**2 / (2*(np.std(X[:,1]))**2))" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "plt.hist(X[:,0], 25, density=True)\n", - "plt.plot(xx, yy[0])" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "plt.hist(X[:,1], 25, density=True)\n", - "plt.plot(xx, yy[1])" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "exclude=np.where(np.sqrt(X[:,1]**2) > 2*np.std(X[:,1]))" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "Xnew.shape" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "exclude[0].shape" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "X.shape[0]-(exclude[0].shape[0])" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "Xnew = np.empty((X.shape[0]-(exclude[0].shape[0]), 2))" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "Xnew[:,1] = np.delete(X[:,1], exclude)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "Xnew[:,0] = np.delete(X[:,0], exclude)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "plt.hist(Xnew[:,1], 25, density=True)\n", - "plt.plot(xx, yy[1])" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "plt.scatter(Xnew[:, 0], Xnew[:, 1], color='blue', facecolor='none', s=200);\n", - "plt.scatter(X[:, 0], X[:, 1], color='red', alpha=0.6);" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# Pandas" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "import pandas as pd" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "cr_temp = pd.read_csv('./materials/CR_Temp.csv')\n", - "cr_temp = pd.read_csv('./materials/CR_Temp.csv', index_col=1)\n", - "cr_temp.head()" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "#Remove the 'N' column\n", - "cr_temp = cr_temp.drop(columns='N')\n", - "cr_temp.head()" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "cr_temp.loc[:, 'ENE':'DIC'] ;" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "#Compute the mean along the axis 1, on a sliced subset.\n", - "my_mean = (cr_temp.loc[:, 'ENE':'DIC']).mean(axis=1)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Check if the two mean columns agree.\n", - "np.mean(my_mean - cr_temp.AVERAGE)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "years = pd.date_range('1958', '2017', freq='Y')" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "years\n", - "\n", - "print(years.size)\n", - "\n", - "cr_temp.index.size" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "crr = cr_temp" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "crr['time'] = years" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "crr = crr.set_index('time')\n", - "crr.head()" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "crr.asfreq('10Y')" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "crr.asfreq('10Y').plot()" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "crr.AVERAGE.plot()" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [] - } - ], - "metadata": { - "hide_input": false, - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.7.3" - }, - "toc": { - "base_numbering": 1, - "nav_menu": {}, - "number_sections": true, - "sideBar": true, - "skip_h1_title": false, - "title_cell": "Table of Contents", - "title_sidebar": "Contents", - "toc_cell": false, - "toc_position": {}, - "toc_section_display": true, - "toc_window_display": false - } - }, - "nbformat": 4, - "nbformat_minor": 2 -} diff --git a/python/Answers-to-exercises.ipynb b/python/Answers-to-exercises.ipynb deleted file mode 100644 index 86ddd32..0000000 --- a/python/Answers-to-exercises.ipynb +++ /dev/null @@ -1,496 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# 5 Answers to Exercises\n", - "\n", - "---\n", - "\n", - "> Author: Samuel Farrens \n", - "> Year: 2017 \n", - "> Email: [samuel.farrens@gmail.com](mailto:samuel.farrens@gmail.com) \n", - "> Website: https://sfarrens.github.io\n", - "\n", - "---\n", - "\n", - "This notebook contains some example answers to the exercises.\n", - "\n", - "---\n", - "\n", - "## Contents\n", - "\n", - "1. [Native Python](#Native-Python)\n", - " * [Execise 1.1](#Exercise-1.1)\n", - " * [Execise 1.2](#Exercise-1.2)\n", - " * [Execise 1.3](#Exercise-1.3)\n", - "1. [Numpy](#Numpy)\n", - " * [Execise 2.1](#Exercise-2.1)\n", - " * [Execise 2.2](#Exercise-2.2)\n", - "1. [Matplotlib](#Matplotlib)\n", - " * [Execise 3.1](#Exercise-3.1)\n", - " * [Execise 3.2](#Exercise-3.2)\n", - "1. [Other Packages](#Other-Packages)\n", - " * [Exercise 4.1](#Exercise-4.1)\n", - " * [Exercise 4.2](#Exercise-4.2) \n", - "\n", - "---\n", - "\n", - "## Native Python\n", - "\n", - "### Exercise 1.1\n", - "\n", - "The equation $F = \\frac{Gm_1m_2}{r^2}$ can be implemented as follows" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "def force(m1, m2, r):\n", - " \"\"\"Grvaitational Force\n", - " \n", - " Function to calculate the gravitational attraction between two bodies.\n", - " \n", - " \"\"\"\n", - " \n", - " return 6.674e-11 * m1 * m2 / r ** 2" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "and the force between the Earth and the Moon is given by" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "print(force(5.9742e24, 7.36e22, 384402 * 1e3), 'N')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Exercise 1.2\n", - "\n", - "The variance of the sample $x = [2, 4, 4, 4, 5, 5, 7, 9]$ can be calculated as follows" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Define the list of values \n", - "x = [2, 4, 4, 4, 5, 5, 7, 9]\n", - "\n", - "# Calculate the mean value of x\n", - "mean = sum(x) / len(x)\n", - "\n", - "# Calculate the square differences between each value and the mean\n", - "square_diff = [(i - mean) **2 for i in x]\n", - "\n", - "# Calculate the variance\n", - "variance = sum(square_diff) / len(x)\n", - "\n", - "print('variance =', variance)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "You can check if the value of the variance is in $x$ and count how many times it occurs as follows" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "print('The variance value is in x:', variance in x)\n", - "print('The variance value occurs', x.count(variance), 'times in x.')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Exercise 1.3\n", - "\n", - "You can write a class to convert between degrees and radians as follows" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "class angleConverter(object):\n", - " \"\"\"Angle Converter Class\n", - " \n", - " This class converts angles between degrees and radians.\n", - " \n", - " \"\"\"\n", - " \n", - " def __init__(self):\n", - " \n", - " self.pi = 3.14159\n", - " \n", - " def deg2rad(self, angle):\n", - " \"\"\"Convert input angle from degrees to radians\n", - "\n", - " \"\"\"\n", - " \n", - " return angle * self.pi / 180.0\n", - " \n", - " def rad2deg(self, angle):\n", - " \"\"\"Convert input angle from radians to degrees\n", - "\n", - " \"\"\"\n", - " \n", - " return angle * 180.0 / self.pi" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "and you can implement the class on the angles as follows" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Define an instance of the class\n", - "ac = angleConverter()\n", - "\n", - "# Use the class methods\n", - "print('45 deg is', ac.deg2rad(45), 'radians')\n", - "print('30 deg is', ac.deg2rad(30), 'radians')\n", - "print('135 deg is', ac.deg2rad(135), 'radians')\n", - "print('pi radians is', ac.rad2deg(ac.pi), 'degrees')\n", - "print('2pi/3 radians is', ac.rad2deg(2 * ac.pi / 3), 'degrees')\n", - "print('5pi/6 radians is', ac.rad2deg(5 * ac.pi / 6), 'degrees')" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "collapsed": true - }, - "source": [ - "---\n", - "\n", - "## Numpy\n", - "\n", - "### Exercise 2.1\n", - "\n", - "You can read in the file and determine the standard deviation of the distances of the stars with spectral type M as follows" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Import the numpy package\n", - "import numpy as np\n", - "\n", - "# Read the ASCII file\n", - "data = np.genfromtxt('materials/nearest_stars.txt', names=True, dtype=None, delimiter=',')\n", - "\n", - "# Calculate the standard deviation of the M type stars\n", - "print('The standard deviation is', data[data['Spectral_Type'] == 'M']['Distance'].std())" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Exercise 2.2\n", - "\n", - "You can generate the array of random floats and calculate the mean $l_1$-norm for each 4x4 matrix as follows" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Import the numpy package\n", - "import numpy as np\n", - "\n", - "# Set a random seed of 8\n", - "np.random.seed(8)\n", - "\n", - "# Generate an array of random floats\n", - "x = np.random.ranf((3, 4, 4))\n", - "\n", - "# Calculate the mean l1-norm of each 4x4 matrix\n", - "print('The mean l1-norms are', np.linalg.norm(x, ord=1, axis=(1, 2)))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "---\n", - "\n", - "## Matplotlib\n", - "\n", - "### Exercise 3.1\n", - "\n", - "You can generate an array of Gaussian distributed random numbers and plot a histogram as follows" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# display plot in notebook\n", - "%matplotlib inline\n", - "\n", - "# import matplotlib\n", - "import matplotlib.pyplot as plt\n", - "\n", - "# generate the Gaussian random numbers\n", - "x = np.random.randn(1000)\n", - "\n", - "# set the plot size\n", - "plt.figure(figsize=(8, 6))\n", - "\n", - "# plot the histogram\n", - "plt.hist(x, bins=50)\n", - "plt.title('Gaussian distributed random numbers')\n", - "\n", - "# display the plot\n", - "plt.show()" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Exercise 3.2\n", - "\n", - "You can read the file and plot the earnings vs the ratings as follows" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# display plot in notebook\n", - "%matplotlib inline\n", - "\n", - "# import matplotlib\n", - "import matplotlib.pyplot as plt\n", - "\n", - "# read the data from the text file\n", - "data = np.genfromtxt('materials/marvel_movies.txt', names=True, dtype=None, delimiter=',')\n", - "\n", - "# set the plot size\n", - "plt.figure(figsize=(8, 6))\n", - "\n", - "# plot the rating as function of the earnings\n", - "plt.plot(data['Earnings'], data['Rating'], 'm^')\n", - "plt.xlabel('Earnings in $', fontsize=20)\n", - "plt.ylabel('Rating', fontsize=20)\n", - "plt.title('MCU Movies', fontsize=20)\n", - "plt.xscale('log')\n", - "\n", - "# display the plot\n", - "plt.show()" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Exercise 3.3\n", - "\n", - "You can generate the chequered image as follows" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# display plot in notebook\n", - "%matplotlib inline\n", - "\n", - "# import matplotlib\n", - "import matplotlib.pyplot as plt\n", - "\n", - "# generate an 8x8 array of ones\n", - "x = np.ones((8, 8))\n", - "\n", - "# set alternate elements to 0\n", - "x[::2, ::2] = 0\n", - "x[1::2, 1::2] = 0\n", - "\n", - "# set the plot size\n", - "plt.figure(figsize=(8, 6))\n", - "\n", - "# plot the image\n", - "plt.imshow(x, interpolation='none', cmap='bone')\n", - "\n", - "# display the plot\n", - "plt.show()" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "---\n", - "\n", - "## Other Packages\n", - "\n", - "\n", - "### Exercise 4.1" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# import the necessary packages\n", - "%matplotlib inline\n", - "import matplotlib.pyplot as plt\n", - "from scipy import fftpack\n", - "from scipy.misc import ascent\n", - "from scipy.ndimage import gaussian_filter\n", - "from astropy.convolution import convolve_fft\n", - "\n", - "# set the input image\n", - "image = ascent()\n", - "\n", - "# set the Gaussian kernel\n", - "kernel = np.zeros(image.shape)\n", - "kernel[tuple(np.array(kernel.shape) // 2)] = 1\n", - "kernel = gaussian_filter(kernel, 10)\n", - "\n", - "# define the convolution function\n", - "def convolve(x, y):\n", - " \"\"\"Convolve\n", - " \n", - " This method returns the convolution of x with y.\n", - " \n", - " \"\"\"\n", - " \n", - " return np.real(fftpack.fftshift(fftpack.ifftn(fftpack.fftn(x) * fftpack.fftn(y))))\n", - "\n", - "# convolve the image with the kernel\n", - "convolved_image = convolve(image, kernel)\n", - "\n", - "# convolve the image with the kernel using astropy\n", - "convolved_image_astropy = convolve_fft(image, kernel, boundary='wrap')\n", - "\n", - "# display the images\n", - "plt.figure(figsize=(16, 6))\n", - "plt.subplot(131)\n", - "plt.imshow(image, cmap='gray')\n", - "plt.title('Original Image')\n", - "plt.subplot(132)\n", - "plt.imshow(convolved_image, cmap='gray')\n", - "plt.title('My Convolved Image')\n", - "plt.subplot(133)\n", - "plt.imshow(convolved_image_astropy, cmap='gray')\n", - "plt.title('Astropy Convolved Image')\n", - "plt.show()" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Exercise 4.2" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# define the deconvolution function\n", - "def deconvolve(x, y):\n", - " \"\"\"Deconvolve\n", - " \n", - " This method returns the deconvolution of x by y.\n", - " \n", - " \"\"\"\n", - " \n", - " return np.real(fftpack.fftshift(fftpack.ifftn(fftpack.fftn(x) / fftpack.fftn(y))))\n", - "\n", - "# deconvolve the blurry image\n", - "deconvolved_image = deconvolve(convolved_image, kernel)\n", - "\n", - "# display the images\n", - "plt.figure(figsize=(16, 6))\n", - "plt.subplot(121)\n", - "plt.title('Original Image')\n", - "plt.imshow(image, cmap='gray')\n", - "plt.subplot(122)\n", - "plt.imshow(deconvolved_image, cmap='gray')\n", - "plt.title('Deconvolved Image')\n", - "plt.show()" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.7.2" - } - }, - "nbformat": 4, - "nbformat_minor": 1 -} diff --git a/python/Classes_I.ipynb b/python/Classes_I.ipynb deleted file mode 100644 index c3f9906..0000000 --- a/python/Classes_I.ipynb +++ /dev/null @@ -1,1684 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\"CosmoStat\n", - "
\n", - "
\n", - "
\n", - "
\n", - "\n", - "# The Anatomy of a Python Class (Part I)\n", - "---\n", - "\n", - "> Author: Samuel Farrens \n", - "> Email: samuel.farrens@cea.fr \n", - "> Year: 2019 \n", - "> Version: 1.0\n", - "\n", - "---\n", - "
\n", - "\n", - "Classes are one of the fundamental building blocks of Python and are essential for object-oriented programming. In this tutorial we will explore how classes work and look at tips and tricks for getting the most out of them. By the end you should have, not only a much better understanding of what Python classes are, but also some new ideas for writing better code.\n", - "\n", - "This first notebook presents some introductory and intermidate topics. The focus being on a intimate understanding of how classes work in Python. You are expected to already have a good grasp on functions, dictionaries, decorators and ... . More advanced topics are discussed in the [second notebook](./Classes_II.ipynb). This tutorial is in no way exhaustive and you are encouraged to suppliment your understanding with further reading. To this end, some links to some additional resources are provided.\n", - "\n", - "If you are new to Jupyter notebooks note that cells are executed by pressing SHIFT+ENTER (⇧+ ⏎). See the Jupyter documentation for more details.\n", - "\n", - "
\n", - "\n", - "## Contents\n", - "---\n", - "\n", - "1. [Set-Up](#1-Set-Up)\n", - "1. [Object-Oriented Programming](#2-Object-Oriented-Programming)\n", - " 1. [A Simple Example](#A-Simple-Example)\n", - "1. [Disecting a Class](#3-Disecting-a-Class)\n", - " 1. [The Class Dictionary](#The-Class-Dictionary)\n", - " 1. [Instantiation](#Instantiation)\n", - "1. [Methods](#4-Methods)\n", - " 1. [Functions as Attributes](#Functions-as-Attributes)\n", - " 1. [Static Methdos](#Static-Methods)\n", - " 1. [Class Methods](#Class-Methods)\n", - " 1. [Instance Methods](#Instance-Methods)\n", - " 1. [Mixing Methods](#Mixing-Methods)\n", - "1. [Properties](#5-Properties)\n", - " 1. [Getters and Setters](#Getters-and-Setters)\n", - " 1. [Class Properties](#Class-Properties)\n", - "1. [Operator Overloading](#6-Operator-Overloading)\n", - " 1. [Class String](#Class-String)\n", - " 1. [Class Representation](#Class-Representation)\n", - " 1. [Mathematical Operators](#Mathematical-Operators)\n", - " 1. [Further Options](#Further-Options)\n", - "1. [Exercises](#7-Exercises)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 1 Set-Up\n", - "\n", - "The following cell contains some set-up commands. Be sure to execute this cell before continuing." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Notebook Set-Up Commands\n", - "\n", - "from math import pi, sqrt\n", - "\n", - "def print_error(error):\n", - " \"\"\" Print Error\n", - " \n", - " Function to print exceptions in red.\n", - " \n", - " Parameters\n", - " ----------\n", - " error : string\n", - " Error message\n", - " \n", - " \"\"\"\n", - " print('\\033[1;31m{}\\033[1;m'.format(error))" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "slideshow": { - "slide_type": "-" - } - }, - "source": [ - "## 2 Object-Oriented Programming\n", - "---\n", - "\n", - "Object-Oriented Programming (OOP) is based around the concept of *[objects](https://en.wikipedia.org/wiki/Object_(computer_science))*, which in turn are usually generated by *[classes](https://en.wikipedia.org/wiki/Class_(computer_programming))*. The philosophy of OOP is generally summarised by four core concepts:\n", - "\n", - "* [abstraction](https://en.wikipedia.org/wiki/Abstraction_(computer_science)) - *only showing certain features to the user*\n", - "* [encapsulation](https://en.wikipedia.org/wiki/Encapsulation_(computer_programming)) - *grouping together certain features*\n", - "* [inheritance](https://en.wikipedia.org/wiki/Inheritance_(object-oriented_programming)) - *reusing and extending features*\n", - "* [polymorphism](https://en.wikipedia.org/wiki/Polymorphism_(computer_science)) - *allow objects to take various forms*\n", - "\n", - "Some of the main benfits that OOP provides, which are directly linked to these core concepts are:\n", - "\n", - "* **reusability** - *minimising the amount of copying and pasting of code*\n", - "* **maintainability** - *minimising the amount of code that needs to be changed to fix a bug*\n", - "* **security** - *minimising the amount of misuse by a user*\n", - "\n", - "In this tutorial we will take an in-depth look at Python classes, which should provide you with all of the tools you need to take advantage of OOP. \n", - "\n", - "
\n", - "\n", - "### A Simple Example\n", - "\n", - "Before diving into what a class is and how it works let's take a moment to understand why you would ever need a class in the first place by looking at a very simple example.\n", - "\n", - "Imagine you are working on a project where you need to keep track of various properties of the planets in the Solar System. Specifically, you want to know the radius (in $\\textrm{km}$) at the equator, the mass (in $\\textrm{kg}$) and the gravitational acceleration at the surface (in $\\textrm{ms}^{-2}$). There are obviously many different ways of doing this. The simplest (in terms of coding) would be to define a variable for each parameter." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Set individual parameter values for the Earth\n", - "earth_radius = 6371\n", - "earth_mass = 5.97e24\n", - "earth_gravity = 9.8\n", - "\n", - "# Print the Earth properties\n", - "print('planet Earth')\n", - "print(' - Radius: {}km'.format(earth_radius))\n", - "print(' - Mass: {}kg'.format(earth_mass))\n", - "print(' - Gravity: {}m/s'.format(earth_gravity))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "This approach would require us to define 21 more variables if we extended it for all of the other planets in the Solar System. It's pretty clear that this becomes quite messy and difficult to maintain as the number of parameters increases.\n", - "\n", - "Another approach would be to define a single variable for each planet, and a simple way to store all of the paremeters would be using a list (or a tuple)." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Set list values for the Earth\n", - "earth = [6371, 5.97e24, 9.8]\n", - "\n", - "# Print the Earth properties\n", - "print('planet Earth')\n", - "print(' - Radius: {}km'.format(earth[0]))\n", - "print(' - Mass: {}kg'.format(earth[1]))\n", - "print(' - Gravity: {}m/s'.format(earth[2]))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "This certainly makes things easier, as we only need to define 7 more variables. The problem here is that we now need to keep track of the order of the paremers in the list, which, in addition to be being less readable, also becomes complicated as the number of parameters increases.\n", - "\n", - "A third option would be to use a dictionary. This allows us to keep the simplicty of a list-like structure, with a single variable for each planet, but also allows us to add labels to the parameter values." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Set dictionary keys and values for the Earth\n", - "earth = {'radius': 6371, 'mass': 5.97e24, 'gravity': 9.8}\n", - "\n", - "# Print the Earth properties\n", - "print('planet Earth')\n", - "print(' - Radius: {}km'.format(earth['radius']))\n", - "print(' - Mass: {}kg'.format(earth['mass']))\n", - "print(' - Gravity: {}m/s'.format(earth['gravity']))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "This solution is pretty good but we still need to manually define a dictionary for each planet.\n", - "\n", - "Let's try tackling this problem using a an object-oriented approach. For this solution we can define a generic class for handling planets. Each planet then constitutes an *object* or instance of the class.\n", - "\n", - "> For now don't worry too much about the syntax and content of this class as we will explore this in depth in the following sections." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Define a planet class\n", - "class Planet:\n", - " \n", - " # class variable\n", - " object_type = 'planet'\n", - " \n", - " # class initialiser\n", - " def __init__(self, name, radius, mass, gravity):\n", - " \n", - " # instance variables\n", - " self.name = name\n", - " self.radius = radius\n", - " self.mass = mass\n", - " self.gravity = gravity\n", - " \n", - " # class instance method\n", - " def show(self):\n", - " \n", - " # Print the object properties\n", - " print(self.object_type, self.name)\n", - " print(' - Radius: {}km'.format(self.radius))\n", - " print(' - Mass: {}kg'.format(self.mass))\n", - " print(' - Gravity: {}m/s'.format(self.gravity))\n", - " print()\n", - " \n", - "# create class instances\n", - "earth = Planet('Earth', 6371, 5.97e24, 9.8)\n", - "earth.show()\n", - "\n", - "jupiter = Planet('Jupiter', 69911, 1.89e27, 24.79)\n", - "jupiter.show()\n", - "\n", - "saturn = Planet('Saturn', 58232, 5.68e26, 10.45)\n", - "saturn.show()" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "This approach is a bit more complicated in terms of the amount of code required initially but once the class is defined it is very easy to generate instances for each planet. We can even define properties for all instances (*e.g.* the `object_type`).\n", - "\n", - "Hopefully this simple example has given you a sneak preview of the power of classes. In the following sections we will take a deeper look at how classes work and the various ways in which they can be used. Once you have mastered these concepts you may find that you will want to rethink the way in which you structure your codes moving towards a more object-oriented implementation." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 3 Disecting a Class\n", - "\n", - "---\n", - "\n", - "To get a better grip on what a class is and how it works lets have a look at what is going on \"inside\". We can start by defining a dummy class and looking at its properties.\n", - "\n", - "Classes are defined with the keyword `class`, much like the keyword `def` is used for defining functions. In the following cell we define a class without any attributes (note the use of the null `pass` statement)." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Define a dummy class\n", - "class myClass:\n", - " pass" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "This object clearly has little use, but it is a good starting place to look at the structure of classes in general. Let's look at the `help` for this class." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Print help for dummy class\n", - "help(myClass)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We can see that despite not setting attributes we have two default attributes predefined, namely `__dict__` and `__weakref__`. `__weakref__` is a special attribute that lists *weak references* to the class object (a topic outside the scope of this tutorial). Instead, we will focus on the `__dict__` attribute.\n", - "\n", - "
\n", - "\n", - "### The Class Dictionary\n", - "\n", - "Let's begin by printing the contents and type of `__dict__` for our dummy class." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Print the class dictionary\n", - "print('myClass.__dict__ =', myClass.__dict__)\n", - "print()\n", - "print('__dict__ is of type', type(myClass.__dict__))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "This attribute is a *mappingproxy*, which is a special type of dictionary that does not have a `__setattr__` method (we will come back to this). In addition to the default attributes discussed above we can see:\n", - "\n", - "* `'__doc__'` : The class docstring (defaults to `None`).\n", - "* `'__module__'` : The name of module in which the class was defined.\n", - "\n", - "Now, let's see what happens when we assign a new attribute to the class. Class attributes are accessed and set using a dot (`.`) following the class name.\n", - "\n", - "> Note that any Python object can be assigned as a class attribute. We will come back to this concept later!" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Assign the attribute myattr with value True to the class\n", - "myClass.myattr = True\n", - "\n", - "# Print the class dictionary\n", - "print('myClass.__dict__ =', myClass.__dict__)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We can see that the class dictionary has a new entry with the key `myattr` and corresponding value `True`. We can even demonstrate that this dictionary behaves as any other..." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Print the class dictionary with key 'myattr'\n", - "print('myattr =', myClass.__dict__['myattr'])" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "...except that it does not allow assignment, given the absence of the `__setattr__` method." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Assign a new value to 'myattr' in the class dictionary\n", - "try:\n", - " myClass.__dict__['myattr'] = False\n", - "except Exception as error:\n", - " print_error(error)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Now let's define a new class with some predefined attributes and see what changes." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Define a new dummy class with some attributes\n", - "class myClass:\n", - " \"\"\" This is my class \n", - " \"\"\"\n", - "\n", - " mybool = True\n", - " myint = 1\n", - " myfloat = 1.0\n", - " mystring = 'string'\n", - " \n", - "# Print the class dictionary \n", - "print(myClass.__dict__)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "All of the attributes can be found in the class dictionary. Notice that `__doc__` is now a string. \n", - "\n", - "All of these attributes can be accessed..." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Print the class attributes\n", - "print('mybool =', myClass.mybool)\n", - "print('myint =', myClass.myint)\n", - "print('myfloat =', myClass.myfloat)\n", - "print('mystring =', myClass.mystring)\n", - "print('mydoc =', myClass.__doc__)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "...and modified." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Modify the value of the string attribute\n", - "myClass.mystring = 'my new string'\n", - "\n", - "# Print the string attribute\n", - "print('mystring =', myClass.mystring)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "In summary the class `__dict__` attribute is useful for understanding how class attributes are stored, but should not really be used directly.\n", - "\n", - "
\n", - "\n", - "### Instantiation\n", - "\n", - "While accessing class attributes directly can be useful, in most applications it will be necessary to create a *class instance* (also reffered to simply as *objects*). Every time a class is isntantiated a unique Python class object is created. These objects retain the global class attributes but can also be assigned unique instance attributes.\n", - "\n", - "We can create a class instance by calling the class name followed by `()`." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Create an instance of the class\n", - "myinst = myClass()\n", - "\n", - "# Print the instance dictionary\n", - "print('myinst.__dict__ =', myinst.__dict__)\n", - "print()\n", - "print('__dict__ is of type', type(myinst.__dict__))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Here we see that the instance dictionary is a true dictionary and is empty, which simply means that no *instance attributes* have been set. \n", - "\n", - "However, instances still have access the class attributes." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Print the instance class dictionary\n", - "print(myinst.__class__.__dict__)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We can clearly demonstrate that istances are unique." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Create two instances of the class\n", - "myinst1 = myClass()\n", - "myinst2 = myClass()\n", - "\n", - "# Check if instances are equal\n", - "print('Instances are equal?', myinst1 == myinst2)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "As with the class itself, attributes can be assigned to the instance object. " - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Assign an attribute to the instance\n", - "myinst.newbool = False\n", - "\n", - "# Print the instance and dictionaries\n", - "print('myinst.__dict__ =', myinst.__dict__)\n", - "print()\n", - "print('myClass.__dict__ =', myClass.__dict__)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We can see that the instance attribute has no impact on the class attributes.\n", - "\n", - "In order to preassign instance attributes we will need to define a special initialisation (or constructor) method (`__init__`), which takes the special variable `self` as an argument." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Define a new dummy class with an init method\n", - "class myClass:\n", - " \n", - " # Initialisation method called when instance is created\n", - " def __init__(self):\n", - " \n", - " # Set the instance attribute\n", - " self.myfloat = 3.5\n", - " \n", - "# Create an instance of the class and print the attribute properties\n", - "myinst = myClass()\n", - "print('myinst.__dict__ =', myinst.__dict__)\n", - "print('myfloat =', myinst.myfloat)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "`self` represents any given class instance and attributes are assigned to it as usual with a dot. Printing the instance dictionary (which is equivalent to `self.__dict__`) reveals the attributes that have been assigned to it.\n", - "\n", - "As with class attributes, instance attribute values can be modified." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "myinst.myfloat = 7.2\n", - "print('myinst.__dict__ =', myinst.__dict__)\n", - "print('myfloat =', myinst.myfloat)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Additional agruments can be passed to `__init__` allowing for a more instance specific initialisation of the class." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Define a new dummy class with a more dynamic init method\n", - "class myClass:\n", - " \n", - " # Pass the argument value when creating an instance\n", - " def __init__(self, value):\n", - " \n", - " # Set the instance attribute\n", - " self.myfloat = value\n", - "\n", - "# Create instances of the class and print the attribute properties\n", - "myinst1 = myClass(3.14)\n", - "myinst2 = myClass(6.28)\n", - "print('myfloat =', myinst1.myfloat)\n", - "print('myfloat =', myinst2.myfloat)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "An important detail to take into account is that instance attributes will override class attributes with the same name, but only for that instance object." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Define a new dummy class with a class attribute\n", - "class myClass:\n", - " \n", - " myfloat = 6.15\n", - " \n", - " def __init__(self):\n", - " pass\n", - "\n", - "# Create 2 instances of the class\n", - "myinst1 = myClass()\n", - "myinst2 = myClass()\n", - "print('myClass.myfloat =', myClass.myfloat)\n", - "print('myinst1.myfloat =', myinst1.myfloat)\n", - "print('myinst1.__dict__', myinst1.__dict__)\n", - "print('myinst2.myfloat =', myinst2.myfloat)\n", - "print('myinst2.__dict__', myinst2.__dict__)\n", - "print()\n", - "\n", - "# Modify the myfloat attribute of the first instance\n", - "myinst1.myfloat = 8.18\n", - "print('myClass.myfloat =', myClass.myfloat)\n", - "print('myinst1.myfloat =', myinst1.myfloat)\n", - "print('myinst1.__dict__', myinst1.__dict__)\n", - "print('myinst2.myfloat =', myinst2.myfloat)\n", - "print('myinst2.__dict__', myinst2.__dict__)\n", - "print()\n", - "\n", - "# Show the first instance class dictionary\n", - "print('myinst1.__class__.__dict__', myinst1.__class__.__dict__)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "By now you should have a pretty clear idea of the differences between a class and a clsss instance. When designing a class it is important to think about which attributes need to be assigned to the class and which to the class instances.\n", - "\n", - "In the following section we will focus on the various types of method attributes.\n", - "\n", - "> Further Reading \n", - "> https://codesachin.wordpress.com/2016/06/09/the-magic-behind-attribute-access-in-python/ \n", - "> https://rushter.com/blog/python-class-internals/" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 4 Methods\n", - "\n", - "---\n", - "\n", - "### Functions as Attributes\n", - "\n", - "As briefly mentioned in the previous section, any Python object can be allocated as class attribute. This of course includes functions. To do so we could define a class and a function and manually assign the function object to the a class attribute as follows." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Define a new dummy class\n", - "class myClass:\n", - " pass\n", - "\n", - "# Define a simple function\n", - "def say_hello():\n", - " print('Hello!')\n", - " \n", - "# Assign the function to the class\n", - "myClass.myfunc = say_hello\n", - "# Execute the function\n", - "myClass.myfunc()\n", - "\n", - "# Print the class dictionary\n", - "print()\n", - "print(myClass.__dict__)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We can see that the class is able to call the function and we can also see the object in the class dictionary.\n", - "\n", - "If we only want the function to exist as a class attribute we can simply define the function inside the class. These functions are called *class methods*.\n", - "\n", - "In the following example we define a calculator class with four methods." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Define a calculator class\n", - "class Calculator:\n", - " \n", - " def add(x, y):\n", - " return x + y\n", - " \n", - " def substract(x, y):\n", - " return x - y\n", - "\n", - " def multiply(x, y):\n", - " return x * y\n", - " \n", - " def divide(x, y):\n", - " return x / y\n", - " \n", - "# Print the class dictionary\n", - "print(Calculator.__dict__)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "These methods work the same as any function..." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "print('1 + 2 =', Calculator.add(1, 2))\n", - "print('5 - 2 =', Calculator.substract(5, 2))\n", - "print('7 * 4 =', Calculator.multiply(7, 4))\n", - "print('6 / 2 =', Calculator.divide(6, 2))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "... but only exist inside the class." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "try:\n", - " print('1 + 2 =', add(1, 2))\n", - "except Exception as error:\n", - " print_error(error)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "While this may be useful for grouping certain methods together we don't gain any new functionality. We could have simply defined the functions in separate module.\n", - "\n", - "
\n", - "\n", - "### Static Methods\n", - "\n", - "If we create an instance of `Calculator` we cannot use its methods, as the instance object is also passed as an argument (we will come back to this)." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Create an instance of the class\n", - "calc = Calculator()\n", - "\n", - "try:\n", - " print(calc.add(1, 2))\n", - "except Exception as error:\n", - " print_error(error)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "To avoid this problem we can declare that the methods are *static*. This can be done using the `@staticmethod` decorator, which essentially tells the method not to expect the instance object as a first argument." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "class staticCalculator:\n", - " \n", - " @staticmethod\n", - " def add(x, y):\n", - " return x + y\n", - " \n", - " @staticmethod\n", - " def substract(x, y):\n", - " return x - y\n", - " \n", - " @staticmethod\n", - " def multiply(x, y):\n", - " return x * y\n", - " \n", - " @staticmethod\n", - " def divide(x, y):\n", - " return x / y\n", - " \n", - "# Print the class dictionary\n", - "print(staticCalculator.__dict__)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We can see that the methods are accessable direcly from the class or from an instance." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Create an instance of the class\n", - "calc = staticCalculator()\n", - "print('1 + 2 =', staticCalculator.add(1, 2))\n", - "print('1 + 2 =', calc.add(1, 2))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "This is an improvement, but we are still not getting much from the class that we could not already get from individual functions." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Class Methods\n", - "\n", - "Similarly to static methods, we can use the `@classmethod` decorator to define a class method. This tells the method to expect the class object (`cls`) as the first argument to a given method rather than an instance object. This means class methods have access to all class attributes.\n", - "\n", - "In the following example we define a class for implementing the Stefan-Boltzmann law. " - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "class StefBoltz:\n", - " \"\"\" The Stefan–Boltzmann law\n", - " \"\"\"\n", - " \n", - " # The Stefan-Boltzmann constant\n", - " sigma = 5.670367e-8 # Wm^−2K^−4\n", - " \n", - " @classmethod\n", - " def luminosity(cls, radius, effective_temp):\n", - " \n", - " return 4 * pi * radius ** 2 * cls.sigma * effective_temp ** 4\n", - " \n", - " @classmethod\n", - " def radius(cls, luminosity, effective_temp):\n", - " \n", - " return sqrt(luminosity / (4 * pi * cls.sigma * effective_temp ** 4))\n", - " \n", - " @classmethod\n", - " def temperature(cls, radius, luminosity):\n", - " \n", - " return (luminosity / (4 * pi * radius ** 2 * cls.sigma)) ** 0.25\n", - " \n", - "# Print the class dictionary\n", - "print(StefBoltz.__dict__)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We can see that all three methods have access to the class attribute `sigma`, which means that this variable does not have be redefined or passed to each method." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "print('The luminosity of the Sun is {:.2e}W'.format(StefBoltz.luminosity(7e8, 5800)))\n", - "print('The radius of Alpha Centauri is {:.2e}m'.format(StefBoltz.radius(4e28, 9700)))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Class methods can also be called by class instances." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "sb = StefBoltz()\n", - "print('The effective temperature of the Earth is {:.2f}K'\n", - " ''.format(sb.temperature(6.37e6, 1.75e+17)))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Finally, class methods can also access static methods and/or other class methods, meaning that a more complicated class structure can be designed that does not require instantitaion." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "class Newton:\n", - " \n", - " @staticmethod\n", - " def velocity(displacement, time):\n", - " return displacement / time\n", - " \n", - " @classmethod\n", - " def acceleration(cls, displacement, time):\n", - " return cls.velocity(displacement, time) / time\n", - " \n", - " @classmethod\n", - " def force(cls, mass, displacement, time):\n", - " return mass * cls.acceleration(displacement, time)\n", - " \n", - "print('Force = {}N'.format(Newton.force(5, 2, 1)))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "While this certainly adds some new functionality, we are still not using class methods to their full potential.\n", - "\n", - "
\n", - "\n", - "### Instance Methods\n", - "\n", - "In the previous section you learned how to instantiate a class using the `self` variable. You also saw that (if we don't add a special decorator) when an instance calls a method it expects an extra agrument. You might have already guessed that this argument is `self` and by defining methods with this argument we can pass instance attributes.\n", - "\n", - "This is the default and arguably most powerful way of using methods. In the following example we demonstrate a very simply class for doubling and tripling an input value.\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "class simpleClass:\n", - " \n", - " def __init__(self, value):\n", - " self.value = value\n", - " \n", - " def double(self):\n", - " return self.value * 2\n", - " \n", - " def triple(self):\n", - " return self.value * 3\n", - "\n", - "# Create a class instance\n", - "sc1 = simpleClass(3)\n", - "sc2 = simpleClass(7)\n", - "print('Instance 1:', sc1.value, sc1.double(), sc1.triple())\n", - "print('Instance 2:', sc2.value, sc2.double(), sc2.triple())" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "This may seem trivial, but this allows us to use methods that are automatically tailored to the class instance. The input value only needs to passed once, meaning the code is easier to debug and a single Python object ends up with a lot of functionality (*i.e.* encapsulation).\n", - "\n", - "
\n", - "\n", - "### Mixing Methods\n", - "\n", - "In the next example we show that various class methods can be combined to provide more complex behaviour." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "class listHandler:\n", - " \"\"\" List Handler\n", - " \n", - " Class for handling list properties.\n", - " \n", - " \"\"\"\n", - " \n", - " def __init__(self, mylist):\n", - " \n", - " self._list = mylist\n", - " self.get_length()\n", - " self.get_first()\n", - " self.get_last()\n", - " \n", - " @staticmethod\n", - " def _get_length(_list):\n", - " \n", - " return len(_list)\n", - " \n", - " @staticmethod\n", - " def _get_element(_list, index):\n", - " \n", - " return _list[index]\n", - " \n", - " @classmethod\n", - " def _get_first(cls, _list):\n", - " \n", - " return cls._get_element(_list, 0)\n", - " \n", - " @classmethod\n", - " def _get_last(cls, _list):\n", - " \n", - " return cls._get_element(_list, -1)\n", - " \n", - " def get_length(self):\n", - " \n", - " self.len = self._get_length(self._list)\n", - " \n", - " def get_first(self):\n", - " \n", - " self.first = self._get_first(self._list)\n", - " \n", - " def get_last(self):\n", - " \n", - " self.last = self._get_last(self._list)\n", - "\n", - "print('listHandler.__dict__', listHandler.__dict__)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "> Note that an underscore is the convention used to denote a \"private\" attribute. These attributes can be accessed in the usual way, but it is assumed that the user should not need to (*i.e.* abstraction)." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "lh = listHandler(list(range(10)))\n", - "print('Length:', lh.len)\n", - "print('First:', lh.first)\n", - "print('Last:', lh.last)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "This is a very complicated way of doing something simple, but it demonstrates how static, class and instance methods can be used together." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 5 Properties\n", - "---\n", - "\n", - "Now that you are familiar with basic class attributes and methods we can look at class properties.\n", - "\n", - "
\n", - "\n", - "### Getters and Setters\n", - "\n", - "As you have already seen, in Python attributes can direcly be assigned to classes or instances. In some cases, however, it may be useful to formally define methods for getting and setting attributes." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "class myClass:\n", - " \n", - " def __init__(self):\n", - " self.myattr = None\n", - " \n", - " def get_attr(self):\n", - " return self.myattr\n", - " \n", - " def set_attr(self, value):\n", - " self.myattr = value\n", - " \n", - "myinst = myClass()\n", - "print('myinst.myattr =', myinst.get_attr())\n", - "myinst.set_attr(5)\n", - "print('myinst.myattr =', myinst.get_attr())" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "This may seem silly, as we could have just as easily done the following to get the same result without defining two additional methods." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "myinst = myClass()\n", - "print('myinst.myattr =', myinst.myattr)\n", - "myinst.myattr = 5\n", - "print('myinst.myattr =', myinst.myattr)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "But now let's assume that we want `myattr` to be a positive integer, but returned as a float and if you attemp to set something else the code will raise an exception." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "class myClass:\n", - " \n", - " def __init__(self):\n", - " self.myattr = 1\n", - " \n", - " def get_attr(self):\n", - " return float(self.myattr)\n", - " \n", - " def set_attr(self, value):\n", - " if not isinstance(value, int) or value < 1:\n", - " raise ValueError('myattr must be a positive integer! '\n", - " 'Input value {} is of type {}.'.format(value, type(value)))\n", - " self.myattr = value\n", - " \n", - "myinst = myClass()\n", - "print('myinst.myattr =', myinst.get_attr())\n", - "myinst.set_attr(5)\n", - "print('myinst.myattr =', myinst.get_attr())" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We can see that everything works as before, but when we try setting a negative integer " - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "try:\n", - " myinst.set_attr(-4)\n", - "except Exception as error:\n", - " print_error(error)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "or a float the code raises an exception." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "try:\n", - " myinst.set_attr(5.6)\n", - "except Exception as error:\n", - " print_error(error)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We can see that getters and setters provide a means of adding conditions to the assignment or retrieval of an attribute.\n", - "\n", - "
\n", - "\n", - "### Class Properties\n", - "\n", - "In the previous example we saw that we could add conditions to the assignment of an attribute using getters and setters, however there is nothing to prevent a user from directly accessing the attribute and assigning an inccorect value. One way to help avoid this is by using the special `property` decorator." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "class myClass:\n", - " \n", - " def __init__(self):\n", - " self._myattr = 1\n", - " \n", - " def get_attr(self):\n", - " return float(self._myattr)\n", - " \n", - " def set_attr(self, value):\n", - " if not isinstance(value, int) or value < 1:\n", - " raise ValueError('myattr must be a positive integer! '\n", - " 'Input value is of type {}.'.format(type(value)))\n", - " self._myattr = value\n", - " \n", - " myattr = property(get_attr, set_attr)\n", - " \n", - "myinst = myClass()\n", - "print('myinst.myattr =', myinst.myattr)\n", - "myinst.myattr = 5\n", - "print('myinst.myattr =', myinst.myattr)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "This allows us to return to the more natural syntax of simply assigning a value to the attribute via the intermediate private attribute `_myattr`.\n", - "\n", - "We can rewrite the same class in the following way for a simpler implementation." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "class myClass:\n", - " \n", - " def __init__(self):\n", - " self._myattr = 1\n", - " \n", - " @property\n", - " def myattr(self):\n", - " return float(self._myattr)\n", - " \n", - " @myattr.setter\n", - " def myattr(self, value):\n", - " if not isinstance(value, int) or value < 1:\n", - " raise ValueError('myattr must be a positive integer! '\n", - " 'Input value {} is of type {}.'.format(value, type(value)))\n", - " self._myattr = value\n", - " \n", - "myinst = myClass()\n", - "print('myinst.myattr =', myinst.myattr)\n", - "myinst.myattr = 5\n", - "print('myinst.myattr =', myinst.myattr)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Which retains all of the same functionality." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "try:\n", - " myinst.myattr = 5.6\n", - "except Exception as error:\n", - " print_error(error)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 6 Operator Overloading\n", - "---\n", - "\n", - "Now that you have mastered the basic properties of classes you may want to add more functionality. One way of doing so is to customise your class by defining some special methods that overload standard Python operators.\n", - "\n", - "
\n", - "\n", - "### Class String\n", - "\n", - "As you are no doubt aware you can print any Python object, however the output is not always user friendly. For example, if we were print a class instance..." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "class myClass:\n", - " pass\n", - "\n", - "print(myClass())" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "... the output tells us the name of the class but little more.\n", - "\n", - "The special `__str__` method can be used to customise this output." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "class Wolf:\n", - " \n", - " def __init__(self, age, name):\n", - " self.age = age\n", - " self.name = name\n", - " \n", - " def __str__(self):\n", - " return 'A {} year old wolf named {}'.format(self.age, self.name)\n", - "\n", - "inst1 = Wolf(3, 'Grey Wind')\n", - "inst2 = Wolf(2, 'Nymeria')\n", - "\n", - "print('inst1 =', inst1)\n", - "print('inst2 =', inst2)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "This can be used to provide the user with details of how the class was instantiated.\n", - "\n", - "
\n", - "\n", - "### Class Representation\n", - "\n", - "Similar to `__str__`, `__repr__` can be provided to customise the class representation. This is intended to be more explicit and more for the benefit of developers." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "class Wolf:\n", - " \n", - " def __init__(self, age, name):\n", - " self.age = age\n", - " self.name = name\n", - " \n", - " def __repr__(self):\n", - " return 'Wolf({}, \"{}\")'.format(self.age, self.name)\n", - "\n", - "inst1 = Wolf(3, 'Lady')\n", - "inst2 = Wolf(2, 'Ghost')\n", - "\n", - "print('inst1 =', repr(inst1))\n", - "print('inst2 =', repr(inst2))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "> Note: The representation is what is called simply by typing the object." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "inst1" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Mathematical Operators\n", - "\n", - "In some cases we may need to define ways in which class instances can interact with each other. For example, what happens if we add two instances or multiply them? How can we determine if a given intance is bigger or smaller than another?\n", - "\n", - "This can be occomplished using special methods such as `__add__`, `__mul__`, `__lt__` and `__gt__` (see the [Python data model](https://docs.python.org/3/reference/datamodel.html#emulating-numeric-types) for a more comprehensive list)." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "class Building:\n", - " \n", - " def __init__(self, name, year, height):\n", - " self.name = name\n", - " self.year = year\n", - " self.height = height\n", - " \n", - " def __add__(self, inst):\n", - " \n", - " return self.height + inst.height\n", - " \n", - " def __mul__(self, inst):\n", - " \n", - " return self.height * inst.height\n", - " \n", - " def __lt__(self, inst):\n", - " \n", - " return self.height < inst.height\n", - " \n", - " def __gt__(self, inst):\n", - " \n", - " return self.height > inst.height\n", - "\n", - "inst1 = Building('Burj Khalifa', 2010, 828)\n", - "inst2 = Building('Empire State Building', 1931, 381)\n", - "\n", - "print('inst1.height =', inst1.height)\n", - "print('inst2.height =', inst2.height)\n", - "print('inst1 + inst2 =', inst1 + inst2)\n", - "print('inst1 * inst2 =', inst1 * inst2)\n", - "print('inst1 < inst2 =', inst1 < inst2)\n", - "print('inst1 > inst2 =', inst1 > inst2)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Calling\n", - "\n", - "It is even possible to define a `__call__` method for a class instance. This effectively allows the instance to behave like a function." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "class Greeting:\n", - " \n", - " def __init__(self, greet):\n", - " \n", - " self._greet = greet\n", - " \n", - " def __call__(self, name):\n", - " \n", - " return '{} {}!'.format(self._greet, name)\n", - " \n", - "inst = Greeting('Howdy')\n", - "print(inst('Arnold'))\n", - "print(inst('Sylvester'))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Further Options \n", - "\n", - "This mechanism is quite extensive. You will find there may different ways in which you can customise your class and we can't possibly cover all of them. If, however, you understand the principle you will find it very easy to extrapolate this to other special methods.\n", - "\n", - "Here is an example of some things that can be done." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "class BirthDates:\n", - " \n", - " def __init__(self):\n", - " self._names = []\n", - " self._years = []\n", - " \n", - " def add(self, name, year):\n", - " self._names.append(name)\n", - " self._years.append(year)\n", - " \n", - " # define the length of an instance\n", - " def __len__(self):\n", - " return len(self._names)\n", - " \n", - " # define what the instance contains\n", - " def __contains__(self, name):\n", - " return name in self._names\n", - " \n", - " # define what the indexing an instance returns\n", - " def __getitem__(self, index):\n", - " return (self._names[index], self._years[index])\n", - " \n", - "inst = BirthDates()\n", - "inst.add('Isaac Newton', 1643)\n", - "inst.add('Marie Curie', 1867)\n", - "inst.add('Albert Einstein', 1879)\n", - "\n", - "print('len(inst) =', len(inst))\n", - "print('Isaac Newton' in inst)\n", - "print('inst[1] =', inst[1])" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 7 Exercises\n", - "---\n", - "\n", - "1. Write a class to calculate the distance between two massive objects using using Newton's law of universal gravitation. \n", - " 1. Use it to determine the distance between the Earth and the Moon in kilometres. \n", - " 1. Then use it to determine the distance between the Earth and the Sun in kilometres.\n", - "\n", - "$$F=\\frac{Gm_1m_2}{r^2}$$\n", - "\n", - "\n", - "\n", - "> You may find the following values useful:\n", - "* Gravitational Constant: $G = 6.674\\times 10^{-11}~\\textrm{m}^3 \\textrm{kg}^{-1} \\textrm{s}^{-2}$\n", - "* Mass of the Earth: $M_\\oplus = 5.972\\times 10^{24}~\\textrm{kg}$ \n", - "* Mass of the Moon: $m = 7.348\\times 10^{22}~\\textrm{kg}$ \n", - "* Mass of the Sun: $M_\\odot = 1.989\\times 10^{30}~\\textrm{kg}$\n", - "* Force of attraction between the Earth and the Moon: $F = 1.986\\times 10^{20}~\\textrm{N}$\n", - "* Force of attraction between the Earth and the Sun: $F = 3.6\\times 10^{22}~\\textrm{N}$" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Add your solution here" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "2. Write a class that generates a numpy array of whole numbers from 0 to a given limit and has a method that returns a given metric on this array.\n", - " 1. Use it to return the median of an array of 20 values.\n", - " 1. Then use it to return the standard deviation of an array of 50 values." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Add your solution here" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "3. Write a class that can be used to store galaxy properties, in particular right ascension, declination and redshift. \n", - " 1. Make sure you only permit appropriate values for these parameters. \n", - " 1. Make it such that your class instances can be added and subtracted to create a new instance for which the all attributes are updated. \n", - " 1. Include an appropriate representation for your class instances. \n", - " 1. Finally, create instances to demonstrate that your class works." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Add your solution here" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.7.0" - } - }, - "nbformat": 4, - "nbformat_minor": 2 -} diff --git a/python/Classes_II.ipynb b/python/Classes_II.ipynb deleted file mode 100644 index 0a74d6e..0000000 --- a/python/Classes_II.ipynb +++ /dev/null @@ -1,1038 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\"CosmoStat\n", - "
\n", - "
\n", - "
\n", - "
\n", - "\n", - "# The Anatomy of a Python Class (Part II)\n", - "---\n", - "\n", - "> Author: Samuel Farrens \n", - "> Email: samuel.farrens@cea.fr \n", - "> Year: 2019 \n", - "> Version: 1.0\n", - "\n", - "---\n", - "
\n", - "\n", - "This notebook introduces some more advanced concepts in the manipulation of Python classes. Before starting you should make sure you have completed the [first notebook](./Classes_I.ipynb) or are at least familiar with the topics covered therein. As with the first part, this tutorial is in no way exhaustive and you are encouraged to suppliment your understanding with further reading.\n", - "\n", - "If you are new to Jupyter notebooks note that cells are executed by pressing SHIFT+ENTER (⇧+ ⏎). See the Jupyter documentation for more details.\n", - "\n", - "\n", - "\n", - "## Contents\n", - "\n", - "---\n", - "\n", - "1. [Set-Up](#1-Set-Up)\n", - "1. [Inheritance](#2-Inheritance)\n", - " 1. [Inherting Attributes](#Inherting-Attributes)\n", - " 1. [Overriding](#Overriding)\n", - " 1. [Handling Instantiation](#Handling-Instantiation)\n", - " 1. [Multiple Parents](#Multiple-Parents)\n", - " 1. [Method Resolution Order](#Method-Resolution-Order)\n", - "1. [Composition](#3-Composition)\n", - "1. [Abstract Classes](#4-Abstract-Classes)\n", - " 1. [Abstract Methods](#Abstract-Methods)\n", - " 1. [Abstract Properties](#Abstract-Properties)\n", - "1. [Exercises](#5-Exercises)\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 1 Set-Up\n", - "\n", - "The following cell contains some set-up commands. Be sure to execute this cell before continuing." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Notebook Set-Up Commands\n", - "\n", - "def print_error(error):\n", - " \"\"\" Print Error\n", - " \n", - " Function to print exceptions in red.\n", - " \n", - " Parameters\n", - " ----------\n", - " error : string\n", - " Error message\n", - " \n", - " \"\"\"\n", - " print('\\033[1;31m{}\\033[1;m'.format(error))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 1 Inheritance\n", - "\n", - "---\n", - "\n", - "A powerful property of classes is the ability to inherit attributes and methods from other classes. This helps avoid repetition of code, which makes it easier to develop and maintain. \n", - "\n", - "
\n", - "\n", - "### Inherting Attributes\n", - "\n", - "We will begin by defining a simple class with a single static method that will act as a *parent* class." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Define a parent class\n", - "class Parent:\n", - " \n", - " @staticmethod\n", - " def add(a, b):\n", - " return a + b\n", - "\n", - "# Print the parent dictionary\n", - "print('Parent.__dict__ =', Parent.__dict__)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Note that this class has no special properties, it the same as classes we saw in the previous notebook.\n", - "\n", - "Now we will define a second class, also with a single static function. This class will serve as a *child* class that will inherit from the parent class. This is done by simply passing the parent class name in `()` when defining the child class. " - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Define a child class\n", - "class Child(Parent):\n", - " \n", - " @staticmethod\n", - " def subtract(a, b):\n", - " return a - b\n", - "\n", - "print('Child.__dict__ =', Child.__dict__)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We can see that the contents of the child class dictionary are only what is expected from a normal class definition. Using the special `__bases__` attribute, however, we can see a tuple of classes that the child inherits from." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "print('Child.__bases__ =', Child.__bases__)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Finally, we can demonstrate that the child has indeed inherited new attributes from the parent as follows." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "print('1 + 2 =', Child.add(1, 2))\n", - "print('3 - 2 =', Child.subtract(3, 2))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We can clearly see that `Child` has inhertited the static `add` method from `Parent`.\n", - "\n", - "In the following example, we can see that this works for any type of class attribute." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "class Parent:\n", - " \n", - " x = 1\n", - " \n", - "class Child(Parent):\n", - " \n", - " @classmethod\n", - " def show(cls):\n", - " return print('x =', cls.x)\n", - "\n", - "Child.show()" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Overriding\n", - "\n", - "We saw in the previous notebook that class intance attributes will override class attributes of the same name. The same happens with child attributes with respect to parent attributes. " - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "class Parent:\n", - " \n", - " x = 1\n", - " \n", - "class Child(Parent):\n", - " \n", - " x = 2\n", - " \n", - " @classmethod\n", - " def show(cls):\n", - " return print('x =', cls.x)\n", - "\n", - "Child.show()" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Hierachy\n", - "\n", - "It is possible to define a hierchy of parent classes, the attributes of which will be inheried by a given child class." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "class GrandParent:\n", - " \n", - " x = 1\n", - " name = 'Grandparent'\n", - "\n", - "class Parent(GrandParent):\n", - " \n", - " y = 2\n", - " name = 'Parent'\n", - " \n", - "class Child(Parent):\n", - " \n", - " z = 3\n", - " name = 'Child'\n", - "\n", - "print(Child.name, Child.x, Child.y, Child.z)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Note that overriding will also act hierachically, meaning that the last class in the chain will be given precedence.\n", - "\n", - "
\n", - "\n", - "### Handling Instantiation \n", - "\n", - "If the parent class contains an `__init__` method, this too can be inherited by the child." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "class Parent:\n", - " \n", - " def __init__(self, value):\n", - " self.myattr = value\n", - " \n", - "class Child(Parent):\n", - " \n", - " def show(self):\n", - " return print('myattr =', self.myattr)\n", - " \n", - "Child('A string').show()" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "In fact it also works the other way around." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "class Parent:\n", - " \n", - " def show(self):\n", - " return print('myattr =', self.myattr)\n", - " \n", - "class Child(Parent):\n", - " \n", - " def __init__(self, value):\n", - " self.myattr = value\n", - " \n", - "Child('A string').show()" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "This is because the child will lookup any attributes not listed in its own dictionary in all the parent (or base) classes.\n", - "\n", - "If both classes have an `__init__` method only the child class will be initialised (see [Overriding](#Overriding))." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "class Parent:\n", - " \n", - " def __init__(self):\n", - " self.pval = 'parent value'\n", - " \n", - "class Child(Parent):\n", - " \n", - " def __init__(self):\n", - " self.cval = 'child value'\n", - " \n", - "inst = Child()\n", - "print('cval =', inst.cval)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "try:\n", - " print('pval =', inst.pval)\n", - "except Exception as error:\n", - " print_error(error)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We can fix this by manually instantiating the parent class." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "class Parent:\n", - " \n", - " def __init__(self):\n", - " self.pval = 'parent value'\n", - " \n", - "class Child(Parent):\n", - " \n", - " def __init__(self):\n", - " self.cval = 'child value'\n", - " Parent.__init__(self)\n", - " \n", - "inst = Child()\n", - "print('cval =', inst.cval)\n", - "print('pval =', inst.pval)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Python has a useful shortcut to make this process easier using the [`super`](https://docs.python.org/3/library/functions.html?highlight=super#super)function. This allows us to instantiate the parent without naming it, this particularly useful if the name of the parent class were to change." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "class Parent:\n", - " \n", - " def __init__(self):\n", - " self.pval = 'parent value'\n", - " \n", - "class Child(Parent):\n", - " \n", - " def __init__(self):\n", - " self.cval = 'child value'\n", - " super().__init__()\n", - " \n", - "inst = Child()\n", - "print('cval =', inst.cval)\n", - "print('pval =', inst.pval)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Multiple Parents\n", - "\n", - "It is possible for a child to inherit attributes from multiple parents." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "class Mother:\n", - " \n", - " def __init__(self, value):\n", - " self.mval = value\n", - " \n", - " def show_mother(self):\n", - " print('mother value =', self.mval)\n", - " \n", - "class Father:\n", - " \n", - " def __init__(self, value):\n", - " self.fval = value\n", - " \n", - " def show_father(self):\n", - " print('father value =', self.fval)\n", - " \n", - "class Child(Mother, Father):\n", - " \n", - " def __init__(self, value1, value2, value3):\n", - " self.cval = value3\n", - " Mother.__init__(self, value1)\n", - " Father.__init__(self, value2)\n", - " \n", - " def show_child(self):\n", - " print('child value =', self.cval)\n", - " \n", - "inst = Child(1, 2, 3)\n", - "inst.show_mother()\n", - "inst.show_father()\n", - "inst.show_child()" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Note that the parent classes were explicitly instantiated in the previous example. Attempting the same thing using the `super` function will raise an error." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "class Mother:\n", - " \n", - " def __init__(self, value):\n", - " self.mval = value\n", - " \n", - " def show_mother(self):\n", - " print('mother value =', self.mval)\n", - " \n", - "class Father:\n", - " \n", - " def __init__(self, value):\n", - " self.fval = value\n", - " \n", - " def show_father(self):\n", - " print('father value =', self.fval)\n", - " \n", - "class Child(Mother, Father):\n", - " \n", - " def __init__(self, value1, value2):\n", - " self.cval = value2\n", - " super().__init__(value1)\n", - " \n", - " def show_child(self):\n", - " print('child value =', self.cval)\n", - " \n", - "inst = Child(1, 2)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "try:\n", - " inst.show_mother()\n", - " inst.show_father()\n", - " inst.show_child()\n", - "except Exception as error:\n", - " print_error(error)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "As we can see the `Mother` parent class was properly instantiated, but the `Father` parent class was not. The reasons for this are explained in the following subsection.\n", - "\n", - "> In general, caution should be used when building a class archtecture that requires inheritance for multiple parent classes.\n", - "\n", - "
\n", - "\n", - "### Method Resolution Order\n", - "\n", - "The Method Resolution Order (MRO) dictates the order in which the child class will search the base classes for a given attribute. We can see the MRO of the child class from the previous example using the `__mro__` attribute as follows." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "print('Child.__mro__ =', Child.__mro__)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We can see that the `Mother` class appears before the `Father` class in the MRO, hence the `super` method in `Child` instantiates `Mother`. \n", - "\n", - "In order to make the previous example work, we would need to add another `super` in the `__init__` of the `Mother` class as follows.\n", - "\n", - "> Note: Don't do this in your code!" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "class Mother:\n", - " \n", - " def __init__(self, value):\n", - " self.mval = value\n", - " super().__init__(value)\n", - " \n", - " def show_mother(self):\n", - " print('mother value =', self.mval)\n", - " \n", - "class Father:\n", - " \n", - " def __init__(self, value):\n", - " self.fval = value\n", - " \n", - " def show_father(self):\n", - " print('father value =', self.fval)\n", - " \n", - "class Child(Mother, Father):\n", - " \n", - " def __init__(self, value1, value2):\n", - " self.cval = value2\n", - " super().__init__(value1)\n", - " \n", - " def show_child(self):\n", - " print('child value =', self.cval)\n", - " \n", - "inst = Child(1, 2)\n", - "inst.show_mother()\n", - "inst.show_father()\n", - "inst.show_child()" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "This, however, is not recommended as it adds unncessary ambiguity to the code and makes debugging extremely difficult.\n", - "\n", - "
\n", - "\n", - "## 3 Composition\n", - "---\n", - "\n", - "Inhertitance is not the only way in which a class can access attributes from other classes. This comes back one of main take-away messages of this tutorial, namely that everything in Python is an object and any object can be assigned as an attribute.\n", - "\n", - "This means that a class attribute can actually be an instance of another class." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "class Composer:\n", - " \n", - " def __init__(self):\n", - " self.myattr = 'composer value'\n", - " \n", - "class myClass:\n", - " \n", - " def __init__(self):\n", - " self.comp = Composer()\n", - "\n", - "inst = myClass()\n", - "print(\"myattr =\", inst.comp.myattr)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "This can particularly useful as we can pass classes or class intances to our new class without knowing anything about the other classes. " - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "class Star:\n", - " \n", - " def __init__(self):\n", - " self.whoami = 'I am a star!'\n", - " \n", - "class Galaxy:\n", - " \n", - " def __init__(self):\n", - " self.whoami = 'I am a Galaxy!'\n", - " \n", - "class myClass:\n", - " \n", - " def __init__(self, composer):\n", - " self.comp = composer()\n", - "\n", - "inst1 = myClass(Star)\n", - "inst2 = myClass(Galaxy)\n", - "print(\"whoami =\", inst1.comp.whoami)\n", - "print(\"whoami =\", inst2.comp.whoami)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We can even achieve inheritance-like properties if we know the name of a given attribute that the composer class should have." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "class Star:\n", - " \n", - " def __init__(self):\n", - " self.whoami = 'I am a star!'\n", - " \n", - "class Galaxy:\n", - " \n", - " def __init__(self):\n", - " self.whoami = 'I am a Galaxy!'\n", - " \n", - "class myClass:\n", - " \n", - " def __init__(self, composer):\n", - " self.whoami = composer().whoami\n", - "\n", - "inst1 = myClass(Star)\n", - "inst2 = myClass(Galaxy)\n", - "print(\"whoami =\", inst1.whoami)\n", - "print(\"whoami =\", inst2.whoami)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "> Note: This will break if the composer does not have an attribute called `whoami`.\n", - "\n", - "Composition also alows preinitialised class instances to be passed." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "class Star:\n", - " \n", - " def __init__(self, mag):\n", - " self.mag = mag\n", - " \n", - "class Galaxy:\n", - " \n", - " def __init__(self, mag):\n", - " self.mag = mag\n", - " \n", - "class myClass:\n", - " \n", - " def __init__(self, composer):\n", - " self.comp = composer\n", - " self.mag = self.comp.mag\n", - "\n", - " \n", - "star = Star(11.05)\n", - "inst1 = myClass(star)\n", - "inst2 = myClass(Galaxy(8.2))\n", - "print(\"mag =\", inst1.mag)\n", - "print(\"mag =\", inst2.mag)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "This can be useful, especially if we want to be able to change the original instance attributes." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "inst1.mag = 12.3\n", - "inst1.comp.mag = 13.1\n", - "print(\"inst1 mag =\", inst1.mag)\n", - "print(\"star mag =\", star.mag)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Notice how the `mag` attribute of the instance `star` changed!" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 4 Abstract Classes\n", - "---\n", - "\n", - "Abstract classes are classes that contain abstract methods. So it makes sense to move right along and explain what an abstract method is.\n", - "\n", - "
\n", - "\n", - "### Abstract Methods\n", - "\n", - "An abstract method is a method that is defined but not implemented. This is useful for definining parent classes that impose some conditions on the child." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "from abc import ABC, abstractmethod\n", - "\n", - "class Parent(ABC):\n", - " \n", - " @abstractmethod\n", - " def get_id(self):\n", - " pass\n", - "\n", - "print ('Parent.__dict__', Parent.__dict__)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We can see that the method `get_id` has been defined but does nothing.\n", - "\n", - "We now define a child class that inherits this parent." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "class Child(Parent):\n", - " \n", - " def __init__(self):\n", - " self.whoami = 'I am a child!'" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "try:\n", - " inst = Child()\n", - "except Exception as error:\n", - " print_error(error)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "This raises an error when trying to create an instance. This is because the child class needs to override the abstract method `get_id`. " - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "class Child(Parent):\n", - " \n", - " def __init__(self):\n", - " self.get_id()\n", - " \n", - " def get_id(self):\n", - " self.whoami = 'I am a child!'\n", - " \n", - "inst = Child()\n", - "print(inst.whoami)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "This allows the parent class, which may be written with no knowledge of the properties of the child class, to impose some conditions on what a given child class should do.\n", - "\n", - "
\n", - "\n", - "### Abstract Properties\n", - "\n", - "In the previous notebook we learned how class properties can be defined to impose conditions on the values of certain class attributes. Using Abstract methods we can combine these tools to ensure that a given class has the properties we want.\n", - "\n", - "We start by defining an abstract class with the abstract property `whoami` and its corresponding setter." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "class AstroObject(ABC):\n", - " \n", - " @property\n", - " @abstractmethod\n", - " def whoami(self):\n", - " pass\n", - " \n", - " @whoami.setter\n", - " @abstractmethod\n", - " def whoami(self, value):\n", - " pass" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We can then define a child class that inherits this abstract class." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "class Star(AstroObject):\n", - " \n", - " def __init__(self):\n", - " self.whoami = 'I am a star!'" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "try:\n", - " star = Star()\n", - "except Exception as error:\n", - " print_error(error)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We can see that while a value for the attribute is provided the property is not explicitly defined. This way we can impose that the `Star` class must define a getter and setter for the attribute `whoami`." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "class Star(AstroObject):\n", - " \n", - " def __init__(self):\n", - " self.whoami = 'I am a star!'\n", - " \n", - " @property\n", - " def whoami(self):\n", - " return self._whoami\n", - " \n", - " @whoami.setter\n", - " def whoami(self, value):\n", - " if not isinstance(value, str):\n", - " raise ValueError('whoami must be a string')\n", - " self._whoami = value\n", - "\n", - "star = Star()\n", - "print(\"whoami =\", star.whoami)\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "This in turn could be used in conjunction with what we learned about class composition to ensure that composers have the properties needed to work with the child class." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 5 Exercises\n", - "---\n", - "\n", - "1. Create a parent class and a child class that can identify its progenitor. \n", - " 1. Your parent class should have the class attribute `parent_name` with the value of your choice.\n", - " 1. Your child class should have the attribute `name` with the value of your choice.\n", - " 1. Printing an instance of your child class should contain its name and its parent's name. *e.g.* \n", - " \n", - " ```python\n", - " print(Child('Thor'))\n", - " Thor Odinson\n", - " ```" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Add your solution here" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "2. Define a class that can be initialised with composer classes that have been constrained by an abstract class.\n", - " 1. Define an abstract class called `EarthAttr` that has the abstract method `whatami`, whcih should return a string of your choice.\n", - " 1. Define at least two composer classes (*e.g.* `Moon` and `Core`) that satisfy the requirements of `EarthAttr`.\n", - " 1. Define a class called `Earth` that composes these classes to get the `whatami` attribute.\n", - " 1. Printing an instance of your `Earth` class should include the value of `whatami`. *e.g.*\n", - " \n", - " ```python\n", - " print(Earth(Moon))\n", - " The Earth has a moon!\n", - " ```\n", - " \n", - " 5. Finally, define a final composer class (*e.g.* `Lake`) and demonstrate that it will not instantiate if not correctly constrained by `EarthAttr`. You should get the following error:\n", - " \n", - " ```bash\n", - " 'Cant instantiate abstract class Lake with abstract methods whatami'\n", - " ```" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Add your solution here" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.7.0" - } - }, - "nbformat": 4, - "nbformat_minor": 2 -} diff --git a/python/Intro-2.ipynb b/python/Intro-2.ipynb deleted file mode 100644 index 4414769..0000000 --- a/python/Intro-2.ipynb +++ /dev/null @@ -1,54 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# Introduction to Python \n", - "## Tutorial 2: Intermediate and Advanced Topics\n", - "\n", - "by Samuel Farrens and Santiago Casas\n", - "- 16/07/2019 first version\n", - "\n", - "![](https://www.python.org/static/community_logos/python-logo.png)\n", - "\n", - "## Contents\n", - "\n", - "1. [Pythonic Thinking](./Pythonic.ipynb)\n", - "\n", - "1. [Numpy](./Numpy-Intro.ipynb)\n", - "\n", - "1. [Pandas](./Pandas-Intro.ipynb)\n", - "\n", - "1. [Scipy-Cosmo](./Scipy-Cosmo.ipynb)\n", - "\n", - "1. [The Anatomy of a Python Class: Part I](./Classes_I.ipynb)\n", - "\n", - "1. [The Anatomy of a Python Class: Part II](./Classes_II.ipynb)\n", - "\n", - "1. [Anwers to Exercises](./Answers-to-exercises-2.ipynb)" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.7.3" - } - }, - "nbformat": 4, - "nbformat_minor": 1 -} diff --git a/python/Intro.ipynb b/python/Intro.ipynb deleted file mode 100644 index 796d04d..0000000 --- a/python/Intro.ipynb +++ /dev/null @@ -1,51 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# Introduction to Python\n", - "## Tutorial 1: Beginner Topics\n", - "\n", - "by Samuel Farrens\n", - "- 2019 python 3 revision\n", - "- 2017 first version\n", - "\n", - "![](https://www.python.org/static/community_logos/python-logo.png)\n", - "\n", - "## Contents\n", - "\n", - "1. [Native Python](./Native.ipynb)\n", - "\n", - "1. [Basic Numpy](./Numpy.ipynb)\n", - "\n", - "1. [Matplotlib](./Matplotlib.ipynb)\n", - "\n", - "1. [Other Packages](./Other-Packages.ipynb)\n", - "\n", - "1. [Anwers to Exercises](./Answers-to-exercises.ipynb)" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.7.3" - } - }, - "nbformat": 4, - "nbformat_minor": 1 -} diff --git a/python/Matplotlib.ipynb b/python/Matplotlib.ipynb deleted file mode 100644 index 0b5c58e..0000000 --- a/python/Matplotlib.ipynb +++ /dev/null @@ -1,457 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# 3 Matplotlib\n", - "\n", - "---\n", - "\n", - "> Author: Samuel Farrens \n", - "> Year: 2017 \n", - "> Email: [samuel.farrens@gmail.com](mailto:samuel.farrens@gmail.com) \n", - "> Website: https://sfarrens.github.io\n", - "\n", - "---\n", - "\n", - "To get started with your research you will most likely want to be able to plot your results. After installing numpy, the second package you will want to install is Matplotlib.\n", - "\n", - "As with numpy, there is plenty of documentation and examples on the matplotlib website. Much more than I could put in this notebook, so these examples are just meant to get you started.\n", - "\n", - "---\n", - "\n", - "## Contents\n", - "\n", - "1. [Installation](#Installation)\n", - "1. [Importing Matplotlib](#Importing-Matplotlib)\n", - "1. [Line Plots](#Line-Plots)\n", - "1. [Sub-Plots](#Sub-Plots)\n", - "1. [Histograms](#Histograms)\n", - "1. [Matrix Images](#Matrix-Images)\n", - "1. [Exercises](#Exercises)\n", - " * [Exercise 3.1](#Exercise-3.1)\n", - " * [Exercise 3.2](#Exercise-3.2)\n", - "\n", - "---\n", - "\n", - "## Installation\n", - "\n", - "To install matplotlib simply run the following command in a terminal (providing you have already set up pip)\n", - "\n", - "```bash\n", - "\n", - "$ pip install matplotlib\n", - "\n", - "```" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Importing Matplotlib\n", - "\n", - "To import numpy use the **`import`** command as follows" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "import matplotlib" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "In general, you will not need to use the majority of the vast number of modules available in matplotlib. Therefore, it is convenient to directly import the modules you intend to use such as **`pyplot`** (matplotlib.pyplot)." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "import matplotlib.pyplot as plt" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "**TIP:** When working in a Jupyter notebook like this one, it is very handy to display the plots inside the notebook. This can be done with the following \"magic\" **`inline`** command." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "%matplotlib inline" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - " > **NOTE:** this will only work in a Jupyter (iPython) notebook." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "---\n", - "\n", - "## Line Plots\n", - "\n", - "The most basic command in **`pyplot`** is **`plot()`**, which can be used to plot lines or points from lists or arrays.\n", - "\n", - "For example, to plot $x^2$ as a function of $x$" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# import numpy\n", - "import numpy as np\n", - "\n", - "# create a range of points in x from 0 to 1\n", - "x = np.arange(10) * 0.1\n", - "\n", - "# calculate x^2\n", - "y = x ** 2\n", - "\n", - "# set the figure size\n", - "plt.figure(figsize=(8, 6))\n", - "\n", - "# plot x vs x^2 with a blue (b) line (-)\n", - "plt.plot(x, y, 'b-')\n", - "\n", - "# set the x axis label with a fontsize of 20\n", - "plt.xlabel('$x$', fontsize=20)\n", - "# set the y axis label with a fontsize of 20\n", - "plt.ylabel('$x^2$', fontsize=20)\n", - "\n", - "# display the plot\n", - "plt.show()\n", - "\n", - "# close the plot\n", - "plt.close()" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "In this example the `'b-'` option specifies that the plot shoudl display a blue line. The same data could be plotted as red circles using the option `'ro'` as follows " - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# set the figure size\n", - "plt.figure(figsize=(8, 6))\n", - "\n", - "# plot x vs x^2 with a blue (b) line (-)\n", - "plt.plot(x, y, 'ro')\n", - "\n", - "# set the x axis label with a fontsize of 20\n", - "plt.xlabel('$x$', fontsize=20)\n", - "# set the y axis label with a fontsize of 20\n", - "plt.ylabel('$x^2$', fontsize=20)\n", - "\n", - "# display the plot\n", - "plt.show()\n", - "\n", - "# close the plot\n", - "plt.close()" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "You can also plot multple lines in a single figure." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# set the figure size\n", - "plt.figure(figsize=(8, 6))\n", - "\n", - "# plot x vs x^2 with a blue (b) line (-)\n", - "plt.plot(x, 3 * x ** 2, 'b:', label='$3x^2$')\n", - "plt.plot(x, 4 * x ** 2, 'r--', label='$4x^2$')\n", - "\n", - "# set the x axis label with a fontsize of 20\n", - "plt.xlabel('$x$', fontsize=20)\n", - "# set the y axis label with a fontsize of 20\n", - "plt.ylabel('$y$', fontsize=20)\n", - "\n", - "# display plot legend()\n", - "plt.legend(loc=0, fontsize=20)\n", - "\n", - "# display the plot\n", - "plt.show()\n", - "\n", - "# close the plot\n", - "plt.close()" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "---\n", - "\n", - "## Sub-Plots\n", - "\n", - "The **`subplot()`** command can be used to display mutiple plots simultaneously." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "scrolled": false - }, - "outputs": [], - "source": [ - "# set the figure size\n", - "plt.figure(figsize=(12, 5))\n", - "\n", - "\n", - "# subplot row 1 of 1 column 1 of 2\n", - "plt.subplot(121)\n", - "\n", - "# plot x vs 2x with a blue (b) line (-)\n", - "plt.plot(x, 2 * x, 'm-')\n", - "plt.xlabel('$x$', fontsize=20)\n", - "plt.ylabel('$2x$', fontsize=20)\n", - "\n", - "# subplot row 1 of 1 column 2 of 2\n", - "plt.subplot(122)\n", - "plt.plot(x, 3 * x ** 3, 'g-')\n", - "plt.xlabel('$x$', fontsize=20)\n", - "plt.ylabel('$3x^3$', fontsize=20)\n", - "\n", - "# automatically adjust layout of subplots\n", - "plt.tight_layout()\n", - "\n", - "# display the plot\n", - "plt.show()\n", - "\n", - "# close the plot\n", - "plt.close()" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "---\n", - "\n", - "# Histograms\n", - "\n", - "Histograms can be produced using the **`hist()`** command." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# generate some random data\n", - "x = np.random.randint(0, 9, 100)\n", - "\n", - "# set the figure size\n", - "plt.figure(figsize=(15, 6))\n", - "\n", - "# plot a histogram of the data with 10 bins (bars)\n", - "plt.subplot(121)\n", - "plt.hist(x, bins=10)\n", - "plt.title('Histogram of random values')\n", - "\n", - "# plot a histogram of the data with 10 bins (step)\n", - "plt.subplot(122)\n", - "plt.hist(x, bins=10, histtype='step')\n", - "plt.title('Histogram of random values')\n", - "\n", - "# display the plot\n", - "plt.show()\n", - "\n", - "# close the plot\n", - "plt.close()" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "---\n", - "\n", - "## Matrix Images\n", - "\n", - "The **`imshow()`** command can be used to display the contents of a matrix as an image.\n", - "\n", - "For example, to create the Swedish flag you could do the following." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# generate and array of zeros\n", - "x = np.zeros((9, 15))\n", - "\n", - "# set certain values to one\n", - "x[4] = 1\n", - "x[:, 5] = 1\n", - "\n", - "# set the figure size\n", - "plt.figure(figsize=(12, 6))\n", - "\n", - "# plot the matrix image\n", - "plt.imshow(x, interpolation='none', cmap='plasma')\n", - "\n", - "# display the plot\n", - "plt.show()\n", - "\n", - "# close the plot\n", - "plt.close()" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "In this example the **`interpolation`** option specifies that each pixel should displayed independently and the **`cmap`** option sets the colour map." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "---\n", - "\n", - "## Exercises\n", - "\n", - "### Exercise 3.1\n", - "\n", - "Generate an array of 1000 Gaussian distributed random numbers and plot a histogram of the array." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Implement code here" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Exercise 3.2\n", - "\n", - "Read in the data from the file `materials/marvel_movies.txt` and plot the film earnings against the ratings.\n", - "\n", - "> If you have not already done so, you will need to download the [Materials](https://minhaskamal.github.io/DownGit/#/home?url=https://github.com/sfarrens/notebooks/tree/master/Python/materials) directory and unzip it. \n", - "> You can use the following command in a terminal: \n", - "> ``` bash\n", - "> $ unzip materials.zip\n", - "> ``` \n", - "> **NOTE:** you need to place the `Materials` folder in the same directory as this notebook.\n", - "\n", - "- Display the points as magenta triangles\n", - "- Plot the earnings in log scale\n", - "- Make sure you label your axes and title your plot\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Implement code here" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Exercise 3.3\n", - "\n", - "Generate a chequered image (*e.g.* like a chess board).\n", - "\n", - "* It should have $8\\times8$ patches\n", - "* half of the patches should be one colour and the other half a different colour\n", - "* Use and your favourite colour map" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Implement code here" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "You can find some example answers to the exercises [here](./Answers-to-exercises.ipynb).\n", - "\n", - "---\n", - "\n", - "> **Continue to [next topic](./Other-Packages.ipynb)**" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.7.2" - } - }, - "nbformat": 4, - "nbformat_minor": 1 -} diff --git a/python/Native.ipynb b/python/Native.ipynb deleted file mode 100644 index eb3170b..0000000 --- a/python/Native.ipynb +++ /dev/null @@ -1,1578 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# 1 Native Python\n", - "\n", - "---\n", - "\n", - "> Author: Samuel Farrens \n", - "> Year: 2017 \n", - "> Email: [samuel.farrens@gmail.com](mailto:samuel.farrens@gmail.com) \n", - "> Website: https://sfarrens.github.io\n", - "\n", - "---\n", - "\n", - "In the context of Python the term \"native\" refers to the core commands, operators and functions that come with a default installation. The following subsections will cover some of the most essential elements provided in native Python (version 3.5 or higher).\n", - "\n", - "---\n", - "\n", - "## Contents\n", - "\n", - "1. [Comments](#Comments)\n", - "1. [Operators](#Operators)\n", - " * [Mathematical Operators](#Mathematical-Operators)\n", - " * [Logic Operators](#Logic-Operators)\n", - " * [Scientific Notation](#Scientific-Notation)\n", - "1. [Built in Functions](#Built-in-Functions)\n", - " * [Print](#Print)\n", - " * [Variables](#Variables)\n", - " * [Integers and Floats](#Integers-and-Floats)\n", - " * [Strings](#Strings)\n", - " * [Type](#Type)\n", - " * [Boolean](#Boolean)\n", - " * [Absolute Value](#Absolute-Value)\n", - "1. [Arrays](#Arrays)\n", - " * [Lists](#Lists)\n", - " * [List Operations](#List-Operations)\n", - " * [Tuples](#Tuples)\n", - " * [Dictionaries](#Dictionaries)\n", - "1. [Loops](#Loops)\n", - " * [Nested Loops](#Nested-Loops)\n", - " * [List Comprehension](#List-Comprehension)\n", - "1. [Functions](#Functions)\n", - "1. [Classes](#Classes) \n", - " * [Class Type](#Class-Type)\n", - "1. [Reading and Writing Files](#Reading-and-Writing-Files)\n", - "1. [Exercises](#Exercises)\n", - " * [Execise 1.1](#Exercise-1.1)\n", - " * [Execise 1.2](#Exercise-1.2)\n", - " * [Execise 1.3](#Exercise-1.3)\n", - " \n", - "---\n", - "\n", - "## Comments\n", - "\n", - "Reading and understanding coding languages can be challenging especially when reading code written by someone else. One of the most important habits one should try to establish early on (and this goes for any language) is adding clear and detailed comments throughout the code. Python is no exception! This practive will improve the readability and faciliate the understanding of the code. \n", - "\n", - "Commenting in Python is very easy. Every comment line simply needs to begin with a hash (**`#`**)." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# This is an example of a comment." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Alternatively comments can be placed between single or double quotes." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "''' This is another comment. '''\n", - "\"\"\" This is also a comment. \"\"\"" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "This latter approach is often implemented for documenting files (modules), functions (methods) and classes, but this will be discussed later." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "---\n", - "\n", - "## Operators\n", - "\n", - "Native python supports various [mathematical operators](https://docs.python.org/2/reference/lexical_analysis.html#operators) (*e.g.* addition, subtraction, multiplication, division, *etc.*) and [logic operators](https://docs.python.org/2/library/stdtypes.html?highlight=boolean) (*e.g.* and, or, not, *etc.*).\n", - "\n", - "These operators do not require any declaration of variables. In fact, one can easily treat a given Python session like a sophisticated calculator.\n", - "\n", - "### Mathematical Operators\n", - "\n", - "The following table lists the basic mathematical operations in Python. These are the operations that most people will be familiar with and one simply needs to learn the appropriate symbols to perform these operations in Python.\n", - "\n", - "| Operation | Mathematical Operator | Python Operator |\n", - "|:--------------------------:|:---------------------:|:---------------:|\n", - "| Addition | $+$ | + |\n", - "| Subtraction | $-$ | - |\n", - "| Multiplication | $\\times$ | * |\n", - "| Division | $\\div$ | / |\n", - "| Exponentiation | $x^n$ | ** |\n", - "| Equality | $=$ | == |\n", - "| Inequality | $\\neq$ | != |\n", - "| Greater Than | $>$ | > |\n", - "| Less Than | $<$ | < |\n", - "| Greater Than or Equal to | $\\geq$ | >= |\n", - "| Less Than or Equal to | $\\leq$ | <= |\n", - "\n", - "Here are a few examples:\n", - "\n", - "$$1+1=2$$" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "1 + 1" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "$$12\\div4=3$$" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "12 / 4" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "> **NOTE:** integer division returns a float." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "$$3^2=9$$" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "3 ** 2" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "$$2<3$$" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "2 < 3" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "In addition to these standard operations, native Python includes some standard computational operations which are shown in the following table. These operations will be familiar to those with prior knowledge of virtually any programming language.\n", - "\n", - "| Operation | Python Operator |\n", - "|:--------------------------:|:---------------:|\n", - "| Modulo | % |\n", - "| Floor Division | // |\n", - "\n", - "The modulo operator returns the remainder of the division of one number by another. For example, in the following operation:\n", - "\n", - "$$5 \\div 2 = 2.5 = 2 \\text{ remainder } 1$$\n", - "\n", - "the remainder of the division of 5 by 2 is 1. So, in Python the modulo operator will return 1.\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "5 % 2" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The floor division operator, on the other hand, returns the result of the division without the remainder. In other words, only the values before the decimal place are returned. So, for the previous example, the floor division operator will return 2." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "5 // 2" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Logic Operators\n", - "\n", - "In Python logic or boolean operators return values that are either *True* or *False*. The following tables lists the logic operators available in native Python.\n", - "\n", - "| Python Operator |\n", - "|:---------------:|\n", - "| and |\n", - "| or |\n", - "| not |\n", - "\n", - "The **`and`** operator tests whether or not both statements are true." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "True and False" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The **`or`** operator tests whether or not one of the statements is true." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "True or False" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The **`not`** operator reverse the statement it preceedes." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "True and not False" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Scientific Notation\n", - "\n", - "Larger numbers can be inputted using scientific notation with the letter **`e`** followed by the power (n) which represents $10^n$. For example, for $n=5$\n", - "\n", - "$$200000 = 2 \\times 10^5$$" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "2e5 " - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "---\n", - "\n", - "## Built-in Functions\n", - "\n", - "A default Python installation includes some standard [built-in functions](https://docs.python.org/2/library/functions.html). \n", - "\n", - "The following subsections will cover some of the most essential native Python functions.\n", - "\n", - "### Print\n", - "\n", - "The first lesson with any programming language is learing how to print output to the terminal. In python this is done with the **`print`** command. Text (strings of characters) are represented either by single (**`''`**) or double quotes (**`\"\"`**). For example, one can display the standard `\"Hello world!\"` output as follows:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "print('Hello world!')" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "print(\"Hello world again!\")" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Single and double quotes can be combined to display the inner set of quotes in the text." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "print('\"What is hell? I maintain that it is the suffering of being unable to love.\" ― Fyodor Dostoyevsky')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The print command also can be used to output the result of an operation. A comma (**`,`**) is used to separate the elements in a print statement." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "print('1 + 1 =', 1 + 1)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Variables\n", - "\n", - "In python virtually any combination of letters, numbers and undescores can be used to define a variable. The variable must, however, start with a letter. For example to set some variable $x = 1$" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "x = 1\n", - "\n", - "print('x =', x)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The variables can of course also be used in conjunction with operators." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "x = 1\n", - "y = 2\n", - "\n", - "print('x + y =', x + y)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "In the simple examples above using very short variables names such as $x$ and $y$ if perfectly adequate, but it is generally good practice to use longer more expressive variable names in codes in order to avoid ambiguity and faciliate readability. For example," - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "speed_of_light_c = 2.99e8 #m/s\n", - "mass_of_human_m = 79.5 #kg\n", - "\n", - "energy = mass_of_human_m * speed_of_light_c ** 2\n", - "\n", - "print('The energy of this human is', energy, 'Joules.')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Apart from numbers, variables can also be used to reference other Python objects such as strings of text." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "x = 'Bob'\n", - "\n", - "print('Hi, my name is', x)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Variables can be removed using the **`del`** command." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "x = 17\n", - "print('x =', x)\n", - "\n", - "del x\n", - "print('x =', x)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "It is important to note that variables are assigned as pointers to objects." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Integers and Floats\n", - "\n", - "The commands **`int()`** and **`float()`** can be used to convert numbers between integers (`int`) and floating point numbers (`float`, *i.e.* real numbers). Python by default sets any number with a decimal to a float and without a decimal to an int. So, the number $1$, for example, is an integer and the number $1.0$ is a float." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# x is an int\n", - "x = 1\n", - "\n", - "# y is a float\n", - "y = 1.0\n", - "\n", - "# convert x to a float and y to an int\n", - "float(x), int(y)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Strings\n", - "\n", - "As already discussed strings of characters are defined between single or double quotes, however numerical values can also be converted to strings using the command **`str()`**. This is useful because in Python strings can be concatenated using the **`+`** operator." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# x is an int\n", - "x = 7\n", - "\n", - "# convert x to a string and concatenate with other strings\n", - "string = 'Have you ever watched the ' + str(x) + ' Samuari?'\n", - "\n", - "# print the new string\n", - "print(string)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Type\n", - "\n", - "The **`type()`** command can be used to determine the \"type\" of object in question. For example, if a variable has been set and you wish to find out if the object it refers to is an int, a float or a string" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "x = 5.6\n", - "y = 'Bob'\n", - "z = 12\n", - "\n", - "print('x is a', type(x))\n", - "print('y is a', type(y))\n", - "print('z is a', type(z))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Boolean\n", - "\n", - "A boolean is a special data type that can only have one of two possible values **`True`** (1) or **`False`** (0). This data type can be very useful when performing logic operations. The function **`bool()`** can be used to convert the integers values $0$ and $1$ to boolean values." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# x and y are ints\n", - "x = 1\n", - "y = 0\n", - "\n", - "# convert x and y to booleans\n", - "bool(x), bool(y)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The reverse process is also possible." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# x and y are booleans\n", - "x = True\n", - "y = False\n", - "\n", - "# convert x and y to ints\n", - "int(x), int(y)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Absolute Value\n", - "\n", - "The function **`abs()`** simply returns the absolute value of a number." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "x = -16.6\n", - "\n", - "print('The absolute value of', x, 'is', abs(x))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "---\n", - "\n", - "## Arrays \n", - "\n", - "There are various ways to store multiple values in a single object. Some of the most basic types are **`list`**, **`tuple`** and **`dict`**." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "collapsed": true - }, - "source": [ - "### Lists\n", - "\n", - "Lists are mutable (*i.e.* can be changed) objects in Python and can be speficied using square brackets **`[]`** or converted using the **`list()`** command." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# x is a list\n", - "x = [1, 2, 3]\n", - "\n", - "print('x =', x)\n", - "print('This second element of x is', x[1])\n", - "print('')\n", - "\n", - "# change the second element of x\n", - "x[1] = 5\n", - "\n", - "print('This second element of x has been changed to', x[1])\n", - "print('x =', x)" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "collapsed": true - }, - "source": [ - "List (and tuple) objects also allow concatenation." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "a = [1, 2]\n", - "b = [3, 4]\n", - "a + b" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### List Operations\n", - "\n", - "Python has many built in functions for dealing with lists." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "x = [9, 3, 4, 1, 2, 3, 7, 5, 4, 5, 2, 6]\n", - "\n", - "print('x =', x)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "You can check the length of the list with the **`len()`** command." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "print('len(x) =', len(x))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "You can find the maximum value in the list with the **`max()`** command." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "print('max(x) =', max(x))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "You can find the minimum value in the list with the **`min()`** command." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "print('min(x) =', min(x))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "You can sum all of the elements in the list with the **`sum()`** command." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "print('sum(x) =', sum(x))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "You can check if a certain value is in the list with the **`in`** statement." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "print('3 in x:', 3 in x)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "You can also check if a certain value is not in the list with the **`not in`** statement." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "print('8 not in x:', 8 not in x)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "You can add elements to a list using the **`append()`** command." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "x.append(5)\n", - "\n", - "print('x =', x)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "You can find the index of a certain element in the list with the **`index()`** command." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "print('the index of value 7 is', x.index(7))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "You can count the number of times a certain value occurs in a list with the **`count()`** command." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "print('the value 5 occurs', x.count(5), 'times')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "It is possible to create lists of ordered numbers in a given range using the **`range()`** command. For example, to produce a list of numbers between 4 and 11 (*i.e.* the half open interval $[4, 12)$)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "print('integers between 4 and 11 are', list(range(4, 12)))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Nested Lists\n", - "\n", - "Most Python objects can be stored in lists such as strings" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "x = ['this', 'is', 'a', 'list']\n", - "print(x)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "but also lists can be stored in lists" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "x = [[1, 2], [3, 4]]\n", - "\n", - "print('x =', x)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "To index items in a nested list you have to index the lists sequentially. For example, to retrieve the value `3`, you would take the first item (*i.e.* index 0) of the second list (*i.e.* index 1) as follows" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "print('the first item of the second list is', x[1][0])" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Tuples\n", - "\n", - "Tuples are similar to lists and are specified using brackets **`()`** or converted using the **`tuple()`** command." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "x = (1, 2, 3)\n", - "\n", - "print('x =', x)\n", - "print('This second element of x is', x[1])" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Tuples, however, are immutable." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# you cannot change the elements of a tuple\n", - "x[1] = 5" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "It is possible to convert lists to tuples and vice versa." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# x is a list\n", - "x = [1, 2, 3]\n", - "\n", - "print('x is a', type(x))\n", - "print('')\n", - "\n", - "# convert x to a tuple\n", - "x = tuple(x)\n", - "\n", - "print('x is a', type(x))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "If variable values are specified without any brackets and separated by commas then the object will automatically be a tuple" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "x = 1, 2, 3\n", - "\n", - "print('x =', x)\n", - "print('x is a', type(x))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "unless you define several variables at the same time" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "x, y, z = 1, 2, 3\n", - "\n", - "print('x =', x)\n", - "print('x is a', type(x))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Dictionaries\n", - "\n", - "Dictionaries are used for storing *key : value* pairs and are specified using curly brackets **`{}`** or converted using the **`dict()`** command." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "a = {'day' : 15, 'month' : 'March', 'year' : 2017}\n", - "print('This cell was written on the ' + str(a['day']) + 'th of ' + a['month'] + ' ' + str(a['year']) + '.')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "---\n", - "\n", - "## Loops\n", - "\n", - "Loops can be implemented in a fairly familiar way (for those who have some coding experience in another language) using a **`for`** statement." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "a = [1, 2, 3, 4]\n", - "\n", - "for i in range(len(a)):\n", - " print('element', i, 'of a is', a[i])" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Here **`len`** provides the length of the list a and **`range`** produces a list of integers from 0 to 3 (*i.e.* the indices of the list). \n", - "\n", - "> **NOTE:** the indentation after the **`for`** statement line is essential. \n", - "\n", - "A more elegant way of implementing a loop would be to simply iterate through the list. Fortunately Python allows this." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "a = [1, 2, 3, 4]\n", - "\n", - "for value in a:\n", - " print('value =', value)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Here `value` is simply a dummy variable.\n", - "\n", - "> **NOTE:** This also works for tuples.\n", - "\n", - "### Nested Loops\n", - "\n", - "All *code blocks* (begin and end statements) in Python are specified with indentation. For example, to implement a nested loop\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "a = [[1, 2], [3, 4]]\n", - "\n", - "for sublist in a:\n", - " print('sublist =', sublist)\n", - " for value in sublist:\n", - " print('value =', value)\n", - " print('')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "In this example the lines\n", - "\n", - "```Python\n", - "print('sublist =', sublist)\n", - "for value in sublist:\n", - "print('')\n", - "```\n", - "\n", - "all occur within the first **`for`** loop, while the line\n", - "\n", - "```Python\n", - "print('value =', value)\n", - "```\n", - "\n", - "occurs in the second **`for`** loop." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### List Comprehension\n", - "\n", - "Another elegant option provided in Python is list comprehension, which automatically creates a list object after iterating." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "a = [1, 2, 3, 4]\n", - "b = [value ** 2 for value in a]\n", - "print('b =', b)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "In this example the operation iterates through the elements in the list `a`, stores each element in a dummy variable `value` and finally adds the square of the variable to a new list `b`.\n", - "\n", - "You can iterate through multiple lists using the **`zip()`** command." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "a = range(1, 5)\n", - "b = ['a', 'b', 'c', 'd']\n", - "\n", - "c = [i * j for i, j in zip(a, b)]\n", - "\n", - "print(c)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "---\n", - "\n", - "## Functions\n", - "\n", - "Functions are an essential part of any programming language and should be used as often as possible to avoid repeating lines of code (which can lead to bugs).\n", - "\n", - "Functions (or methods) are defined using the **`def`** statement and results are returned with the **`return`** statement. For example, to define a function that adds two values you would do the following" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Here we define a function\n", - "def add(a, b):\n", - " \n", - " return a + b\n", - "\n", - "# Here we implement the function\n", - "print('3 plus 5 is', add(3, 5))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "> **NOTE:** as with loops, the indentation after the **`def`** statement line is essential. Only lines indented after the function definition will be included in the function. \n", - "\n", - "\n", - "Here is another example function that multiplies three numbers." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "def multiply(a, b, c):\n", - " \n", - " \"\"\" Multiply\n", - " \n", - " This function mutliplies three input values.\n", - " \n", - " This is an example of how you can add comments to functions.\n", - " \n", - " \"\"\"\n", - " \n", - " return a * b * c\n", - "\n", - "print('The answer is', multiply(3, 7, 2))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Virtually all Python objects can be passed to functions including other functions. For example, we can pass the `add` function we just defined to a new function as follows" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "def operate(a, b, func):\n", - " \n", - " return func(a, b)\n", - "\n", - "print('the operation \"add\" of 6 and 4 is', operate(6, 4, add))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "---\n", - "\n", - "## Classes\n", - "\n", - "Classes can be an extremely powerful tool when used correctly in your code.\n", - "\n", - "Classes are defined using the **`class`** statement. A simple example of a class (which need not be a class) is given below" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Here we define a class\n", - "class exampleClass:\n", - " \"\"\"Example Class\n", - " \n", - " This is an example of how to add comments to a class.\n", - " \n", - " \"\"\"\n", - "\n", - " def __init__(self, a):\n", - " \n", - " self.a = a\n", - "\n", - " def add(self, b):\n", - " \"\"\" Add\n", - " \n", - " Add input to self.a and return result.\n", - " \n", - " \"\"\"\n", - " \n", - " return self.a + b\n", - "\n", - " def multiply(self, b):\n", - " \"\"\" Multiply\n", - " \n", - " Multiply input by self.a and return result.\n", - " \n", - " \"\"\"\n", - " \n", - " return self.a * b\n", - "\n", - "# Here we initalise a class instance\n", - "x = exampleClass(3)\n", - "\n", - "# Here we implement the class methods\n", - "print('add 7 gives', x.add(7))\n", - "print('')\n", - "print('multiply 4 gives', x.multiply(4))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "In this example a class called `exampleClass` is defined. Then a special **`__init__`** method is defined that takes two arguments: the class instance **`self`** and an input value `a`. This method sets the value of the instance variable `self.a`. The **`add()`** method adds the value of `self.a` to an input value b. The **`multiply()`** method multiplies the value of `self.a` by an input value b.\n", - "\n", - "`x` is a class instance with `self.a=3`. \n", - "\n", - "All instance properties are accessible." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "print('self.a for instance x is', x.a)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "> **NOTE:** as with loops and functions, the indentation after the **`class`** statement line is essential." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "---\n", - "\n", - "## Reading and Writing Files\n", - "\n", - "There many ways to read and write information to and from files using Python. Personally, I prefer some of the options offered by numpy, which will be discussed in the next section, to those available in native Python. Therefore, I will only briefly mention how to read and write files here.\n", - "\n", - "> To run the following cells you will need to download the [Materials](https://minhaskamal.github.io/DownGit/#/home?url=https://github.com/sfarrens/notebooks/tree/master/Python/materials) directory and unzip it. \n", - "> You can use the following command in a terminal: \n", - "> ``` bash\n", - "> $ unzip materials.zip\n", - "> ``` \n", - "> **NOTE:** you need to place the `Materials` folder in the same directory as this notebook.\n", - "\n", - "You can read an ASCII (plain text) file using the **`open()`** command." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# open the text file with reading permission only\n", - "data_file = open('materials/star_wars.txt', 'r')\n", - "\n", - "# print each line in the file\n", - "for line in data_file:\n", - " print(line)\n", - "\n", - "# close the file\n", - "data_file.close()" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "You can write to an ASCII file using the **`write()`** command." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# open the text file with writing permission\n", - "data_file = open('materials/empty.txt', 'w')\n", - "\n", - "# write string to the file\n", - "data_file.write('This is a text string')\n", - "\n", - "# close the file\n", - "data_file.close()" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "---\n", - "\n", - "## Exercises" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Exercise 1.1\n", - "\n", - "Write a function that implements the following equation.\n", - "\n", - "$$F = \\frac{Gm_1m_2}{r^2}$$\n", - "\n", - "where $G = 6.674 \\times 10^{−11}\\,\\textrm{N (m/kg)}^2$ and $1\\,\\textrm{N} = 1\\,\\textrm{kg m/s}^2$." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Write your function here" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Now use your function to calculate the force of attraction between the Earth and the Moon using the following facts:\n", - "\n", - "- Mass of the Earth: $5.9742 \\times 10^{24}\\,\\textrm{kg}$\n", - "- Mass of the Moon: $7.36 \\times 10^{22}\\,\\textrm{kg}$\n", - "- Distance between the Earth and the Moon: $384,402\\,\\textrm{km}$" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Implement your function here" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "**Hint:** think about how many inputs your funciton should have and what it should return." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Exercise 1.2\n", - "\n", - "Calculate the variance ($\\sigma^2$) of the following set of values using native Python only.\n", - "\n", - "$$x = [2, 4, 4, 4, 5, 5, 7, 9]$$\n", - "\n", - "Recall that\n", - "\n", - "$$\\sigma^2 = \\frac{\\sum(x-\\bar{x})^2}{N}$$" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Implement your code here" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Check if the value you obtained for the variance is in $x$ and if so count how many times it occurs." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Implement your code here" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Exercise 1.3\n", - "\n", - "Write a class for converting angles between degrees and radians. Your class should have one method to convert from \n", - "degrees to radians and another method to convert from radians to degrees." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Write your class here" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Now use your class to do the following conversions:\n", - "\n", - "- 45° in radians\n", - "- 30° in radians\n", - "- 135° in radians\n", - "- $\\pi$ radians in degrees\n", - "- $2\\pi/3$ radians in degrees\n", - "- $5\\pi/6$ radians in degrees" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Implement your class here" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "You can find some example answers to the exercises [here](./Answers-to-exercises.ipynb).\n", - "\n", - "---\n", - "\n", - "> **Continue to [next topic](./Numpy.ipynb)**" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.7.2" - } - }, - "nbformat": 4, - "nbformat_minor": 1 -} diff --git a/python/Numpy-Intro.ipynb b/python/Numpy-Intro.ipynb deleted file mode 100644 index 074e8ef..0000000 --- a/python/Numpy-Intro.ipynb +++ /dev/null @@ -1,1931 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\n", - "\"CosmoStat\n", - "
\n", - "
\n", - "
\n", - "
\n", - "\n", - "# Numpy Intro\n", - "\n", - "---\n", - "\n", - "> Author: Santiago Casas \n", - "> Email: santiago.casas@cea.fr \n", - "> Year: 2019 \n", - "> Version: 1.0\n", - "\n", - "---\n", - "
\n", - "\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "So far we have seen how to use and define *lists, dictionaries, functions* and some other *pythonic* tools. However, in scientific research one often needs more than simple algorithms and one needs specialized libraries for working with arrays, math functions, databases and graphics." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Let's start by importing the necessary libraries" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "import numpy" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "It more convenient to assign the numpy package contents to an alias to avoid having longer expressions." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "import numpy as np" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "In this example the **`as`** statement assigns the numpy package contents to the object `np`.\n", - "\n", - "---\n", - "\n", - "## Arrays\n", - "\n", - "### The Basics\n", - "\n", - "The most essential numpy object is the numpy array (numpy.ndarray)." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# a is a list\n", - "a = [1, 2, 3, 4]\n", - "print('a is', type(a))\n", - "\n", - "# b is a numpy array\n", - "b = np.array(a)\n", - "print('b is', type(b))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Accessing and printing a single entry works exactly the same." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "print('first element of a is', a[0])\n", - "print('first element of b is', b[0])" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "print('last element of a is', a[-1])\n", - "print('last element of b is', b[-1])" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "However, their printed forms are slightly different." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "print('list: ', a)\n", - "print('np array: ',b)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Moreover, while lists can contain different object types" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "a = [1, 1.0, 'a', True]\n", - "print(a)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "numpy arrays are of a single type only, which is one of the reasons why they are so efficient.\n", - "\n", - "---\n", - "\n", - "\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "For example, in this case it will convert all entries to strings (upcasting)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "np.array(a)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Or in this case, all entries to floats" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "np.array([1, 2.5, 23, 100.0, np.pi])" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "One can also specify the type directly with the optional argument **dtype** and the entries will be converted to the specified type." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "np.array([1, 2.5, 23, 100.0, np.pi], dtype='int32')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Did you notice the $\\pi$ constant in the list above? Here you can find a list of available (constants). Another one useful in science is" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "#Euler's constant\n", - "np.e" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Creating arrays from scratch" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Sometimes it is useful to create a numpy array in a fast way from scratch. Numpy offers several neat methods." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Zeros and Ones" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Create a length-10 integer array filled with zeros\n", - "np.zeros(10, dtype=int)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Create a length-5 floating-point array filled with ones\n", - "np.ones(5, dtype=float)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### A range and linspace" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "One of the most useful arrays for everyday scientific work is to create arrays that contain evenly-spaced numbers within an interval. " - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "#A range of floats up to 3.0 with default step 1.0\n", - "print(np.arange(3.0))\n", - "# A range from start to stop, with a given step\n", - "print(np.arange(5.0, 405., 50))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "> **NOTE:** Note that with **`arange`** the endpoint is not included !\n", - "\n", - "Remember to check the documentation within the Jupyter notebook running on a cell: **`?np.arange`**" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "If one needs to specify the number of samples and also include the endpoint, then linspace is the right tool. It even contains an optional argument `endpoint`, which defaults to `True`." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "#Three floats evenly spaced in the interval 0. to 3. \n", - "print(np.linspace(0.,3.,3))" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "#With endpoint=False, we get the same behavior as `np.arange`\n", - "print(np.linspace(0.,3.,3, endpoint=False))" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "#default number of samples is 50\n", - "np.linspace(0,100)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Another important array for scientists is a **logarithmically-spaced** interval. The default logarithm is base 10, but that can be changed with the `base` optional argument. The initial and final values of the interval have to be specified in their logarithms." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# A log10-spaced interval from 10^-2 to 10^3 of size 5.\n", - "np.logspace(-2, 3, 5)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Applying a $\\log_{10}$ on the whole array shows that it is indeed log-spaced." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "np.log10(np.logspace(-2, 3, 5))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "> **Notice** how we are using `numpy` internal functions, called uFuncs to calculate properties on the entire array. We will explain that better a bit later." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "#A ln-spaced interval from e^-1 to e^4 of size 4.\n", - "e_array = np.logspace(np.log(np.exp(-1)), np.log(np.exp(4)), 4, base=np.e)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "> **Puzzle 1:** What is the ouput of `np.log(e_array)[-1]` ?" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "#Answer Puzzle 1:\n", - "#Uncomment to see the answer\n", - "#print('the result is: ', np.log(e_array)[-1])" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Multi-dimensional arrays" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Arrays can also be multi-dimensional. And their shape can be specified at creation." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "#2-dimensional array of size 3x5\n", - "np.ones((3,5))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Creating the ***identity*** matrix of size 5" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "np.identity(5)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "`np.eye` is a generalization of the identity, with arguments `numpy.eye(N, M=None, k=0, dtype=, order='C')`. `N` is the number of rows of the array, `M` defaults to `N` and is the number of columns, while `k` shifts the diagonal by a positive or negative integer with respect to the main diagonal. The other arguments can be looked up in the documentation." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "#rectangular matrix\n", - "np.eye(3,4)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "#shifted diagonal\n", - "np.eye(5, k=2)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Sometimes one just needs an array with garbage numbers which is to be filled later on. `np.empty` does the job:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Create an uninitialized array of three integers\n", - "# The values will be whatever happens to already exist at that memory location\n", - "np.empty((3,6))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Very useful in science is the creation of arrays with **random numbers** following a given distribution. Check the extensive documentation of (numpy.random) for much more information on all the available methods." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Create a 3x3 array of normally distributed random values\n", - "# with mean 0 and standard deviation 1\n", - "np.random.normal(0, 1, (3, 3))" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Create a 6x6 array of uniformly distributed\n", - "# random integers between 0 and 10\n", - "rand_mat = np.random.randint(0, 10, (6, 6))\n", - "print(rand_mat[:,0])" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "One can create a random set of points following a Gaussian distribution with a given covariance." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "mean = [0, 0]\n", - "cov = [[1, 0.5],\n", - " [0.5, 2]]\n", - "X = np.random.multivariate_normal(mean, cov, 1000)\n", - "X.shape" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "> **NOTE:** We will see more details about matplotlib in the next session!" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "%matplotlib inline\n", - "import matplotlib.pyplot as plt\n", - "#import seaborn; seaborn.set() # for plot styling\n", - "\n", - "plt.scatter(X[:, 0], X[:, 1]);" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "To construct multi-dimensional arrays, one can also reshape 1-dimensional arrays, using the useful method `reshape(i,j)`. The arguments indicate the rows and the columns of the new array." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Convert 1-dim array into 2x2 matrix\n", - "e_array.reshape((2,2))" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Reshape a 2-dim array\n", - "rand_mat.reshape(9,4)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "If the second argument is `-1` then the size of the second axis is inferred from the previous array." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "rand_mat.reshape(3,-1)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "> **Puzzle 2:** The attribute `shape` returns the shape of a numpy array in the form of a tuple. What is the output of `rand_mat.reshape(2,-1).shape[1]` ?" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "#Answer Puzzle 2:\n", - "#Uncomment to see the answer\n", - "#print('the answer is: ', rand_mat.reshape(2,-1).shape[1])" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Another useful method is `np.ravel` which is roughly the \"inverse\" of reshape in this case. It returns a flattened 1-d array from a 2-d array, equivalent in most cases to `np.flatten`." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "e_array.reshape((2,2)).ravel()" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "e_array.reshape((2,2)).flatten()" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "And for scientific purposes, the ***transpose*** is a very important attribute" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "e_array.reshape((2,2)).T" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "(e_array.reshape((2,2)).T)[0,1]==(e_array.reshape((2,2)))[1,0]" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Other available attributes are:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "print(\"Number of dimensions, ndim: \", rand_mat.ndim)\n", - "print(\"Array shape:\", rand_mat.shape)\n", - "print(\"Array size: \", rand_mat.size)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Slicing and accessing elements" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "As you might now, slicing works for lists, using the `:` operator" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "list = [1,2,3,4,5,6]" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "#Take the first two elements of the list\n", - "list[:2]" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "print(list[2:])\n", - "#Omitting the number after the semicolon is equivalent to indicating the list size:\n", - "list[2:6] == list[2:]" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "#Take every second element\n", - "list[::2]" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "#Reverse the list\n", - "list[::-1]" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "> Slicing in visual form:\n", - "\n", - "" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Here we extend our `e_array` by concatenating it with the useful method `concatenate`, three times." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "extended_array = np.concatenate([e_array, e_array, e_array])\n", - "print(extended_array)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "To obtain three times the same number, we just need to get every fourth element" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "extended_array[::4]" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "> **Puzzle 3:** Obtain the array `[4., 4., 4.]` from `extended_array` by using slicing and `log`." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "#Answers can be found in the Answers.ipynb" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "> **NOTE:** Slicing performs a view of the element and not a copy !" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Slicing: View vs. Copy" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Nested list comprehension\n", - "array2d = np.array([[ ii+jj for jj in range((ii-1)*4,(ii-1)*4+5)] for ii in range(1,1+4)])\n", - "print(array2d)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "For multi-dimensional arrays, the different axis are accessed by separating the slices with commas." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "#All rows, every other column\n", - "array2d[:,::2]" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "#Fourth row, all columns\n", - "array2d[3,:]" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "#Obtain shape\n", - "array2d.shape" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "#Obtain central array\n", - "array2d_center=array2d[1:array2d.shape[0]-1, 1:array2d.shape[1]-1]\n", - "print(array2d_center)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "#Now if we modify this subarray, we'll see that the original array is changed! \n", - "array2d_center[0,0]=1001.5" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "print(array2d_center)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "print(array2d)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "> **NOTE:** Notice not only the big array was changed, but also how numpy converted automatically the type of the variable to `int`, to match the other variable types !" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "From the `Python Data Science Handbook`: This default behavior is actually quite useful: it means that when we work with large datasets, we can access and process pieces of these datasets without the need to copy the underlying data buffer." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "However, if we don't want this behavior, because it can be confusing and introduce possible bugs (believe me, it has happened to me), we can use the `copy` method." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "sub_array = array2d[2:,:3].copy()\n", - "print(sub_array)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "#Modify an element of the subarray\n", - "sub_array[1,1] = 999\n", - "print(sub_array)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Now the large array is not touched:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "print(array2d)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Useful trick: Many elements of the large array can be modified through the small array using slicing." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "array2d_center[:2,:2]=[[42,42],[42,42]]" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "print(array2d)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "> **Puzzle 4:** Change all the last two columns by zeros using slicing, np.shape and np.zeros (assuming you don't know the size of the array beforehand)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "#Answers.ipynb" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Numpy integrated universal functions" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The power of **numpy** lies in its speed and efficiency in performing operations on large arrays." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Functions that operate on the entire array are called ***universal functions*** or for short, uFuncs." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Arithmetic" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "x = np.arange(5)\n", - "print(\"x =\", x)\n", - "print(\"x + 5 =\", x + 5)\n", - "print(\"x - 5 =\", x - 5)\n", - "print(\"x * 2 =\", x * 2)\n", - "print(\"x / 2 =\", x / 2)\n", - "print(\"x // 2 =\", x // 2) # floor division\n", - "print(\"-x = \", -x)\n", - "print(\"x ** 2 = \", x ** 2)\n", - "print(\"x % 2 = \", x % 2) # modulo" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "These operators are actually wrappers to the method form:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "np.add(x,5)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "np.floor_divide(x,2)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "More functions like this can be found in the documentation of the uFuncs ." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Trigonometric functions" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Numpy can compute mathematical functions very efficiently over a large array. They are computed to withon machine precision, therefore tiny values instead of exact zeros can appear." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# An array of 3000 elements between 0 and Pi.\n", - "theta = np.linspace(0, np.pi, 3000)\n", - "print(\"theta = \", theta[0])\n", - "print(\"sin(theta) = \", np.sin(theta)[-1])\n", - "print(\"cos(theta) = \", np.cos(theta)[0])\n", - "print(\"tan(theta) = \", np.tan(theta)[-1])" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Numpy also offers mathematical functions like `exp` and `log` and versions that are more accurate for tiny numbers, like **`expm1`** and **`log1p`**. " - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# For tiny x values, log(1+x) and exp(x)-1 are very very close to x.\n", - "x = np.array([0., 1e-10, 1e-12, 1e-14])\n", - "print(\" exp(x) - 1 =\", np.expm1(x))\n", - "print(\"std: exp(x) - 1 =\", np.exp(x)-1.0)\n", - "print(\" log(1 + x) =\", np.log1p(x))\n", - "print(\"std: log(1 + x) =\", np.log(1.0+x))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "There are tons of functionalities more, but this is just a rough overview. In the documentation and many other excellent tutorials and books, such as the \n", - " Python Data Science Handbook one can find much more on these topics." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Slowness of loops and lists vs. Numpy " - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "In a standard programming language, like C, we would have to define the following non-pythonic function in order to compute the reciprocal of a list" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "def compute_reciprocals(values):\n", - " output = []\n", - " for i in range(len(values)):\n", - " output.append(1.0 / values[i])\n", - " return output" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "a_list = np.random.randint(1,10,10000000).tolist() #notice the tolist method here\n", - "print(a_list[0])" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Computing the reciprocal will be slow, since each element of the python list is a data structure object." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "%%time\n", - "b_list = compute_reciprocals(a_list)\n", - "#print(b_list[0])" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "> `%time` is one of these magic functions we will see explained in the next session." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "a_array = np.random.randint(1,10,10000000)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "In Numpy we don't even need to define a function, we just calculate 1.0/list." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "%%time\n", - "b_array = 1.0 / a_array\n", - "#print(b_array)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "> 25 times faster?" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Broadcasting" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "One of the most powerful features of `numpy` is, as we just saw, the fact that one can operate directly on entire arrays, element by element, without the need of cumbersome loops." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "***Broadcasting*** is a way of applying uFuncs to arrays of different sizes. In the simplest case, addition, multiplication and so on, but actually it can be done with any uFunc that receives 2 arguments." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Add a scalar to an array\n", - "s=5\n", - "ones = np.ones((3,5))\n", - "sixes = ones + s" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "print(sixes)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Add two arrays\n", - "ones + sixes" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "> **NOTE:** Lists do not behave like that, when added, they concatenate." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "scrolled": true - }, - "outputs": [], - "source": [ - "#Convert numpy arrays to lists\n", - "ones.tolist() + sixes.tolist()" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "One can also add arrays of different sizes, thanks to broadcasting." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "a = np.array([0,1,2])\n", - "b = np.ones((3,3))\n", - "print('a= ',a)\n", - "print('b=', b)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "a+b" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "print('shape: ', a.shape, '| dimensions: ', a.ndim, '| size: ', a.size)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "print('shape: ', b.shape, '| dimensions: ', b.ndim, '| size: ', b.size)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "In this case what `numpy` has done is to compare the shape of the arrays. \n", - " - If they differ in the number of dimensions the shape gets augmented on the left (padded) with 1s. \n", - " - Then, if the arrays do not match in all their dimensions, the array with the padded 1s, is stretched in that dimension to match the size of the other array. \n", - " - Finally the two arrays can be combined.\n", - "" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "For the third graphical example we can create row and column vectors using slicing." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "x = np.array([0,1,2])\n", - "# row vector\n", - "row = x[np.newaxis, :]\n", - "print(row)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "What `np.newaxis` does is to increase the dimensions of the array by 1 on the axis where it is used." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "row.shape" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "#column vector\n", - "col = x[:,np.newaxis]\n", - "print(col)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "col.shape" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "row + col" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Notice that with simple 1-d arrays, we do get the expected scalar in the dot product:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "np.dot(x,x)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "While with the row and column vectors we get a matrix:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "np.dot(row,col)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "np.dot(col,row)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Boradcasting is much more powerful and has many more subtleties that we cannot cover here. I refer again to Python Data Science Handbook for a deeper treatment of this." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\n", - "## Fancy indexing and masking" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Finally, let's look at a couple of important tricks and methods available for numpy arrays, which make a scientist's life much easier." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Obtaining several entries of an array at once" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Fancy indexing is simply the property of accessing arrays not only with slicing, but with integers or booleans." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "a100 = np.arange(100)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Let's say we wanted to obtain entries 1, 21, 41 and 61 and put them in an array. In other languages one would do something like:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "[a100[1], a100[21], a100[41] , a100[61]]" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "**Option 1:** In python we could use slicing:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "a100[1:62:20]" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "**Option 2:** We could use integers:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "#accessing entries with integers\n", - "a100[[1,21,41,61]]" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "**Option 3:** Or we could define a list and extract only those that match using booleans (masking)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "extract = [1,21,41,61]\n", - "mask = np.array([aa in extract for aa in a100])\n", - "print(mask)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "a100[mask]" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Fancy indexing is very powerful, because it can be used for assigning:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "a100[[10,20,30]] = -1000" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "a100" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "And it can also change the shape of the original array, if it is indexed with an array of integers:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "idx = np.array([[40,50],[60,70]])" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "a100[idx]" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Masking and comparisons" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "## Obtain booleans for values smaller than 10\n", - "a100 < 0" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Get the values\n", - "a100[a100 < 0]" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "#Get all odd values\n", - "a100[a100 %2 != 0]" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "> **NOTE:** Note how powerful this masking and comparison is compared to other languages." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Other functions like `np.sum` or `np.count_nonzero` can be used with these logical operators (True==1, False==0):" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "> **Puzzle 5:** What is the result of np.count_nonzero(a100 > 90)? " - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "#Answer Puzzle 5:\n", - "#Uncomment to see the answer:\n", - "#np.count_nonzero(a100 > 90)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "#Sum all entries larger than 90\n", - "np.sum(a100[a100>90])" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "#Sum all boolean entries larger than 90, equivalent to count_nonzero\n", - "np.sum(a100>90)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "One can also combine comparisons with logical operators:\n", - "\n", - "|Operator|uFunc|\n", - "|------|---------------|\n", - "|&|`np.bitwise_and`|\n", - "|^|`np.bitwise_xor`|\n", - "|||`np.bitwise_or`|\n", - "|~|`np.bitwise_not`|" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "#Number of integers larger than 80, which are odd:\n", - "np.sum((a100 > 80) & ~(a100%2 == 0))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "One can also convert an array of `1` and `0` to booleans right at creation." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "identity_mask=np.identity(5, dtype=bool)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Which can be neatly used as a mask" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "#Create a matrix out of its diagonal using np.diag and replace the zeros by 7.\n", - "\n", - "int_arr = np.diag(np.arange(1,6))\n", - "int_arr[int_arr == 0] = 7\n", - "print(int_arr)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "#Apply the mask and return a 1-d array.\n", - "int_arr[identity_mask]" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Exercises" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Exercise 1:" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "> Using broadcasting and in max. 2 lines of code, construct a multiplication table of the numbers from 1 to 10, i.e., where for each column corresponding to 1,2,3,..., the rows correspond to their integer multiples. For 1 to 3, it looks like this:\n", - "\n", - "---\n", - "\n", - "|1|2|3|\n", - "|--|--|--|\n", - "|2|4|6|\n", - "|3|6|9|\n", - "\n", - "> Then, using masking, remove all multiples of 3, in one line of code. So that the result (in this case for 3x3) looks like this:\n", - "\n", - "|1|2|0|\n", - "|--|--|--|\n", - "|2|4|0|\n", - "|0|0|0|\n", - "\n", - "> Finally, compute the sum over each column and write it into a list:\n", - "\n", - " [3,6,0]\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### **Exercise 2:** \n", - " * Produce a random set of points, following a normal distribution with mean 0 and covariance matrix `cov=[[1, 3.0/5], [3/5.0, 2]]`.\n", - " * Plot the set of points using `plt.scatter(X, Y)`\n", - " * Compute the marginal distributions for the x and y coordinates and using numpy functions, check that they follow a Gaussian distribution when compared to their histograms. Gaussian distribution:\n", - " $$ \\mathcal{N}(x) = \\frac{1}{\\sqrt{2 \\pi \\sigma^2}} \\exp(-(x-\\mu)^2/ 2 \\sigma^2)$$\n", - " * Here $\\mu$ is the mean and $\\sigma$ the standard deviation.\n", - " * (Hints: `plt.hist(points, bins, density=True)`, `np.exp`, `np.sqrt`, `np.std`, `np.mean`). Use 25 bins for the histogram.\n", - " * Using masking and `np.where` remove all points whose y-coordinate is more than 2-sigma away from the mean.\n", - " * Plot the remaining points in a scatter plot with blue color, with `facecolor='none'` and size `s=200`, together with the old set of points in red color with a transparency of alpha=0.6." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "> **[Answers](./Answers.ipynb) to puzzles and exercises**" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "> **[Next Chapter: Pandas](./Pandas-Intro.ipynb)**" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [] - } - ], - "metadata": { - "hide_input": false, - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.7.3" - }, - "toc": { - "base_numbering": 1, - "nav_menu": {}, - "number_sections": true, - "sideBar": true, - "skip_h1_title": false, - "title_cell": "Table of Contents", - "title_sidebar": "Contents", - "toc_cell": false, - "toc_position": {}, - "toc_section_display": true, - "toc_window_display": false - } - }, - "nbformat": 4, - "nbformat_minor": 2 -} diff --git a/python/Numpy.ipynb b/python/Numpy.ipynb deleted file mode 100644 index fa8003f..0000000 --- a/python/Numpy.ipynb +++ /dev/null @@ -1,891 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# 2 Numpy\n", - "\n", - "---\n", - "\n", - "> Author: Samuel Farrens \n", - "> Year: 2017 \n", - "> Email: [samuel.farrens@gmail.com](mailto:samuel.farrens@gmail.com) \n", - "> Website: https://sfarrens.github.io\n", - "\n", - "---\n", - "\n", - "Numpy is a numerical python package for scientific computing. If you only ever install one Python package this should be it.\n", - "\n", - "The website contains extensive documentation on all the modules available with examples. This notebook barely skims the surface of what you cand do with numpy, but I have provided a few examples to help get you started.\n", - "\n", - "---\n", - "\n", - "## Contents\n", - "\n", - "1. [Installation](#Installation)\n", - "1. [Importing Numpy](#Importing-Numpy)\n", - "1. [Arrays](#Arrays)\n", - " * [The Basics](#The-Basics)\n", - " * [Multidimensional Arrays](#Multidimensional-Arrays)\n", - " * [Mathematical Operations](#Mathematical-Operations)\n", - "1. [Random Numbers](#Random-Numbers)\n", - "1. [Reading and Writing Files](#Reading-and-Writing-Files)\n", - " * [ASCII Files](#ASCII-Files)\n", - " * [Binary Files](#Binary-Files)\n", - "1. [Linear Alegbra](#Linear-Algebra)\n", - "1. [Exercises](#Exercises)\n", - " * [Exercise 2.1](#Exercise-2.1)\n", - " * [Exercise 2.2](#Exercise-2.1)\n", - "\n", - "---\n", - "\n", - "## Installation\n", - "\n", - "To install numpy simply run the following command in a terminal (providing you have already set up pip)\n", - "\n", - "```bash\n", - "\n", - "$ pip install numpy\n", - "\n", - "```" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "---\n", - "\n", - "## Importing Numpy\n", - "\n", - "To import numpy use the **`import`** command as follows" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "import numpy" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "It more convenient to assign the numpy package contents to an alias to avoid having longer expressions." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "import numpy as np" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "In this example the **`as`** statement creates an alias of numpy called `np`.\n", - "\n", - "---\n", - "\n", - "## Arrays\n", - "\n", - "### The Basics\n", - "\n", - "The most essential numpy object is the numpy array (numpy.ndarray)." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# a is a list\n", - "a = [1, 2, 3, 4]\n", - "print('a is', type(a))\n", - "\n", - "# b is a numpy array\n", - "b = np.array(a)\n", - "print('b is', type(b))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The **`np.array()`** command converts the list `a` to a numpy array `b`.\n", - "\n", - "Numpy arrays have several fantastic properties. Some basic examples are shown below.\n", - "\n", - "For a given array:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# a is 1D numpy array\n", - "a = np.array([4, 5, 6, 8, 7, 5, 2, 1, 0, 4, 0, 6, 5, 4, 2, 4, 1, 7, 1, 4])\n", - "\n", - "# print a\n", - "print('a =', a)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "You can print a given element of the array." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# print the 10th element of a\n", - "print('the 10th element of a is', a[10])" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "You can also index from the end of the array." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# print the last element of a\n", - "print('the last element of a is', a[-1])\n", - "\n", - "# print the second to last element of a\n", - "print('the second to last element of a is', a[-2])" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "You can specfiy every element in an array with a colon **`:`**. This can also be used for *slicing* (*i.e.* to specify every element before or after a certain point)." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# print every element in a\n", - "print('a =', a[:])\n", - "\n", - "# print every element up to index 10\n", - "print('first half of elements in a are', a[:10])\n", - "\n", - "# print every element after index 10\n", - "print('second half of elements in a are', a[10:])\n", - "\n", - "# print every element between index 2 and 6\n", - "print('the 3rd through 6th elements of a are', a[2:6])\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "You can print every $i^{th}$ element using a double set of colons **`::`**" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# print every 5th element\n", - "print('every fifth element of a is', a[::5])" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "You can also use double colons to reverse the entire array." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# print a backwards\n", - "print('a backwards is', a[::-1])" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "It is possible to subample a list using a list of indices." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# extract elements 1 3 11 and 18 from a\n", - "print('elements at indices 1, 3, 11 and 18 of a are', a[[1, 3, 11, 18]])" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "You can also mask out items using booleans and hence conditions." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "print('the elements of a greater than 3 are', a[a > 3])\n", - "print('the elements of a excluding 5 are', a[a != 5])" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "You can check if any or all elements in an array have certain properties using the **`any()`** and **`all()`** commands." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "print('are there any elements in a greater than 10?', np.any(a > 10))\n", - "print('are there all of elements in a less than 10?', np.all(a < 10))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "There several built-in functions for calculating the properties of Numpy arrays." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "print('a has size', a.size)\n", - "print('a has mean', a.mean())\n", - "print('a has standard deviation', a.std())" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "In this example the size of the numpy array is displayed using the **`size`** command. The mean and standard deviation of the array elements are calculated using the **`mean()`** and **`std()`** commands, respectively." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The **`where()`** command can be used to find the index of a given elements." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "print('a = 8 at index', np.where(a == 8)[0])" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Numpy includes several nice ways to quickly generate arrays using the commands **`zeros()`**, **`ones()`** and **`arange()`**." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "print('- Generate an array of zeros.')\n", - "print(' ', np.zeros(10))\n", - "print('')\n", - "\n", - "print('- Generate an array of ones.')\n", - "print(' ', np.ones(10))\n", - "print('')\n", - "\n", - "print('- Generate a range of integers from 0 to 9.')\n", - "print(' ', np.arange(10))\n", - "print('')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Multidimensional Arrays\n", - "\n", - "The **`ndim`** command can be used to display the number of dimensions of a given array. while **`shape`** command can be used to display the size of each of the dimensions." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# a is a 2D numpy array\n", - "a = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]])\n", - "\n", - "print('a =', a)\n", - "print('a has', a.ndim, 'dimensions')\n", - "print('the shape of a is', a.shape)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "> **NOTE:** notice the extra set of square brackets!\n", - "\n", - "It is also possible to change the shape of an existing array using the **`reshape()`** command." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# x is a 1D array\n", - "x = np.arange(16)\n", - "print('x =', x)\n", - "print('')\n", - "\n", - "# x is now a 2D array\n", - "x = x.reshape(4, 4)\n", - "print('x =', x)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "All of the indexing options shown for 1D arrays are equally applicable to multidimensional arrays. For example, for a 5x5 array" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "x = np.arange(25).reshape(5, 5)\n", - "\n", - "print('x =', x)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "You can specify whole rows or columns." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# first row of x\n", - "print('the first row of x is', x[0])\n", - "\n", - "# 3rd column of x\n", - "print('the third column of x is', x[:, 2])" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Or you can specify single elements" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# 2nd element in the 4th row of x\n", - "print('the second element in the 4th row of x is', x[3, 1])" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Slicing also works." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# print every other element in the 1st two rows of x\n", - "print('every other element in the first two rows of x are')\n", - "print(x[:2, ::2])" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "There several handy numpy commands for transforming multidimensional arrays. The command **`rot90()`** rotates a matrix by 90 degrees." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# rotate x by 180 degrees\n", - "print('x rotated by 180 degrees is')\n", - "print(np.rot90(x, 2))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The command **`transpose()`** will transpose an array." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# transpose x\n", - "print('x transpose is')\n", - "print(np.transpose(x))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Mathematical Operations\n", - "\n", - "One of the most useful features of numpy is that it allows element-wise mathematical operations. It also has built-in mathematical operators.\n", - "\n", - "You can multiply every element in an array by a single number." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "x = np.arange(5) + 1\n", - "\n", - "print('x =', x)\n", - "print('2x =', 2 * x)\n", - "print('x^2 =', x ** 2)\n", - "print('sqrt(x) =', np.sqrt(x))\n", - "print('x * pi =', np.pi * x)\n", - "print('log(x) =', np.log(x))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "You can also multiply each element in one array by each element in another of the same size." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "y = np.array([7, 9, 2, 3, 2])\n", - "print('y =', y)\n", - "\n", - "# check if x and y have the same size\n", - "print('x is the same size as y:', x.size == y.size)\n", - "\n", - "# add the elements of x to those of y\n", - "print('x + y =', x + y)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "This also works for multidimensional arrays with the same shape." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "a = np.ones((2, 2)).astype(int) * 2\n", - "b = np.arange(4).reshape(2, 2)\n", - "print('a =', a)\n", - "print('b =', b)\n", - "print('')\n", - "\n", - "# check if a and b have the same shape\n", - "print('a has the same shape as b:', a.shape == b.shape)\n", - "print('')\n", - "\n", - "# multiply the elements of a by those of b\n", - "print('a * b =')\n", - "print(a * b)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "---\n", - "\n", - "## Random Numbers\n", - "\n", - "The numpy **`random`** module (numpy.random) contain various methods for generating random numbers." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "print('- Generate an array of 10 random integers with values between 2 and 9.')\n", - "print(' ', np.random.randint(2, 9, 10))\n", - "print('')\n", - "\n", - "print('- Generate an array of 5 random floats with values between 0 and 1.')\n", - "print(' ', np.random.ranf(5))\n", - "print('')\n", - "\n", - "print('- Generate an array of 5 Gaussian distributed random floats with values between -1 and 1.')\n", - "print(' ', np.random.randn(5))\n", - "print('')\n", - "\n", - "print('- Generate an array of 5 Poisson distributed random integers.')\n", - "print(' ', np.random.poisson(1, 5))\n", - "print('')\n", - "\n", - "print('- Generate an 2D array of random numbers.')\n", - "print(' ', np.random.rand(5, 5))\n", - "print('')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "---\n", - "\n", - "## Reading and Writing Files\n", - "\n", - "Numpy provides some advatages over native Python in terms of reading and writing files.\n", - "\n", - "> If you have not already done so, you will need to download the [Materials](https://minhaskamal.github.io/DownGit/#/home?url=https://github.com/sfarrens/notebooks/tree/master/Python/materials) directory and unzip it. \n", - "> You can use the following command in a terminal: \n", - "> ``` bash\n", - "> $ unzip materials.zip\n", - "> ``` \n", - "> **NOTE:** you need to place the `Materials` folder in the same directory as this notebook.\n", - "\n", - "### ASCII Files\n", - "\n", - "ASCII (plain text) files can be read using the **`genfromtxt()`** command (numpy.genfromtxt) as follows" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Read in the file\n", - "data = np.genfromtxt('materials/ninja_turtles.txt', names=True, dtype=None, encoding='ascii')\n", - "\n", - "# Print the column names\n", - "print(data.dtype.names)\n", - "print('')\n", - "\n", - "# Print the rows\n", - "for line in data:\n", - " print(line)\n", - "\n", - "print('')\n", - "\n", - "# Calculate the average age\n", - "print('The average age is', data['Age'].mean())" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "In the example above a file called `ninja_turtles.txt` which is in the directory `materials` is read in. The first row\n", - "of this file provides column names which are stored in `dtype.names` using the **`names`** option. Setting the **`dtype`** as `None` allows the command to assign the appropriate data type to each column in the file.\n", - "\n", - "Data can be saved to an ASCII file using the **`savetxt()`** command (numpy.savetext) as follows " - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# create a new string\n", - "string = 'average turtle age is ' + str(data['Age'].mean())\n", - "\n", - "# save the string to a file\n", - "np.savetxt('materials/turtle_age.txt', [string], fmt='%s')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "In this example we need to pass the string as a list (a numpy array would also work) to **`savetext()`**. Also, we need to define the format of the data using the **`fmt`** option." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Binary Files\n", - "\n", - "Numpy arrays can also be read in or saved to special numpy binary files (`.npy` extension) using the **`load()`** and **`save()`** commands (numpy.save, numpy.load), respectively." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Read in the numpy binary file\n", - "data = np.load('materials/ninja_turtles.npy')\n", - "\n", - "# Print the array contents\n", - "print(data)\n", - "print('')\n", - "\n", - "# Increase ages by 1\n", - "data['Age'] += 1\n", - "print(data)\n", - "\n", - "# Save new data to a binary file\n", - "np.save('materials/ninja_turtles_new.npy', data)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Numpy binaries have the fantastic property of retaining formatting and data structures. In addition, they can be read and written much faster than text files." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "---\n", - "\n", - "## Linear Algebra\n", - "\n", - "Numpy also handles linear algebra (numpy.linalg). For example, you can multiply two matrices using the **`dot()`** command." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "x = np.random.randint(0, 9, (3, 3))\n", - "print('x =')\n", - "print(x)\n", - "print('')\n", - "\n", - "y = np.random.randint(0, 9, (3, 3))\n", - "print('y =')\n", - "print(y)\n", - "print('')\n", - "\n", - "print('xy =')\n", - "print(np.dot(x, y))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "You can also calculate the inverse of a matrix" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "a = np.array([[1., 2.], [3., 4.]])\n", - "print('a =')\n", - "print(a)\n", - "print('')\n", - "\n", - "print('a^-1 =')\n", - "print(np.linalg.inv(a))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "or you can calculate the $l_2$-norm of a matrix using the **`norm`** command" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "x = np.random.ranf((3, 3))\n", - "print('x =', x)\n", - "print('')\n", - "print('||x|| =', np.linalg.norm(x))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "---\n", - "\n", - "## Exercises\n", - "\n", - "### Exercise 2.1\n", - "\n", - "Read in the file called `nearest_stars.txt` in the directory `materials` and determine the standard deviation of the distances of the stars with spectral type `M`." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Implement your code here" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Exercise 2.2\n", - "\n", - "Generate a three dimensional array of random floats with shape (3, 4, 4) using a random seed of 8. For each 4 x 4 matrix in the array calculate the $l_1$-norm and then calculate the mean of these values.\n", - "\n", - "**HINT:** Make sure you check the documentation pages for commands that are not mentioned in this notebook." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Implement your code here" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "You can find some example answers to the exercises [here](./Answers-to-exercises.ipynb).\n", - "\n", - "---\n", - "\n", - "> **Continue to [next topic](./Matplotlib.ipynb)**" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.7.2" - } - }, - "nbformat": 4, - "nbformat_minor": 1 -} diff --git a/python/Other-Packages.ipynb b/python/Other-Packages.ipynb deleted file mode 100644 index 821c45e..0000000 --- a/python/Other-Packages.ipynb +++ /dev/null @@ -1,419 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# 4 Other Packages\n", - "\n", - "---\n", - "\n", - "> Author: Samuel Farrens \n", - "> Year: 2017 \n", - "> Email: [samuel.farrens@gmail.com](mailto:samuel.farrens@gmail.com) \n", - "> Website: https://sfarrens.github.io\n", - "\n", - "---\n", - "\n", - "Now that you have mastered the basics of Python and the essential packages you can begin to explore some of the other packages that are available. Here I will briefly introduce a few that I use often.\n", - "\n", - "---\n", - "\n", - "## Contents\n", - "\n", - "1. [Installation](#Installation)\n", - "1. [Scipy](#Scipy)\n", - " * [Fourier Transforms](#Fourier-Transforms)\n", - " * [SVD](#SVD)\n", - "1. [Astropy](#Astropy)\n", - " * [FITS Files](#FITS-Files)\n", - " * [Convolution](#Convolution)\n", - "1. [Exercises](#Exercises)\n", - " * [Exercise 4.1](#Exercise-4.1)\n", - " * [Exercise 4.2](#Exercise-4.2)\n", - " \n", - "---\n", - "\n", - "## Installation\n", - "\n", - "To install the packages discussed in this notebook simply run the following commands in a terminal\n", - "\n", - "```bash\n", - "\n", - "$ pip install scipy\n", - "$ pip install astropy\n", - "\n", - "```\n", - "---\n", - "\n", - "## Scipy\n", - "\n", - "Scipy is an open source package for scientific computing that contains a collection of useful algorithms and tools. Scipy overlaps with some Numpy modules. Scipy, however, tends to be a bit more comprehensive in certain areas.\n", - "\n", - "### Fourier Transforms\n", - "\n", - "The **`fftpack`** module in scipy (scipy.fftpack) contains virtually all the tools you will need for working in Fourier space. \n", - "\n", - "You can import the package as follows" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "from scipy import fftpack" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "and since it is easier to understand these operations visually, we can also import matplotlib." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "%matplotlib inline\n", - "import numpy as np\n", - "import matplotlib.pyplot as plt\n", - "from matplotlib.image import imread" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We can start with a simple sine curve time series." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# set the signal properties\n", - "sample_rate = 150.0\n", - "signal_freq = 5\n", - "\n", - "# generate a range of time values\n", - "t = np.arange(0, 1, 1 / sample_rate)\n", - "\n", - "# calculate the sin curve\n", - "y = np.sin(2 * np.pi * signal_freq * t)\n", - "\n", - "# plot the curve\n", - "plt.figure(figsize=(8, 6))\n", - "plt.plot(t, y, 'b-')\n", - "plt.ylim(-1.2, 1.2)\n", - "plt.xlabel('$t$', fontsize=20)\n", - "plt.ylabel('$y(t)$', fontsize=20)\n", - "plt.show()" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Now we can use the command **`fft()`** to transform the data." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# define the frequency range\n", - "frq = (np.arange(y.size) * sample_rate / y.size)[:int(y.size / 2)]\n", - "\n", - "# calculate the FFT of the sine curve\n", - "abs_y = np.abs((fftpack.fft(y) / y.size)[:frq.size])\n", - "\n", - "# display the FFT data\n", - "plt.figure(figsize=(8, 6))\n", - "plt.plot(frq, abs_y, 'b-')\n", - "plt.ylim(0, 0.6)\n", - "plt.xlabel('$\\omega$', fontsize=20)\n", - "plt.ylabel('$|y(\\omega)|$', fontsize=20)\n", - "plt.show()" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### SVD\n", - "\n", - "Singular Value Decomposition can be performed using the **`svd()`** command in the **`linalg`** module.\n", - "\n", - "> If you have not already done so, you will need to download the [Materials](https://minhaskamal.github.io/DownGit/#/home?url=https://github.com/sfarrens/notebooks/tree/master/Python/materials) directory and unzip it. \n", - "> You can use the following command in a terminal: \n", - "> ``` bash\n", - "> $ unzip materials.zip\n", - "> ``` \n", - "> **NOTE:** you need to place the `Materials` folder in the same directory as this notebook.\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# import the linalg module\n", - "from scipy import linalg\n", - "\n", - "# read in an image\n", - "x = imread('materials/r2d2.jpg')[:, :, 0]\n", - "\n", - "# calculate the svd\n", - "u, s, v = linalg.svd(x)\n", - "\n", - "# keep only singular values >= 1.0\n", - "s_new = s * (np.abs(s) >= 1.0)\n", - "\n", - "# reconstruct the image with thresholded singular values\n", - "s_new = linalg.diagsvd(s_new, *x.shape)\n", - "x_new = np.dot(u, np.dot(s_new, v))\n", - "\n", - "# dsiplay the two images\n", - "plt.figure(figsize=(10, 8))\n", - "plt.subplot(121)\n", - "plt.imshow(x, interpolation='none', cmap='gray')\n", - "plt.title('Original Image')\n", - "plt.subplot(122)\n", - "plt.imshow(x_new, interpolation='none', cmap='gray')\n", - "plt.title('Reconstructed Image')\n", - "plt.show()" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "In this example the **`diagsvd()`** command diagonalises and reshapes the array of singular values." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "---\n", - "\n", - "## Astropy\n", - "\n", - "Astropy is a package that contains tools designed for the analysis of astronomical data.\n", - "\n", - "### FITS Files\n", - "\n", - "One of the most useful modules in Astropy is **`io.fits`** (astropy.io.fits), which handles the reading and writing FITS binary files.\n", - "\n", - "You can import the module as follows" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "from astropy.io import fits" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "and use it to read a fits image as follows" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# read image data from FITS file\n", - "x = fits.getdata('materials/r2d2.fits')\n", - "\n", - "# Display the image\n", - "plt.figure(figsize=(6, 6))\n", - "plt.imshow(x)\n", - "plt.show()" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "You can write image data to a new file as follows" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# generate an array of random numbers\n", - "x = np.random.ranf((10, 10))\n", - "\n", - "# save array to a FITS file\n", - "fits.PrimaryHDU(x).writeto('materials/random_data.fits')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Convolution\n", - "\n", - "You can convolve vectors or matrices using the **`convolve_fft()`** command in the **`convolution`** module (astropy.convoluion)." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# read in image from scipy.misc\n", - "from scipy.misc import face\n", - "\n", - "# import Gaussian filter function\n", - "from scipy.ndimage import gaussian_filter\n", - "\n", - "# import astropy methods\n", - "from astropy.convolution import convolve_fft\n", - "\n", - "# set the image as a 2D array\n", - "image = face()[:, :, 0]\n", - "\n", - "# create a 2D Gaussian kernel with sigma=0.5\n", - "kernel = np.zeros(image.shape)\n", - "kernel[tuple(np.array(kernel.shape) // 2)] = 1\n", - "kernel = gaussian_filter(kernel, 20)\n", - "\n", - "# convolve the image by the kernel\n", - "convolved_image = convolve_fft(image, kernel, boundary='wrap')\n", - "\n", - "# display the images\n", - "plt.figure(figsize=(16, 6))\n", - "plt.subplot(131)\n", - "plt.imshow(image, cmap='gray')\n", - "plt.subplot(132)\n", - "plt.imshow(kernel)\n", - "plt.subplot(133)\n", - "plt.imshow(convolved_image, cmap='gray')\n", - "plt.show()\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "---\n", - "\n", - "## Exercises\n", - "\n", - "### Exercise 4.1\n", - "\n", - "Write your own convolution function and compare its performance to the astroy module. Your function should:\n", - "\n", - "- Take an image and a convolution kernel as inputs\n", - "- Output a convolved image\n", - "- The convolution kernel should be a Gaussian with sufficiently large $\\sigma$ to visually distort the image\n", - "- You can use the scipy **`fftpack`** module\n", - "- You can use any input image you like\n", - "\n", - "Recall the *convolution theorem*\n", - "\n", - "$$\\mathcal{F}\\{x * y\\} = \\mathcal{F}\\{x\\} \\cdot \\mathcal{F}\\{y\\}$$\n", - "\n", - "where $\\mathcal{F}\\{x\\}$ denotes the Fourier transform of $x$, and $x * y$ denotes the convolution of $x$ and $y$.\n", - "\n", - "Display your input image next to the convolved image you obtain with your function and with Astropy. Make sure you title your images.\n", - "\n", - "**Hints:**\n", - "- Make sure the output of your function is *real*-istic. There might be a Numpy function to help with this.\n", - "- If your convolved image is blurry but some pices are in the wrong places, you made need to check the fftpack documentation for the correct way to *shift* the data." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Implement your code here" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Exercise 4.2\n", - "\n", - "Now write a deconvolution function (*i.e.* a function that removes the kernel distortion from the convolved image) to recover the original image. Your function should:\n", - "\n", - "- Take a distorted image and convolution kernel as inputs\n", - "- Output a deconvolved image\n", - "- You can use the scipy **`fftpack`** module\n", - "- Your input image should be the output of Exercise 4.1\n", - "\n", - "Display your deconvolved image next to the original image. Make sure you title your images.\n", - "\n", - "**Hints:**\n", - "- If your deconvolution function is not working make sure the noise level set in the previous exercise is not too high." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Implement your code here" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "You can find some example answers to the exercises [here](./Answers-to-exercises.ipynb).\n", - "\n", - "---\n", - "\n", - "> **Continue to [next topic](./Answers-to-exercises.ipynb)**" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.7.2" - } - }, - "nbformat": 4, - "nbformat_minor": 1 -} diff --git a/python/Pandas-Intro.ipynb b/python/Pandas-Intro.ipynb deleted file mode 100644 index 4aa822c..0000000 --- a/python/Pandas-Intro.ipynb +++ /dev/null @@ -1,1310 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\n", - "\"CosmoStat\n", - "
\n", - "
\n", - "
\n", - "
\n", - "\n", - "# Pandas Intro\n", - "\n", - "---\n", - "\n", - "> Author: Santiago Casas (based on the Python Data Science Handbook)\n", - "> Email: santiago.casas@cea.fr \n", - "> Year: 2019 \n", - "> Version: 1.0\n", - "\n", - "\n", - "\"CosmoStat\n", - "\n", - "---\n", - "
" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Let's start by importing the necessary libraries\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "import numpy as np" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "import pandas as pd" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# Numpy with structured arrays" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Numpy is quite powerful because it can host structured data into the array, using a dictionary structure." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Imagine we have this data about people and their age and weight." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "name = ['Alice', 'Bob', 'Cathy', 'Doug']\n", - "age = [25, 45, 37, 19]\n", - "weight = [55.0, 85.5, 68.0, 61.5]" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We can create a structured array using a compound data type specification" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Use a compound data type for structured arrays\n", - "data = np.zeros(4, dtype={'names':('name', 'age', 'weight'),\n", - " 'formats':('U10', 'int32', 'float32')})\n", - "print(data.dtype)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We have a data type that contains unicode strings, ints and floats." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We can now fill this structure data array using `keys` as we did for dictionaries." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "data['name'] = name\n", - "data['age'] = age\n", - "data['weight'] = weight\n", - "print(data)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "This allows us to access the array with indices" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Get first row of data\n", - "data[0]" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "#Get name of the last row of data\n", - "data[-1]['name']" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Masking with structured arrays" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Using the powerful masking we saw before, we can ask questions such as: What is the name of all the persons below an age of 30?" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Get names where age is under 30\n", - "data[data['age'] < 30]['name']" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Despite the power of numpy, once we start dealing with more complicated structures, we need to resort to Pandas. A python package specialized for databases, structured data and statistical analysis." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# Enter Pandas" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Pandas Series" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "A Pandas `Series` is a one-dimensional array of indexed data. It is the most basic structure. It can be created from a list or array as follows:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "data = pd.Series([0.25, 0.5, 0.75, 1.0])\n", - "data" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Notice how we now get an index column, next to the array values. This index column is by default just a numbering of the entries." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The values are simply a numpy array:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "data.values" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "While the index is just an array-type object, special to `Pandas`" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "data.index" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Data can be accessed like in a numpy array" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "data[1:3]" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "data[1]" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The Pandas Series has an explicitly defined index associated with the values.\n", - "\n", - "This explicit index definition gives the Series object additional capabilities. For example, the index need not be an integer, but can consist of values of any desired type. For example, if we wish, we can use strings as an index:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "data = pd.Series([0.25, 0.5, 0.75, 1.0],\n", - " index=['a', 'b', 'c', 'd'])\n", - "data" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Now we can access the data through an index\n", - "data['b']" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Series as specialized dictionaries" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Instead of specifying the data and the indices seprately, we can inmediatly create a series from a dictionary." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "population_dict = {'California': 38332521,\n", - " 'Texas': 26448193,\n", - " 'New York': 19651127,\n", - " 'Florida': 19552860,\n", - " 'Illinois': 12882135}\n", - "population = pd.Series(population_dict)\n", - "population" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "So we can simply ask, What is the population of California?" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "population['California']" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "And we can even perform slicing on the `'string'` indices:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "population['California':'New York']" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Notice that the indices can be non-contiguous numbers in any order and the values can be strings" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "series = pd.Series({2:'a', 1:'b', 3:'c', 6:'d'})" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "series[1]" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Which can lead to counter-intuitive slicing such as" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "series[1:6].values" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Data Frames as generalized numpy arrays and special dictionaries" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "From several dictionaries one can also construct a `DataFrame` object, basically what we scientists call tables, with a row of headers and a column of indices." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "area_dict = {'California': 423967, 'Texas': 695662, 'New York': 141297,\n", - " 'Florida': 170312, 'Illinois': 149995}" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Create a Series object\n", - "area = pd.Series(area_dict)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "area" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Now use population and area as two different columns in a `\"database\"`, which corresponds to a DataFrame." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "states = pd.DataFrame({'population': population,\n", - " 'area': area})" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "states" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "> **NOTE:** Notice the nice printing of a DataFrame in the form of a table !" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "DataFrame also has an index attribute" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "states.index" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "But now also the columns, have a generalized index" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "states.columns" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "A `DataFrame` behaves like a dictionary in which the `keys` are the columns" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "states['area']" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "And as opposed to a `numpy` array it does not return the rows when indexed with numbers" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# This will raise an error, uncomment to see it\n", - "#states[0]" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "A `DataFrame` can be created from a list of dictionaries and `Pandas` will merge the indices accordingly." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "list_of_dicts = [{'a': 1, 'b': 2}, {'b': 3, 'a': 4}]" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "pd.DataFrame(list_of_dicts)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "If some indices are not present in one or more of the dicts, `Pandas` will fill the columns with `naN`s." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "list_of_dicts = [{'a': 1, 'b': 2, 'c':3}, {'a': 3, 'd': 4, 'b':6}]\n", - "pd.DataFrame(list_of_dicts)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "A useful tool for scientists working with numerical data, is to construct a `Pandas` `DataFrame` out of a numpy array, in which we label the rows and columns with strings." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "xy_data=pd.DataFrame(np.random.rand(3, 2),\n", - " columns=['X', 'Y'],\n", - " index=['a', 'b', 'c'])\n", - "xy_data" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "In the same way that dictionaries can be \"filled-up\" by providing a new \"key-value\" pair, the same applies for `DataFrames`:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "xy_data['Z'] = pd.Series([1.,1.,1.], index=xy_data.index) # We pass the same index of the original DataFrame\n", - "xy_data" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "> **Puzzle 6:** Do you think data frames can also be transposed like numpy arrays? \n", - "What is the output of `(xy_data.T).columns[2]` ?\n", - "\n", - " * Option a): `c`\n", - " * Option b): `X`\n", - " * Option c): `0.303641`" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "#Answer Puzzle 5:\n", - "#Uncomment to see the result\n", - "#(xy_data.T).columns[2]" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Manipulating Data Frames" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "A very useful tool for scientists is to be able to compute derived quantities from given data.\n", - "In our `states` DataFrame:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "states" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We can create quickly a new `density` column by operating on the available data columns" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "states['density'] = states['population']/states['area']" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "states" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Notice that now the columns are attributes of the `DataFrame`, which return `Series`." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "states.area" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "states.density" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "*Masking* also works for `DataFrames`" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Which states have a density larger than 100?\n", - "states[states.density > 100]" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "As you can easily see, this properties are quite powerful when one has to analyze large datasets with thousands of columns and rows." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Indexing with integers: loc and iloc" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "data = pd.Series(['a', 'b', 'c', 'd'], index=[1, 2, 5, 6])" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Notice that when the indices are integers, some confusions might appear." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# explicit index when indexing\n", - "data[2]" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# This will raise an error: (uncomment to see it and comment back)\n", - "#data[3]" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "> **Puzzle 6:** What do you expect from the following slice: `data[2:4]` ? Which values will be returned?" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "#Answer Puzzle 6:\n", - "#Uncomment to see the answer\n", - "#data[2:4]" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The maybe surprising result, is because `Pandas` uses the explicit indices in `index` when indexing, while the implicit (standard numpy) indices when slicing." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Due to this confusion, it is better to use the attributes `loc`, `iloc`." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### loc" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "First, the `loc` attribute allows indexing and slicing that always references the ***explicit*** index:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "data.loc[1]" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "data.loc[1:3]" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "states.loc['California':'Texas']" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### iloc" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The `iloc` attribute allows indexing and slicing that always references the implicit Python-style index:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "data.iloc[1]" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "data.iloc[1:3]" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "> **NOTE:** Notice the difference with `loc` above !" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "states.iloc[0:2]" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "One guiding principle of Python code is that \"explicit is better than implicit.\" The explicit nature of `loc` and `iloc` make them very useful in maintaining clean and readable code; especially in the case of integer indexes, I recommend using these both to make code easier to read and understand, and to prevent subtle bugs due to the mixed indexing/slicing convention. (from the PDSH)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Subtleties of indexing" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "For `DataFrames`, the situation is a bit more complicated. Passing a single index to the values, as we saw above, returns a row:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "states.values[0:2]" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Passing a column name as a key, returns a column:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "states['area']" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "But slicing is always performed on the rows:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "states['California':'Texas']" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "states.iloc[:3, :2]\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Equivalently:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "states.loc[:'New York',:'area']" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Operations between DataFrames and Series" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "In the same way as boradcasting for `numpy` where we could have operations between arrays of different dimensions, here we can operate between `DataFrames` and `Series`." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "This is quite useful again for treatment of numerical and statistical data." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Create a `DataFrame` of random ints of size (3,4), with given column names." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "df = pd.DataFrame(np.random.randint(0, 10, (3, 4)),\n", - " columns=['A', 'B', 'C', 'D'])\n", - "df" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Apply a complicated function on the whole `DataFrame`" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "np.sin(df * np.pi / 4)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Internally, this has been done using `uFuncs` and `broadcasting`." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "For numpy arrays, the operation works `row-wise` (notice the first row always being zero):" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "A = np.random.randint(10, size=(3, 4))\n", - "A - A[0]" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Create a `DataFrame` from `A`:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "df = pd.DataFrame(A, columns=list('ABCD'))\n", - "df" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Subtract the `0` column:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "df - df.iloc[0]" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "If you wish to operate column-wise, use `loc` and the corresponding `uFunc` along the wanted axis." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "df.subtract(df['A'], axis=0)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Pandas has lots more of functionality that we can't cover in this session. But a look at the Python Data Science Handbook can reveal many more methods, attributes and tricks." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# Importing CSV files" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "A useful property of Pandas, is that it allows us to work with spreadsheets in a natural way." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "#Import CSV data of average monthly Temperatures in Turrialba, Costa Rica from 1958 until 2016" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "cr_temp = pd.read_csv('./materials/CR_Temp.csv')" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Look at the first 5 rows with head()\n", - "cr_temp.head()" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "If we want to avoid having an index column which is just an integer count of the rows, we can specify it at import time." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "cr_temp = pd.read_csv('./materials/CR_Temp.csv', index_col=1)\n", - "cr_temp.head()" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "#Get the columns\n", - "cr_temp.columns" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "#Get the index\n", - "cr_temp.index" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "#Get the temperature of the 'ENE' month (January) in 1965\n", - "cr_temp.ENE[1965]" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "> **Exercise 1:** Compute the yearly average and check if it matches what is given in the last column of the csv file. You can drop the `N` column with pd.drop(columns=...)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "> **Exercise 2:** Averaging over all the years, which is the hottest month in Turrialba, CR? *Hints:* np.mean, np.max, `loc` and slicing." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "> **Exercise 3:** Using the built-in plotting routine of Pandas, can you notice any effect of climate change over the years? *Hint:* plot is a built-in method of a Pandas `DataFrame`. You can plot any column or any row using the masking and slicing seen above." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "---" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "> **[Answers](./Answers.ipynb) to puzzles and exercises**" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [] - } - ], - "metadata": { - "hide_input": false, - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.7.3" - }, - "toc": { - "base_numbering": 1, - "nav_menu": {}, - "number_sections": true, - "sideBar": true, - "skip_h1_title": false, - "title_cell": "Table of Contents", - "title_sidebar": "Contents", - "toc_cell": false, - "toc_position": {}, - "toc_section_display": true, - "toc_window_display": false - } - }, - "nbformat": 4, - "nbformat_minor": 2 -} diff --git a/python/Pythonic.ipynb b/python/Pythonic.ipynb deleted file mode 100644 index 3b213a4..0000000 --- a/python/Pythonic.ipynb +++ /dev/null @@ -1,1611 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\"CosmoStat\n", - "
\n", - "
\n", - "
\n", - "
\n", - "\n", - "# Pythonic Thinking\n", - "---\n", - "\n", - "> Author: Samuel Farrens \n", - "> Email: samuel.farrens@cea.fr \n", - "> Year: 2019 \n", - "> Version: 1.0\n", - "\n", - "---\n", - "
\n", - "\n", - "Python is a high-level interepreted programming language. It provides the simplicity needed for quick development, while retaining the power needed for dedicated software development. One of the core tenets of Python is the emphasis on readability, *i.e.* the ability to implement complex functionality with simple and elegant syntax.\n", - "\n", - "In order to get the most out of using Python, it is essntial to know how to code in a Pythonic way. This notebook will introduce some useful objects and operators that are central to how Python works.\n", - "\n", - "If you are new to Jupyter notebooks note that cells are executed by pressing SHIFT+ENTER (⇧+ ⏎). See the Jupyter documentation for more details.\n", - "\n", - "
\n", - "\n", - "---\n", - "\n", - "There is plenty to cover in this tutorial but first let's take a deep breath and take in *The Zen of Python.*\n", - "\n", - "" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "import this" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Contents\n", - "---\n", - "\n", - "1. [Set-Up](#1-Set-Up)\n", - "1. [A Quick Recap](#2-A-Quick-Recap)\n", - "1. [Unpacking](#3-Unpacking)\n", - " 1. [Basic Unpacking](#Basic-Unpacking)\n", - " 1. [Positional Expansion](#Positional-Expansion)\n", - " 1. [Unpacking with Functions](#Unpacking-with-Functions)\n", - "1. [Dictionaries](#4-Dictionaries)\n", - " 1. [First Example](#First-Example)\n", - " 1. [Dictionary Unpacking](#Dictionary-Unpacking)\n", - " 1. [Keyword Arguments](#Keyword-Arguments)\n", - " 1. [Tips and Tricks](#Tips-and-Tricks)\n", - "1. [List Comprehension](#5-List-Comprehension)\n", - " 1. [Loops](#Loops)\n", - " 1. [List Comprehension](#List-Comprehension)\n", - " 1. [Absolute Magnitude Example](#Absolute-Magnitude-Example)\n", - " 1. [Redshift Example](#Redshift-Example)\n", - " 1. [Conditions](#Conditions)\n", - " 1. [Nested Lists](#Nested-Lists)\n", - "1. [Iterators & Generators](#6-Iterators-&-Generators)\n", - " 1. [Iterators](#Iterators)\n", - " 1. [Built-In Functions](#Built-In-Functions)\n", - " 1. [Generators](#Generators)\n", - " 1. [Generator Expressions](#Generator-Expressions)\n", - "1. [Exercises](#7-Exercises)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 1 Set-Up\n", - "---\n", - "\n", - "The following cell contains some set-up commands. Be sure to execute this cell before continuing." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Notebook Set-Up Commands\n", - "\n", - "import math\n", - "\n", - "def print_error(error):\n", - " \"\"\" Print Error\n", - " \n", - " Function to print exceptions in red.\n", - " \n", - " Parameters\n", - " ----------\n", - " error : string\n", - " Error message\n", - " \n", - " \"\"\"\n", - " print('\\033[1;31m{}\\033[1;m'.format(error))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 2 A Quick Recap\n", - "---\n", - "\n", - "Before getting into the specifics of Pythonic coding, let's take a second to refresh the basics. In order to understand the topics covered in this notebook you will need to know the following:\n", - "\n", - "### Basic Python object types " - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Basic Python objects\n", - "\n", - "myint = 1\n", - "print('Object is of type:', type(myint))\n", - "\n", - "myfloat = 1.0\n", - "print('Object is of type:', type(myfloat))\n", - "\n", - "mybool = True\n", - "print('Object is of type:', type(mybool))\n", - "\n", - "mystring = 'hello'\n", - "print('Object is of type:', type(mystring))\n", - "\n", - "mylist = [1, 2, 3]\n", - "print('Object is of type:', type(mylist))\n", - "\n", - "mytuple = (1, 2, 3)\n", - "print('Object is of type:', type(mytuple))\n", - "\n", - "myset = set([1, 1, 2])\n", - "print('Object is of type:', type(myset))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Standard operators" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Standard operators\n", - "\n", - "print('Addition: 1 + 2 = ', 1 + 2)\n", - "print('Subtraction: 1 - 2 = ', 1 - 2)\n", - "print('Multiplication: 4 * 2 = ', 4 * 2)\n", - "print('Division: 4 / 2 = ', 4 / 2)\n", - "print('Floor Division: 4 // 2 = ', 4 // 2)\n", - "print('Exponentiation: 4 ** 2 = ', 4 ** 2)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Simple logic operations" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Simple logic operations\n", - "\n", - "print('True and True = ', True and True)\n", - "print('True and False = ', True and False)\n", - "print('True or True = ', True or True)\n", - "print('True or False = ', True or False)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Functions" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Functions\n", - "\n", - "def say_hello():\n", - " print('Hello world!')\n", - " \n", - "def show_value(value):\n", - " print('My value is {}.'.format(value))\n", - " \n", - "say_hello()\n", - "show_value(5)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 3 Unpacking\n", - "---\n", - "\n", - "A fundamental first step on your journey of Pythonic coding is understanding how handle array-like objects (*i.e.* lists and tuples).\n", - "\n", - "### Basic Unpacking\n", - "\n", - "As you are no doubt aware lists and tuples can be indexed by passing an element number inside `[]`..." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Create a tuple\n", - "mytuple = (1, 2, 3)\n", - "\n", - "# Get index 1 from the tuple\n", - "print('mytuple[1] =', mytuple[1])" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "... however, it is also possible to *unpack* these objects." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Unpack a tuple\n", - "val1, val2, val3 = (1, 2, 3)\n", - "\n", - "print('val1 = {} ; val2 = {} ; val3 = {}'.format(val1, val2, val3))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Notice that when three objects are assigned to the tuple each value is unpacked automatically, but if we attempt to unpack three values into two objects we raise an exception." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Try to unpack values into two objects\n", - "try:\n", - " val1, val2 = (1, 2, 3)\n", - "except Exception as error:\n", - " print_error(error)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Positional Expansion\n", - "\n", - "We can bypass the problem using the *splat* (`*`) operator. In addition to be used as the standard operator for multiplication, `*` also has special unpacking features." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Print mytuple\n", - "print('mytuple =', mytuple)\n", - "\n", - "# Print mytuple after unpacking\n", - "print('unpacked mytuple =', *mytuple)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Using this operator we can now unpack our three-element tuple into just two objects." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Unpack a tuple\n", - "val1, *val2 = mytuple\n", - "\n", - "print('val1 = {} ; val2 = {}'.format(val1, val2))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We can see that the first value was unpacked and the remaining values were packed into a list.\n", - "\n", - "> **Puzzle 1:** Guess the value of `val2` in the following example." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Create a list\n", - "mylist = [7, 2, 8, 2, 6]\n", - "\n", - "# Unpack the list\n", - "val1, *val2, val3 = mylist\n", - "\n", - "# Uncomment to see the answer\n", - "# print(val2)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We can also recursively unpack array-like objects.\n", - "\n", - "> **Puzzle 2:** Guess the value of `b` in the following cell." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Create a list of tuples\n", - "list_of_tuples = [(1, 2, 3), (4, 5, 6), (7, 8, 9)]\n", - "\n", - "# Unpack the list\n", - "*A, B = list_of_tuples\n", - "\n", - "# Unpack one of the tuples\n", - "*a, b = A[1]\n", - "\n", - "# Uncomment to see the answer\n", - "# print('b =', b)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "> Further Reading\n", - "> - Understanding the Asterisk of Python" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Unpacking with Functions\n", - "\n", - "Unpacking can be particularly useful for passing arguments to functions. *e.g.* if you want to pass a single object to a function that expects multiple arguments." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Define a function that takes multiple arguments\n", - "def myfunc(a, b, c):\n", - " \n", - " return a + b - c\n", - "\n", - "# Set a tuple of values\n", - "myvalues = (4, 5, 3)\n", - "\n", - "# Unpack the values into the function\n", - "print('The result of myfunc is {}.'.format(myfunc(*myvalues)))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Alternatively, you could pass multiple values to a function that does not know how many arguments to expect." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Define a function that has an unknown number of arguments\n", - "def count_agrs(*args):\n", - " \n", - " return len(args)\n", - "\n", - "# Pass multiple values to the function\n", - "print('count_agrs received {} arguments.'.format(count_agrs(1, 2, 3)))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 4 Dictionaries\n", - "---\n", - "\n", - "Python provides many different ways of storing multiple values in a single object (*e.g.* lists, tuples, sets, *etc*). One of the most useful, and certainly one of the most important, are *dictionaries*. \n", - "\n", - "Dictionaries are designated with `{}` and are comprised of two main components: *keys* and *values*. The benefit that dictionaries provide with respect to simpler objects like lists is the ability to label values. This makes it a lot easier to store a large number of values in a single object without losing track of what they are.\n", - "\n", - "### First Example\n", - "\n", - "Let's look at a concrete example. Imagine we want to keep track of the colours associated to the Teenage Mutant Ninja Turtles.\n", - "\n", - "\n", - "\n", - "We could start by defining a unique object for each turtle." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Unique objects for turtle colours\n", - "Leonardo = 'blue'\n", - "Raphael = 'red'\n", - "Donatello = 'purple'\n", - "Michelangelo = 'orange'\n", - "\n", - "print('Leonardo wears {}.'.format(Leonardo))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "It might be nicer instead to define a single object." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# List of turtle colours\n", - "turtles = ['blue', 'red', 'purple', 'orange']\n", - "\n", - "print('Raphael wears {}.'.format(turtles[1]))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "This, however, assume you will remember the order and names of the turtles. A dictionary makes it possible to retain both the simplicty of a list, but the details of the single objects.\n", - "\n", - "To define a dictionary object we need to specify a series of keys followed by a colon (`:`) and then the corresponding values, all comma separated and within `{}`." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Dictionary of turtle names and colours\n", - "turtles = {'Leo': 'blue', 'Raph': 'red', 'Donny': 'purple', 'Mickey': 'orange'}" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We can look at the pieces that make up the dictionary by looking at the `keys`, `values` and `items`." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Show dictionary components\n", - "print('dict:', turtles)\n", - "print('keys:', turtles.keys())\n", - "print('values:', turtles.values())\n", - "print('items:', turtles.items())" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Now we can use the keys (*i.e.* the turtles' names) to look up the values (*i.e.* the corresponding colours). *e.g.* To look up what colour Donatello wears we simply need to pass the key `'Donny'` to the dictionary `turtles` as follows.\n", - "\n", - "```python\n", - "turtles['Donny']\n", - "```" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Look up dictionary value|\n", - "print('Donatello wears {}.'.format(turtles['Donny']))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "> Note that dictionary look up in Python is highly optimised, but it only works one way (*i.e.* keys → values).\n", - "\n", - "It is possible to add entries to a dictionary by specifying a new key and assigning a corresponding value." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Add dictionary entry\n", - "turtles['Splinter'] = 'wine'\n", - "print(turtles)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "It is also possible to modify existing entry values by takinging an existing key and assigning a new value." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Modify dictionary value\n", - "turtles['Splinter'] = 'yellow'\n", - "print(turtles)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "You will raise an exception if you try to index a dictionary or look up a key for a entry that doesn't exist." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "try:\n", - " turtles['Rocksteady']\n", - "except Exception as error:\n", - " print_error('KeyError: {}'.format(error))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Dictionary Unpacking\n", - "\n", - "Similarly to array-like objects, dictionaries can be unpacked." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Create a dictionary\n", - "mydict = {'a' : 1, 'b' : 2, 'c' : 3}\n", - "\n", - "# Unpack the dictionary\n", - "val1, val2, val3 = mydict\n", - "\n", - "print('val1 = {} ; val2 = {} ; val3 = {}'.format(val1, val2, val3))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Note that by default unpacking is performed over the keys. To unpack the values you have to be more explicit." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Unpack the dictionary values\n", - "val1, val2, val3 = mydict.values()\n", - "\n", - "print('val1 = {} ; val2 = {} ; val3 = {}'.format(val1, val2, val3))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "> **Puzzle 3:** What values will you get when you unpack the dictionary items?" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Unpack the dictionary values\n", - "val1, val2, val3 = mydict.items()\n", - "\n", - "# Uncomment to see the answer\n", - "# print('val1 = {} ; val2 = {} ; val3 = {}'.format(val1, val2, val3))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Keyword Arguments\n", - "\n", - "In addition to standard position arguments, Python functions accept *keyword arguments* (or simply *kwagrs*), *i.e.* a dictionary of keys and values. \n", - "\n", - "kwargs can be used to define default values in functions." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Define a function with defaults\n", - "def myfunc(a=1, b=2, c=2):\n", - " \n", - " return a + b * 3\n", - "\n", - "print('myfunc =', myfunc())" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "They can also be used to pass arguments in any given order." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "print('myfunc =', myfunc(c=5, a=3, b=7))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "There is also a special keyword expansion operator `**` thay behaves similarly to the splat operator `*` for use in functions." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Create dictionary of values\n", - "mydict = {'a': 3, 'b': 4, 'c': 5}\n", - "\n", - "# Unpack the dictionary kwargs into the function\n", - "print('myfunc =', myfunc(**mydict))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Finally, you can define functions for which an idefinite number of kwargs can be passed." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Define a function with optional kwargs\n", - "def subtract(a, b, **kwargs):\n", - " \n", - " res = a - b\n", - " \n", - " if 'return_abs' in kwargs and kwargs['return_abs']:\n", - " res = abs(res)\n", - " \n", - " if 'add_value' in kwargs:\n", - " res += kwargs['add_value']\n", - "\n", - " return res\n", - " \n", - "print('1 - 2 = {}'.format(subtract(1, 2)))\n", - "print('|1 - 2| = {}'.format(subtract(1, 2, return_abs=True)))\n", - "print('1 - 2 + 3 = {}'.format(subtract(1, 2, add_value=3, dummy_arg=None)))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Tips and Tricks\n", - "\n", - "There are plenty of useful tricks that make dictionary handling a lot easier.\n", - "\n", - "For example, you can convert a list of tuples to a dictionay." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Tuples of film names and release dates\n", - "film1 = ('alien', '1979')\n", - "film2 = ('the shining', '1980')\n", - "film3 = ('the evil dead', '1981')\n", - "film4 = ('blade runner', '1982')\n", - "\n", - "# List of films\n", - "films = [film1, film2, film3, film4]\n", - "print('List: ', films)\n", - "print()\n", - "\n", - "# Dictionary of films\n", - "films = dict(films)\n", - "print('Dict: ', films)\n", - "print()\n", - "print('Alien came out in {}.'.format(films['alien']))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\n", - "\n", - "You can concatinate two dictionaries in a single line." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Make another dictionary\n", - "more_films = dict([('WarGames', '1983'), ('Amadeus', '1984')])\n", - "\n", - "# Concatinate the dictionaries\n", - "films_80s = {**films, **more_films}\n", - "print('Concatination: ', films_80s)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "You can clear the contents of an existing dictionary using the `clear` method." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Clear the films dictionary\n", - "films.clear()\n", - "print('Clear: ', films)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Finally, you can add, look up and remove dicttionary entries using the `update`, `get` and `pop` methods respectively." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Add a value to the dictionary\n", - "films.update({'Back to the Future': '1985'})\n", - "print('Update:', films)\n", - "print()\n", - "\n", - "# Get the value for the key\n", - "print('Get: Back to the Future came out in {}.'.format(films.get('Back to the Future')))\n", - "print()\n", - "\n", - "# Pop the entry out of the dictionary\n", - "films.pop('Back to the Future')\n", - "print('Pop: ', films)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "> Note that the `get` method will not raise an error if the key is not found, instead it will simply return `None`." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "> Further Reading \n", - "> - Python dicts" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 5 List Comprehension\n", - "---\n", - "\n", - "This section introduces the concept of *list comprehension*, a tool that you will likely have seen implemented but perhaps not understood. The most sensible way to begin understanding this concept is by looking at basic loops.\n", - "\n", - "### Loops\n", - "\n", - "Loops are a core component of any programming language and Python is certainly no different." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Simple loop\n", - "for i in (1, 2, 3):\n", - " print('i =', i)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Here we can see the use of a basic `for` statement using the variable `i`, which iterates through the values in a tuple.\n", - "\n", - "That being said, Python does offer an extremely powerful way to generate lists from loops using a process called list comprehension. In lower level lanaguages (*e.g.* C, Fortran, *etc.*) arrays are usually built by declaring an empty object of a given size and type and then iteratively filling this array. It is possible to do something similar in Python, in fact, this is often done by people who come to Python from one of these languages." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Create a list of size 3\n", - "my_int_list = [None, None, None]\n", - "\n", - "# Run a loop\n", - "for i in (1, 2, 3):\n", - " # Fill the list values\n", - " my_int_list[i - 1] = i\n", - " \n", - "print('my_int_list =', my_int_list)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "While this works, it goes against most of what Python aims to achieve. Using Python list properties we can improve a little bit." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Create an empty list\n", - "my_int_list = []\n", - "\n", - "# Run a loop\n", - "for i in range(1, 4):\n", - " # Add list values\n", - " my_int_list.append(i)\n", - " \n", - "print('my_int_list =', my_int_list)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "> Note that `range` is a built-in function that produces a list-like object of values within a given range.\n", - "\n", - "This approach may be appropriate in certain situations, particularly if we plan to *break* the loop when a condition is met. For this particular example, however, there is a much easier way to generate the desired object.\n", - "\n", - "### List Comprehension\n", - "\n", - "To perform list comprehension we need the following structure: `[final_expression for_loop_and_conditions]`. For example, to generate the same list object as the previous cell we can do this:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Create an object through list comprehension\n", - "my_int_list = [i for i in range(1, 4)]\n", - "\n", - "print('my_int_list =', my_int_list)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "In this example the first `i` is the final expression we want (*i.e.* simply the value), this is followed by a for loop that is identical to that in the loop example.\n", - "\n", - "Using list comprehension it is possible to create a list object in one line! This may seem trivial, but it is actually a very useful and highly optimised tool in Python.\n", - "\n", - "### Absolute Magnitude Example\n", - "\n", - "Imagine you need to calculate the absolute magnitude \n", - "\n", - "$$M = m - 5\\log_{10}(d_{\\textrm{pc}}) + 5$$\n", - "\n", - "of various stars:\n", - "\n", - "|Star|$m_v$|$d_{\\textrm{pc}}$|\n", - "|----|-----|-----------------|\n", - "|Alpha Centauri|-0.3|1.3|\n", - "|Canopus|-0.72|30.1|\n", - "|Rigel|0.14|276.1|\n", - "|Deneb|1.26|490.8|\n", - "\n", - "\n", - "\n", - "One way would be be to store the information about the stars in a dictionary (as we just learned about them)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Dictionary of star properties\n", - "stars = dict([('alpha centauri', {'mv': -0.3, 'dpc': 1.3}), \n", - " ('canopus', {'mv': -0.72, 'dpc': 30.1}), \n", - " ('rigel', {'mv': 0.14, 'dpc': 276.1}), \n", - " ('deneb', {'mv': 1.26, 'dpc': 490.8})])\n", - "\n", - "print('The apparent magnitude of Rigel is {}.'.format(stars['rigel']['mv']))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "and to loop through the stars." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Create an empty list\n", - "abs_mags = []\n", - "\n", - "# Loop through the stars\n", - "for star in stars.values():\n", - " # Calculate absolute magnitude\n", - " mag = round(star['mv'] - 5 * math.log10(star['dpc']) + 5, 2)\n", - " # Add value to list\n", - " abs_mags.append(mag)\n", - " \n", - "print('Absolute magnitude values =', abs_mags)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "A more Pythonic approach, however, would be to define a reusable function and generate the final object via list comprehension." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Define a function to calculate absolute magnitude\n", - "def abs_mag(mv, dpc):\n", - " \n", - " return round(mv - 5 * math.log10(dpc) + 5, 2)\n", - "\n", - "# Produce a list of absolute magnitude values\n", - "abs_mags = [abs_mag(star['mv'], star['dpc']) for star in stars.values()]\n", - "\n", - "print('Absolute magnitude values =', abs_mags)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "This way, we not only have a function that can be used for future calculations, but we also generate our object in one line.\n", - "\n", - "### Redshift Example\n", - "\n", - "Let's look at another example. Imagine you want to calculate the redshift of an object given the emitted and observed wavelenths.\n", - "\n", - "$$z = \\frac{\\lambda_{\\textrm{obs}}-\\lambda_{\\textrm{emit}}}{\\lambda_{\\textrm{emit}}}$$\n", - "\n", - "|$\\lambda_{\\textrm{emit}}$|$\\lambda_{\\textrm{obs}}$|\n", - "|-------------------------|------------------------|\n", - "|450|679|\n", - "|348|956|\n", - "|579|264|\n", - "\n", - "We can approach this in a very similar way to the previous example, except in this case let's try without using a dictionary." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Create lists of wavelength values\n", - "lambda_emit = [450., 348., 579.]\n", - "lambda_obs = [679., 956., 264.]\n", - "\n", - "# Define a function to calculate redshift\n", - "def redshift(emit, obs):\n", - " \n", - " return round((obs - emit) / emit, 3)\n", - "\n", - "# Generate redshift values via list comprehension\n", - "z = [redshift(e, o) for e, o in zip(lambda_emit, lambda_obs)]\n", - "\n", - "print('Redshifts =', z)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "> **Puzzle 4:** Note the use of the `zip` function. Any guess for what it does?" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Uncomment to see the answer\n", - "# print('The result of zip is:', list(zip(lambda_emit, lambda_obs)))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Conditions\n", - "\n", - "It is possible to embed a complex series of conditions (*e.g.* `if` statements) into list comprehension." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# A list of names\n", - "names = ['paul', 'jessica', 'leto', 'gunrey', 'duncan', 'piter']\n", - "\n", - "# A new list of names satisfying various conditions\n", - "new_names = [name for name in names if 'e' in name and name[0] < 'k']\n", - "\n", - "print('Names containing the letter \"e\" and starting with a letter lower than \"k\":', new_names)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Nested Lists\n", - "\n", - "It is also possible to flatten nested lists using unpacking and conditions.\n", - "\n", - "" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Nested list\n", - "cats = [['lion-o', 'cheetara', 'tygra', 'panthro'], ['wilykit', 'wilycat'], ['snarf'], ['mumm-ra']]\n", - "\n", - "# Flatten list with condition\n", - "cats = [cat for group in cats for cat in group if cat!='mumm-ra']\n", - "\n", - "print('Thundercats:', cats)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "> Further Reading \n", - "> - List Comprehension in Python" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 6 Iterators & Generators\n", - "---\n", - "\n", - "Some very useful tools in Python, particularly when it comes to memory management, are iterators and generators.\n", - "\n", - "### Iterators\n", - "\n", - "The `iter` function can be used to convert objects to iterators, such as lists..." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Create list iterator\n", - "myiter = iter([1, 2, 3, 4])\n", - "\n", - "print('myiter =', myiter)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "... or tuples." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Create tuple iterator\n", - "myiter = iter((1, 2, 3, 4))\n", - "\n", - "print('myiter =', myiter)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "> **Puzzle 5:** How would you go about converting an iterator into a list?" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Uncomment and edit the following lines appropriately\n", - "# Convert iterator into a list\n", - "# mylist = myiter\n", - "# \n", - "# print('mylist =', mylist)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "It is not possible to index an iterator." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Try to index iterator\n", - "try:\n", - " myiter[0]\n", - "except Exception as error:\n", - " print_error(error)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Instead, iterator values are accessed one at a time using the `next` function." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "print('next(myiter) =', next(myiter))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "> Try re-running the previous cell multiple times to see what happens.\n", - "\n", - "### Built-In Functions\n", - "\n", - "Most built-in functions that work on lists (*e.g.* `sum`, `min`, `max`, *etc.*) will also work directly on iterators." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Create a list\n", - "mylist = [1, 2, 3, 4, 5]\n", - "\n", - "# Create an iterator\n", - "myiter = iter(mylist)\n", - "\n", - "print('sum(mylist) =', sum(mylist))\n", - "print('sum(myiter) =', sum(myiter))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "But note that, unlike a list, once an interator has been fully run it remains empty." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "print('sum(mylist) =', sum(mylist))\n", - "print('sum(myiter) =', sum(myiter))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "In addtion, many built-in function return iterator-like objects by default (*e.g.* `zip`)." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Create zip object\n", - "myzip = zip([1, 2], [3, 4])\n", - "\n", - "print('Next:', next(myzip))\n", - "print('Next:', next(myzip))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Generators\n", - "\n", - "Generators can be thought of as functions that behave like iterators. \n", - "\n", - "Imagine you want a function that can provide the square of a list of values. One approach would be to write a function that takes all of the inputs and returns a list of the corresponding squared values. " - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Function to calculate square of input values\n", - "def square(values):\n", - " \n", - " return [value ** 2 for value in values]\n", - "\n", - "print('Squared values =', square([1, 2, 3, 4, 5]))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "So, while this certainly works, we are storing all of the outputs in memory. In some situations we may only wish to calculate the output value when it's needed. To do so we can define a *generator* using a `yield` statement. " - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Define a generator function to calculate square of input values\n", - "def square(values):\n", - " \n", - " for value in values:\n", - " \n", - " yield value ** 2\n", - " \n", - "# Create a generator object\n", - "mygen = square([1, 2, 3, 4, 5])\n", - "\n", - "# Run the generator\n", - "print('next(mygen) =', next(mygen))\n", - "print('next(mygen) =', next(mygen))\n", - "print('next(mygen) =', next(mygen))\n", - "print('next(mygen) =', next(mygen))\n", - "print('next(mygen) =', next(mygen))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "In this case the function is only evaluated when the `next` function is called.\n", - "\n", - "An important feature of generators is that mutiple `yield` statements can be included in a single function." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Define a generator function with multiple yields\n", - "def peasant():\n", - " \n", - " yield \"Ready to work.\"\n", - " yield \"Job's done.\"\n", - " yield \"Alright.\"\n", - " yield \"You're the king? Well, I didn't vote for you.\"\n", - "\n", - "# Create a generator object\n", - "p = peasant()\n", - "\n", - "# Run the generator\n", - "print(next(p))\n", - "print(next(p))\n", - "print(next(p))\n", - "print(next(p))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Generator Expressions\n", - "\n", - "Similarly to list comprehension, it is possible to produce generator objects in one line using `()`." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Define a generator expression to square values from 1 to 5\n", - "mygen = (value ** 2 for value in range(1, 6))\n", - "\n", - "print('next(mygen) =', next(mygen))\n", - "print('next(mygen) =', next(mygen))\n", - "print('next(mygen) =', next(mygen))\n", - "print('next(mygen) =', next(mygen))\n", - "print('next(mygen) =', next(mygen))" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Calculate the sum of cubes using a generator expression\n", - "soc = sum((value ** 3 for value in range(10)))\n", - "\n", - "print('The sum of cubes from 0 to 9 is {}.'.format(soc))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "> Further Reading\n", - "> - Python Practice Book: Iterators" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 7 Exercises\n", - "---\n", - "\n", - "1. Create a list of your favourite superheros and another list of their secret identities.\n", - " 1. Convert your two lists into a dictionary. (Can you do it in one line?)\n", - " 1. Remove one of your heroes and add a villain to your dictionary.\n", - " 1. Add a character that has multiple identities to your dictionary. (What kind of object should this be?)\n", - " 1. Demonstrate that you can look up one of your character's identities.\n", - " \n", - "\n", - "| Superhero | Identity |\n", - "|:-----------:|:---------------------------:|\n", - "| Iron Man | Tony Stark |\n", - "| The Thing | Ben Grimm |\n", - "| Storm | Ororo Munroe |\n", - "| Spider-Man | Peter Parker, Miles Morales |\n", - "\n", - " \n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Add your solution here" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "2. Use Hubble's law ($v=H_{0}\\,D$) to calculate the distance (in Mpc) of the galaxies in the following table.\n", - "\n", - "|Galaxy|Velocity (km/s)|\n", - "|------|---------------|\n", - "|NGC 123|1320|\n", - "|NGC 2342|5690|\n", - "|NGC 4442|8200|\n", - "\n", - "Remember that $H_0 \\approx 70$ km/s/Mpc." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Add your solution here" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "3. Flatten the following list using list comprehension.\n", - "\n", - "```python\n", - "mylist = [[[1, 2], [3, 4, 5]], [[6], [7, 8]]]\n", - "```" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Add your solution here" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "4. Write a generator function that can be used to calculate the Fibonacci sequence.\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Add your solution here" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.7.3" - } - }, - "nbformat": 4, - "nbformat_minor": 2 -} diff --git a/python/README.md b/python/README.md deleted file mode 100644 index cf34aeb..0000000 --- a/python/README.md +++ /dev/null @@ -1,58 +0,0 @@ - - -[![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/CosmoStat/Tutorials/master) - -## Introduction to Python ---- - -> Authors: [Samuel Farrens](http://www.cosmostat.org/people/sfarrens), [Santiago Casas](http://www.cosmostat.org/people/santiago-casas) -> Year: 2019 -> Email: [samuel.farrens@cea.fr](mailto:samuel.farrens@cea.fr), [santiago.casas@cea.fr](mailto:santiago.casas@cea.fr) - - - -This tutorial is comprised of a series of Jupyter notebooks with simple demonstrations and exercises. - -All code blocks are provided in Python (support for >=3.5) and the number of external packages required to run the examples has be kept to a minimum. All of the code has been clearly presented inside the notebook at least once to avoid the use of any "black boxes" for solving the problems presented. - -### Requirements - -In order to run the tutorial notebooks tutees will need to have the following installed: - -* Python (require >=3.5) -* Astropy (recommend >=3.1.2) -* Jupyter (recommend >=1.0.0) -* Matplotlib (recommend >=3.0.3) -* NumPy (recommend >=1.16.2) -* Pandas (recommend >= 0.24.0) -* scikit-learn (recommend >= 0.21.0) -* SciPy (recommend >=1.2.1) -* Seaborn (recommend >= 0.9.0) - -### Notebooks - -#### Tutorial 1: Beginner Topics - -The objective of this tutorial is to provide a first look at Python for beginners. The level is aimed at individuals with little or no experience whatsoever with Python. Experienced users are unlikely to benefit from this tutorial. - -1. [Table of Contents](./Intro.ipynb) -1. [Native Python](./Native.ipynb) -1. [Basic Numpy](./Numpy.ipynb) -1. [Matplotlib](./Matplotlib.ipynb) -1. [Other Packages](./Other-Packages.ipynb) -1. [Answers to Exercises](./Answers-to-exercises.ipynb) - -#### Tutorial 2: Intermediate and Advanced Topics - -The objective of this tutorial is to provide a more in-depth look at object-oriented -and pythonic coding. The level is aimed at individuals with some experience with -Python and good knowledge of basic object types. This tutorial will likely benefit all except the most advanced users. - -1. [Table of Contents](./Intro-2.ipynb) -1. [Pythonic Thinking](./Pythonic.ipynb) -1. [Numpy](./Numpy-Intro.ipynb) -1. [Pandas](./Pandas-Intro.ipynb) -1. [Scipy-Cosmo](./Scipy-Cosmo.ipynb) -1. [The Anatomy of a Python Class: Part I](./Classes_I.ipynb) -1. [The Anatomy of a Python Class: Part II](./Classes_II.ipynb) -1. [Answers to Exercises](./Answers-to-exercises-2.ipynb) diff --git a/python/Scipy-Cosmo.ipynb b/python/Scipy-Cosmo.ipynb deleted file mode 100644 index baea7f2..0000000 --- a/python/Scipy-Cosmo.ipynb +++ /dev/null @@ -1,1270 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\"CosmoStat\n", - "
\n", - "
\n", - "
\n", - "
\n", - "\n", - "# Scipy, Matplotlib and useful tricks applied to Cosmology\n", - "\n", - "---\n", - "\n", - "> Author: Santiago Casas\n", - "\n", - "> Email: santiago.casas@cea.fr \n", - "> Year: 2019 \n", - "> Version: 1.0\n", - "\n", - "\n", - "---\n", - "
" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Let's import some useful libraries" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Tell Jupyter to display plots in this notebook.\n", - "%matplotlib inline\n", - "##\n", - "import numpy as np #numpy for arrays and numerics\n", - "import pandas as pd #pandas for working with ordered datasets\n", - "import scipy as sci # for scientific functions\n", - "from scipy import interpolate #for interpolation\n", - "from scipy import integrate #for integration\n", - "import matplotlib.pyplot as plt # for plotting\n", - "# Import interaction module.\n", - "from ipywidgets.widgets import *\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "> In this tutorial we will do some basic cosmology use functions, dictionaries, numpy arrays, matplotlib and scipy, to illustrate some of the common tools used by researchers in their everyday life." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# Cosmological redshift: easy plots" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The cosmological redshift we observe in galaxies (their spectral lines move towards larger wavelengths) $z$ is related to the relative size of the observable Universe $a$, by $$a = \\frac{1}{1+z}$$ \n", - "\n", - "This can be obtained from the fact that light travels in null-geodesics, since it is described as a relativistic particle." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "> Let's code this in a simple function" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "def a_of_z(z):\n", - " a=1/(1+z)\n", - " return a " - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "z_arr = np.linspace(0.001,3, 100)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "a_arr = a_of_z(z_arr)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "a_arr.shape" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We can plot this using matplotlib's function `plot`, which takes an `x` and a `y` array as arguments, a format argument `fmt = '[marker][line][color]'` and many other optional keyword arguments, like `color`, linestyle, opacity and many more. Available markers, such as `o` for circles, `-.` for dot-dashed linestyles and `'r'` for colors can be checked in the documentation ." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "plt.plot(z_arr, a_arr)\n", - "plt.show()" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "For plots in log-log space, we can use `plt.loglog`. And it is always better to start the plot with a figure creation, since it simplifies exporting and adjustments later on." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "scrolled": true - }, - "outputs": [], - "source": [ - "fig = plt.figure()\n", - "#Let's add some color and linestyle, r for red and a label for the curve.\n", - "plt.loglog(z_arr, a_arr, '-.r', label='$a(z)$') #notice how you can use LaTeX here\n", - "#Let's add a plot title\n", - "plt.title(\"Redshift\")\n", - "#Let's add a legend\n", - "plt.legend()\n", - "#Let's add an x-label, with some font-size\n", - "plt.xlabel(\"z\", fontsize=15)\n", - "# and a y-label, rotated by 90 degrees (which is the default in this case anyway)\n", - "plt.ylabel(\"a\", rotation=90, fontsize=15) \n", - "plt.show()" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# The Hubble function: working with dictionaries and optional parameters" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The Hubble function $H(z)$ can be found solving Einstein's field equations of General Relativity under the Friedmann-Lemaître-Robertson-Walker metric, i.e. under the assumption of homogeneity and isotropy. It describes the expansion rate (speed) of an observable Universe of size a, $H(a) = \\dot{a}/a$." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\"Cosmic" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "If we know the energy and matter components of the Universe today:\n", - " * Cold Dark Matter density $\\Omega_c$ \n", - " * Baryonic (normal) matter density $\\Omega_b$\n", - " * Radiation density $\\Omega_r$\n", - "we can write down the Hubble function as:\n", - "$$ H(z) = H_0 \\sqrt{(\\Omega_c(1+z)^3 + \\Omega_b(1+z)^3) + \\Omega_r(1+z)^4 + \\Omega_{\\Lambda}}$$" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "where this mysterious component $\\Omega_{\\Lambda}$, is what we think is causing the accelerated expansion of the Universe and we think it comes from a *Cosmological constant*. In the general case where we are looking for alternatives to the cosmological constant, we call it also \"Dark Energy\", for the lack of a better name." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "If the Universe is flat, then $\\Omega_{\\Lambda}$ is defined by whatever we can't observe and it's simply given by: $$ \\Omega_{\\Lambda} = 1 - \\Omega_c - \\Omega_b - \\Omega_r $$" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "> Let's code first this constraint:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "def Omega_L(Omega_c, Omega_b, Omega_r):\n", - " oL = 1 - Omega_c - Omega_b - Omega_r\n", - " return oL" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "> Now let's write a function for $H(z)$. If you still don't know or haven't asked, $H_0$ is the rate of expansion of the universe today and it is measured in km/s/Mpc. Its value can be inferred in many ways, from the Cosmic Microwave Background, from Gravitational Lensing or by measuring the distance to Supernovae. Currently it is a hot topic in cosmology, since there are strong tensions in measurements from different probes.\n", - "A value of $70$ km/s/Mpc, means that a galaxy at a distance of $1$Mpc from us, moves away from us at a speed of $70$km/s." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "> To be fancy and practice some dictionary techniques, let's avoid defining so many arguments in the function and let's just pass a dictionary instead, that contains the cosmological parameters." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "> This function will return default parameters when nothing is passed and can return either all the cosmological parameters or a single one." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "def cosmological_parameters(cosmo_pars=dict(), which='all'):\n", - " H0 = cosmo_pars.get('H0', 67.32) # Planck 2018 cosmological parameters as default\n", - " Oc = cosmo_pars.get('Omega_c', 0.26641)\n", - " Ob = cosmo_pars.get('Omega_b', 0.04939)\n", - " Or = cosmo_pars.get('Omega_r', 0.0000930479)\n", - " Om = Ob+Oc\n", - " OL = Omega_L(Oc, Ob, Or)\n", - " default_dict = {'Omega_b': Ob, \n", - " 'Omega_c':Oc,\n", - " 'Omega_m': Om,\n", - " 'Omega_r':Or,\n", - " 'Omega_L':OL,\n", - " 'H0':H0}\n", - " if which=='all':\n", - " return H0, Oc, Ob, Or, Om, OL\n", - " else: # this code is prone to bugs if which is not a string or not a parameter\n", - " return default_dict[which]" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "cosmological_parameters()" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "cosmological_parameters(which='Omega_m')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "> Here we have used `dict.get()` instead of `dict['key']`, since `get('key', default)` allows us to define default values in case the keys are not present in the dictionary. And we have created a keyword argument for cosmo_pars, so we can evaluate $H(z)$ with a single redshift argument:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "def Hubble(z, cosmo_pars=dict()):\n", - " H0, Oc, Ob, Or, Om, OL = cosmological_parameters(cosmo_pars)\n", - " H = H0 * np.sqrt(Om*(1+z)**3 + Or*(1+z)**4 + OL)\n", - " return H" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Notice how we recover the default H_0 when we evaluate at z=0.\n", - "Hubble(0.)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "#Or how we recover another H0 that we passed as dictionary \n", - "Hubble(0., {'H0':73})" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Let's define a dictionary of cosmological parameters" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "mycosmo = {'H0': 72, \n", - " 'Omega_c':0.32,\n", - " 'Omega_b':0.05}" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Now we get the H_0 we actually provided\n", - "Hubble(0., mycosmo)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "> **Exercise 1**: Plot the Hubble function $H(z)$, for a newly created cosmology of your choice, next to `mycosmo` and the default Planck cosmology, using the colors `red`, `green` and `blue`, respectively. Change also the line styles (`'--', or '-' or ':'`) (You can use the `fmt` keywords or the `color` optional argument). Use a range in redshift from 0. to 10. Use linear scales for the plot `plot`. Hint: Copy as much as possible from the plotting code above." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Matter, Radiation and Dark Energy densities as a function of time: Complicated subplots, annotations and labeling." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The density as a function of time of any of those components can be obtained with the Hubble function as follows: \n", - "$$ \\Omega_i(z) = \\frac{H_0^2 \\Omega_i (1+z)^p}{H^2(z)}$$ \n", - "where $i$ is the component, and $p$ is the exponent corresponding to the specific component.\n", - "For matter, $p=3$, for radiation, $p=4$, for the cosmological constant $p=0$." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Let's code this one up, by first creating a dictionary of exponents:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "cosmo_pexp = {'Omega_b': 3, \n", - " 'Omega_c':3,\n", - " 'Omega_m':3,\n", - " 'Omega_r':4,\n", - " 'Omega_L':0}" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "def Omega_of_z(z, component='Omega_c', cosmo_pars=dict({None:0}) ):\n", - " O_generic = cosmological_parameters(cosmo_pars, which=component)\n", - " H0 = cosmological_parameters(cosmo_pars, which='H0')\n", - " p = cosmo_pexp[component]\n", - " O_of_z = (H0**2 * O_generic*(1+z)**p)/np.power(Hubble(z, cosmo_pars),2)\n", - " return O_of_z" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Let's check that the function works for a given array" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "z_arr = np.logspace(-3,2.5, 100)\n", - "Omega_of_z(z_arr)[1:5]" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "crazycosmo = {'H0': 75, \n", - " 'Omega_c':0.58,\n", - " 'Omega_b':0.06, \n", - " 'Omega_r':0.005}" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "> Now let's plot the evolution of matter and radiation species as a function of redshift for two different cosmologies. For this we will use the subplot functionality. There are many ways of doing this, but the most common one is `plt.subplot(Rows, Cols, *kwargs) `, where some of the optional keyword arguments are for sharing axes, for setting the figure size or for adjusting interplot spaces. It will return a figure instance and an array of axis instances, which can be indexed." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "z_arr = np.logspace(-3,2.5, 100)\n", - "#return a figure object instance and an axis object instance\n", - "fig, ax = plt.subplots(2, 2, sharey='row', sharex='col', figsize=(15,8)) #all plots in the same row, share the y-axis.\n", - "\n", - "# once you specify an axis, it is in this instance where plots are performed\n", - "ax[0,0].semilogx(z_arr, Omega_of_z(z_arr, 'Omega_m'), '-', label='$\\Omega_m$', color='orange') \n", - "ax[0,0].semilogx(z_arr, Omega_of_z(z_arr, 'Omega_L'), '-.', label='$\\Omega_{\\Lambda}$', color='violet')\n", - "ax[0,0].legend()\n", - "#Let's add a yellow shade for the matter domination era\n", - "ax[0,0].axvspan(0.279, 300, alpha=0.3, color='yellow')\n", - "# and let's annotate it using text()\n", - "ax[0,0].text(5,0.8, \"matter domination era\", size=10, color='darkorange')\n", - "ax[1,1].set_xlabel('redshift $z$', fontsize=15) #notice the change from plt.xlabel() -> ax.set_xlabel()\n", - "ax[1,0].set_xlabel('redshift $z$', fontsize=15)\n", - "ax[0,1].semilogx(z_arr, Omega_of_z(z_arr, 'Omega_m', crazycosmo), '-', label='$\\Omega_m$', color='orange') \n", - "ax[0,1].semilogx(z_arr, Omega_of_z(z_arr, 'Omega_L', crazycosmo), '-.', label='$\\Omega_{\\Lambda}$', color='violet')\n", - "# Let's add an annotation with an arrow. Check the documentation for more about the settings.\n", - "ax[0,1].annotate('no dark energy domination', xy=(0.01,0.66), xytext=(3,0.4), \n", - " arrowprops=dict(arrowstyle=\"->\", connectionstyle=\"angle3, angleA=0, angleB=90\"))\n", - "ax[0,1].legend()\n", - "ax[1,0].semilogx(z_arr, Omega_of_z(z_arr, 'Omega_c'), '-', label='$\\Omega_c$', color='darkorange') \n", - "ax[1,0].semilogx(z_arr, Omega_of_z(z_arr, 'Omega_r'), ':', label='$\\Omega_{r}$', color='darkblue')\n", - "ax[1,0].legend(loc='upper left') #notice the different location compared to the default\n", - "ax[1,1].set_xlabel('redshift $z$', fontsize=15) #notice the change from plt.xlabel() -> ax.set_xlabel()\n", - "ax[1,0].set_xlabel('redshift $z$', fontsize=15)\n", - "ax[1,1].semilogx(z_arr, Omega_of_z(z_arr, 'Omega_c', crazycosmo), '-', label='$\\Omega_c$', color='darkorange') \n", - "ax[1,1].semilogx(z_arr, Omega_of_z(z_arr, 'Omega_r', crazycosmo), ':', label='$\\Omega_{r}$', color='darkblue')\n", - "ax[1,1].legend()\n", - "# Let's add an annotation with an arrow. Check the documentation for more about the settings.\n", - "ax[1,1].annotate('radiation dominated era', xy=(250,0.66), xytext=(0.001,0.4), \n", - " arrowprops=dict(arrowstyle=\"->\", connectionstyle=\"angle3, angleA=0, angleB=90\"))\n", - "ax[1,1].set_xlabel('redshift $z$', fontsize=15) #notice the change from plt.xlabel() -> ax.set_xlabel()\n", - "ax[1,0].set_xlabel('redshift $z$', fontsize=15) #notice that we only label the bottom 2 plots\n", - "ax[0,0].set_ylabel('Fractional energy density $\\Omega_i$', fontsize=15) #notice the change from plt.xlabel() -> ax.set_xlabel()\n", - "ax[1,0].set_ylabel('Fractional energy density $\\Omega_i$', fontsize=15) #notice that we only label the bottom 2 plots\n", - "#let's save the figure\n", - "fig.savefig('./materials/cosmological_energy_densities.png')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "> **NOTE**: This plotting code seems a bit clumsy and extended, another way of doing this, without so many lines of code, is through a proper loop and some zip() and list comprehensions." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# Hubble function: Working with external data, scipy interpolation and fit" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "One very useful function in numpy is `np.loadtxt` which reads very efficiently data tables and automatically ignores headers (which start usually with `#`)." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "data_h_ofz=np.loadtxt('./materials/alldataEz-H0planck-real-H0planck.txt')\n", - "data_h_ofz[1:5]" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "This data contains measurements of the dimensionless Hubble function $E(z) = H(z)/H_0$ at different redshifts and with given error bars. This data was taken using supernovae, cepheids or cosmic chronometers (measurements of the age of very massive galaxies)." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Let's plot the data using the function `plt.errorbar`" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "fig = plt.figure()\n", - "plt.errorbar(data_h_ofz[:,0], data_h_ofz[:,1], data_h_ofz[:,2], fmt='o', color='black', ecolor='black', capsize=2)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "> **Exercise 2:** Overplot the dimensionless Hubble function $E(z)$, using the default cosmological parameters." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Scipy Interpolation" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Scipy is a package that includes many useful tools for scientific research, such as interpolation, fitting and integration." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "First, let's compute an interpolation function through the given points." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Let's use interp1d as the simplest one first\n", - "interp1_h_ofz = interpolate.interp1d(data_h_ofz[:,0], data_h_ofz[:,1])" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# The domain of the interpolation is\n", - "data_range=[data_h_ofz[0,0],data_h_ofz[-1,0]]\n", - "data_range" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Now we have a function of E(z)\n", - "interp1_h_ofz(0.5)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "#Let's plot it together with the data\n", - "fig = plt.figure()\n", - "z_arr = np.linspace(data_range[0], data_range[1], 100)\n", - "plt.errorbar(data_h_ofz[:,0], data_h_ofz[:,1], data_h_ofz[:,2], fmt='o', color='black', ecolor='black', capsize=2)\n", - "plt.plot(z_arr, interp1_h_ofz(z_arr), '-', color='tomato', label='Interpolation')" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "#Now let's try a cubic and a UnivariateSpline interpolation\n", - "interp2_h_ofz = interpolate.interp1d(data_h_ofz[:,0], data_h_ofz[:,1], kind='cubic')\n", - "interp3_h_ofz = interpolate.UnivariateSpline(data_h_ofz[:,0], data_h_ofz[:,1])" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "#Let's plot it together with the data\n", - "fig = plt.figure(figsize=(8,8))\n", - "z_arr = np.linspace(data_range[0], data_range[1], 100)\n", - "plt.errorbar(data_h_ofz[:,0], data_h_ofz[:,1], data_h_ofz[:,2], fmt='o', color='black', ecolor='black', capsize=2, \n", - " markersize=2)\n", - "plt.plot(z_arr, interp1_h_ofz(z_arr), '-', color='tomato', label='linear')\n", - "plt.plot(z_arr, interp2_h_ofz(z_arr), ':', color='indigo', label='cubic')\n", - "plt.plot(z_arr, interp3_h_ofz(z_arr), ':', color='forestgreen', label='spline')\n", - "plt.legend()" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "One can see that the result of the interpolation might differ a lot between different methods" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The `UniVariateSpline` method has a `weight` optional argument `w[i]`, which sets a smoothing factor `s`. If this smoothing factor is set to `0`, the interpolation will pass through all points. By default it will be set to `s=len(w)` if `1/w[i]` is a good estimate of the standard deviation of the `y[i]` data." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "> Let's use a neat trick of IPython notebooks, **`interact`**, which allows us to play with sliders on the notebook itself, to control some of the parameters." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "z_arr = np.linspace(data_range[0], data_range[1], 100)\n", - "# Let's define a wrapper function for the above plot with the smoothing factor as an interactive parameter\n", - "def plot_interpol_data(smoothing):\n", - " interp4_h_ofz = interpolate.UnivariateSpline(data_h_ofz[:,0], data_h_ofz[:,1], s=0)\n", - " interp5_h_ofz = interpolate.UnivariateSpline(data_h_ofz[:,0], data_h_ofz[:,1], w=(1/data_h_ofz[:,2]) )\n", - " interp6_h_ofz = interpolate.UnivariateSpline(data_h_ofz[:,0], data_h_ofz[:,1], s=smoothing)\n", - " fig = plt.figure(figsize=(8,8))\n", - " plt.errorbar(data_h_ofz[:,0], data_h_ofz[:,1], data_h_ofz[:,2], fmt='o', color='black', ecolor='black', capsize=2, \n", - " markersize=2)\n", - " plt.plot(z_arr, interp3_h_ofz(z_arr), '-', color='orange', lw=3,label='auto spline')\n", - " plt.plot(z_arr, interp4_h_ofz(z_arr), '-.', color='firebrick', lw=3,label='s=0 spline')\n", - " plt.plot(z_arr, interp5_h_ofz(z_arr), ':', color='violet', lw=3,label='weigthed spline')\n", - " plt.plot(z_arr, interp6_h_ofz(z_arr), '-.', color='teal', lw=3, label='smoothed spline')\n", - " plt.legend()\n", - " plt.show()" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "#Simply passing the function and the argument, together with a tuple, specifying (min, max, step), \n", - "#already produces nice interactive plots.\n", - "interact(plot_interpol_data, smoothing=(0.0,1.0,0.05));" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "> **NOTE**: Interpolating through data points with error bars is an active area of research and a not completely solved problem. The best is to test several methods and find the most appropriate one for the given case." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Numpy polynomial fit" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "A very useful tool for data is to perform a polynomial fit by minimizing the equation:\n", - "$$ E = \\sum_{j=0}^k |p(x_j)-y_j|^2 $$\n", - "where $p$ is a polynomial of degree $d$ for a dataset of length $k$." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# numpy.polyfit(x, y, deg, rcond=None, full=False, w=None, cov=False)\n", - "# The weights should be w=1/sigma\n", - "# deg is the degree of the polynomial\n", - "coeffs = np.polyfit(data_h_ofz[:,0], data_h_ofz[:,1], 3, w=1/(data_h_ofz[:,2]))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The coefficients are returned from larger to smaller degree." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "print(coeffs)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "One can use `poly1d` to deal with polynomials easily:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "poly = np.poly1d(coeffs)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "poly(np.arange(0.0,3,0.1))[0:5]" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "fig=plt.figure()\n", - "plt.plot(np.arange(0.0,3,0.1), poly(np.arange(0.0,3,0.1)), color='red', label='polyfit')\n", - "plt.legend();" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "> **Exercise 3:** Create an interactive plot, that shows the data with errorbars, the smoothing spline (with the same steps as before) and the polyfit with degrees 1-8, with steps of 1." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Gaussian Processes with Scikit-Learn" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "import numpy as np\n", - "from matplotlib import pyplot as plt\n", - "\n", - "from sklearn.gaussian_process import GaussianProcessRegressor\n", - "from sklearn.gaussian_process.kernels import RBF, ConstantKernel as C\n", - "\n", - "x = np.atleast_2d(np.linspace(data_range[0], data_range[1], 1000)).T\n", - "\n", - "X = data_h_ofz[:,0]\n", - "X = np.atleast_2d(X).T\n", - "\n", - "# Observations and noise\n", - "y = data_h_ofz[:,1]\n", - "dy = data_h_ofz[:,2]\n", - "\n", - "kernel = C(1.0, (1e-3, 1e3)) * RBF(10, (1e-2, 1e2))\n", - "# Instantiate a Gaussian Process model\n", - "gp = GaussianProcessRegressor(kernel=kernel, alpha=dy ** 2,\n", - " n_restarts_optimizer=10)\n", - "\n", - "# Fit to data using Maximum Likelihood Estimation of the parameters\n", - "gp.fit(X, y)\n", - "\n", - "# Make the prediction on the meshed x-axis (ask for MSE as well)\n", - "y_pred, sigma = gp.predict(x, return_std=True)\n", - "\n", - "# Plot the function, the prediction and the 95% confidence interval based on\n", - "# the MSE\n", - "plt.figure(figsize=(9,6))\n", - "plt.plot(x, poly(x), 'r:', lw=3, label=r'$E(z) polyfit$')\n", - "plt.errorbar(X.ravel(), y, dy, fmt='r.', markersize=10, label='Observations')\n", - "plt.plot(x, y_pred, 'b-', label='Prediction')\n", - "plt.fill(np.concatenate([x, x[::-1]]),\n", - " np.concatenate([y_pred - 1.9600 * sigma,\n", - " (y_pred + 1.9600 * sigma)[::-1]]),\n", - " alpha=.5, fc='b', ec='None', label='95% confidence interval')\n", - "plt.xlabel('$z$')\n", - "plt.ylabel('$E(z)$')\n", - "plt.ylim()\n", - "plt.legend(loc='upper left')\n", - "\n", - "plt.show()" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Comoving and angular diameter distance: Scipy integration" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The comoving distance [in Mpc] to a galaxy far away from us, is defined as\n", - "$$ d_{\\rm com}(z) = \\int_0^z \\frac{c \\,\\, \\rm{d}\\tilde{z}}{H(\\tilde{z})}$$\n", - "where $c$ is the speed of light and $H(z)$ the Hubble function in km/s/Mpc. " - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# speed of light\n", - "import scipy.constants as sciconst\n", - "clight = sciconst.speed_of_light/1000 #in km/s\n", - "clight" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Let's code up the integral:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "def drdz(zp, cosmo_pars=dict()):\n", - " return (clight)/Hubble(zp, cosmo_pars)\n", - "\n", - "def d_comoving(zc, cosmo_pars=dict()):\n", - " dd = lambda zc : drdz(zc, cosmo_pars=cosmo_pars) \n", - " #notice the use of a lambda function to pass the optional arguments\n", - " int = integrate.romberg(dd,0.,zc) #notice the use of a romberg integration\n", - " return int\n", - "\n", - "#notice the use of np.vectorize to allow for array arguments\n", - "vect_dcom = np.vectorize(d_comoving)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The distance to a galaxy at redshift $z=0.5$ is almost 2 Gpc!" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "print(vect_dcom(0.5))\n", - "print(d_comoving(0.5))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "While the angular diameter distance $d_A$, is defined as the distance to an object of size x, that is viewed under an angle $\\theta$. Doing the math, one gets simply:\n", - "$$ d_{\\rm A}(z) = \\frac{1}{1+z} \\int_0^z \\frac{c \\,\\, \\rm{d}\\tilde{z}}{H(\\tilde{z})}$$" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "def d_angular(zc, cosmo_pars=dict()):\n", - " da=(1/(1+zc)) * vect_dcom(zc, cosmo_pars=cosmo_pars)\n", - " return da" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "d_angular(0.5)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "> Notice how this distance is smaller than the comoving one." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Let's plot both distances together, for a new cosmology!" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "newcosmo = {'H0': 72, \n", - " 'Omega_c':0.32,\n", - " 'Omega_b':0.05,\n", - " 'Omega_r':0.00001}\n", - "z_arr = np.linspace(0.,10., 100)\n", - "fig = plt.figure()\n", - "#Let's add some color and linestyle, r for red and a label for the curve.\n", - "plt.plot(z_arr, d_angular(z_arr, newcosmo), '-r', label='$d_A$') #notice how you can use LaTeX here\n", - "plt.plot(z_arr, vect_dcom(z_arr, newcosmo), '--g', label='$d_{com}$') #notice how you can use LaTeX here\n", - "plt.plot(z_arr, vect_dcom(z_arr), '-.b', label='$d_{com}$ Planck cosmo') #notice how you can use LaTeX here\n", - "\n", - "#Let's add a plot title\n", - "plt.title(\"cosmological distances\")\n", - "#Let's add a legend\n", - "plt.legend()\n", - "#Let's add an x-label, with some font-size\n", - "plt.xlabel(\"z\", fontsize=15)\n", - "# and a y-label, rotated by 90 degrees (which is the default in this case anyway)\n", - "plt.ylabel(\"d(z)\", rotation=90, fontsize=15) " - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Integration of arrays" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "As we saw before, obtaining an interpolating function from sparse and noisy data is a bit tricky." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "For this reason, sometimes one is forced to integrate directly the given array. Let's try this with the data we imported before. The tool for this is called `integrate.simps`. (See the documentation)." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "def data_drdz(zc, e_data, cosmo_pars=dict()): #accepting the {z, E(z)} data\n", - " xx = e_data[:,0]\n", - " maxi = np.where(xx <= zc)[0][-1] + 1\n", - " #print(maxi)\n", - " xx = xx[:maxi]\n", - " h0 = cosmological_parameters(cosmo_pars, which='H0')\n", - " yy = (clight/h0)*(1/e_data[:,1])\n", - " yy = yy[:maxi]\n", - " return xx, yy\n", - "\n", - "def data_comoving_distance(zc, e_data, cosmo_pars=dict()): #remember the data we have is in E(z)\n", - " xx, yy = data_drdz(zc, e_data, cosmo_pars)\n", - " dc = integrate.simps(yy, xx)\n", - " return dc \n", - "\n", - "#this function can't be easily vectorized, because of np.where" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "data_comoving_distance(0.5,data_h_ofz)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "z_range = np.linspace(0.,2.5,10)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "d_com_data = np.array([data_comoving_distance(zz, data_h_ofz) for zz in z_range])" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "newcosmo = {'H0': 72,\n", - " 'Omega_r':0.00001}\n", - "z_arr = np.linspace(0.,5., 100)\n", - "fig = plt.figure()\n", - "#Let's add some color and linestyle, r for red and a label for the curve.\n", - "plt.plot(z_range, d_com_data, 'o-r', label='$d_{com}$ from data') #notice how you can use LaTeX here\n", - "plt.plot(z_arr, vect_dcom(z_arr, newcosmo), '--g', label='$d_{com} \\,\\,H_0=72$km/s/Mpc') #notice how you can use LaTeX here\n", - "plt.plot(z_arr, vect_dcom(z_arr), '-.b', label='$d_{com}$ Planck cosmo') #notice how you can use LaTeX here\n", - "\n", - "#Let's add a plot title\n", - "plt.title(\"cosmological distances\")\n", - "#Let's add a legend\n", - "plt.legend()\n", - "#Let's add an x-label, with some font-size\n", - "plt.xlabel(\"z\", fontsize=15)\n", - "# and a y-label, rotated by 90 degrees (which is the default in this case anyway)\n", - "plt.ylabel(\"d(z)\", rotation=90, fontsize=15) " - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "> **NOTE**: Notice that despite the data being evaluated with $H_0$ from Planck (the default one), the distance at higher redshifts seems to indicate a higher value of $H_0 \\approx 72$km/s/Mpc. Does this indicate some sort of tension with Planck? " - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "> **Puzzle:** Can you propagate the errors from $E(z)$ onto $d_{com}(z)$ and add the error bars to the red dots above?" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### **Homework Exercise 4:** \n", - "* In the materials folder you can find a file called `Fs8dataclean`.It contains `z`, $f\\sigma_8 (z)$ data and its error bars.\n", - "* The function $f\\sigma_8 (z)$ is an indication of how fast structures have grown in the universe.\n", - "* For the $\\Lambda$CDM model, this function can be written as:\n", - "$$ f\\sigma_8(z) = f(z)D(z)\\sigma_{8,0}$$\n", - "* Where the growth rate is:\n", - "$$f(z) = \\Omega_m (z)^{\\gamma}$$ with $\\gamma=0.55$\n", - "* And the growth function is:\n", - "$$D(z) = \\exp \\big(- \\int_0^z \\frac{f(\\tilde{z})}{(1+\\tilde{z})} \\big) $$\n", - "* Finally, the free parameter is $\\sigma_{8,0} = 0.83$\n", - "\n", - " 1. Import the data and plot it with its error bars.\n", - " 2. Plot the data together with a couple of interpolation functions of your choice and a polynomial fit of order 3.\n", - " 3. Compute the theoretical curve, according to the points above, using integration. Does the theoretical model pass through the data points? try out different cosmological values.\n", - " 4. Compute the integral of the data alone, to obtain $D(z)$. Is it a good approximation or is the data too noisy?" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "> **[Answers](./Answers.ipynb) to puzzles and exercises**" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Bonus Chapter on Seaborn plots" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - " > Notice that despite matplotlib being a really nice library, the standard settings\n", - "produce a very rough plot and it needs a lot of tweaking to get publication-ready plots. A way to improve this quickly, is to use `seaborn`." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "import seaborn as sns\n", - "sns.set()" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Consider the example we had before in the `Numpy` lesson:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - " data = np.random.multivariate_normal([0, 0], [[5, 2], [2, 2]], size=2000)\n", - "data = pd.DataFrame(data, columns=['x', 'y'])" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "#matplotlib with seaborn style\n", - "for col in 'xy':\n", - " plt.hist(data[col], density=True, alpha=0.5)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# pure seaborn\n", - "sns.distplot(data['x'])\n", - "sns.distplot(data['y']);" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - " sns.kdeplot(data['x'], data['y']);" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "with sns.axes_style('white'):\n", - " sns.jointplot(\"x\", \"y\", data, kind='kde');" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [] - } - ], - "metadata": { - "hide_input": false, - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.7.3" - }, - "toc": { - "base_numbering": 1, - "nav_menu": {}, - "number_sections": true, - "sideBar": true, - "skip_h1_title": false, - "title_cell": "Table of Contents", - "title_sidebar": "Contents", - "toc_cell": false, - "toc_position": {}, - "toc_section_display": true, - "toc_window_display": false - } - }, - "nbformat": 4, - "nbformat_minor": 2 -} diff --git a/python/materials/02.05-broadcasting.png b/python/materials/02.05-broadcasting.png deleted file mode 100644 index 2699996..0000000 Binary files a/python/materials/02.05-broadcasting.png and /dev/null differ diff --git a/python/materials/CR_Temp.csv b/python/materials/CR_Temp.csv deleted file mode 100644 index 4c0f32d..0000000 --- a/python/materials/CR_Temp.csv +++ /dev/null @@ -1,60 +0,0 @@ -N,YEAR,ENE,FEB,MAR,ABR,MAY,JUN,JUL,AGO,SET,OCT,NOV,DIC,AVERAGE -1,1958,20.5,21.5,23.2,23.3,24.4,24.2,23.4,23.3,23.6,23.6,22.7,21.6,22.9 -2,1959,20.8,21.1,22.0,22.4,23.0,23.0,22.5,22.5,23.0,22.7,22.0,21.2,22.2 -3,1960,21.0,21.2,21.9,22.8,23.8,23.5,22.9,23.2,23.0,23.5,22.2,21.9,22.6 -4,1961,21.1,21.6,22.1,22.5,22.5,22.4,22.7,23.1,22.7,22.8,22.4,22.7,22.4 -5,1962,20.8,20.7,21.9,22.0,23.0,23.3,22.8,22.8,23.2,23.0,21.9,20.9,22.2 -6,1963,20.0,20.8,21.5,22.0,22.6,23.1,22.6,22.9,23.1,22.6,22.0,21.2,22.0 -7,1964,20.6,21.6,22.3,22.5,22.9,22.7,22.8,23.0,22.7,22.3,22.1,20.8,22.2 -8,1965,20.1,20.5,21.5,20.9,21.8,22.0,21.8,21.4,21.5,21.5,21.5,20.3,21.2 -9,1966,21.2,20.8,21.0,22.1,22.3,22.8,22.5,22.4,22.3,22.1,20.8,20.5,21.7 -10,1967,20.6,20.6,19.9,21.1,21.7,21.9,21.4,21.7,21.7,21.7,21.1,20.6,21.2 -11,1968,19.8,19.6,19.9,20.9,21.7,21.7,21.4,21.5,22.0,21.9,21.4,20.1,21.0 -12,1969,20.8,20.7,22.1,22.7,23.4,22.8,21.9,22.0,22.0,22.0,21.2,20.9,21.9 -13,1970,20.9,20.1,21.6,22.1,21.9,22.4,21.9,21.7,21.8,21.9,20.5,20.3,21.4 -14,1971,20.1,20.0,20.2,21.5,21.8,21.4,21.1,21.4,21.5,21.6,21.2,19.9,21.0 -15,1972,20.0,20.3,21.0,21.5,22.3,22.1,22.5,21.6,22.0,21.6,21.8,21.0,21.5 -16,1973,21.0,20.6,22.1,22.4,22.2,22.2,21.6,21.9,22.0,22.0,21.3,19.5,21.6 -17,1974,19.7,19.6,20.8,21.2,22.1,21.9,21.0,21.6,21.9,21.2,20.8,20.3,21.0 -18,1975,19.8,19.9,21.4,21.3,22.2,21.4,20.9,21.1,21.5,21.5,21.4,19.4,21.0 -19,1976,19.2,19.3,20.4,21.5,21.8,21.8,21.5,21.4,21.8,22.1,21.4,20.6,21.1 -20,1977,19.8,20.9,21.1,21.5,22.9,22.3,21.7,21.9,21.8,22.0,21.7,21.1,21.6 -21,1978,20.0,20.8,21.2,22.0,22.7,22.0,21.8,22.0,21.9,22.1,21.9,21.1,21.6 -22,1979,20.5,20.6,21.6,22.0,22.6,22.5,22.5,22.2,22.6,22.7,21.6,21.0,21.9 -23,1980,21.1,20.3,21.6,22.2,23.0,22.7,22.5,22.2,22.1,22.1,21.6,20.2,21.8 -24,1981,20.0,20.6,21.3,21.4,22.9,22.3,21.6,21.9,21.8,22.2,21.2,21.0,21.5 -25,1982,21.1,21.0,21.2,22.1,22.9,22.8,21.8,22.0,22.2,21.4,21.2,20.9,21.7 -26,1983,21.4,22.2,22.6,23.1,22.9,23.3,22.8,22.8,22.7,21.8,21.9,21.0,22.4 -27,1984,19.9,20.7,21.5,22.9,21.9,21.8,22.0,22.1,22.0,21.8,20.8,19.8,21.4 -28,1985,20.0,20.2,20.8,21.6,22.8,22.3,21.8,21.5,22.2,21.8,21.2,20.4,21.4 -29,1986,19.7,20.8,20.6,21.5,22.5,22.2,21.9,22.1,22.1,22.0,21.8,21.2,21.5 -30,1987,20.7,21.2,22.7,21.6,22.6,23.0,22.7,22.3,22.7,22.1,22.1,21.8,22.1 -31,1988,21.0,21.0,21.0,22.7,22.8,22.7,22.0,22.2,22.5,21.9,21.6,20.4,21.8 -32,1989,20.3,19.6,20.6,21.7,22.1,22.0,21.8,22.2,22.1,21.8,22.0,20.6,21.4 -33,1990,20.5,20.5,20.9,22.4,22.4,22.4,22.1,22.2,22.4,22.3,22.0,20.7,21.7 -34,1991,20.5,20.3,22.2,22.0,22.4,22.8,22.2,21.8,22.5,21.9,21.1,20.3,21.7 -35,1992,20.6,21.3,21.6,22.5,22.4,23.0,21.9,21.5,22.0,21.5,22.0,20.9,21.8 -36,1993,20.7,21.0,21.6,23.0,23.4,22.9,22.7,23.6,23.3,23.9,23.1,22.5,22.6 -37,1994,21.8,21.2,22.2,22.2,23.0,22.7,21.9,22.1,22.8,22.5,21.8,21.3,22.1 -38,1995,21.4,22.7,22.7,24.2,23.5,22.5,22.6,22.9,22.9,22.4,22.2,22.4,22.7 -39,1996,21.2,20.9,21.6,22.4,22.8,22.8,22.5,22.2,22.7,22.4,21.4,20.4,21.9 -40,1997,20.7,21.2,21.4,22.7,22.1,22.4,22.7,23.0,22.9,22.7,22.6,21.6,22.2 -41,1998,21.7,22.8,22.6,22.7,23.3,23.0,22.3,22.4,23.2,22.8,22.1,21.0,22.5 -42,1999,21.0,20.7,21.0,22.0,22.6,22.3,21.7,22.0,22.1,21.8,21.3,19.7,21.5 -43,2000,19.6,20.0,21.1,21.6,22.3,22.5,22.0,22.3,22.6,22.2,22.0,20.9,21.6 -44,2001,20.0,20.6,21.7,21.6,23.0,22.1,22.1,22.7,22.7,23.0,21.6,21.6,21.9 -45,2002,21.0,21.2,21.5,21.6,22.5,23.1,22.4,22.2,23.0,22.4,21.8,21.5,22.0 -46,2003,21.0,21.7,22.5,22.9,22.8,22.6,22.7,22.5,23.0,23.0,22.3,21.0,22.3 -47,2004,21.0,21.2,21.2,22.6,22.2,22.6,22.8,22.3,22.3,23.0,21.6,20.6,22.0 -48,2005,19.9,20.5,23.1,22.8,23.6,23.5,23.0,22.6,22.8,22.7,21.4,21.3,22.3 -49,2006,20.9,20.7,21.0,22.1,22.8,22.7,22.5,23.0,22.5,22.9,21.7,21.7,22.0 -50,2007,20.8,21.4,21.9,22.8,23.5,23.4,22.4,22.9,22.6,22.4,21.5,21.0,22.2 -51,2008,21.0,21.0,21.4,22.3,22.8,22.9,22.6,22.9,22.9,22.2,21.1,20.8,22.0 -52,2009,20.4,20.0,20.7,22.1,22.6,23.0,22.5,22.5,23.1,22.7,21.6,21.7,21.9 -53,2010,20.9,22.0,22.2,23.1,23.3,22.9,22.9,22.9,22.5,22.5,21.2,20.0,22.2 -54,2011,20.6,21.1,21.2,22.3,23.0,23.2,22.5,22.8,23.0,22.0,21.5,20.6,22.0 -55,2012,20.3,20.8,20.8,22.1,23.1,23.1,22.2,22.8,22.6,22.4,21.1,21.4,21.9 -56,2013,21.2,21.0,21.4,22.7,22.6,22.7,22.7,22.9,23.3,23.0,22.7,21.7,22.3 -57,2014,21.2,21.4,22.1,22.9,23.4,23.8,23.8,23.7,22.8,24.0,23.7,23.3,23.0 -58,2015,22.7,22.6,22.6,23.7,22.4,23.1,22.3,22.8,22.7,23.2,22.7,23.0,22.8 -59,2016,22.1,21.5,23.0,23.7,24.1,23.2,22.6,23.2,23.2,23.2,22.5,21.9,22.9 diff --git a/python/materials/Fs8covariance.txt b/python/materials/Fs8covariance.txt deleted file mode 100644 index 6768492..0000000 --- a/python/materials/Fs8covariance.txt +++ /dev/null @@ -1,20 +0,0 @@ -0.003025 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. -0. 0.0225 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. -0. 0. 0.0036 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. -0. 0. 0. 0.00339889 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. -0. 0. 0. 0. 0.004489000000000001 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. -0. 0. 0. 0. 0. 0.009409 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. -0. 0. 0. 0. 0. 0. 0.00142884 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. -0. 0. 0. 0. 0. 0. 0. 0.002025 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. -0. 0. 0. 0. 0. 0. 0. 0. 0.0046240000000000005 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. -0. 0. 0. 0. 0. 0. 0. 0. 0. 0.0064 0. 0. 0. 0.0038441958326807446 0. 0. 0. 0. 0. 0. -0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.001444 0. 0. 0. 0. 0. 0. 0. 0. 0. -0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.001444 0. 0. 0. 0. 0. 0. 0. 0. -0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.0036 0. 0. 0. 0. 0. 0. 0. -0. 0. 0. 0. 0. 0. 0. 0. 0. 0.0038441958326807446 0. 0. 0. 0.0088775 0. 0.003933091290066886 0. 0. 0. 0. -0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.0011560000000000001 0. 0. 0. 0. 0. -0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.003933091290066886 0. 0.005556500000000001 0. 0. 0. 0. -0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.0064 0. 0. 0. -0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.0121 0. 0. -0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.00905 0. -0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.013456000000000001 \ No newline at end of file diff --git a/python/materials/Fs8dataclean.txt b/python/materials/Fs8dataclean.txt deleted file mode 100644 index 2c89591..0000000 --- a/python/materials/Fs8dataclean.txt +++ /dev/null @@ -1,20 +0,0 @@ -0.067 0.423 0.055 -0.15 0.48999999999999994 0.15 -0.17 0.5099999999999999 0.06 -0.25 0.3512 0.0583 -0.3 0.366 0.067 -0.35 0.445 0.097 -0.37 0.46019999999999994 0.0378 -0.38 0.497 0.045 -0.4 0.394 0.068 -0.44 0.41600000000000004 0.08 -0.51 0.458 0.038 -0.55 0.44400000000000006 0.038 -0.57 0.48799999999999993 0.06 -0.6 0.4295160397021415 0.09422048609511628 -0.61 0.43600000000000005 0.034 -0.7284999999999999 0.3712262215423378 0.07454193450669228 -0.8 0.47000000000000003 0.08 -0.85 0.45000000000000007 0.11 -0.86 0.468950276243094 0.09513148795220225 -1.36 0.48200000000000004 0.116 \ No newline at end of file diff --git a/python/materials/alldataEz-H0planck-real-H0planck.txt b/python/materials/alldataEz-H0planck-real-H0planck.txt deleted file mode 100644 index 67c1d89..0000000 --- a/python/materials/alldataEz-H0planck-real-H0planck.txt +++ /dev/null @@ -1,33 +0,0 @@ -0. 1. 0.008796960890725113 -0.07 0.997 0.023 -0.09 1.0207100591715974 0.17593921781450225 -0.12 1.0147928994082838 0.3841339588949966 -0.17 1.2278106508875737 0.11729281187633485 -0.199 1.1094674556213013 0.07330800742270928 -0.27 1.1390532544378695 0.20526242078358597 -0.28 1.3136094674556211 0.5366146143342319 -0.35 1.128 0.037 -0.38 1.2056213017751478 0.027857042820629525 -0.4 1.405325443786982 0.24924722523721154 -0.4247 1.288461538461538 0.16420993662686875 -0.44 1.22189349112426 0.11436049157942647 -0.4783 1.1967455621301775 0.1319544133608767 -0.51 1.3372781065088757 0.027857042820629525 -0.55 1.364 0.063 -0.593 1.538461538461538 0.1906008192990441 -0.61 1.439349112426035 0.030789363117537896 -0.68 1.3609467455621298 0.11729281187633485 -0.73 1.4393491124260351 0.10263121039179299 -0.781 1.5532544378698225 0.17593921781450225 -0.875 1.8491124260355027 0.24924722523721154 -0.9 1.52 0.12 -1.037 2.278106508875739 0.2932320296908371 -1.3 2.4852071005917153 0.24924722523721154 -1.363 2.366863905325443 0.49262980988060634 -1.43 2.618343195266272 0.2639088267217534 -1.5 2.67 0.675 -1.53 2.0710059171597623 0.20526242078358597 -1.75 2.9881656804733723 0.5864640593816742 -1.965 2.7588757396449695 0.7389447148209094 -2.34 3.2840236686390525 0.10263121039179299 -2.36 3.343195266272189 0.11729281187633485 \ No newline at end of file diff --git a/python/materials/cosmological_energy_densities.png b/python/materials/cosmological_energy_densities.png deleted file mode 100644 index 469b145..0000000 Binary files a/python/materials/cosmological_energy_densities.png and /dev/null differ diff --git a/python/materials/empty.txt b/python/materials/empty.txt deleted file mode 100644 index 266be60..0000000 --- a/python/materials/empty.txt +++ /dev/null @@ -1 +0,0 @@ -This is a text string \ No newline at end of file diff --git a/python/materials/marvel_movies.txt b/python/materials/marvel_movies.txt deleted file mode 100644 index 1b79ce0..0000000 --- a/python/materials/marvel_movies.txt +++ /dev/null @@ -1,15 +0,0 @@ -Title,Earnings,Rating -Iron Man,585174222,94 -The Incredible Hulk,263427551,67 -Iron Man 2,623933331,72 -Thor,449326618,77 -Captain America: The First Avenger,370569774,80 -Marvel's The Avengers,1518812988,92 -Iron Man 3,1214811252,79 -Thor: The Dark World,644571402,66 -Captain America: The Winter Soldier,714264267,89 -Guardians of the Galaxy,773328629,91 -Avengers: Age of Ultron,1405403694,75 -Ant-Man,519311965,81 -Captain America: Civil War,1153304495,90 -Doctor Strange,676867562,90 \ No newline at end of file diff --git a/python/materials/nearest_stars.txt b/python/materials/nearest_stars.txt deleted file mode 100644 index d370b13..0000000 --- a/python/materials/nearest_stars.txt +++ /dev/null @@ -1,11 +0,0 @@ -Order,Name,Spectral Type,Distance -1,Proxima Centauri,M,4.2421 -2,Alpha Centauri a,G,4.3650 -3,Alpha Centauri b,K,4.3650 -4,Barnard’s Star,M,5.9630 -5,Luhman 16A,L,6.59 -6,Luhman 16B,T,6.59 -7,Wolf 359,M,7.7825 -8,Lalande 21185,M,8.2905 -9,Sirius A,A,8.5828 -10,Sirius B,D,8.5828 diff --git a/python/materials/ninja_turtles.npy b/python/materials/ninja_turtles.npy deleted file mode 100644 index 81d5d97..0000000 Binary files a/python/materials/ninja_turtles.npy and /dev/null differ diff --git a/python/materials/ninja_turtles.txt b/python/materials/ninja_turtles.txt deleted file mode 100644 index c913795..0000000 --- a/python/materials/ninja_turtles.txt +++ /dev/null @@ -1,5 +0,0 @@ -Name Age Colour Weapon -Leonardo 15 blue Katana -Raphael 15 Red Sai -Donatello 14 Purple Bo-staff -Michelangelo 13 Orange Nunchucks diff --git a/python/materials/ninja_turtles_new.npy b/python/materials/ninja_turtles_new.npy deleted file mode 100644 index 4e71713..0000000 Binary files a/python/materials/ninja_turtles_new.npy and /dev/null differ diff --git a/python/materials/numpy_indexing.png b/python/materials/numpy_indexing.png deleted file mode 100644 index 72b5eb2..0000000 Binary files a/python/materials/numpy_indexing.png and /dev/null differ diff --git a/python/materials/planck_cosmic_pie.jpg b/python/materials/planck_cosmic_pie.jpg deleted file mode 100644 index 97461c8..0000000 Binary files a/python/materials/planck_cosmic_pie.jpg and /dev/null differ diff --git a/python/materials/r2d2.fits b/python/materials/r2d2.fits deleted file mode 100644 index ae02e75..0000000 Binary files a/python/materials/r2d2.fits and /dev/null differ diff --git a/python/materials/r2d2.jpg b/python/materials/r2d2.jpg deleted file mode 100644 index 3b68fdd..0000000 Binary files a/python/materials/r2d2.jpg and /dev/null differ diff --git a/python/materials/random_data.fits b/python/materials/random_data.fits deleted file mode 100644 index 752626d..0000000 Binary files a/python/materials/random_data.fits and /dev/null differ diff --git a/python/materials/star_wars.txt b/python/materials/star_wars.txt deleted file mode 100644 index 03cc1bb..0000000 --- a/python/materials/star_wars.txt +++ /dev/null @@ -1,3 +0,0 @@ -It is a period of civil war. Rebel spaceships, striking from a hidden base, have won their first victory against the evil Galactic Empire. -During the battle, Rebel spies managed to steal secret plans to the Empire's ultimate weapon, the DEATH STAR, an armored space station with enough power to destroy an entire planet. -Pursued by the Empire's sinister agents, Princess Leia races home aboard her starship, custodian of the stolen plans that can save her people and restore freedom to the galaxy.... \ No newline at end of file diff --git a/python/materials/turtle_age.txt b/python/materials/turtle_age.txt deleted file mode 100644 index 879f5e5..0000000 --- a/python/materials/turtle_age.txt +++ /dev/null @@ -1 +0,0 @@ -average turtle age is 14.25 diff --git a/ada/sparsity_1.ipynb b/sparsity_1.ipynb similarity index 66% rename from ada/sparsity_1.ipynb rename to sparsity_1.ipynb index b34c116..f2ad686 100644 --- a/ada/sparsity_1.ipynb +++ b/sparsity_1.ipynb @@ -2,20 +2,32 @@ "cells": [ { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "slide" + } + }, "source": [ + "
\n", + "\n", "# Introduction to Sparsity I\n", "\n", "---\n", "\n", "> Author: Samuel Farrens \n", - "> Year: 2018 \n", + "> Year: 2018 (updated in 2023) \n", "> Email: [samuel.farrens@cea.fr](mailto:samuel.farrens@cea.fr) \n", - "> Website: https://sfarrens.github.io\n", - "\n", - "---\n", - "\n", - "\n", + "> Website: https://sfarrens.github.io" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "slideshow": { + "slide_type": "skip" + } + }, + "source": [ "## Contents\n", "\n", "1. [Set-Up](#Set-Up)\n", @@ -41,10 +53,12 @@ }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "slide" + } + }, "source": [ - "---\n", - "\n", "## Set-Up\n", "\n", "Here we will import a couple of packages that will be needed throughout the notebook. \n", @@ -52,87 +66,86 @@ "Users new to Jupyter notebooks should note that cells are executed by pressing SHIFT+ENTER (⇧+ ⏎). See here for more details." ] }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "scrolled": true - }, - "outputs": [], - "source": [ - "# Tell Jupyter to display plots in this notebook.\n", - "%matplotlib inline\n", - "\n", - "# Import the numpy package with the alias np.\n", - "import numpy as np\n", - "\n", - "# Import function to set size of figures.\n", - "from matplotlib.pylab import rcParams\n", - "\n", - "# Import interaction module.\n", - "from ipywidgets.widgets import *\n", - "\n", - "# Import tutorial plotting functions.\n", - "from sparsity_tutorial import plot" - ] - }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ - "# For Python 2 users only\n", - "# Uncomment the following commands\n", - "\n", - "# Import Python 3 behaviour\n", - "# from __future__ import print_function" + "# Uncomment the following lines if you are running this on Google Colab\n", + "# !git clone -b ada https://github.com/CosmoStat/Tutorials.git\n", + "# import sys\n", + "# sys.path.insert(0,'/content/Tutorials')" ] }, { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "scrolled": true, + "slideshow": { + "slide_type": "fragment" + } + }, "outputs": [], "source": [ - "# Set size of figures.\n", - "rcParams['figure.figsize'] = (14.0, 8.0)" + "# Import the numpy package with the alias np.\n", + "import numpy as np\n", + "\n", + "# Import interactive widgets.\n", + "from ipywidgets.widgets import interact, IntSlider, FloatSlider, Select\n", + "\n", + "# Import tutorial plotting functions.\n", + "from sparsity_tutorial import plot" ] }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "slide" + } + }, "source": [ - "---\n", "\n", "## Introduction\n", "\n", - "### Objective\n", + "### Objective\n", "\n", "The objective of this notebook is to introduce the concept of sparsity, particularly in the context of solving inverse problems. To that end, the following examples focus on providing some basic practical applications of the tools and techniques used in sparse analysis. This tutorial does not provide all of mathematical background and readers are encouraged to refer to the following resources for more in-depth explanations.\n", "\n", - "* Sparse Image and Signal Processing (2015)\n", - "\n", - "### Sparsity Definition\n", + "* Sparse Image and Signal Processing (2015)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, + "source": [ + "### Sparsity Definition\n", "\n", "A sparse signal is one that is comprised mostly of zeros when expressed in the appropriate basis, in other words the only a few coefficients are required to represent the entire signal. This is an inherent propery of the signal that can be expolited to great effect.\n", "\n", "A signal $x\\in\\mathbb{R}^N$, *i.e.* a discrete signal with $N$ coefficients $x_i$, is exactly $k$-sparse if only $k$ of its coefficients are non-zero. In practice, however, one generally deals with approximately sparse signals in which the majority of the coefficients are very small with respect to those that carry the majority of the information. Additionally, some signals that are not naturally sparse can be \"sparsified\" with an appropriate transform basis.\n", "\n", - "### Notes\n", - "\n", - "By default the term signal will be used to refer to discrete signals." + "> Note that by default the term signal will be used to refer to discrete signals." ] }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "slide" + } + }, "source": [ - "---\n", "\n", "## Time-Frequency Example\n", "\n", - "### Signal\n", + "### Signal\n", "\n", "Let's start with a simple sine wave example. We have a time series signal where\n", "\n", @@ -145,7 +158,10 @@ "cell_type": "code", "execution_count": null, "metadata": { - "scrolled": false + "scrolled": false, + "slideshow": { + "slide_type": "subslide" + } }, "outputs": [], "source": [ @@ -165,11 +181,24 @@ }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "source": [ - "We can see that out of the 30 coefficients only 2 are equal to zero and therefore this signal is clearly not sparse according the definition provided.\n", - "\n", - "### Discrete Fourier Transform\n", + "We can see that out of the 30 coefficients only 2 are equal to zero and therefore this signal is clearly not sparse according the definition provided." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, + "source": [ + "### Discrete Fourier Transform\n", "\n", "We can transform our signal using the Discrete Fourier Transform (DFT) in order to examine its frequency components. \n", "\n", @@ -177,13 +206,17 @@ "\n", "$$X_k = \\sum_{n=0}^{N-1}x_n e^{-i2\\pi kn/N}$$\n", "\n", - "which can be implemented as follows." + "which can be implemented as follows:" ] }, { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "outputs": [], "source": [ "# Calculate the DFT of a given signal\n", @@ -194,12 +227,16 @@ " n = np.arange(N)\n", " k = n.reshape((N, 1))\n", " \n", - " return np.dot(np.exp(-2j * np.pi * k * n / N), xn)" + " return np.exp(-2j * np.pi * k * n / N) @ xn" ] }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "fragment" + } + }, "source": [ "Using this function we can compute the normalised DFT of the time series signal." ] @@ -208,7 +245,10 @@ "cell_type": "code", "execution_count": null, "metadata": { - "scrolled": false + "scrolled": false, + "slideshow": { + "slide_type": "subslide" + } }, "outputs": [], "source": [ @@ -219,13 +259,24 @@ "y_fft = np.abs(dft(y)) / y.size \n", "\n", "# Plot the signal\n", - "plot.stem_plot(y_fft, x_vals=w, title='Fourier Transform of Sine', imag=False, ylim=(-0.1, 0.6),\n", - " xlab='$\\omega$', ylab=r'$\\frac{|y(\\omega)|}{n}$')" + "plot.stem_plot(\n", + " y_fft, \n", + " x_vals=w, \n", + " title='Fourier Transform of Sine', \n", + " imag=False, \n", + " ylim=(-0.1, 0.6),\n", + " xlab=r'$\\omega$', \n", + " ylab=r'$\\frac{|y(\\omega)|}{n}$',\n", + ")" ] }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "source": [ "Now, we have only two coefficients which are non-zero (in fact we only need one as the signal is symmetric in Fourier space). So, although in its natural domain our signal does not appear to be sparse, we were able to find a sparse representation in a different basis.\n", "\n", @@ -234,9 +285,13 @@ }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "source": [ - "### Fast Fourier Transform\n", + "### Fast Fourier Transform\n", "\n", "Note that we can more easily convert signals back and forth between the direct and Fourier space using the Fast Fourier Transform (FFT)." ] @@ -244,7 +299,11 @@ { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "fragment" + } + }, "outputs": [], "source": [ "# FFT\n", @@ -260,9 +319,37 @@ }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, + "source": [ + "For a more complete understanding of the Fourier transform, I recommend you checkout this YouTube video." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "slideshow": { + "slide_type": "fragment" + } + }, + "outputs": [], + "source": [ + "from IPython.display import HTML, IFrame\n", + "IFrame(src=\"https://www.youtube.com/embed/spUNpyF58BY\", width=560, height=315)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "slideshow": { + "slide_type": "slide" + } + }, "source": [ - "---\n", "\n", "## Sparsity\n", "\n", @@ -272,31 +359,59 @@ }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "source": [ - "### Sparse Representation\n", + "### Sparse Representation\n", "\n", "A signal $x$ can be respresented as\n", "\n", "$$x = \\phi\\alpha = \\sum_{i=1}^n\\phi_i\\alpha_i$$\n", "\n", - "where $\\alpha$ are the sparse coefficients of $x$ in the dictionary $\\phi$ and $n$ is the sparsity of the signal. Each $\\phi_i$ corresponds to an *atom* of the dictionay $\\phi$. In the case of Fourier decomposition the atoms are sinusoids and when $\\phi$ is a matrix the atoms are the columns of the matrix." + "where $\\alpha$ are the sparse coefficients of $x$ in the dictionary $\\phi$ and $n$ is the sparsity of the signal. Each $\\phi_i$ corresponds to an *atom* of the dictionay $\\phi$. In the case of Fourier decomposition, the atoms are sinusoids and, when $\\phi$ is a matrix, the atoms are the columns of the matrix." ] }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "source": [ - "### Measures of Sparsity\n", + "### Measures of Sparsity\n", "\n", "The true sparsity of a signal can be measured with the $l_0$ pseudo-norm.\n", "\n", "$$\\|x\\|_0 = \\text{Card}(\\text{Supp}(x))$$\n", "\n", - "*i.e.* the number of non-zero elements in the signal. However, this function is non-convex and is computationally hard to solve. Therefore, in practice the convex $l_1$ norm is used instead.\n", - "\n", - "$$\\|x\\|_1 = \\sum_{i=1}^n |x_i|$$\n", + "*i.e.* the number of non-zero elements in the signal. However, this function is non-convex and is computationally hard to solve." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "slideshow": { + "slide_type": "fragment" + } + }, + "source": [ + "Therefore, in practice the convex $l_1$ norm is used instead.\n", "\n", + "$$\\|x\\|_1 = \\sum_{i=1}^n |x_i|$$" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, + "source": [ "The $l_1$ norm is the closest convex norm to $l_0$ and Donoho and Huo (1999) have shown that for sufficiently sparse signals the two norms are equivalent.\n", "\n", "We can use this to measure the sparsity of our time-frequency example." @@ -306,7 +421,10 @@ "cell_type": "code", "execution_count": null, "metadata": { - "scrolled": true + "scrolled": true, + "slideshow": { + "slide_type": "fragment" + } }, "outputs": [], "source": [ @@ -322,36 +440,75 @@ }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "fragment" + } + }, "source": [ "As expected the frequency signal is significantly more sparse." ] }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "source": [ - "### Promoting Sparsity\n", + "### Promoting Sparsity\n", "\n", "In order to actually produce solutions that are more sparse one typically performs a thresholding operation in the sparse domain.\n", "\n", - "Here we introduce the soft thresholding operator and compare it with a more commonly known hard thresholding operator.\n", - "\n", + "Here we introduce the soft thresholding operator and compare it with a more commonly known hard thresholding operator." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, + "source": [ "**Hard Thresholding**\n", "\n", - "$$HT_{\\lambda}(x_i) = \\begin{cases} x_i & \\text{if}\\ |x_i| \\geq \\lambda \\\\ 0 & \\text{otherwise} \\\\ \\end{cases}$$\n", - "\n", + "$$HT_{\\lambda}(x_i) = \\begin{cases} x_i & \\text{if}\\ |x_i| \\geq \\lambda \\\\ 0 & \\text{otherwise} \\\\ \\end{cases}$$" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "slideshow": { + "slide_type": "fragment" + } + }, + "source": [ "**Soft Thresholding**\n", "\n", - "$$ST_{\\lambda}(x_i) = \\begin{cases} x_i - \\lambda\\text{sign}(x_i) & \\text{if}\\ |x_i| \\geq \\lambda \\\\ 0 & \\text{otherwise} \\\\ \\end{cases}$$\n", - "\n", + "$$ST_{\\lambda}(x_i) = \\begin{cases} x_i - \\lambda\\text{sign}(x_i) & \\text{if}\\ |x_i| \\geq \\lambda \\\\ 0 & \\text{otherwise} \\\\ \\end{cases}$$" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, + "source": [ "These operations can be implemented with the following functions." ] }, { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "fragment" + } + }, "outputs": [], "source": [ "# Function for performing hard thresholding.\n", @@ -367,7 +524,11 @@ }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "source": [ "In the following cell you can see how these functions work on the following matrix for different values of the threshold, $\\lambda$.\n", "\n", @@ -377,25 +538,23 @@ "11 & 12 & 13 & 14 & 15 \\\\\n", "6 & 17 & 18 & 19 & 20 \\\\\n", "21 & 22 & 23 & 24 & 25 \n", - "\\end{bmatrix}$$\n", - "\n", - "
\n", - "INTERACTIVE CELL" + "\\end{bmatrix}$$" ] }, { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "skip" + } + }, "outputs": [], "source": [ - "######################################\n", - "# YOU SHOULD INTERACT WITH THIS CELL #\n", - "######################################\n", + "# Define the matrix M\n", + "matrix = np.arange(25).reshape(5, 5) + 1\n", "\n", "def test_thresh(thresh_type=None, lambda_val=None):\n", - "\n", - " matrix = np.arange(25).reshape(5, 5) + 1\n", " \n", " if thresh_type == 'Hard':\n", " thresh_matrix = hard_thresh(matrix, lambda_val)\n", @@ -409,16 +568,46 @@ " print('The l1 norm of the matrix:', l1_norm(thresh_matrix))\n", " \n", "lambda_slider = IntSlider(value=0, min=0, max=26, step=1)\n", - "thresh_option = Select(options=['Hard', 'Soft'], value='Hard', description='Threshold Type', disabled=False)\n", - " \n", + "thresh_option = Select(options=['Hard', 'Soft'], value='Hard', description='Threshold Type', disabled=False)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, + "source": [ + "#### INTERACTIVE CELL" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "slideshow": { + "slide_type": "fragment" + } + }, + "outputs": [], + "source": [ + "######################################\n", + "# YOU SHOULD INTERACT WITH THIS CELL #\n", + "######################################\n", + "\n", "interact(test_thresh, thresh_type=thresh_option, lambda_val=lambda_slider)" ] }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "source": [ - "### Median Absolute Deviation\n", + "### Median Absolute Deviation\n", "\n", "The threshold value, $\\lambda$, will vary depending on the type of problem. In the case of denoising $\\lambda=n\\sigma$, where $\\sigma$ an estimate of the noise standard deviation in the sparse domain and $n$ is multiplicative factor (usually around 3 or 4).\n", "\n", @@ -432,7 +621,11 @@ { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "fragment" + } + }, "outputs": [], "source": [ "# Function to calculate the noise standard deviation using the median absolute deviation.\n", @@ -443,24 +636,32 @@ }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "source": [ "The following cell demonstrates how well this function recovers the value of $\\sigma$.\n", "\n", - "
\n", - "INTERACTIVE CELL" + "#### INTERACTIVE CELL" ] }, { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "fragment" + } + }, "outputs": [], "source": [ "######################################\n", "# YOU SHOULD INTERACT WITH THIS CELL #\n", "######################################\n", "\n", + "@interact(sigma=FloatSlider(value=0.2, min=0.2, max=2, step=0.2))\n", "def test_sigma_mad(sigma=None):\n", "\n", " # Generate random signal\n", @@ -468,18 +669,18 @@ " random_signal = np.random.randn(1000) * sigma\n", "\n", " # Calcualte sigma from MAD\n", - " print('sigma estimate from MAD =', sigma_mad(random_signal))\n", - " \n", - "sigma_slider = FloatSlider(value=0.2, min=0.2, max=2, step=0.2)\n", - "\n", - "interact(test_sigma_mad, sigma=sigma_slider)" + " print('sigma estimate from MAD =', sigma_mad(random_signal))" ] }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "source": [ - "### Sparse Regularisation\n", + "### Sparse Regularisation\n", "\n", "For solving an inverse problem of the form\n", "\n", @@ -487,25 +688,58 @@ "\n", "we aim to minimise a function of the following form\n", "\n", - "$$\\begin{aligned} & \\underset{x}{\\text{argmin}} & \\frac{1}{2}\\|y-Hx\\|_2^2\\end{aligned}$$\n", - "\n", + "$$\\begin{aligned} & \\underset{x}{\\text{argmin}} & \\frac{1}{2}\\|y-Hx\\|_2^2\\end{aligned}$$" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, + "source": [ "Now, if we know that $x$ is sparse in the domain $x = \\phi\\alpha$, we can update our minimisation problem as follows\n", "\n", "$$\\begin{aligned} & \\underset{\\alpha}{\\text{argmin}} & \\frac{1}{2}\\|y-H\\phi\\alpha\\|_2^2 + \\lambda\\|\\alpha\\|_1\\end{aligned}$$\n", "\n", - "where we now seek a value of $\\alpha$ that both minimises the data fidelity term and requires that $\\alpha$ be sparse. This formulation is generally referred to as a *synthesis prior*. This problem can alternatively be formulated as follows\n", + "where we now seek a value of $\\alpha$ that both minimises the data fidelity term and requires that $\\alpha$ be sparse. This formulation is generally referred to as a *synthesis prior*. " + ] + }, + { + "cell_type": "markdown", + "metadata": { + "slideshow": { + "slide_type": "fragment" + } + }, + "source": [ + "This problem can alternatively be formulated as follows\n", "\n", "$$\\begin{aligned} & \\underset{x}{\\text{argmin}} & \\frac{1}{2}\\|y-Hx\\|_2^2 + \\lambda\\|\\phi^{T}x\\|_1\\end{aligned}$$\n", "\n", - "which is referred to as an *analysis prior*. When $\\phi$ is orthogonal these two formulations are equivalent, when $\\phi$ is not orthogonal this is no longer the case. Each approach has certain benefits when applied to different problems." + "which is referred to as an *analysis prior*. " ] }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, + "source": [ + "When $\\phi$ is orthogonal these two formulations are equivalent, when $\\phi$ is not orthogonal this is no longer the case. Each approach has certain benefits when applied to different problems." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "slideshow": { + "slide_type": "slide" + } + }, "source": [ - "---\n", - "\n", "## Denoising Example\n", "\n", "This example provides a very simple demonstration of how sparse representations of signals can be extremely useful. The objective is to recover the original signal from a noisy observation by denoising it in a appropriate sparse domain.\n", @@ -517,7 +751,9 @@ "cell_type": "code", "execution_count": null, "metadata": { - "scrolled": false + "slideshow": { + "slide_type": "subslide" + } }, "outputs": [], "source": [ @@ -530,30 +766,50 @@ "# Define the signal.\n", "x = np.cos(t + 11 * np.pi / 1000)\n", "\n", + "plot.line_plot(x, title='True Signal', ylim=(-1.1, 1.1))" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "scrolled": false, + "slideshow": { + "slide_type": "subslide" + } + }, + "outputs": [], + "source": [ "# Add noise to the signal.\n", "np.random.seed(1)\n", "y = x + sigma * np.random.randn(x.size)\n", "\n", - "plot.line_plot(x, title='True Signal', ylim=(-1.1, 1.1))\n", "plot.line_plot(y, title='Noisy Observation')" ] }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "source": [ - "### Choose Sparse Representation\n", + "### Choose Sparse Representation\n", "\n", "Your first task is to identify an appropriate sparse domain for the signals.\n", "\n", - "
\n", - "EXERCISE CELL " + "### EXERCISE " ] }, { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "fragment" + } + }, "outputs": [], "source": [ "##############################\n", @@ -567,7 +823,11 @@ }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "fragment" + } + }, "source": [ "Now we can have a look at the siganls in the sparse domain you chose." ] @@ -575,36 +835,69 @@ { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "outputs": [], "source": [ - "# Plot the signals\n", + "# Plot the x signal in the sparse domain\n", "if x_sparse is not None:\n", - " plot.line_plot(np.abs(x_sparse), title='True Signal in Sparse Domain')\n", + " plot.line_plot(np.abs(x_sparse), title='True Signal in Sparse Domain')" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, + "outputs": [], + "source": [ + "# Plot the y signal in the sparse domain\n", "if y_sparse is not None:\n", " plot.line_plot(np.abs(y_sparse), title='Noisy Observation in Sparse Domain')" ] }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "source": [ "Can you identify and similarities between the original and noisy signals in the sparse domain?\n", "\n", - "If not, you may want to rethink your choice.\n", - "\n", - "### Denoising\n", + "If not, you may want to rethink your choice." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, + "source": [ + "### Denoising\n", "\n", "Your next task is to determine the appropriate threshold value, $\\lambda$, and use this to remove the noise from the observed signal. \n", "\n", - "
\n", - "EXERCISE CELL " + "### EXERCISE" ] }, { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "fragment" + } + }, "outputs": [], "source": [ "##############################\n", @@ -620,7 +913,11 @@ }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "source": [ "Now let's look at the signal you have recovered." ] @@ -628,7 +925,11 @@ { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "fragment" + } + }, "outputs": [], "source": [ "# Plot the denoised signal\n", @@ -638,16 +939,24 @@ }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "source": [ "Did it work?" ] }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "source": [ - "### Residual\n", + "### Residual\n", "\n", "Finally, we can examine the residual of your recovered signal to get a sense of how close to the original signal it is.\n", "\n", @@ -657,7 +966,11 @@ { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "fragment" + } + }, "outputs": [], "source": [ "# Plot the residual\n", @@ -667,7 +980,11 @@ }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "source": [ "Which threshold produces the best residual? Why do we have a residual? Try decreasesing the noise added to the original signal and see what happens." ] @@ -675,14 +992,17 @@ { "cell_type": "markdown", "metadata": { - "collapsed": true + "collapsed": true, + "slideshow": { + "slide_type": "skip" + } }, "source": [ "---\n", "\n", - "## Hints\n", + "## Hints\n", "\n", - "** Denoising Exercise Hints **\n", + "**Denoising Exercise Hints**\n", "\n", "1. For your sparse representation you may want to use soemthing *fast and f(o)urious*.\n", "1. Try not to get *mad* when choosing $\\lambda$.\n", @@ -693,7 +1013,11 @@ }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "skip" + } + }, "source": [ "[Open next notebook ->](./sparsity_2_compressed_sensing.ipynb)" ] @@ -701,7 +1025,7 @@ ], "metadata": { "kernelspec": { - "display_name": "Python 3", + "display_name": "sparsity", "language": "python", "name": "python3" }, @@ -715,7 +1039,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.7.2" + "version": "3.14.2" } }, "nbformat": 4, diff --git a/ada/sparsity_2_compressed_sensing.ipynb b/sparsity_2_compressed_sensing.ipynb similarity index 70% rename from ada/sparsity_2_compressed_sensing.ipynb rename to sparsity_2_compressed_sensing.ipynb index 5ebd368..298af3a 100644 --- a/ada/sparsity_2_compressed_sensing.ipynb +++ b/sparsity_2_compressed_sensing.ipynb @@ -4,17 +4,26 @@ "cell_type": "markdown", "metadata": {}, "source": [ + "
\n", + "\n", "# Introduction to Sparsity II: Compressed Sensing\n", "\n", "---\n", "\n", "> Author: Samuel Farrens \n", - "> Year: 2018 \n", + "> Year: 2018 (updated in 2023) \n", "> Email: [samuel.farrens@cea.fr](mailto:samuel.farrens@cea.fr) \n", - "> Website: https://sfarrens.github.io\n", - "\n", - "---\n", - "\n", + "> Website: https://sfarrens.github.io" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "slideshow": { + "slide_type": "skip" + } + }, + "source": [ "## Contents\n", "\n", "1. [Set-Up](#Set-Up)\n", @@ -40,10 +49,12 @@ }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "slide" + } + }, "source": [ - "---\n", - "\n", "## Set-Up\n", "\n", "Here we will import a couple of packages that will be needed throughout the notebook. \n", @@ -57,78 +68,98 @@ "metadata": {}, "outputs": [], "source": [ - "# Tell Jupyter to display plots in this notebook.\n", - "%matplotlib inline\n", - "\n", - "# Import the numpy package with the alias np.\n", - "import numpy as np\n", - "\n", - "# Import function to set size of figures.\n", - "from matplotlib.pylab import rcParams\n", - "\n", - "# Import interaction module.\n", - "from ipywidgets.widgets import *\n", - "\n", - "# Import tutorial functions.\n", - "from sparsity_tutorial import plot\n", - "from sparsity_tutorial.functions import l1_norm, sigma_mad" + "# Uncomment the following lines if you are running this on Google Colab\n", + "# !git clone -b ada https://github.com/CosmoStat/Tutorials.git\n", + "# !cp -r /content/Tutorials/data .\n", + "# import sys\n", + "# sys.path.insert(0,'/content/Tutorials')" ] }, { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "fragment" + } + }, "outputs": [], "source": [ - "# For Python 2 users only\n", - "# Uncomment the following commands\n", + "# Import the numpy package with the alias np.\n", + "import numpy as np\n", "\n", - "# Import Python 3 behaviour\n", - "# from __future__ import print_function" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Set size of figures.\n", - "rcParams['figure.figsize'] = (14.0, 8.0)" + "# Import interaction module.\n", + "from ipywidgets.widgets import interact, IntSlider\n", + "\n", + "# Import tutorial functions.\n", + "from sparsity_tutorial import plot\n", + "from sparsity_tutorial.functions import l1_norm, sigma_mad" ] }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "slide" + } + }, "source": [ - "---\n", - "\n", "## Introduction\n", "\n", - "### Objective\n", + "### Objective\n", "\n", "This notebook aims to give a brief introduction to how Compressed Sensing (CS) works in practice without getting into too much mathematical detail. For more comprehensive details please see *e.g.*\n", "\n", "* Candès et al. (2006b)\n", "* Candès and Tao (2006)\n", - "* Donoho (2006a)\n", - "\n", - "### Nyquist-Shannon Sampling Theorem\n", + "* Donoho (2006a)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, + "source": [ + "### Nyquist-Shannon Sampling Theorem\n", "\n", "The Nyquist-Shannon Sampling Theorem dictates that a bandlimited singal can perfectly be recovered if the sample frequency, $f_s$, (*i.e.* number of sample taken per unit time/space) is at least twice the highest frequency contained in the signal, $f_c$.\n", "\n", - "$$f_s \\geq 2f_c$$\n", - "\n", - "In the following cell you can see a sine wave (green dotted line) that we wish to recover. Initially this signal is sampled well below the Nyquist rate (blue dots). The resulting effect, known as aliasing, creates the illusion of a signal with a lower frequency (red line). Try increasing the sampling rate and/or frequency to see what happens.\n", - "\n", - "
\n", - "INTERACTIVE CELL" + "$$f_s \\geq 2f_c$$" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "slideshow": { + "slide_type": "fragment" + } + }, + "source": [ + "In the following cell you can see a sine wave (green dotted line) that we wish to recover. Initially this signal is sampled well below the Nyquist rate (blue dots). The resulting effect, known as aliasing, creates the illusion of a signal with a lower frequency (red line). Try increasing the sampling rate and/or frequency to see what happens." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, + "source": [ + "#### INTERACTIVE CELL" ] }, { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "fragment" + } + }, "outputs": [], "source": [ "######################################\n", @@ -136,38 +167,49 @@ "######################################\n", "\n", "# function to plot signal of given frequency with a given sampling rate\n", + "@interact(sampling_rate=IntSlider(value=9, min=2, max=42, step=1), frequency=IntSlider(value=10, min=1, max=10, step=1))\n", "def sampling(sampling_rate, frequency):\n", "\n", " t = np.arange(0, 1, 1.0 / sampling_rate)\n", " y = np.sin(2 * np.pi * frequency * t)\n", - " plot.stem_plot(y, x_vals=t, title='Aliasing', imag=False, ylim=(-1.1, 1.1), line=True, f=frequency)\n", - " \n", - "sampling_slider = IntSlider(value=9, min=2, max=42, step=1)\n", - "frequency_slider = IntSlider(value=10, min=1, max=10, step=1)\n", - " \n", - "interact(sampling, sampling_rate=sampling_slider, frequency=frequency_slider)" + " plot.stem_plot(y, x_vals=t, title='Aliasing', imag=False, ylim=(-1.1, 1.1), line=True, f=frequency)" ] }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "source": [ - "### Compressed Sensing\n", + "### Compressed Sensing\n", "\n", "Compressed (or compressive) sensing is a paradigm that allows one to sample certain singals at a rate lower than the Nyquist rate by exploiting the sparsity of the signal in a given domain." ] }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "source": [ - "---\n", - "\n", "## 1D Example\n", "\n", - "To get a more intuitive understanding of how the principles of CS work we begin with an example of a very simple 1D signal. This example will demonstrate how to recover the signal from a masked observation.\n", - "\n", - "### Sparse Signal\n", + "To get a more intuitive understanding of how the principles of CS work we begin with an example of a very simple 1D signal. This example will demonstrate how to recover the signal from a masked observation." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "slideshow": { + "slide_type": "fragment" + } + }, + "source": [ + "### Sparse Signal\n", "\n", "First we define a sparse signal, $\\alpha$, with 128 coefficients of which only 5 have non-zero values (*i.e.* a $k$-sparse singal with $k=5$)." ] @@ -175,7 +217,11 @@ { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "outputs": [], "source": [ "# full sample size\n", @@ -196,9 +242,13 @@ }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "source": [ - "### Fourier Transform\n", + "### Fourier Transform\n", "\n", "Next we want to generate a new signal $x$ which is the Fourier transform of $\\alpha$. *i.e.*\n", "\n", @@ -218,7 +268,11 @@ { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "outputs": [], "source": [ "# normalised FFT\n", @@ -229,23 +283,30 @@ "# normalised inverse FFT\n", "def infft(data):\n", "\n", - " return np.sqrt(data.size) * np.fft.fftshift(np.fft.ifft(np.fft.ifftshift(data)))\n" + " return np.sqrt(data.size) * np.fft.fftshift(np.fft.ifft(np.fft.ifftshift(data)))" ] }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "source": [ - "Use the functions above to generate the singal $x$.\n", + "### EXERCISE\n", "\n", - "
\n", - "EXERCISE CELL " + "Use the functions above to generate the singal $x$." ] }, { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "fragment" + } + }, "outputs": [], "source": [ "##############################\n", @@ -262,9 +323,13 @@ }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "source": [ - "### Observation\n", + "### Observation\n", "\n", "To simulate a CS problem we need to randomly subsample $x$ in order to generate an observation (or acquisition) $y$. We can do this using a masking operator $H$ that we define in the cell below." ] @@ -272,7 +337,11 @@ { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "fragment" + } + }, "outputs": [], "source": [ "# Function to mask a signal.\n", @@ -283,20 +352,27 @@ }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "source": [ "For this example we will keep only a quarter of the coefficients in $x$.\n", "\n", - "Define a mask that can be used with above function that will keep only $25\\%$ of the coefficients.\n", + "### EXERCISE \n", "\n", - "
\n", - "EXERCISE CELL " + "Define a mask that can be used with above function that will keep only $25\\%$ of the coefficients." ] }, { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "fragment" + } + }, "outputs": [], "source": [ "##############################\n", @@ -312,23 +388,38 @@ }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "source": [ "Now, we can model $y$ as \n", "\n", - "$$y = H\\phi\\alpha$$\n", - "\n", - "Use the mask you defined to generate the observation $y$.\n", + "$$y = H\\phi\\alpha$$" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, + "source": [ + "### EXERCISE\n", "\n", - "
\n", - "EXERCISE CELL " + "Use the mask you defined to generate the observation $y$." ] }, { "cell_type": "code", "execution_count": null, "metadata": { - "scrolled": false + "scrolled": false, + "slideshow": { + "slide_type": "fragment" + } }, "outputs": [], "source": [ @@ -346,7 +437,11 @@ }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "source": [ "Notice that the signal $y$ is indeed shorter than $x$, however in order to properly visualse this masked signal we need to upsample.\n", "\n", @@ -356,7 +451,11 @@ { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "fragment" + } + }, "outputs": [], "source": [ "# Function to upsample a signal.\n", @@ -369,7 +468,11 @@ }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "source": [ "Now we can display the upsampled version of $y$." ] @@ -377,7 +480,11 @@ { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "fragment" + } + }, "outputs": [], "source": [ "# plot real and imaginary components of y\n", @@ -387,22 +494,30 @@ }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "source": [ - "### Inverse Problem\n", + "### Inverse Problem\n", "\n", "\n", "The problem we aim to solve is to recover the original signal $\\alpha$ from $y$ given that we know both $\\phi$ and $H$. This corresponds to the following optimisation problem\n", "\n", "$$\\begin{aligned} & \\underset{\\alpha}{\\text{argmin}} & \\frac{1}{2}\\|y-H\\phi\\alpha\\|_2^2 + \\lambda\\|\\alpha\\|_1\\end{aligned}$$\n", "\n", - "where we implement sparse regularisation given that we know a priori that $\\alpha$ is sparse. We can implement $l_1$ regularisation via soft thresholding of the sparse coefficients. For this example we also have to manage the complex coefficients of the signal, which we do using the following function." + "where we implement sparse regularisation given that we know a priori that $\\alpha$ is sparse. We can implement $l_1$ regularisation via soft thresholding of the sparse coefficients. For this example, we also have to manage the complex coefficients of the signal, which we do using the following function." ] }, { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "outputs": [], "source": [ "# Function for performing soft thresholding with complex coefficients.\n", @@ -414,7 +529,11 @@ }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "fragment" + } + }, "source": [ "For a given estimate of $\\alpha$ the optimisation problem provides the cost of the solution. To solve this problem we aim to minimise this cost, which we will cacluate with the following function." ] @@ -422,7 +541,11 @@ { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "fragment" + } + }, "outputs": [], "source": [ "# Function to calculate the cost.\n", @@ -433,9 +556,13 @@ }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "source": [ - "### Optimisation Algorithm\n", + "### Optimisation Algorithm\n", " \n", "For this example we will use the forward-backward splitting algorithm with a soft thresholding operation to promote sparse solutions. It has the following form for each iteration\n", "\n", @@ -448,7 +575,11 @@ { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "outputs": [], "source": [ "# Function that performs simple forward backward splitting.\n", @@ -457,7 +588,7 @@ " alpha_rec = first_guess\n", " cost = []\n", " \n", - " for i in range(n_iter):\n", + " for _ in range(n_iter):\n", " \n", " alpha_temp = alpha_rec - gamma * grad(observation, alpha_rec, mask)\n", " alpha_rec = soft_thresh(alpha_temp, lambda_val)\n", @@ -471,25 +602,40 @@ }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "source": [ - "### Gradient\n", + "### Gradient\n", "\n", "The gradient of the above problem is given by\n", "\n", - "$$\\nabla F(\\alpha) = \\phi^{-1}H^T(H\\phi\\alpha-y)$$\n", - "\n", - "write a funciton that implements the gradient calculation.\n", + "$$\\nabla F(\\alpha) = \\phi^{-1}H^T(H\\phi\\alpha-y)$$" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, + "source": [ + "### EXERCISE\n", "\n", - "
\n", - "EXERCISE CELL " + "Write a funciton that implements the gradient calculation." ] }, { "cell_type": "code", "execution_count": null, "metadata": { - "scrolled": false + "scrolled": false, + "slideshow": { + "slide_type": "fragment" + } }, "outputs": [], "source": [ @@ -506,21 +652,36 @@ }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "source": [ - "### Signal Recovery\n", + "### Signal Recovery\n", "\n", - "Now we can use the forward-backward algorithm to recover $\\alpha$. Notice how the solution improves after a given number of iterations until it converges on a very good reconstruction of $\\alpha$.\n", - "\n", - "
\n", - "INTERACTIVE CELL" + "Now we can use the forward-backward algorithm to recover $\\alpha$. Notice how the solution improves after a given number of iterations until it converges on a very good reconstruction of $\\alpha$." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, + "source": [ + "#### INTERACTIVE CELL" ] }, { "cell_type": "code", "execution_count": null, "metadata": { - "scrolled": false + "scrolled": false, + "slideshow": { + "slide_type": "fragment" + } }, "outputs": [], "source": [ @@ -529,26 +690,37 @@ "######################################\n", "\n", "# Interactive function to plot the recovered alpha after a given number of iterations\n", + "@interact(n_iter=IntSlider(value=1, min=1, max=361, step=40))\n", "def solve(n_iter):\n", - " \n", - " alpha_rec, cost = forwardBackward(observation=y, first_guess=np.ones(n_samples), mask=mask, grad=grad,\n", - " lambda_val=0.01, n_iter=n_iter, gamma=1.0, return_cost=True)\n", - " \n", - " plot.stem_plot(alpha_rec, title=r'$\\hat{\\alpha}$')\n", - " plot.cost_plot(cost)\n", " \n", - "slider = IntSlider(value=1, min=1, max=361, step=40)\n", - "if grad is not None:\n", - " interact(solve, n_iter=slider)" + " if grad is not None:\n", + " alpha_rec, cost = forwardBackward(observation=y, first_guess=np.ones(n_samples), mask=mask, grad=grad,\n", + " lambda_val=0.01, n_iter=n_iter, gamma=1.0, return_cost=True)\n", + "\n", + " plot.stem_plot(alpha_rec, title=r'$\\hat{\\alpha}$')\n", + " plot.cost_plot(cost)" ] }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "source": [ - "Notice that the solution converges after roughly 200 iterations.\n", - "\n", - "### Performance\n", + "Notice that the solution converges after roughly 200 iterations." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, + "source": [ + "### Performance\n", "\n", "Finally, we can test the quality of our solution by comparing our recovered $\\hat{\\alpha}$ with the true $\\alpha$. Notice that the difference between the two signals is extrmeley small even for the imaginary components." ] @@ -557,7 +729,10 @@ "cell_type": "code", "execution_count": null, "metadata": { - "scrolled": false + "scrolled": false, + "slideshow": { + "slide_type": "subslide" + } }, "outputs": [], "source": [ @@ -572,7 +747,11 @@ }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "source": [ "The Normalised Mean Squared Error (NMSE) provides a useful measure of the quality of a given signal reconstruction.\n", "\n", @@ -584,7 +763,11 @@ { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "fragment" + } + }, "outputs": [], "source": [ "#normalised mean squared error\n", @@ -595,20 +778,27 @@ }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "source": [ "We can use this function to test solutions to our minimisation problem.\n", "\n", - "Calculate the NMSE of $\\hat{\\alpha}$ after 10 and 200 iterations.\n", + "### EXERCISE\n", "\n", - "
\n", - "EXERCISE CELL " + "Calculate the NMSE of $\\hat{\\alpha}$ after 10 and 200 iterations." ] }, { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "fragment" + } + }, "outputs": [], "source": [ "##############################\n", @@ -628,17 +818,23 @@ }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "source": [ "Can you see an improvement in the NMSE after an appropriate number of iterations?" ] }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "slide" + } + }, "source": [ - "---\n", - "\n", "## 2D Exercise\n", "\n", "Attempt to solve the following problem using the techniques you learned from the first exercise." @@ -646,9 +842,13 @@ }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "source": [ - "### Problem\n", + "### Problem\n", "\n", "An observer with limited time on her detection instrument has been forced to take only few noisy samples of a singal of interest. Fortunately she has been celever and has chosen the samples randomly. She now provides you with her samples and the mask she used to collect her data and asks for help in recoevering the original signal.\n", "\n", @@ -656,8 +856,17 @@ "\n", "The observation can be modelled as\n", "\n", - "$$y = Hx + n$$\n", - "\n", + "$$y = Hx + n$$" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, + "source": [ "**Questions and tasks**\n", "\n", "1. What percentage of the coefficients have been sampled?\n", @@ -673,9 +882,13 @@ }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "source": [ - "### Data\n", + "### Data\n", "\n", "For this exercise you are given a subsampled noisy observation $y$ and a mask $H$. Your task is to recoever the original image $x$. " ] @@ -683,7 +896,11 @@ { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "fragment" + } + }, "outputs": [], "source": [ "# Shape of the true image\n", @@ -693,47 +910,83 @@ "mask2d = np.load('data/cs_mask.npy')\n", "\n", "# Load the observation\n", - "y2d = np.load('data/cs_obs_data.npy')\n", - "\n", + "y2d = np.load('data/cs_obs_data.npy')" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, + "outputs": [], + "source": [ "# Display the mask H\n", - "plot.display(mask2d, 'Mask', shape=image_shape, cmap='bone')\n", - "\n", + "plot.display(mask2d, 'Mask', shape=image_shape, cmap='bone')" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, + "outputs": [], + "source": [ "# Display the observation y\n", "plot.display(y2d, r'$y$', shape=(25, 20))" ] }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "source": [ - "### Solutions\n", + "### Solutions\n", "\n", - "EXERCISE: provide your solutions in the cell(s) below.\n" + "### EXERCISE\n", + "\n", + "Provide your solutions in the cell(s) below." ] }, { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "fragment" + } + }, "outputs": [], "source": [] }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "skip" + } + }, "source": [ "---\n", "\n", - "## Hints\n", + "## Hints\n", "\n", - "** 1D Example Hints**\n", + "**1D Example Hints**\n", "\n", "1. The mask should only contain zeros and ones.\n", "1. The mask should have the same number of coefficients as the original signal.\n", "1. To define $y$ note that $x$ has already been defined. \n", "1. To define the gradient make sure you use ALL your operators in the right order.\n", "\n", - "** 2D Exercise Hints **\n", + "**2D Exercise Hints**\n", "\n", "1. You can work out the percentage of coefficients retained using only the mask.\n", "1. Note that, although the signal is a 2D image, data have been provided in the form of 1D vectors. Think about how this compares to the 1D example.\n", @@ -743,7 +996,11 @@ }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "skip" + } + }, "source": [ "[Open next notebook ->](./sparsity_3_deconvolution.ipynb)" ] @@ -751,7 +1008,7 @@ ], "metadata": { "kernelspec": { - "display_name": "Python 3", + "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, @@ -765,7 +1022,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.7.2" + "version": "3.10.8" } }, "nbformat": 4, diff --git a/ada/sparsity_3_deconvolution.ipynb b/sparsity_3_deconvolution.ipynb similarity index 69% rename from ada/sparsity_3_deconvolution.ipynb rename to sparsity_3_deconvolution.ipynb index 1193b20..1e7d12e 100644 --- a/ada/sparsity_3_deconvolution.ipynb +++ b/sparsity_3_deconvolution.ipynb @@ -2,19 +2,32 @@ "cells": [ { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "slide" + } + }, "source": [ + "
\n", + "\n", "# Introduction to Sparsity III: Deconvolution\n", "\n", "---\n", "\n", "> Author: Samuel Farrens \n", - "> Year: 2018 \n", + "> Year: 2018 (updated in 2023) \n", "> Email: [samuel.farrens@cea.fr](mailto:samuel.farrens@cea.fr) \n", - "> Website: https://sfarrens.github.io\n", - "\n", - "---\n", - "\n", + "> Website: https://sfarrens.github.io" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "slideshow": { + "slide_type": "skip" + } + }, + "source": [ "## Contents\n", "\n", "1. [Set-Up](#Set-Up)\n", @@ -34,10 +47,12 @@ }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "slide" + } + }, "source": [ - "---\n", - "\n", "## Set-Up\n", "\n", "Here we will import a couple of packages that will be needed throughout the notebook. \n", @@ -51,21 +66,30 @@ "metadata": {}, "outputs": [], "source": [ - "# Tell Jupyter to display plots in this notebook.\n", - "%matplotlib inline\n", - "\n", + "# Uncomment the following lines if you are running this on Google Colab\n", + "# !git clone -b ada https://github.com/CosmoStat/Tutorials.git\n", + "# !cp -r /content/Tutorials/data .\n", + "# !pip install --upgrade scipy\n", + "# import sys\n", + "# sys.path.insert(0,'/content/Tutorials')" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "slideshow": { + "slide_type": "fragment" + } + }, + "outputs": [], + "source": [ "# Import the numpy package with the alias np.\n", "import numpy as np\n", "\n", "# Import tools from Scipy\n", "from scipy.ndimage import gaussian_filter\n", - "from scipy.misc import face\n", - "\n", - "# Import function to set size of figures.\n", - "from matplotlib.pylab import rcParams\n", - "\n", - "# Import interaction module.\n", - "from ipywidgets.widgets import *\n", + "from scipy.datasets import face\n", "\n", "# Import tutorial functions.\n", "from sparsity_tutorial import plot\n", @@ -73,62 +97,63 @@ ] }, { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], + "cell_type": "markdown", + "metadata": { + "slideshow": { + "slide_type": "slide" + } + }, "source": [ - "# For Python 2 users only\n", - "# Uncomment the following commands\n", + "## Introduction\n", "\n", - "# Import Python 3 behaviour\n", - "# from __future__ import print_function" + "### Objective\n", + "\n", + "The objective of this notebook is to demonstrate how sparsity can be used for deconvolution. A very simple image is used to simpify the implementation. More realistic applications to astrophysical images are provided in\n", + "\n", + "* Farrens et al. (2017)" ] }, { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], + "cell_type": "markdown", + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "source": [ - "# Set size of figures.\n", - "rcParams['figure.figsize'] = (14.0, 8.0)" + "### Point Spread Function\n", + "\n", + "All optical systems are subject to imperfections that distort the images obtained. The sum of these aberrations is commonly referred to as the point spread function (PSF) or instrumental response. Removing the effects of the PSF from images of interest if a fundamental aspect of image processing, particularly in fields such as astrophysics where accurate measurments of the shapes and flux of objects has a significant impact on the science.\n", + "\n", + "The blurring introduced by the PSF can be modelled as a convolution of the PSF with the image. Consequently, the reverse process, deconvolution, removes this effect." ] }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "source": [ - "---\n", - "\n", - "## Introduction\n", - "\n", - "### Objective\n", - "\n", - "The objective of this notebook is to demonstrate how sparsity can be used for deconvolution. A very simple image is used to simpify the implementation. More realistic applications to astrophysical images are provided in\n", - "\n", - "* Farrens et al. (2017)\n", - "\n", - "### Point Spread Function\n", - "\n", - "All optical systems are subject to imperfections that distort the images obtained. The sum of these aberrations is commonly referred to as the point spread function (PSF) or instrumental response. Removing the effects of the PSF from images of interest if a fundamental aspect of image processing, particularly in fields such as astrophysics where accurate measurments of the shapes and flux of objects has a significant impact on the science.\n", - "\n", - "The blurring introduced by the PSF can be modelled as a convolution of the PSF with the image. Consequently, the reverse process, deconvolution, removes this effect.\n", - "\n", "In the following cell we define a function for generating a Gaussian PSF." ] }, { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "fragment" + } + }, "outputs": [], "source": [ "# Function to generate a Gaussian PSF.\n", "def make_psf(shape, sigma):\n", " \n", " psf = np.zeros(shape)\n", - " psf[list(zip(np.array(psf.shape) // 2))] = 1\n", + " psf[tuple(np.array(psf.shape) // 2)] = 1\n", " \n", " return gaussian_filter(psf, sigma)\n", "\n", @@ -137,11 +162,15 @@ }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "slide" + } + }, "source": [ "## Deconvolution Example\n", "\n", - "### Convolution\n", + "### Convolution\n", "\n", "The convolution theorem shows that a signal $a$ can be convolved with a signal $b$ (under certain conditions) as follows\n", "\n", @@ -153,7 +182,11 @@ { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "fragment" + } + }, "outputs": [], "source": [ "# This function convolves an image with a kernel using FFT.\n", @@ -167,7 +200,11 @@ }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "source": [ "The following cell demonstrates what happens to an image when it is convolved with the PSF we defined." ] @@ -175,7 +212,11 @@ { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "fragment" + } + }, "outputs": [], "source": [ "# Load example image\n", @@ -185,20 +226,46 @@ "psf_1 = make_psf(image.shape, 10)\n", "\n", "# Convolve image with PSF\n", - "image_conv = fftconvolve(image, psf_1)\n", - "\n", + "image_conv = fftconvolve(image, psf_1)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, + "outputs": [], + "source": [ "# Display the original image\n", - "plot.display(image, cmap='Greys_r', title='Original Image')\n", - "\n", + "plot.display(image, cmap='Greys_r', title='Original Image')" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, + "outputs": [], + "source": [ "# Display the convolved image\n", "plot.display(image_conv, cmap='Greys_r', title='Convolved Image')" ] }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "slide" + } + }, "source": [ - "### Deconvolution\n", + "### Deconvolution\n", "\n", "Deconvolution is the process of reversing the effects of convolution.\n", "\n", @@ -210,7 +277,11 @@ { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "fragment" + } + }, "outputs": [], "source": [ "# This function deconvolves an image with a kernel using FFT.\n", @@ -224,7 +295,11 @@ }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "source": [ "Using this function we can recover our original image given that we know the PSF." ] @@ -232,7 +307,11 @@ { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "fragment" + } + }, "outputs": [], "source": [ "image_deconv = fftdeconvolve(image_conv, psf_1)\n", @@ -243,16 +322,24 @@ }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "source": [ "As you can see the all the blurring effects have been removed from the convolved image." ] }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "source": [ - "### Convolution with Noise\n", + "### Convolution with Noise\n", "\n", "In practical applications all observations contain some degree of noise. We can imagine that our Raccoon was photographed with a very good instrument with only the tiniest amount of noise ($\\sigma=10^{-6}$) as follows." ] @@ -260,7 +347,11 @@ { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "outputs": [], "source": [ "# Add Gaussian noise to convolved image\n", @@ -272,11 +363,24 @@ }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "source": [ - "Given the amount of noise, the convolved image looks virtually the same.\n", - "\n", - "### Deconvolution with Noise\n", + "Given the amount of noise, the convolved image looks virtually the same." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, + "source": [ + "### Deconvolution with Noise\n", "\n", "Let's see what happens when we try to recoever the original image from the noisy convolved image." ] @@ -284,7 +388,11 @@ { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "outputs": [], "source": [ "image_deconv_noise = fftdeconvolve(image_conv_noise, psf_1)\n", @@ -295,17 +403,23 @@ }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "source": [ "The image is completely destroyed! Why?" ] }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "slide" + } + }, "source": [ - "---\n", - "\n", "## Ill-Posed Problem\n", "\n", "A problem is considered well-posed if \n", @@ -314,9 +428,18 @@ "2. the solution is unique,\n", "3. the solution's behavior changes continuously with the initial conditions.\n", "\n", - "Any problem that does meet all three of this requirements is considered \"ill-posed\".\n", - "\n", - "### Ill-Conditioned Problem\n", + "Any problem that does meet all three of this requirements is considered \"ill-posed\"." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, + "source": [ + "### Ill-Conditioned Problem\n", "\n", "A problem (or matrix) with a large condition number is considered to be \"ill-coniditioned\". This relates to the 3rd point on our list of requirements for a well-posed problem.\n", "\n", @@ -332,7 +455,11 @@ { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "outputs": [], "source": [ "# Data for first problem\n", @@ -346,7 +473,11 @@ }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "source": [ "Two solve both problems we simply need to calculate $x = H^{-1}y$ as follows." ] @@ -354,21 +485,29 @@ { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "fragment" + } + }, "outputs": [], "source": [ "# Solution to the first problem\n", - "x1 = np.dot(np.linalg.inv(H1), y1)\n", + "x1 = np.linalg.inv(H1) @ y1\n", "print('x1 =', x1)\n", "\n", "# Solution to the second problem\n", - "x2 = np.dot(np.linalg.inv(H2), y2)\n", + "x2 = np.linalg.inv(H2) @ y2\n", "print('x2 =', x2)" ] }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "source": [ "Now, let's see what happens if we add a small change to $y$ (e.g. mimicking the effects of noise).\n", "\n", @@ -380,28 +519,45 @@ { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "fragment" + } + }, "outputs": [], "source": [ "# Noisy versions of y\n", "y1_noise = np.array([4.0, 7.01])\n", "y2_noise = np.array([3.0, 1.48])\n", "\n", - "x1_noise = np.dot(np.linalg.inv(H1), y1_noise)\n", + "x1_noise = np.linalg.inv(H1) @ y1_noise\n", "print('x1 =', x1_noise)\n", "\n", - "x2_noise = np.dot(np.linalg.inv(H2), y2_noise)\n", + "x2_noise = np.linalg.inv(H2) @ y2_noise\n", "print('x2 =', x2_noise)" ] }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "source": [ "Notice that for the first problem the solution has changed only as much as the input data, while for the second problem the solution is completely different. \n", "\n", - "It is clear from these results that the first problem is well-conditioned well the second is ill-conditioned. This means that the first problem can be solved analytically, even with noise, well the second cannot.\n", - "\n", + "It is clear from these results that the first problem is well-conditioned well the second is ill-conditioned. This means that the first problem can be solved analytically, even with noise, while the second cannot." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "slideshow": { + "slide_type": "fragment" + } + }, + "source": [ "We can relate this property back to our deconvolution problem, which can also be modelled as a linear inverse problem of the form\n", "\n", "$$y=Hx + n$$\n", @@ -411,26 +567,47 @@ }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "slide" + } + }, "source": [ - "---\n", "\n", "## Deconvolution Exercise\n", "\n", - "### Deconvolution with Sparsity\n", + "### Deconvolution with Sparsity\n", "\n", "Given that we can model our observations as $y=Hx + n$, we aim to solve an optimisation problem of the form\n", "\n", "$$\\begin{aligned} & \\underset{\\alpha}{\\text{argmin}} & \\frac{1}{2}\\|y-H\\phi\\alpha\\|_2^2 + \\lambda\\|\\alpha\\|_1\\end{aligned}$$\n", "\n", - "where we assume we know that our data is sparse in the dictionary $\\phi$. Notice that this is exactly the same problem we needed to solve for the CS exercises, only now $H$ is a convolution operator rather than a masking operator.\n", - "\n", - "### Problem\n", + "where we assume we know that our data is sparse in the dictionary $\\phi$. Notice that this is exactly the same problem we needed to solve for the CS exercises, only now $H$ is a convolution operator rather than a masking operator." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, + "source": [ + "### Problem\n", "\n", "An observer with has obtained a very blurry and noisy image with instrument. Fortunately she is celever and knows the exact PSF of her instrument. She now provides you with her image and the PSF her instrument and asks for help in deconvoving the PSF effects.\n", "\n", - "You, being an expert in sparsity and having some experience with this type of data, know that this signal is sparse in Fourier space. You also know that $\\lambda=0.003$ would be an appropriate value for this problem.\n", - "\n", + "You, being an expert in sparsity and having some experience with this type of data, know that this signal is sparse in Fourier space. You also know that $\\lambda=0.003$ would be an appropriate value for this problem." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, + "source": [ "**Questions and tasks**\n", "1. Try to deconvolve the image analytically.\n", "1. Does it work?\n", @@ -440,9 +617,18 @@ "1. Deconvolve the image using the Forward-Backward algorithm.\n", "1. Display your deconvolved image $\\hat{x}$.\n", "1. Roughly how many iterations did it take to converge?\n", - "1. Finally, load the file `data/cs_true_data.npy` to determine the NMSE of your solution.\n", - "\n", - "### Data\n", + "1. Finally, load the file `data/cs_true_data.npy` to determine the NMSE of your solution." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, + "source": [ + "### Data\n", "\n", "For this exercise you are given a noisy convolved observation $y$ and a PSF. Your task is to deconvolve the psf effects from the observation to recover the original image $x$. " ] @@ -450,27 +636,57 @@ { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "fragment" + } + }, "outputs": [], "source": [ "# Load the observation.\n", "y = np.load('data/deconv_obs_data.npy')\n", "\n", "# Load the PSF.\n", - "psf = np.load('data/deconv_psf.npy')\n", - "\n", + "psf = np.load('data/deconv_psf.npy')" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, + "outputs": [], + "source": [ "# Display the observed image.\n", - "plot.display(y, r'$y$')\n", - "\n", + "plot.display(y, r'$y$')" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, + "outputs": [], + "source": [ "# Display the PSF.\n", "plot.display(psf, 'PSF')" ] }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "source": [ - "### Functions\n", + "### Functions\n", "\n", "Some functions that may be useful for this exercise are provided below." ] @@ -478,7 +694,11 @@ { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "fragment" + } + }, "outputs": [], "source": [ "# H operator.\n", @@ -489,15 +709,26 @@ "# H transpose operator.\n", "def Ht(data, psf):\n", " \n", - " return fftconvolve(data, np.rot90(psf, 2))\n", - "\n", + " return fftconvolve(data, np.rot90(psf, 2))" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, + "outputs": [], + "source": [ "# Function that performs simple forward backward splitting.\n", "def forwardBackward(observation, first_guess, psf, grad, lambda_val, n_iter=300, gamma=1.0, return_cost=False):\n", " \n", " alpha_rec = first_guess\n", " cost = []\n", " \n", - " for i in range(n_iter):\n", + " for _ in range(n_iter):\n", " \n", " alpha_temp = alpha_rec - gamma * grad(observation, alpha_rec, psf)\n", " alpha_rec = soft_thresh(alpha_temp, lambda_val)\n", @@ -511,29 +742,43 @@ }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "source": [ - "### Solutions\n", + "### Solutions\n", + "\n", + "### EXERCISE\n", "\n", - "EXERCISE: provide your solutions in the cell(s) below." + "Provide your solutions in the cell(s) below." ] }, { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "fragment" + } + }, "outputs": [], "source": [] }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "skip" + } + }, "source": [ "---\n", "\n", - "## Hints\n", + "## Hints\n", "\n", - "** Deconvolution Exercise **\n", + "**Deconvolution Exercise**\n", "\n", "1. I wonder if the functions provided will help for defining the cost function and the gradient.\n", "1. Make sure you display your deconvolved image in the correct domain.\n", @@ -542,7 +787,11 @@ }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "skip" + } + }, "source": [ "[Open next notebook ->](./wavelets_1.ipynb)" ] @@ -550,7 +799,7 @@ ], "metadata": { "kernelspec": { - "display_name": "Python 3", + "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, @@ -564,7 +813,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.7.2" + "version": "3.10.8" } }, "nbformat": 4, diff --git a/ada/sparsity_tutorial/__init__.py b/sparsity_tutorial/__init__.py similarity index 100% rename from ada/sparsity_tutorial/__init__.py rename to sparsity_tutorial/__init__.py diff --git a/ada/sparsity_tutorial/functions.py b/sparsity_tutorial/functions.py similarity index 100% rename from ada/sparsity_tutorial/functions.py rename to sparsity_tutorial/functions.py diff --git a/ada/sparsity_tutorial/plot.py b/sparsity_tutorial/plot.py similarity index 68% rename from ada/sparsity_tutorial/plot.py rename to sparsity_tutorial/plot.py index aec8e74..f406ef4 100644 --- a/ada/sparsity_tutorial/plot.py +++ b/sparsity_tutorial/plot.py @@ -1,11 +1,29 @@ import numpy as np import matplotlib.pyplot as plt +from matplotlib import pylab -def regression_plot(x1, y1, x2, y2): +# Set the default size of figures. +pylab.rcParams['figure.figsize'] = (8.0, 6.0) - plt.plot(x1, y1, 'o', color="#19C3F5", label='Data') - plt.plot(x2, y2, '--', color='#FF4F5B', label='Model') + +def regression_plot(data, model): + """Regression Plot. + + This function plots the data points (in blue) and a the corresponding + model line (in red). + + Parameters + ---------- + data : tuple + Data points in (x, y) + model : tuple + Model points in (x, y) + + """ + + plt.plot(*data, 'o', color="#19C3F5", label='Data') + plt.plot(*model, '--', color='#FF4F5B', label='Model') plt.title('Best Fit Line', fontsize=20) plt.xlabel('x', fontsize=18) plt.ylabel('y', fontsize=18) @@ -13,12 +31,34 @@ def regression_plot(x1, y1, x2, y2): plt.show() -def grad_plot(x, y1, y2, dy, point): - - plt.plot(x, y1, 'b-', label='$||x||_2$') - plt.plot(x, y2, 'g-', label='$||x||_2^2$') - plt.plot(x[point], y2[point], 'ro') - plt.plot(x, dy, 'r--', label='Grad $||x_i||_2^2$') +def grad_plot(x, x_norm, x_norm2, grad, tangent, index): + """Gradient Plot. + + This function plots the values of x against the corresponding l2-norm + (in blue) and squared l2-norm (in green). The gradient of the squared + l2-norm is shown at a specified point (in red). + + Parameters + ---------- + x : numpy.ndarray + Array of data values in the x-axis + x_norm : numpy.ndarray + Array of l2-norm values + x_norm2 : numpy.ndarray + Array of squared l2-norm values + grad : float + Gradient at the specified point + tangent : numpy.ndarray + Array of tangent line y-axis values + index : int + Index of currently selected point + + """ + + plt.plot(x, x_norm, 'b-', label='$||x_i||_2$') + plt.plot(x, x_norm2, 'g-', label='$||x_i||_2^2$') + plt.plot(x[index], x_norm2[index], 'ro') + plt.plot(x, tangent, 'r--', label=rf'$\nabla ||x_i||_2^2={grad}$') plt.ylim(-0.1, 1.0) plt.title('Convex L2-Norm') plt.xlabel('$x$', fontsize=24) @@ -26,6 +66,57 @@ def grad_plot(x, y1, y2, dy, point): plt.show() +def regression_plot_cost(data, model, cost): + """Regression Plot. + + This function plots the data points (in blue) and a the corresponding + model line (in red). + + Parameters + ---------- + data : tuple + Data points in (x, y) + model : tuple + Model points in (x, y) + + """ + + _, ax = plt.subplots(nrows=1, ncols=2, figsize=(16, 6)) + + ax[0].plot(*data, 'o', color="#19C3F5", label='Data') + ax[0].plot(*model, '--', color='#FF4F5B', label='Model') + ax[0].set_title('Best Fit Line', fontsize=20) + ax[0].set_xlabel('x', fontsize=18) + ax[0].set_ylabel('y', fontsize=18) + ax[0].legend() + + ax[1].plot(cost, linestyle='-', color='#FF4F5B') + ax[1].set_xlabel('Iteration', fontsize=18) + ax[1].set_ylabel('Cost', fontsize=18) + ax[1].set_title('Cost Function', fontsize=20) + + plt.show() + + +def cost_plot(cost): + """Cost Plot. + + This function plots the cost for a set of iterations. + + Parameters + ---------- + cost : list + List of cost values + + """ + + plt.plot(cost, linestyle='-', color='#FF4F5B') + plt.xlabel('Iteration', fontsize=18) + plt.ylabel('Cost', fontsize=18) + plt.title('Cost Function', fontsize=20) + plt.show() + + def stem_plot(data, x_vals=None, title=None, imag=True, ylim=None, xlab=None, ylab=None, line=False, f=None): @@ -77,15 +168,6 @@ def line_plot(data, title=None, ylim=None, xlab=None): plt.show() -def cost_plot(data): - - plt.plot(data, linestyle='-', color='#FF4F5B') - plt.xlabel('Iteration', fontsize=18) - plt.ylabel('Cost', fontsize=18) - plt.title('Cost Function', fontsize=20) - plt.show() - - def display(data, title='example', shape=None, cmap='gist_stern', vmax=None, vmin=None): @@ -149,9 +231,9 @@ def filter_plot(n, Fn, g, Fg, Fg_fft): ax[1].plot(n, g, '-', color='#0764DB') ax[1].set_title('$g[n-m]$', fontsize=24) ax[2].plot(n, Fg, '-', color='#0764DB') - ax[2].set_title('$F \cdot g$', fontsize=24) + ax[2].set_title(r'$F \cdot g$', fontsize=24) ax[3].plot(n, Fg_fft, '-', color='#0764DB') - ax[3].set_title('$\mathcal{F}(F \cdot g)$', fontsize=24) + ax[3].set_title(r'$\mathcal{F}(F \cdot g)$', fontsize=24) for ax_i in ax: ax_i.set_xticks([]) ax_i.set_yticks([]) diff --git a/ada/wavelets_1.ipynb b/wavelets_1.ipynb similarity index 72% rename from ada/wavelets_1.ipynb rename to wavelets_1.ipynb index 510967a..4c05945 100644 --- a/ada/wavelets_1.ipynb +++ b/wavelets_1.ipynb @@ -2,19 +2,32 @@ "cells": [ { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "slide" + } + }, "source": [ + "
\n", + "\n", "# Introduction to Wavelets\n", "\n", "---\n", "\n", "> Author: Samuel Farrens \n", - "> Year: 2018 \n", + "> Year: 2018 (updated in 2023) \n", "> Email: [samuel.farrens@cea.fr](mailto:samuel.farrens@cea.fr) \n", - "> Website: https://sfarrens.github.io\n", - "\n", - "---\n", - "\n", + "> Website: https://sfarrens.github.io" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "slideshow": { + "slide_type": "skip" + } + }, + "source": [ "## Contents\n", " \n", "1. [Set-Up](#Set-Up)\n", @@ -38,10 +51,12 @@ }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "slide" + } + }, "source": [ - "---\n", - "\n", "## Set-Up\n", "\n", "Here we will import a couple of packages that will be needed throughout the notebook. \n", @@ -55,16 +70,27 @@ "metadata": {}, "outputs": [], "source": [ - "# Tell Jupyter to display plots in this notebook.\n", - "%matplotlib inline\n", - "\n", + "# Uncomment the following lines if you are running this on Google Colab\n", + "# !git clone -b ada https://github.com/CosmoStat/Tutorials.git\n", + "# !cp -r /content/Tutorials/data .\n", + "# !pip install --upgrade scipy\n", + "# import sys\n", + "# sys.path.insert(0,'/content/Tutorials')" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "slideshow": { + "slide_type": "fragment" + } + }, + "outputs": [], + "source": [ "# Import the numpy package with the alias np.\n", "import numpy as np \n", "\n", - "# Import the pyplot package from matplotlib with the alias plt.\n", - "import matplotlib.pyplot as plt \n", - "from matplotlib.pylab import rcParams\n", - "\n", "# Import some tools from scipy.\n", "from scipy.signal import gausspulse\n", "\n", @@ -72,62 +98,41 @@ "import time\n", "\n", "# Import interaction module\n", - "from ipywidgets.widgets import *\n", + "from ipywidgets.widgets import interact\n", "\n", "# Import tutorial functions.\n", "from sparsity_tutorial import plot\n", "from sparsity_tutorial.functions import *" ] }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# For Python 2 users only\n", - "# Uncomment the following commands\n", - "\n", - "# Import Python 3 behaviour\n", - "# from __future__ import print_function" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Set size of figures.\n", - "rcParams['figure.figsize'] = (14.0, 8.0)" - ] - }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "slide" + } + }, "source": [ - "---\n", "\n", "## Introduction\n", "\n", - "### Objective\n", + "### Objective\n", "\n", "The objective of this notebook is to provide a very brief introduction to the concept of wavelets with some practical demonstrations of how they work. To keep things as simple and open a possible I have avoided using packages such as PyWavelets or PySAP to implement the wavelet transforms. However, after undertanding the basics, these packages can be very useful for practical applications. \n", - "\n", - "### Wavelets\n", - "\n", - "Wavelets are a class functions that are localised in both time and frequency (or space and scale). In this notebook we will explore some of the basic properties of wavelets and how they compare to other time-frequency analysis techniques.\n" + "\n" ] }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "source": [ - "---\n", - "\n", "## Stationary vs Non-Stationary Signals\n", "\n", - "### Stationary Signal\n", + "### Stationary Signal\n", "\n", "Stationary signals are those that do not change over time. Take for example the following singal that combines 3 sine waves with different frequencies." ] @@ -135,7 +140,11 @@ { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "outputs": [], "source": [ "# Set a range of t values\n", @@ -151,7 +160,11 @@ }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "source": [ "We can clearly see that the pattern produced by the combination of frequencies is periodic. By taking the Fourier transform of this signal we can easily pinpoint the exact frequencies it contains." ] @@ -159,7 +172,11 @@ { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "outputs": [], "source": [ "# Calculate the FFT of the signal\n", @@ -172,11 +189,24 @@ }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "source": [ - "As expected we get three peaks at the corresponding frequencies. In fact, the frequency information tells us everything we need to know about this signal.\n", - "\n", - "### Non-Stationary Signal\n", + "As expected we get three peaks at the corresponding frequencies. In fact, the frequency information tells us everything we need to know about this signal.\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, + "source": [ + "### Non-Stationary Signal\n", "\n", "Non-stationary signals are those that change over time. Let's combine the same 3 sine waves at different time intervals and see what happens." ] @@ -184,7 +214,11 @@ { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "outputs": [], "source": [ "# Define a non-stationary signal\n", @@ -198,7 +232,11 @@ }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "source": [ "In this example the signal is clearly not periodic. As before we can take the Fourier transform to examine the frequency content." ] @@ -206,7 +244,11 @@ { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "outputs": [], "source": [ "# Calculate the FFT of the signal\n", @@ -219,17 +261,23 @@ }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "source": [ - "The frequency space is not as clean as that of the stationary signal due to the sudden change of frequencies, but we can still see wider peaks centred on our original frequencies. For a more realistic signal we would expect something more like this for both cases. The main difference for the non-stationary signal is that even though we can work out the frequencies we don't know when they occurred." + "The frequency space is not as clean as that of the stationary signal due to the sudden change of frequencies, but we can still see wider peaks centred on our original frequencies. For a more realistic signal, we would expect something more like this for both cases. The main difference for the non-stationary signal is that, even though we can work out the frequencies, we don't know when they occurred." ] }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "slide" + } + }, "source": [ - "---\n", - "\n", "## Windowed Fourier Transform\n", "\n", "Most discrete signals, such as the non-stationary time series in the cell below, can be decomposed into a sum of sine and cosine wave with the Discrete Fourier Transform (DFT).\n", @@ -244,7 +292,11 @@ { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "outputs": [], "source": [ "# Define the signal\n", @@ -262,7 +314,11 @@ }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "source": [ "In doing so, however, we lose all of the time information. We can try instead looking at the frequency information contained within smaller time intervals." ] @@ -271,7 +327,10 @@ "cell_type": "code", "execution_count": null, "metadata": { - "scrolled": false + "scrolled": false, + "slideshow": { + "slide_type": "subslide" + } }, "outputs": [], "source": [ @@ -287,7 +346,11 @@ }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "source": [ "Notice that we can identify different frequencies of the signal contained within the time intervals, but what happens if we make the windows smaller?" ] @@ -295,10 +358,12 @@ { "cell_type": "markdown", "metadata": { - "collapsed": true + "slideshow": { + "slide_type": "subslide" + } }, "source": [ - "### Uncertainty Principle\n", + "### Uncertainty Principle\n", "\n", "The **Heisenberg–Gabor limit**, \n", "\n", @@ -306,9 +371,19 @@ "\n", "an interpretation of the uncertainty principle, shows that it is not possible to simultaneously sharply localize a signal in both the time domain and frequency domain.\n", "\n", - "In other words we will always need to make a trade-off between time information and frequency information, but what is the best way to preserve as much of both as possible?\n", - "\n", - "### Gabor Transform\n", + "In other words we will always need to make a trade-off between time information and frequency information, but what is the best way to preserve as much of both as possible?" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "collapsed": true, + "slideshow": { + "slide_type": "subslide" + } + }, + "source": [ + "### Gabor Transform\n", "\n", "The Gabor transform or Short Time Fourier Transform (STFT) can be represented as:\n", "\n", @@ -324,15 +399,41 @@ { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "outputs": [], "source": [ "# Define a Gaussian filter.\n", "def get_filter(m, a):\n", "\n", " return np.exp(-a * (n - m) ** 2)\n", - " return np.exp(-a * (n - m) ** 2)\n", - "\n", + " return np.exp(-a * (n - m) ** 2)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, + "source": [ + "#### INTERACTIVE CELL" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "slideshow": { + "slide_type": "fragment" + } + }, + "outputs": [], + "source": [ "@interact(pos=(0, 10, 0.5), alpha=(0, 20, 1))\n", "def show_filter(pos=0.0, alpha=1):\n", " \n", @@ -351,18 +452,45 @@ }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, + "source": [ + "Try adjusting the filter width and position and see what happens." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "source": [ - "Try adjusting the filter width and position and see what happens.\n", - "\n", "Now, to try to preserve some frequency and time information simultaneously we will slide the filter across the whole signal and store the frequencies at a set of time intervals." ] }, + { + "cell_type": "markdown", + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, + "source": [ + "#### INTERACTIVE CELL" + ] + }, { "cell_type": "code", "execution_count": null, "metadata": { - "scrolled": false + "scrolled": false, + "slideshow": { + "slide_type": "fragment" + } }, "outputs": [], "source": [ @@ -384,7 +512,11 @@ }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "source": [ "Try changing the filter width to see what happens.\n", "\n", @@ -393,18 +525,24 @@ }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "slide" + } + }, "source": [ - "---\n", - "\n", "## Wavelets" ] }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "source": [ - "### Multiresolution Analysis\n", + "### Multiresolution Analysis\n", "\n", "One way we can improve upon the Gabor method is using *Multiresolution Analysis (MRA)*. Rather than choosing a fixed window size that will be more localised in either the time or frequency domain, we can instead do the following:\n", "\n", @@ -416,26 +554,43 @@ }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "source": [ - "### Wavelet Types\n", + "### Wavelet Types\n", "\n", "The wavelet function is often represented as follows:\n", "\n", "$$\\psi_{a,b}(t) = \\frac{1}{\\sqrt{a}} \\psi \\Bigg(\\frac{t-b}{a} \\Bigg)$$\n", "\n", - "where $a$ defines the wavelet scale and $b$ the shift. $\\psi$ is known as the *Mother Wavelet* which must posses certain properties (see *e.g.* [Wikipedia](https://en.wikipedia.org/wiki/Wavelet#Mother_wavelet) for more details). Many different wavelets exist, but we will only discuss a few basic ones.\n", - "\n", + "where $a$ defines the wavelet scale and $b$ the shift. $\\psi$ is known as the *Mother Wavelet* which must posses certain properties (see *e.g.* [Wikipedia](https://en.wikipedia.org/wiki/Wavelet#Mother_wavelet) for more details). Many different wavelets exist, but we will only discuss a few basic ones.\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, + "source": [ "A couple of essential properties we want any given wavelet to have are the following:\n", "\n", "* $\\int_{-\\infty}^\\infty\\psi(t)dt = 0$ (*i.e.* equal positive and negative area under the curve)\n", "* $\\|\\psi(t)\\|_2^2 = 1$\n", - "* Compact support (*i.e.* signal does not last forever)\n" + "* Compact support (*i.e.* signal does not last forever)" ] }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "source": [ "**Haar Wavelet**\n", "\n", @@ -447,7 +602,11 @@ { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "fragment" + } + }, "outputs": [], "source": [ "# Function to define Haar wavelet.\n", @@ -462,7 +621,11 @@ }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "source": [ "**Mexican Hat Wavelet**\n", "\n", @@ -475,7 +638,10 @@ "cell_type": "code", "execution_count": null, "metadata": { - "scrolled": false + "scrolled": false, + "slideshow": { + "slide_type": "fragment" + } }, "outputs": [], "source": [ @@ -483,24 +649,43 @@ "def mex_hat_wt(t, sigma=1.0):\n", "\n", " return (2.0 * np.pi ** -0.25 * (3.0 * sigma) ** -0.5 *\n", - " (1 - (t / sigma) ** 2) * np.exp(-t ** 2 / (2.0 * sigma ** 2)))\n", - "\n", + " (1 - (t / sigma) ** 2) * np.exp(-t ** 2 / (2.0 * sigma ** 2)))" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, + "outputs": [], + "source": [ "# Display\n", "plot.wavelet_plot(n, Fn, haar_wt(2 * n / np.max(n) - 0.5), mex_hat_wt(n / np.max(n) - 0.5, 0.1))" ] }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "source": [ "In general one aims to use the wavelet most suited to the problem at hand." ] }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "source": [ - "### Continuous Wavelet Transform\n", + "### Continuous Wavelet Transform\n", "\n", "The Continuous Wavelet Transform (CWT) is used to decompose a signal into wavelets and can be defined as follows:\n", "\n", @@ -512,7 +697,11 @@ { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "outputs": [], "source": [ "# Function to get wavelet scales\n", @@ -520,8 +709,19 @@ " \n", " J = 12.0 * np.log2(a_max / a_min)\n", " del_a = (a_max - a_min) / (J - 1)\n", - " return a_min + del_a * np.arange(J)\n", - "\n", + " return a_min + del_a * np.arange(J)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, + "outputs": [], + "source": [ "# Function to perform continuous wavelet transform\n", "def cwt(data, wavelet, scales):\n", " \n", @@ -531,8 +731,19 @@ " for scale in scales:\n", " w_ft = fft(wavelet(t / scale) * scale ** -0.5)\n", " out.append(ifft(w_ft * x_ft))\n", - " return np.abs(out)\n", - "\n", + " return np.abs(out)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, + "outputs": [], + "source": [ "# Set wavelet scales\n", "scales = get_scales(1, 100)\n", "\n", @@ -545,11 +756,24 @@ }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "source": [ - "As you can see we are able to preserve detailed information regarding the scale and translation of the signal.\n", - "\n", - "### Discrete Wavelet Transform\n", + "As you can see we are able to preserve detailed information regarding the scale and translation of the signal." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, + "source": [ + "### Discrete Wavelet Transform\n", "\n", "The Discrete Wavelet Transform (DWT) uses a discrete set of wavelet scales and translations that removes certain redundancies introduced in the CWT.\n", "\n", @@ -558,15 +782,33 @@ "\n", "This corresponds to consecutively convolving the signal with high and low pass filters with a downsampling at each level. \n", "\n", - "\n", - "\n", - "### Undecimated Wavelet Transform\n", + "\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, + "source": [ + "### Undecimated Wavelet Transform\n", "\n", "The Undecimated Wavelet Transform (UWT) (also called the Stationary Wavelet Transform) removes the downsampling factor to compensate for the translation-invariance in the DWT at the cost of added redundancy. \n", "\n", - "\n", - "\n", - "### Starlet Transform\n", + "" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, + "source": [ + "### Starlet Transform\n", "\n", "Other specialised wavelet transforms exist such as the **Starlet** (or Isotropic Undecimated Wavelet Transform) that\n", "is particularly well suited to astrophysical images, which are generally isotropic (read more here).\n", @@ -575,15 +817,28 @@ "\n", "$$\\mathbf{x} = \\mathbf{x}^{J} + \\sum_{j=1}^J \\boldsymbol{w}^{j}$$\n", "\n", - "where the first level ($j = 1$) corresponds to the highest frequencies (*i.e.* the finest scale).\n", - "\n", + "where the first level ($j = 1$) corresponds to the highest frequencies (*i.e.* the finest scale)." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, + "source": [ "Have a look at the following starlet decomposition of Venus." ] }, { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "fragment" + } + }, "outputs": [], "source": [ "# Load starlet decomposition of Venus.\n", @@ -595,18 +850,25 @@ }, { "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "subslide" + } + }, "source": [ "Reconstruct the image of venus from the decomposition.\n", "\n", - "
\n", - "EXERCISE CELL" + "### EXERCISE" ] }, { "cell_type": "code", "execution_count": null, - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "fragment" + } + }, "outputs": [], "source": [ "##############################\n", @@ -623,20 +885,17 @@ }, { "cell_type": "markdown", - "metadata": {}, - "source": [ - "This makes the starlet a useful transform for obtaining sparse representations of astronomical images." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, + "metadata": { + "slideshow": { + "slide_type": "skip" + } + }, "source": [ "---\n", "\n", - "## Hints\n", + "## Hints\n", "\n", - "** Starlet Transform **\n", + "**Starlet Transform**\n", "\n", "1. You have just about reached the *sum(m)it* of this tutorial." ] @@ -644,7 +903,7 @@ ], "metadata": { "kernelspec": { - "display_name": "Python 3", + "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, @@ -658,7 +917,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.7.2" + "version": "3.10.8" } }, "nbformat": 4,