{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Introduction to Neural Networks (Psy 5038): Python" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "So far in this course we've tried to emphasize concepts usually with toy examples. We'll now spend a few classes going over tools that can be applied to state-of-the-art problems in cognitive neuroscience. Mathematica is excellent for learning concepts, and for many high-end applications. However, it is not so good for specialized work in certain fields due lack of software. Matlab and increasingly Python have large user communities who are building tools that we can build on. That's the goal. But first an introduction to IPython as an environment for scientific programming." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Starting Python in the middle" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Python code can be run within a notebook, similar in spirit to Mathematica notebooks. Originally IPython, the environment is being replaced by Jupyter, which in addition to Python supports R and Julia among other languages. \n", "\n", "To run a notebook on the CLA server using Microsoft Remote Desktop, bring up a terminal window with cmd and then type: ipython notebook. This will bring up a browser, and from there, you can locate and load any notebooks (*.ipynb) you have downloaded." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In order to run this notebook locally on your computer, you will need to have Python and Jupyter/IPython installed. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "###### Installation" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The hardest, or at least the most frustrating, aspect of Python can be installation. There are a number of package managers. We recommend [Anaconda](https://store.continuum.io/cshop/anaconda/).\n", "\n", "In addition to Python and [IPython](http://ipython.org/), you will need [numpy](http://www.numpy.org/), [matplotlib](http://matplotlib.org/) and the [scipy library](http://www.scipy.org/scipylib/index.html). All of these are part of the [scipy stack](http://scipy.org) for general purpose scientific programming. Later, we'll also need [pymc](http://pymc-devs.github.io/pymc/) for Bayesian computations using MCMC sampling, sckit modules built on scipy, such as the machine learning module [scikit-learn](http://scikit-learn.org/stable/index.html), and [scikit-image](http://scikit-image.org/).\n", "\n", "\n", "These [installation notes](http://iacs-courses.seas.harvard.edu/courses/am207/blog/installing-python.html) from a 2014 course at Harvard may be helpful.\n", "\n", "Once installed, find the directory where you've downloaded this notebook, and then from a terminal command line go the directory (or parent directory) and type: ipython notebook, or jupyter notebook (ipython's successor). This will bring up a browswer window from where you should see this notebook listed in your browser window. You can load it from there." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "###### Rationale for scientific python" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "http://nbviewer.ipython.org/github/jrjohansson/scientific-python-lectures/blob/master/Lecture-0-Scientific-Computing-with-Python.ipynb\n", "\n", "The prefix, nbviewer.ipython.org, in the above link provides an online notebook reader. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Style" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's first cover some style. We'll get to substance--i.e. python code--soon enough. Right off the bat, you can create a Raw NBConvert, various headings, or a [Markdown cell](http://daringfireball.net/projects/markdown/). Try the latter and type in LaTeX commands. E.g. try putting the next line between double dollar signs:\n", "\n", "p(y_i|x)= \\frac{e^{y_i(w.x+b)}}{{1+e^{(w.x+b)}}}\n", "\n", "\n", "\n", "After getting acquainted with the menu items and buttons of the IPython notebook interface, take a look at these notes on: [IPython's Rich Display System]( http://nbviewer.ipython.org/github/ipython/ipython/blob/1.x/examples/notebooks/Part%205%20-%20Rich%20Display%20System.ipynb). Try copying in cell content in this notebook to try out displaying different kinds of content. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Making and plotting lists" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "collapsed": false }, "outputs": [], "source": [ "import numpy as np" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To get started, let's look at some simple python coding examples. We need to load numpy to handle vectors and matrices. To make lists in Mathematica we typically used Table[], e.g. \n", "\n", " Table[Sin[x], {x, 0, 1, .1}];\n", "\n", "\n", "In python, we'll use \"list comprehensions\". Create a code cell and try these:\n", "\n", " [sin(x) for x in arange(0,1,.1)]\n", "\n", " [sin(x) for x in linspace(0,1,10)]\n", "\n", "But wait! Python needs to know where these function definitions came from. arange(), linspace(), and sin() are all numpy functions. We imported numpy functions with the shorthand \"np\", so you will need to write:\n", "\n", " np.sin(x), and np. arange(0,1,.1). \n", "\n", "For more on creating and manipulating numerical lists in NumPy see [scipy page](http://scipy-lectures.github.io/intro/numpy/index.html)" ] }, { "cell_type": "code", "execution_count": 20, "metadata": { "collapsed": false }, "outputs": [], "source": [ "?np.linspace" ] }, { "cell_type": "code", "execution_count": 17, "metadata": { "collapsed": false }, "outputs": [], "source": [ "sl=[np.sin(x) for x in np.arange(0,10,.1)]" ] }, { "cell_type": "code", "execution_count": 29, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "100" ] }, "execution_count": 29, "metadata": {}, "output_type": "execute_result" } ], "source": [ "len(sl)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's plot these values. To do this, we'll need to import matplotlib, and in particular the pyplot module, or plt for short. If you want a more matlab like plotting environment, you can use pylab:\n", "\n", " from pylab import *\n", "\n", "For more information on plotting, see [scipy notes](https://scipy-lectures.github.io/intro/matplotlib/matplotlib.html#ipython-and-the-pylab-mode), and Lecture 4 in the [scientific python notebook series](http://nbviewer.ipython.org/github/jrjohansson/scientific-python-lectures/blob/master/Lecture-4-Matplotlib.ipynb). " ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "collapsed": false }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/Users/kersten/anaconda/lib/python2.7/site-packages/matplotlib/font_manager.py:273: UserWarning: Matplotlib is building the font cache using fc-list. This may take a moment.\n", " warnings.warn('Matplotlib is building the font cache using fc-list. This may take a moment.')\n" ] } ], "source": [ "import matplotlib.pyplot as plt" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "collapsed": false }, "outputs": [], "source": [ "plt.plot(sl)\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "With the magic function in the next cell, your plot should appear in a separate window. Close the figure window and now excecute the next cell below, and then run the above plot cell again." ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "collapsed": false }, "outputs": [], "source": [ "%matplotlib inline" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We made 2D lists in Mathematica using the Table[] function like this:\n", "\n", " Table[Sin[x] Cos[y], {x, 0, 1, .1}, {y, 0, 1, .1}];\n", "\n", "Now try using the list comprehension syntax for python:\n", "\n", " [[sin(x)*cos(y) for x in arange(0,1,.1)] for y in arange(0,1,.1)] or\n", "\n", " [[sin(x)*cos(y) for x in linspace(0,1,10)] for y in linspace(0,1,10)]\n", "\n", "Again remember to specify the numpy class using np. " ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "collapsed": false }, "outputs": [], "source": [ "#If you remove the semicolon and it is the last line, the next line will print out the values\n", "[[np.sin(x)*np.cos(y) for x in np.arange(0,1,.1)] for y in np.arange(0,1,.1)];" ] }, { "cell_type": "code", "execution_count": 30, "metadata": { "collapsed": false }, "outputs": [], "source": [ "#but it won't if you assign a variable name to the array\n", "\n", "sl2=[[np.sin(x)*np.cos(y) for x in np.arange(0,1,.1)] for y in np.arange(0,1,.1)];" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "###### Getting parts of vectors and arrays" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "[Accessing array values or slicing](http://scipy-lectures.github.io/intro/numpy/numpy.html#indexing-and-slicing)" ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[[ 0 1 2 3 4 5]\n", " [ 6 7 8 9 10 11]\n", " [12 13 14 15 16 17]\n", " [18 19 20 21 22 23]\n", " [24 25 26 27 28 29]] \n", "\n", "(#rows, #columns)= (5, 6) # elements: 30\n" ] } ], "source": [ "#make a vector with 5 rows and 6 columns\n", "sl3=np.array([[x+y*6 for x in np.arange(0,6,1)] for y in np.arange(0,5,1)])\n", "#what happens if you don't use np.array() in the above line?\n", "\n", "print np.array(sl3),'\\n\\n(#rows, #columns)=',np.shape(sl3),\"# elements:\",np.size(sl3)" ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "array([0, 1, 2, 3, 4, 5])" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "#note that unlike Mathematica and Matlab (but like C), indexing starts with 0\n", "#first row\n", "sl3[0]" ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "array([ 0, 6, 12, 18, 24])" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "#first column\n", "sl3[:,0]" ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "collapsed": false }, "outputs": [], "source": [ "#Try accessing the element in the 3rd row, 4th column. It isn't sl3[3,4]|\n", "\n", "#Get the first 2 elements of sl. It isn't sl[0:1]." ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "array([[15, 16, 17],\n", " [21, 22, 23]])" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "#Now get the submatrix 3rd and 4th row, 4th through 6th columns\n", "sl3[2:4,3:6]" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "#try flattening s13. And then check its dimensions." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "###### Defining functions" ] }, { "cell_type": "code", "execution_count": 13, "metadata": { "collapsed": false }, "outputs": [], "source": [ "def f(x,y):\n", " return([[np.sin(3*x)*np.cos(4*y) for x in np.arange(0,1,.1)] for y in np.arange(0,1,.1)])" ] }, { "cell_type": "code", "execution_count": 14, "metadata": { "collapsed": false }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPcAAAD7CAYAAAC2TgIoAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAC8VJREFUeJzt3V9onfUdx/HPJ01j06Z/FCFiS+3KTuYQNlvodJPBNgcb\nO7BdDaqOsV3PPygMxTtvdjeGsF4oTsuG22BF2ODIkCH2QvFPZ9yqbWmwOmudwTG0lqbQtN9dJAsu\niTnPyTm/POd8935B8Zzw8MuXpG+fJ+c8+dURIQD5DNU9AIAyiBtIiriBpIgbSIq4gaSIG0hquFcL\n2eY9NaAmEeHFH+tZ3JJ04MCBSse1Wi01m81Kx05NTXUz0oreeuutSsedOHFC119/feV133zzzdWO\n1NYHH3xQ6bhz585pbGys8roXLlxY7UgrGhqqfnE4MzOj0dHRysdv3bp1NSO1tXPnzkrHvf3229q1\na1dHa+/evXsVE63s4MGDy36cy3IgKeIGkqol7kajUcenXbWrr7667hE6NjIyUvcIHRse7ulPicVt\n27at7hFWVEvcExMTdXzaVSPutbF+/fq6R+gIcQOoBXEDSRE3kBRxA0kRN5AUcQNJETeQFHEDSRE3\nkBRxA0kRN5AUcQNJETeQFHEDSRE3kFTf/3b8pUuXiq19/vz5Iut+/PHHRdaVpLNnzxZZd2Zmpsi6\n69atK7Ku1Nn+bJ0o+f27ePFisbUX48wNJEXcQFLEDSRF3EBSxA0kRdxAUsQNJFUpbtv32n7d9t9t\nP2l78DbFBv7PtI3b9rWS7pK0NyK+oLkbX/aXHgxAd6reobZO0ibblyVtlPReuZEA9ELbM3dEvCfp\n55LekXRG0ocR8ZfSgwHoTtszt+1tkr4n6TpJH0k6ZPv2iPjt4mNbrdbC40ajMXD/JhgwCKanpzU9\nPd32uCqX5d+UdCoi/i1Jtp+S9BVJS+JuNpsdjgmgU+Pj4xofH194fvTo0WWPq/Jq+TuSbra9wbYl\n3SrpeC+GBFBOlZ+5X5Z0SNKkpL9JsqRHC88FoEuVXi2PiIckPVR4FgA9xB1qQFLEDSRF3EBSxA0k\nRdxAUsQNJNX3WxuX2r5WkkZGyvzm6hVXXFFkXUnasGFDkXUjosi6Jbc2LvV1Lvn9K/n1WIwzN5AU\ncQNJETeQFHEDSRE3kBRxA0kRN5AUcQNJETeQFHEDSRE3kBRxA0kRN5AUcQNJETeQFHEDSRE3kBRx\nA0kRN5AUcQNJETeQVN/vflpyJ8orr7yyyLo7duwosq4kjY6OFll3ZmamyLold/vcunVrkXWvueaa\nIutK0tjYWLG1F+PMDSRF3EBSxA0kRdxAUsQNJEXcQFKV4ra91fYfbB+3/Ybtm0oPBqA7Vd/nfljS\n0xHxfdvDkjYWnAlAD7SN2/YWSV+NiB9JUkTMSjpbeC4AXapyWf4ZSf+y/YTtV20/arvMbVIAeqZK\n3MOS9ko6EBF7JZ2X9EDRqQB0rcrP3O9KOh0RR+afH5J0/3IHtlqthceNRkMTExNdDwjgf505c0Zn\nzpxpe1zbuCNi2vZp2xMRcVLSrZKOLXdss9nseFAAndm+fbu2b9++8PzIkSPLHlf11fK7JT1pe72k\nU5J+3O2AAMqqFHdE/E3SvsKzAOgh7lADkiJuICniBpIibiAp4gaSIm4gKeIGkur7rY23bNlSbO3L\nly8XWXfz5s1F1pXKbUE8OztbZF3bRdaVpJGRkSLrbtq0qci6UrntmJfDmRtIiriBpIgbSIq4gaSI\nG0iKuIGkiBtIiriBpIgbSIq4gaSIG0iKuIGkiBtIiriBpIgbSIq4gaSIG0iKuIGkiBtIiriBpIgb\nSKrvdz/dtm1bsbVL7UQZEUXWxdoouWNrybUX48wNJEXcQFLEDSRF3EBSxA0kRdxAUpXjtj1k+1Xb\nfyo5EIDe6OTMfY+kY6UGAdBbleK2vUPSdyQ9VnYcAL1S9cz9C0k/lcStV8CAaHv7qe2mpOmIeM32\n1yR96v1zrVZr4XGj0dDExEQvZgTwCSdPntTU1FTb49zuPmjbP5P0A0mzkkYlbZb0VET8cNFxceDA\ngVUP/GlK3qddam3uLR9sg3Zv+Z133qmIWLJw28vyiHgwInZGxG5J+yU9uzhsAP2H97mBpDr6lc+I\nOCzpcKFZAPQQZ24gKeIGkiJuICniBpIibiAp4gaS6vvdT6+66qpia5faWXXTpk1F1pWkkZGRIusO\nDZX5/3zJu/VmZ2eLrHvhwoUi60rSRx99VGztxThzA0kRN5AUcQNJETeQFHEDSRE3kBRxA0kRN5AU\ncQNJETeQFHEDSRE3kBRxA0kRN5AUcQNJETeQFHEDSRE3kBRxA0kRN5AUcQNJETeQVN9vbVxq+2FJ\nGh8fL7JuyZk3btxYZN3h4TJ/FUpubVxqC+Jz584VWVcqtzX1cjhzA0kRN5AUcQNJETeQFHEDSRE3\nkFTbuG3vsP2s7TdsH7V991oMBqA7Vd7cnJV0X0S8ZntM0l9tPxMRJwrPBqALbc/cEfF+RLw2//ic\npOOStpceDEB3OvqZ2/YuSTdKeqnEMAB6p/I9h/OX5Ick3TN/Bl+i1WotPG40GpqYmOh6QAD/a3Jy\nUpOTk22PqxS37WHNhf2biPjjpx3XbDYrDwhgdfbs2aM9e/YsPD948OCyx1W9LH9c0rGIeLjryQCs\niSpvhd0i6Q5J37A9aftV298uPxqAbrS9LI+I5yWtW4NZAPQQd6gBSRE3kBRxA0kRN5AUcQNJETeQ\nVN/vfjo6Olps7c2bNxdZdxB3P123rsy7nSV3Py31d2NoqNw5b2Zmptjai3HmBpIibiAp4gaSIm4g\nKeIGkiJuICniBpIibiAp4gaSIm4gKeIGkiJuICniBpIibiAp4gaSIm4gKeIGkiJuICniBpIibiAp\n4gaSIm4gqb7f2rjkNrPr168fqHUlaXi4zLes1LqXL18usm7JtUt+/0ptIb0cztxAUsQNJEXcQFLE\nDSRF3EBSleK2/W3bJ2yftH1/6aEAdK9t3LaHJP1S0rck3SDpNtvXlx4MQHeqnLm/JGkqIv4RERcl\n/V7S98qOBaBbVeLeLun0J56/O/8xAH2MF9SApKrcc3hG0s5PPN8x/7ElWq3WwuNGo6GJiYmuhgOw\n1CuvvKIjR460Pa5K3K9I+qzt6yT9U9J+Sbctd2Cz2exkRgCrsG/fPu3bt2/h+SOPPLLscW3jjohL\ntu+U9IzmLuN/FRHHezQngEIq/SpQRPxZ0ucKzwKgh3hBDUiKuIGkiBtIqpa4T548WcenXbUXX3yx\n7hE6dvjw4bpH6Nhzzz1X9wgdef755+seYUW1xD01NVXHp1014l4bgzbzCy+8UPcIK+KyHEiKuIGk\nHBG9WcjuzUIAOhYRXvyxnsUNoL9wWQ4kRdxAUmsa96DtxWZ7h+1nbb9h+6jtu+ueqQrbQ7Zftf2n\numepwvZW23+wfXz+a31T3TO1Y/te26/b/rvtJ22P1D3TYmsW94DuxTYr6b6IuEHSlyX9ZABmlqR7\nJB2re4gOPCzp6Yj4vKQvSurr3zq0fa2kuyTtjYgvaO4XsPbXO9VSa3nmHri92CLi/Yh4bf7xOc39\npevrLaZs75D0HUmP1T1LFba3SPpqRDwhSRExGxFnax6rinWSNtkelrRR0ns1z7PEWsY90Hux2d4l\n6UZJL9U7SVu/kPRTSYPyNshnJP3L9hPzP0o8anu07qFWEhHvSfq5pHc0tyvRhxHxl3qnWooX1Cqw\nPSbpkKR75s/gfcl2U9L0/NWG5//0u2FJeyUdiIi9ks5LeqDekVZme5vmrjqvk3StpDHbt9c71VJr\nGXflvdj6yfxl1yFJv4mIP9Y9Txu3SPqu7VOSfifp67Z/XfNM7bwr6XRE/HdTsEOai72ffVPSqYj4\nd0RckvSUpK/UPNMSaxn3wl5s868s7pc0CK/mPi7pWEQ8XPcg7UTEgxGxMyJ2a+7r+2xE/LDuuVYS\nEdOSTtv+726at6r/Xwx8R9LNtjfYtuZm7rsXAcv8i+vLGMS92GzfIukOSUdtT2ru59gH57edQu/c\nLelJ2+slnZL045rnWVFEvGz7kKRJSRfn//tovVMtxe2nQFK8oAYkRdxAUsQNJEXcQFLEDSRF3EBS\nxA0kRdxAUv8Bs4J05cUnAnUAAAAASUVORK5CYII=\n", "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "#Visualize f(x,y) using imshow()\n", "\n", "import matplotlib.cm as cm\n", "\n", "n = 10\n", "x = np.linspace(-3,3,4*n)\n", "y = np.linspace(-3,3,3*n)\n", "X, Y = np.meshgrid(x,y)\n", "plt.imshow(f(X,Y),cmap = cm.Greys_r,origin='lower',interpolation='nearest');" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "###### IPython Examples" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For a directory of [IPython notebook examples](https://github.com/ipython/ipython/wiki/A-gallery-of-interesting-IPython-Notebooks). E.g. here's a demo of [receptive field models]( http://nbviewer.ipython.org/github/jonasnick/ReceptiveFields/blob/master/receptiveFields.ipynb)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "###### Python neural network resources" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For python code, look at [neurolab](https://pythonhosted.org/neurolab/index.html). This has several topics that should look familiar. However, there is some material that we haven't covered, for example see Elmans network for learning temporal sequences.\n", "\n", "For python code to simulate spiking neurons, see http://briansimulator.org. " ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "collapsed": false }, "outputs": [], "source": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Ok, now let's look at a neural network. Try downloading the python file that simulates the two neuron Hopfield net [TwoNeuroHopfield.py](http://gandalf.psych.umn.edu/users/kersten/kersten-lab/courses/Psy5038WF2014/Lectures/Lect_18_Python/TwoNeuroHopfield.py). And then convert it to an IPython notebook.\n", "\n", "For the IPython version, look at this [link]( http://nbviewer.ipython.org/url/gandalf.psych.umn.edu/users/kersten/kersten-lab/courses/Psy5038WF2014/Lectures/Lect_12_Hopfield/HopfieldTwoNeuronDemo.ipynb)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 2", "language": "python", "name": "python2" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 2 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython2", "version": "2.7.12" } }, "nbformat": 4, "nbformat_minor": 0 }