Plotting a Logarithmic Y-Axis from a Pandas Histogram

Note to self: How to plot a histogram from Pandas that has a logarithmic y-axis. By using the "bottom" argument, you can make sure the bars actually show up. Ordinarily a "bottom" of 0 will result in no bars.

In [3]:

import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
%matplotlib inline

In [9]:

df = pd.DataFrame(np.random.randn(1000,1))
df.hist(bins=200, bottom=1)
plt.semilogy()

Out[9]:

[]

In []:

Join my mailing list for topics on scientific computing.

Posts on Scientific Computing

Sleep Quality Over Time

Every morning I log how I felt my sleep was. I log it on a scale of 1 to 5, with a "3" being "meh" sleep, "1" is awful and "5" is something I've never experienced. I've been doing this daily since September of 13 - so about 6 months of data at this point.

Here's a plot of my sleep ratings computed with a 7 day running average - a 7 day boxcar filter, as it were. The raw data is difficult to visualize since it's a very quantized value.

That seems incredible to me that there's such a strong periodicity to my perceived sleep quality. Any ideas why? 

And in case someone thinks the window is the problem, here's the day with weekly averages plotted.

IPython Extension of the Week: ipy_alerts

Have you ever had a IPython cell that was taking a **long** time to finish? Maybe you were importing data or training some model? If you've ever wished you could get an alert when the process finished, here's the extension for you: ipy_alerts from Trent Hauck at Zulilly.

The cell magic will send an email once the cell has finished running. Login credentials can be provided in the cell or via a config file.

IPython Extension of the Week: IPyCache

The IPython "extension of the week" is the IPyChache extention from Cyrille Rossant (rossant). The extension defines a cell magic (%%cache) that will cache the results of a cell into a persistent pickel file. When the cell is re-run the cached values of the variables are loaded instead of having them recomputed.

Here's an example notebook showing the operation.

The extension also has options for forcing the load of variables or forcing the rewrite of the cache.

Go check out IPyCache and let me know how it works! Thanks rossant!

Yann LeCun on Protection Against Privacy Invasion from AI

The best protections against privacy invasion through abusive data mining (whether it uses AI or mundane ML) are strong democratic institutions, an independent judiciary, and low levels of corruption. - Yann LeCun 

Auto-start IPython Notebook Server on Amazon EC2 Instance

Creating an IPython Notebook server on Amazon EC2 is a relatively straightforward process. I simply followed the instructions here. The only catch was that each time I created a new instance I had to SSH into the machine and manually start up the notebook server. That's irritating. I'd rather have them start automatically.

Added the following to my rc.local file:

ipython notebook --profile=nbserver > /tmp/ipynb.out 2>&1

But, when I rebooted the machine and looked at the log file it said, "/etc/rc.local: ipython: not found". Hmmm.

The following command in the rc.local file does work. The problem is that the rc.local file runs as 'root', not as 'ubuntu' where I installed Anaconda and set up the notebook server.

su ubuntu -c 'ipython notebook --profile=nbserver > /tmp/ipynb.out 2>&1'

That also appears to not work. I think this is because the bash.rc file is not loading. This is where Anaconda is assigned to the user's path.

This didn't work either:

su ubuntu -c 'export PATH=/home/ubuntu/anaconda/bin:$PATH'

su ubuntu -c 'ipython notebook --profile=nbserver > /tmp/ipynb.out 2>&1 &'

I'm not sure why that didn't work as two steps.

Here's the final solution. 
It involves two steps.

1) Add the following line to your rc.local file:

su ubuntu -c 'bash /home/ubuntu/startup_ipynb.sh'

2) Create a script called "startup_ipynb.sh" that contains these lines:

export PATH=/home/ubuntu/anaconda/bin:$PATH

ipython notebook --notebook-dir=\writeable\path\ --profile=nbserver > /tmp/ipynb.out 2>&1 &

This will run those two lines as the 'ubuntu' user and start up the notebook server.