Robert Thompson

Customizing Emacs

There’s an entire world of emacs out there that I have yet to explore. Before leaving for Amhurst I took the first step into that world and oh is it wonderful. I’m going to run through a few run of the mill additions to our ~/.emacs which are fairly essential.

(setq backup-by-copying t)

allow for editing files in Transmit or other FTP programs

(add-to-list 'auto-mode-alist '("\\.h\\'" . c-mode))
(add-to-list 'auto-mode-alist '("\\.pyx\\'" . python-mode))
(add-to-list 'auto-mode-alist '("\\.sm\\'" . fortran-mode))

add additional unrecognized file types that will allow for syntax highlighting

(unless window-system
  (xterm-mouse-mode 1)
  (global-set-key [mouse-4] '(lambda ()
                               (interactive)
                               (scroll-down 1)))
  (global-set-key [mouse-5] '(lambda ()
                               (interactive)
                               (scroll-up 1))))

enable mouse interaction in terminal mode

(require 'package)
(add-to-list 'package-archives '("marmalade" . "http://marmalade-repo.org/packages/"))
(package-initialize)

Here we add a package repository. This allows for a package management system very similar to Debian’s apt-get (only for Emacs v24+). We can access these packages via open Emacs after adding this line, then hitting M-x (M stands for meta, default key is ESC), and typing ‘package-list-packages’.

Search the list for ‘jedi’, put your cursor next to it and hit ‘i’ for install, you should see a big I come up next to it. Now hit ‘x’ to execute and it will ask if you would like to install jedi - the obvious answer is ‘yes’. Unfortunately we’re not finished, we have to add some commands to our ~/.emacs and setup a python virtual environment:

(autoload 'jedi:setup "jedi" nil t)
(add-hook 'python-mode-hook 'auto-complete-mode)
(add-hook 'python-mode-hook 'jedi:setup)
(setq jedi:setup-keys t)
(setq jedi:complete-on-dot t)

Now we can proceed to the virtual environment, which can be a bit tricky. Navigate to your jedi.el directory, which is most likely located in ~/.emacs.d/elpa/jedi-xxx. In this directory you will find a Makefile. If you are using any python distribution other than CANOPY, then you can probably just run ‘make requirements’ and be set to go. If you are using CANOPY however, a small change must be made to the makefile since virtualenv is not compatible with 64bit CANOPY on osx. Edit the makefile and change the following line:

virtualenv --python=$(PYTHON) --> venv -s python_virtual

Once this is done run ‘make requirements’ and it should download and compile everything else you need for jedi. Open up a new python document and enjoy the auto complete! ref

Next we can add a syntax highlighter called Flycheck. To install in emacs we M-x package-list-packages and search for flycheck, then follow a similar installation procedure above for the initial install of jedi. Next install pylint and pyflakes to your python install via pip (really not sure which one does the trick, but I believe it is pylint):

pip install pylint
pip install pyflakes

Lastly add this line to your ~/.emacs:

(add-hook 'after-init-hook #'global-flycheck-mode)

Finally we can install autopair/smartpar, which automatically closes parentheses, brackets, etc. Follow the routine above for adding packages, and install ‘autopair’. Then add this to your ~/.emacs and you should be set to go:

(require 'autopair)
(autopair-global-mode)
OR
(smartparens-global-mode t)

refs: 1 2 3 4

EDIT 11/7/13:

I forgot to mention color themes! One of the default Emacs24 themes that is pretty decent is ‘deeper-blue’ which can be enabled by modifying your custom-set-variables by:

(custom-set-variables
 ;; custom-set-variables was added by Custom.
 ;; If you edit it by hand, you could mess it up, so be careful.
 ;; Your init file should contain only one such instance.
 ;; If there is more than one, they won't work right.
 '(custom-enabled-themes (quote (deeper-blue)))
 '(inhibit-startup-screen t))

the ‘(custom-enabled-themes (quote (deeper-blue))) does it. BUT, I’m starting to prefer another theme called solarized. The installation is as simple as adding it via marmalade - ‘M-x package-list-packages’, then search for ‘color-theme-solarized’ and install it. Then in your ~/.emacs add the line:

(load-theme 'solarized-[light|dark] t)

and you should be set to go.

what exactly is in a python object?

When working with other people’s packages tit can sometimes be frustrating to not know exactly what types of things are in the objects you are creating. Especially when the documentation is lacking and you have to dig through the source to find the answers you seek. Luckily in python 2.7+ (I think) there is a way to list methods and attributes of an object. Let’s take a look at one of my galaxy objects - dataSet.galaxies[0]. We can see what lies inside if we tack a __dict__ on the end like so:

In [1]: dataSet.galaxies[0].__dict__
Out[1]:
{'ALPHA': 1.2212573740203678,
 'BETA': -0.9541605597579632,
 'FMR': 82.891999999999996,
 'FMR_cv': 1.9885869336675259,
 'HMR': 44.951360000000001,
 'HMR_cv': 1.9472879450163203,
 'L': array([ -1.81083747e+11,   1.20597990e+11,   4.39587033e+10]),
 'RS_halo_d': 0,
 'RS_halo_gmass': 0,
 'RS_halo_index': -1,
 'RS_halo_isparent': 0,
 'RS_halo_mass': 0,
 'RS_halo_smass': 0,
 'Z': 0.0,
 'central': -1,
 'cm': array([ 10345.936     ,   9542.06628571,   9012.98285714]),
 'gas_mass': 108890025.46897629,
 'glist': [29820L,....],
 'index': 0,
 'max_cv': 2.2245953691479929,
 'max_vphi': 113.64515965485172,
 'sfr': 0.0,
 'slist': [],
 'stellar_mass': 0.0}

Now if you have a ton of information/attributes attached to this object, it can get pretty messy quickly. If we tack the keys() function on the end it cleans things up quite a bit

In [2]: dataSet.galaxies[0].__dict__.keys()
Out[2]:
['RS_halo_gmass',
'slist',
'cm',
'max_cv',
'RS_halo_d',
'index',
'RS_halo_mass',
'HMR',
'FMR_cv',
'HMR_cv',
'stellar_mass',
'sfr',
'FMR',
'L',
'max_vphi',
'BETA',
'glist',
'Z',
'RS_halo_isparent',
'central',
'RS_halo_smass',
'gas_mass',
'ALPHA',
'RS_halo_index']

Now don’t ask me why it’s ordered the way it is; the point is you now know every attribute contained in the object.

But what about methods? Well apparently there is an equivalent function called dir(object) that pretty much does the same thing as .__dirs__.keys()

In [3]: dir(dataSets[0])
Out[3]:
['L_STARS',
'O0',
'Ol',
'RS_dmlist',
'RS_glist',
'RS_haloIndexes',
'RS_haloSeparations',
'RS_halos',
'RS_slist',
'SORAD_PRESENT',
'TIPSY_L_convert',
'TIPSY_M_convert',
'TIPSY_V_convert',
'__class__',
'__delattr__',
'__dict__',
'__doc__',
'__format__',
'__getattribute__',
'__hash__',
'__init__',
'__module__',
'__new__',
'__reduce__',
'__reduce_ex__',
'__repr__',
'__setattr__',
'__sizeof__',
'__str__',
'__subclasshook__',
'__weakref__',
'boxsize',
'boxsize_h',
'deleteParticleData',
'deleteParticleIDs',
'flagAge',
'flagCooling',
'flagFB',
'flagFH2',
'flagMetals',
'flagSFR',
'galaxies',
'getParticleData',
'getParticleIDs',
'gne',
'grho',
'gsigma',
'gu',
'h',
'loadData',
'n_bndry',
'n_bulge',
'n_disk',
'n_dm',
'n_gas',
'n_star',
'num_gals',
'process_galaxies',
'process_halos',
'rconvert',
'read_galaxies',
'read_rockstar',
'read_skid',
'read_sorad',
'redshift',
'saveData',
'skid_dmlist',
'skid_glist',
'skid_slist',
'snap',
'snapdir',
'snapnum',
'time',
'vfactor']

but this time we can see the methods such as read_sorad() and read_rockstar(). I’m not sure if there is a way to isolate the methods yet but regardless either command is quite helpful when dealing with another’s code.

compiling OpenMPI & HDF5 with Fortran support

Fortran…why on Earth would I want something with Fortran support? Apparently it’s fast; and when something is fast people still use it. I’m currently looking into the radiative transfer code Hyperion and it requires both MPI and HDF5 to be compiled with fortran support. It’s fairly simple to do, but can be a pain in the neck to find the syntax if you don’t know exactly where to look. Let’s start with OpenMPI which is currently on version 1.6.5.

?> ./configure --prefix=/Users/bob/local/openmpi-1.6.5 F77=gfortran FC=gfortran
?> make
?> make install

For HDF5 the syntax is similar:

?> ./configure CC=mpicc --enable-fortran --enable-hl --prefix=/Users/bob/local/hdf5-1.8.11 FC=gfortran
?> make
?> make install

And that should do it. To get hyperion to see these you just add the /bin dirs to your $PATH and the lib dirs to your $LD_LIBRARY_PATH. To check if the MPI-fortran bindings worked you can type mpif90 and mpi77 at the terminal and gfortran should error out saying that there are no input files.

Lastly once you recompile OpenMPI you may have to reconfigure/recompile/reinstall mpi4py if you’re using it. This is also super simple, but on OSX it can be troublesome. Before the compilation you have to redirect it to the proper SDK directory; the install should look like so:

?> export SDKROOT=/
?> python setup.py build
?> python setup.py install

I usually end up with an error at the end, but it installs and seems to run without any problems.

Listing files in a directory via python

I’ve come across a few situations where batch processing of snapshots is necessary. Normally I manually key in the number of snaps in the for loop, but is there a better more python way? Of course there is! Enter ‘glob‘. Glob is basically a wrapper for os.listdir, which is equivalent to typing ‘ls -U’ in terminal. If you use the -U however, the ordering is completely arbitrary, so by using say:

import glob
snaps = glob.glob("snapshot\_\*")

You’re going to get an unsorted back. For some this may not be a problem, but for me? I usually need a list ordered 000-N! The solution for this comes in the form of the the python function sorted():

import glob
import os
snaps = sorted(glob.glob("snapshot\_\*"), key=os.path.relpath)

which returns a sorted list! The key (pun intended) here is the key keyword, which tells it which os path to sort by. Many options exist such as ‘os.path.getmtime’ to sort by timestamp, and ‘os.path.getsize’ to sort by size. Thanks to stackoverflow again for the assistance!

check executable library dependencies

When compiling a C code it often requires certain libraries be present. To check which libraries the executable depends on there is a handy linux and OSX command that can show you where the executable thinks these libraries reside:

 ## LINUX ##
 ldd executable\_name

## OSX ##
 otool -L executable\_name

very handy when working on remote machines where things might be a little nutty.

setup an iPython notebook

Let’s setup an iPython notebook shall we? First we need to check what version of iPython we have, this is as easy as typing this in your terminal:

$> ipython --version

as of right now, the latest version is 1.0.0, if you have anything less than that you may want to consider upgrading. The easiest way to do this is via pip.

PIP INSTALLATION

Type pip in terminal and see if you have it, I’ll wait…ok you don’t? Well follow the link and download it, then to install you un-tar the file and cd into the directory, then in terminal:

$> python setup.py build
$> python setup.py install

that second command may require sudo depending on your python installation.

IPYTHON UPGRADE!

Now that you have pip installed you have something very similar to ubuntu’s apt-get or fedora’s yum, except for python! Now we can upgrade iPython via pip:

$> pip install ipython --upgrade

Create a new iPython profile

Next we create a new iPython profile:

$> ipython profile create username

where you replace “username” with a profile name of your choice. Then we edit the file ~/.ipython/profile_username/ipython_notebook_config.py in your editor of choice; We’re concerned with two lines in particular:

c.IPKernelApp.pylab = 'inline'
c.FileNotebookManager.notebook\_dir = u'/Users/bob/Dropbox/research/python/ipyNotebooks'

These two lines are commented out by default. The first says to put matplotlib figures in the notebook rather than open a new window, and the second dictates where the notebooks are saved by default. I like to set this second option to somewhere in my dropbox so that I can automatically share those between my multiple machines.

Launch!

Finally we create an alias to launch the notebook as it makes things much easier. Edit your ~/.bash_profile (mac) or ~/.bashrc (*nix) and add:

alias notebook='ipython notebook --profile=username'

Once this is done close all of your terminals and open a fresh one. Type ‘notebook’ and whala, your browser will open to your very own iPython notebook. I typically minimize this terminal window and keep it open for extended periods of time so the notebook stays alive.

There are numerous other settings you can dink around with in ~/.ipython/profile_username/ipython_notebook_config.py that I didn’t cover today, so feel free mess around in there. Lastly there are some additional security options to setup a password and such for your notebook that were also not covered, more details about this and credit for the bulk of these instructions comes from here.

timing python functions

One normally would use the magic %timeit function() to time how fast a python function is. For my purposes, I need to test nested functions that aren’t simple to call on their own; to do this I usually just print out the start and end time, but I frequently forget the syntax.

import time
start_time = time.strftime("%H:%M:%S",time.localtime())
end_time = time.strftime("%H:%M:%S",time.localtime())
print(r'START: %s END: %s' % (start_time, end_time))

A little manual subtraction is in order…and I’m sure there’s a simpler way to have python do this, but it suites my needs at this point.

multiple legends

It is certainly useful at times to break your legend up into multiple parts, but doing so in python (matplotlib) can be a pain in the neck. Luckily there’s a quick and fairly simple solution!

First thing you need to do, is actually name your plotted lines like so:

 fig=figure()
 ax1=fig.add_subplot(111)

l1 = ax1.plot(x, y, 'bo', label='ONE')
 l2 = ax1.plot(i, j, 'rx', label='TWO')

we now separate these lines and their labels into different variables

legendlines1 = l1
legendlabels1 = [l.get_label() for l in legendlines1]
legendlines2 = l2
legendlabels2 = [l.get_label() for l in legendlines2]

at this point we can’t just plot a legend, then another legend. The second legend always takes precedence and essentially erases the first. So in order to get around this, we name the first legend, and render the second:

legend1 = ax1.legend(legendlines1,legendlabels1,loc=1)
ax1.legend(legendlines2,legendlabels2,loc=3)

if you run the script up until this point, you’ll only have the second legend showing in location 3. In order to get the second legend we add the object via gca():

gca().add_artist(legend1)

and that should do it!

full script below:

 from pylab import *
 from numpy import *

x=linspace(0,5,20)
 y=-x

fig=figure()
 ax1=fig.add_subplot(111)

l1=ax1.plot(x,y,label='ONE')
 l2=ax1.plot(y,x,label='TWO')

legendlines1 = l1
 legendlabels1 = [l.get_label() for l in legendlines1]
 legendlines2 = l2
 legendlabels2 = [l.get_label() for l in legendlines2]

legend1=ax1.legend(legendlines1,legendlabels1,loc=1)
ax1.legend(legendlines2,legendlabels2,loc=3)

gca().add_artist(legend1)

show()

frameless legend()

I keep forgetting how to turn the bounding box OFF for matplotlib’s legend(). It quite simple but I can’t stop forgetting =(

legend(frameon=False)

colorbar for a multipanel plot

I was having some issues tonight getting a colorbar to function properly. If my script simply looked like so:

fig=figure(figsize=(12,6))
p1=fig.add_subplot(121)
column_render(datasets[0],PICKER,logData=LOG,
minIn=datasets[COLOR_SCALE].MIN, maxIn=datasets[COLOR_SCALE].MAX)
axis([XL,XH,YL,YH])
title(r'VSPH')
p2=fig.add_subplot(122)
column_render(datasets[1],PICKER,logData=LOG,
minIn=datasets[COLOR_SCALE].MIN, maxIn=datasets[COLOR_SCALE].MAX)
p2.set_yticklabels([])
axis([XL,XH,YL,YH])
title(r'DISPH')
subplots_adjust(wspace=0,hspace=0,right=0.85,left=0.07)
cb = colorbar()
cb.set_label(r'%s' % datasets[COLOR_SCALE].LABEL,fontsize=18)
show()

I got something that looked like this:

which is obviously not what I was looking for. Apparently I need to add an axis where I will place the colorbar. Tips from stackoverflow and checking out python’s website itself led me to the proper conclusion:

fig=figure(figsize=(12,6))
p1=fig.add_subplot(121)
column_render(datasets[0],PICKER,logData=LOG,
minIn=datasets[COLOR_SCALE].MIN, maxIn=datasets[COLOR_SCALE].MAX)
axis([XL,XH,YL,YH])
title(r'VSPH')
p2=fig.add_subplot(122)
column_render(datasets[1],PICKER,logData=LOG,
minIn=datasets[COLOR_SCALE].MIN, maxIn=datasets[COLOR_SCALE].MAX)
p2.set_yticklabels([])
axis([XL,XH,YL,YH])
title(r'DISPH')
subplots_adjust(wspace=0,hspace=0,right=0.85,left=0.07)
cax = fig.add_axes([0.86, 0.1, 0.03, 0.8])
cb = colorbar(cax=cax)
cb.set_label(r'%s' % datasets[COLOR_SCALE].LABEL,fontsize=18)
show()

by adding an axis for the colorbar to live on lets me customize its location, width, and height. The parameters in add_axis([left,bottom,width,height]) as in it’s drawing a rectangle. With this slight change the plot now looks like this: