Robert Thompson

compiling OpenMPI & HDF5 with Fortran support

Fortran…why on Earth would I want something with Fortran support? Apparently it’s fast; and when something is fast people still use it. I’m currently looking into the radiative transfer code Hyperion and it requires both MPI and HDF5 to be compiled with fortran support. It’s fairly simple to do, but can be a pain in the neck to find the syntax if you don’t know exactly where to look. Let’s start with OpenMPI which is currently on version 1.6.5.

?> ./configure --prefix=/Users/bob/local/openmpi-1.6.5 F77=gfortran FC=gfortran
?> make
?> make install

For HDF5 the syntax is similar:

?> ./configure CC=mpicc --enable-fortran --enable-hl --prefix=/Users/bob/local/hdf5-1.8.11 FC=gfortran
?> make
?> make install

And that should do it. To get hyperion to see these you just add the /bin dirs to your $PATH and the lib dirs to your $LD_LIBRARY_PATH. To check if the MPI-fortran bindings worked you can type mpif90 and mpi77 at the terminal and gfortran should error out saying that there are no input files.

Lastly once you recompile OpenMPI you may have to reconfigure/recompile/reinstall mpi4py if you’re using it. This is also super simple, but on OSX it can be troublesome. Before the compilation you have to redirect it to the proper SDK directory; the install should look like so:

?> export SDKROOT=/
?> python setup.py build
?> python setup.py install

I usually end up with an error at the end, but it installs and seems to run without any problems.

h5py to replace npz?!

Apparently .npz files have a limit of ~4GB, which can be a problem when dealing with large datasets ($$600^3$$), so how do I write binary files of this size?! Looks like I need to migrate to HDF5…lol; here’s an example for read/write similar to npz:

import h5py
a = np.random.rand(1000,1000)
f = h5py.File('/tmp/myfile.hdf5')
f['a'] = a # <-- Save
f.close()

now that will write the file, lets open it

import h5py
f = h5py.File('/tmp/myfile.hdf5')
b = f['a']
#b = b[()]
f.keys() #shows all values in the dataset
f.close()

but the odd thing is, you can’t directly access the data here by just ‘b’. Instead there is a ton of information attached like b.shape, b.dtype, etc etc, all found here. Instead, you simply have to use the command

b.value

and that will output the data of the file.