Port failed to install: adam-macbook ~$ sudo port install py25-matplotlib---> Building py25-hashlib with target build---> Staging py25-hashlib into destroot---> Installing py25-hashlib 2.5.2_0---> Activating py25-hashlib 2.5.2_0---> Cleaning py25-hashlib---> Fetching py25-zlib---> Verifying checksum(s) for py25-zlib---> Extracting py25-zlib---> Configuring py25-zlib---> Building py25-zlib with target build---> Staging py25-zlib into destroot---> Installing py25-zlib 2.5.2_0---> Activating py25-zlib 2.5.2_0---> Cleaning py25-zlib---> Fetching py25-setuptools---> Attempting to fetch setuptools-0.6c8.tar.gz from http://cheeseshop.python.org/packages/source/s/setuptools/---> Verifying checksum(s) for py25-setuptools---> Extracting py25-setuptools---> Applying patches to py25-setuptools---> Configuring py25-setuptools---> Building py25-setuptools with target build---> Staging py25-setuptools into destroot---> Installing py25-setuptools 0.6c8_2---> Activating py25-setuptools 0.6c8_2---> Cleaning py25-setuptools---> Fetching py25-tz---> Attempting to fetch pytz-2007k.tar.bz2 from http://cheeseshop.python.org/packages/source/p/pytz/---> Verifying checksum(s) for py25-tz---> Extracting py25-tz---> Configuring py25-tz---> Building py25-tz with target build---> Staging py25-tz into destroot---> Installing py25-tz 2007k_0---> Activating py25-tz 2007k_0---> Cleaning py25-tz---> Fetching py25-dateutil---> Attempting to fetch python-dateutil-1.4.1.tar.gz from http://labix.org/download/python-dateutil---> Verifying checksum(s) for py25-dateutil---> Extracting py25-dateutil---> Configuring py25-dateutil---> Building py25-dateutil with target build---> Staging py25-dateutil into destroot---> Installing py25-dateutil 1.4.1_0---> Activating py25-dateutil 1.4.1_0---> Cleaning py25-dateutil---> Fetching fftw-3---> Attempting to fetch fftw-3.1.3.tar.gz from http://www.fftw.org/---> Verifying checksum(s) for fftw-3---> Extracting fftw-3---> Configuring fftw-3---> Building fftw-3 with target all---> Staging fftw-3 into destroot---> Installing fftw-3 3.1.3_0---> Activating fftw-3 3.1.3_0---> Cleaning fftw-3---> Fetching py25-nose---> Attempting to fetch nose-0.10.4.tar.gz from http://somethingaboutorange.com/mrl/projects/nose/---> Verifying checksum(s) for py25-nose---> Extracting py25-nose---> Configuring py25-nose---> Building py25-nose with target build---> Staging py25-nose into destroot---> Installing py25-nose 0.10.4_0---> Activating py25-nose 0.10.4_0---> Cleaning py25-nose---> Fetching py25-numpy---> Attempting to fetch numpy-1.2.0.tar.gz from http://downloads.sourceforge.net/numpy---> Verifying checksum(s) for py25-numpy---> Extracting py25-numpy---> Applying patches to py25-numpy---> Configuring py25-numpy---> Building py25-numpy with target buildError: Target org.macports.build returned: shell command " cd "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_python_py25-numpy/work/numpy-1.2.0" && /opt/local/bin/python2.5 setup.py build " returned error 1Command output: setup_package() File "setup.py", line 89, in setup_package configuration=configuration ) File "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_python_py25-numpy/work/numpy-1.2.0/numpy/distutils/core.py", line 184, in setup return old_setup(**new_attr) File "/opt/local/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/distutils/core.py", line 151, in setup dist.run_commands() File "/opt/local/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/distutils/dist.py", line 974, in run_commands self.run_command(cmd) File "/opt/local/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/distutils/dist.py", line 994, in run_command cmd_obj.run() File "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_python_py25-numpy/work/numpy-1.2.0/numpy/distutils/command/build.py", line 37, in run old_build.run(self) File "/opt/local/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/distutils/command/build.py", line 112, in run self.run_command(cmd_name) File "/opt/local/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/distutils/cmd.py", line 333, in run_command self.distribution.run_command(command) File "/opt/local/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/distutils/dist.py", line 994, in run_command cmd_obj.run() File "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_python_py25-numpy/work/numpy-1.2.0/numpy/distutils/command/build_src.py", line 130, in run self.build_sources() File "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_python_py25-numpy/work/numpy-1.2.0/numpy/distutils/command/build_src.py", line 147, in build_sources self.build_extension_sources(ext) File "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_python_py25-numpy/work/numpy-1.2.0/numpy/distutils/command/build_src.py", line 250, in build_extension_sources sources = self.generate_sources(sources, ext) File "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_python_py25-numpy/work/numpy-1.2.0/numpy/distutils/command/build_src.py", line 307, in generate_sources source = func(extension, build_dir) File "numpy/core/setup.py", line 87, in generate_config_h raise SystemError,"Failed to test configuration. "\SystemError: Failed to test configuration. See previous error messages for more information.Error: The following dependencies failed to build: py25-numpy py25-tkinter tcl tkError: Status 1 encountered during processing. I don't want to deal with it so I'm just using Fink.
PSDs, cross-correlation...
scipy is capable of doing fft-base cross-correlation, convolution, etc., but it requires the stsci package, which is not generally easy to install. For that matter, scipy can be a pain some of the time. So agpy now includes a 2D cross-correlation code and a power spectrum / power spectral density code. These are pure-numpy codes that should be easy to use without any other bothersome dependencies. EDIT: I have them check for scipy (which can cause crashes if you have a bad scipy install, e.g. 32 bit executables on a 64 bit system) because scipy uses FFTW and numpy appears not to. Also, this code & related stuff has been discussed on astrobetyter agpy correlate2d psds
pyspeckit: an astronomical spectroscopic toolkit
Jordan and I have been working on our python-based spectroscopic analysis tool for a while now: pyspeckit is a pretty awesome, now functional but incomplete (and incompletely documented) tool.
Python 64-bit on Mac OS X 10.6 Snow Leopard
After yesterday's disastrous attempt to install various python packages, I started from scratch. First, I got rid of all of my python frameworks (backed up but removed from the path). Then, I compiled python 2.7 from scratch:
I got some help from http://blog.mahmoudimus.com/2009/12/python-2-6-4-and-twisted-9-on-os-x-10-6-snow-leopard/ ./configure --enable-framework --enable-universalsdk=/Developer/SDKs/MacOSX10.6.sdk MACOSX_DEPLOYMENT_TARGET=10.6 --with-universal-archs=intel -with-readline-dir=/usr/localmake -j 17make -j 17 test make results:
Python build finished, but the necessary bits to build these modules were not found: _bsddb dl gdbm imageop linuxaudiodev ossaudiodev spwd sunaudiodev To find the necessary bits, look in setup.py in detect_modules() for the module's name.I'm not concerned about these - I don't use any of them and I assume I need to install some other packages to get them to work. During make test, I had two failures that resulted in "python crash" pop-up boxes:
test_subprocess. this bit of output is from a test of stdout in a different process .... this bit of output is from a test of stdout in a different process ... test_sunaudiodev
Then, I got some malloc errors:
test_ioTesting large file ops skipped on darwin.It requires 2147483648 bytes and a long time.Use 'regrtest.py -u largefile test_io' to run it.Testing large file ops skipped on darwin.It requires 2147483648 bytes and a long time.Use 'regrtest.py -u largefile test_io' to run it.python.exe(22914,0x7fff70d3ebe0) malloc: *** mmap(size=9223372036854775808) failed (error code=12)*** error: can't allocate region*** set a breakpoint in malloc_error_break to debugpython.exe(22914,0x7fff70d3ebe0) malloc: *** mmap(size=9223372036854775808) failed (error code=12)*** error: can't allocate region*** set a breakpoint in malloc_error_break to debugpython.exe(22914,0x7fff70d3ebe0) malloc: *** mmap(size=9223372036854775808) failed (error code=12)*** error: can't allocate region*** set a breakpoint in malloc_error_break to debugtest_ioctl
Python magic / advanced numpy indexing
Yeah, indexing python arrays should really be easy. Stefan van der Walt's page In [85]: bi = (f.bolo_indices[np.newaxis,:] + zeros([7751,1])).astype('int')In [86]: whc = (whscan[:,np.newaxis] + zeros([1,107])).astype('int')In [87]: array2d[whc,bi] = temp2d
Python: one-line arrays
Ahhh, refreshing: `` whscan = asarray([arange(scanlen)+i for i in scans_info[:,0]]).ravel()`` Not like IDL, which takes at least 4 lines b/c of the variable declaration. There's probably a better way to do that too.
Python: setting matplotlib defaults
Setting matplotlib defaults is a lot more difficult than it should be. matplotlib.defaultParams['image.origin']='lower'matplotlib.defaultParams['image.interpolation']='nearest'
Rebuilding python from scratch again
I got scipy working a week or two ago, but doing so killed matplotib's tkagg. So, I switched to the MacOSX backend, which worked ok until I realized that the interactive (connect) features of macosx failed miserably. This led me to try to get matplotlib working.... which broke with those awful "symbol not found" errors in ft2font.so and _path.so, which I've determined all have to do with linking to the wrong library files. The most worrisome part of this process was discovering that a full Time Machine recovery of /usr and /Library/Frameworks and /Library/Python did *not* restore python - it stayed dead with IDENTICAL errors. So there are probably additional layers of hidden links. The process below is based on hyperjeff's blog post but differs substantially based on Sam Skillman's recommendations and the very big issue I ran into that my /usr/local files appeared to be corrupted. After this install, my path no longer includes /usr/local/bin and /sw has been moved to /_sw... hopefully one of these days I'll be ballsy enough to delete it.
- Install python 2.6.4
- Needed a clean terminal with no flags set at all. Don't know why - all I had set were a bunch of -arch x86_64 flags. export LD_LIBRARY_PATH="/usr/local/lib:/usr/X11/lib"./configure --enable-framework=/Library/Frameworks MACOSX_DEPLOYMENT_TARGET=10.6 make -j 17sudo make install
- Reset PYTHONPATH to blank
- `` alias clearflags='export CFLAGS=""; export CCFLAGS=""; export CXXFLAGS=""; export LDFLAGS=""; export FFLAGS="";'`` to make sure
- Install FFTW cd ~/tmpcurl -O http://www.fftw.org/fftw-3.2.2.tar.gztar xf fftw-3.2.2.tar.gzcd fftw-3.2.2clearflags./configure CC="gcc -arch x86_64" CXX="g++ -arch x86_64" CPP="gcc -E" CXXCPP="g++ -E"make -j 17sudo make install
- Install UMFPACK cd ~/tmpcurl -O http://www.cise.ufl.edu/research/sparse/umfpack/current/UMFPACK.tar.gzcurl -O http://www.cise.ufl.edu/research/sparse/UFconfig/current/UFconfig.tar.gzcurl -O http://www.cise.ufl.edu/research/sparse/amd/current/AMD.tar.gztar xf AMD.tar.gztar xf UFconfig.tar.gztar xf UMFPACK.tar.gzsed -ibck 's/F77 = f77/F77 = gfortran/' UFconfig/UFconfig.mk sed -ibck '299,303s/# //' UFconfig/UFconfig.mkcp UFconfig/UFconfig.h AMD/Include/cp UFconfig/UFconfig.h UMFPACK/Include/cd UMFPACKmake -j 17make hbmake clean
- Install numpy
- Set environment variables export MACOSX_DEPLOYMENT_TARGET=10.6export CFLAGS="-arch x86_64"export FFLAGS="-m64"export LDFLAGS="-Wall -undefined dynamic_lookup -bundle -arch x86_64"export PYTHONPATH="/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/"echo "[amd]library_dirs = /Users/adam/tmp/AMD/Libinclude_dirs = /Users/adam/tmp/AMD/Includeamd_libs = amd[umfpack]library_dirs = /Users/adam/tmp/UMFPACK/Libinclude_dirs = /Users/adam/tmp/UMFPACK/Includeumfpack_libs = umfpack" > site.cfg
- Setup & Install python setup.py build --fcompiler=gnu95sudo python setup.py install
- Test: python -c "import numpy"
- Install scipy. The important thing is to use g++-4.2 because g++-4.5 doesn't accept the -arch flag. Also, get rid of /sw if it's on your computer at all. sudo mv /System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/numpy /System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/numpyXcd ~/repos/scipy-0.7.1python setup.py cleanrm -rf buildclearflagsFFLAGS="-m64" CFLAGS="-arch x86_64 -I/usr/local/include/freetype2 -I/usr/X11/include -L/usr/X11/lib" LDFLAGS="-Wall -undefined dynamic_lookup -bundle -lpng -arch x86_64" CXX="/usr/bin/g++-4.2" CC="/usr/bin/gcc-4.2" python setup.py buildpython setup.py install Test the install: python -c "import scipy, scipy.fftpack, scipy.interpolate"
- Install matplotlib. MAKE SURE /usr/bin/texbin is in front of /usr/local/bin and /sw/bin so that dvipng comes from MacTEX. I also ended up having to remove /usr/local/bin from my path completely sudo mv /System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/numpy /System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/numpyXcd ~/repos/matplotlib-svnpython setup.py cleanrm -rf buildclearflags
- Do hyperjeff's recommended edits except don't use /usr/local because it's f'd: make.osx: MACOSX_DEPLOYMENT_TARGET=10.6PREFIX=/usrPYTHON=/Library/Frameworks/Python.framework/Versions/Current/bin/python## You shouldn't need to configure past this point (and yet…)PKG_CONFIG_PATH="${PREFIX}/lib/pkgconfig"CFLAGS="-arch i386 -arch x86_64 -I${PREFIX}/include -I${PREFIX}/include/freetype2 -isysroot /Developer/SDKs/MacOSX10.6.sdk"LDFLAGS="-arch i386 -arch x86_64 -L${PREFIX}/lib -syslibroot,/Developer/SDKs/MacOSX10.6.sdk"FFLAGS="-arch i386 -arch x86_64" setup.cfg: wxagg = False
- Do the install (different from hyperjeff b/c I don't want root) `` sudo make -f make.osx fetch deps make -f mpl_build mpl_installpython setup.py install``
- Install setuptools
- easy_install ipython
- install everything else pythonically
- Install python 2.6.4
Reinstalling a system
Python died tonight when I foolishly tried to update numpy or matplotlib. One was not compatible with the other in upgraded form, and the update for matplotlib wouldn't install because of GTK issues that were totally opaque. So, I reinstalled EVERYTHING - fink AND macports - from scratch. If I don't get things running by morning I'm going to have to do a restore, which is ugly as death. Things I need to do post-install: `` sudo port install python_select sudo python_select python25`` test ipython test matplotlib test numpy
SAVE / RESTORE in Python
Save/Restore is probably the single best feature of IDL that, sadly, is very poorly replicated in Python. For 1 or 2 dimensional variables, you can use Pylab's save/load, but I never use such piddling tiny arrays. For higher dimensional objects, either using FITS files (a pain because of header definitions) or pickling ought to work. e.g.: import numpyimport picklex=ones([10,10,10,10],dtype='float64')pickle.dump(x,open('x.pysav','w'))X = pickle.load(open('x.pysav','r'))