Workshop & IDL extensions

Cluster Workshop

This week we had a maxBCG cluster workshop in AA.
It was very productive from my point of view. We clarified
the science we want to do, defined tasks that need to
be completed, and actually did some work. I got to some
of the sub-sampling that had been on the back burner while
I worked on systematics. These were pretty enlightening,
including a feature seen that an L200 luminosity split in a tNgals
2Mpc bin resulted in very weird profiles as you might expect,
while L200 in an N200 bin was much cleaner. At this point
I'm really liking N200 measures and I think the work that
Eduardo and Morad are both doing to improve this will be
fruitful.

I also did some X-ray clusters and, although the sample is
small and S/N not terrible exciting, it points to some interesting
future work, such as going back to the RASS and looking for
lower flux sources like we did back in the 42 clusters paper.

IDL extensions

I've been getting tired of the crappy file reading
support IDL has, and the slowness of fits and its
inability to properly deal with 64-bit integers. A while
back I wrote a suite of routines to read and write structures
to a file. These handle all the basic data types of IDL
and can write/read in either the native byte order
or another. This makes reading at least 3 times faster
than fits on an little endian machine.

But what is really needed is the ability to extract
individual rows and columns for both memory
and speed reasons. You can't do this efficiently
with the build in routines. I wrote binary_read and ascii_read
C routines, linked to IDL with the DLM mechanism, which
allow efficient reading of data from these types of files into
structures. You can extract just the data you need fast.

Also polished off ascii_write which writes a structure to an
ascii file. Many times faster than looping and doing printf statements.
The idlstruct routines can all work without these, but if these are
compiled it makes them much more powerful.

I realized as I was finishing off binary_read that this is
approaching the ability of databases. I thought about how
I could modify those simple Goddard database routines, which
just work with the most basic read/write abilities of IDL,
to make an extremely efficient native IDL database system.
Don't know if it's worth it since I have an interface to postgres,
but it would have the advantage of not requiring another
program. It would never have all the features. And, besides,
I'm supposed to be migrating to Python!

Anyway, the sdss idl (aka sdssidl, umsdss_idl) code, C/C++
and documentation are here:

http://cheops1.uchicago.edu/idlhelp/sdssidl/umich_idl.html

Erin

Intrinsic alignments & Lots 'o talks

Intrinsic Alignments

Finished a measurement of intrinsic alignments
in clusters. Used Berlind's spectroscopic clusters
sample cross correlated with the main sample. I'm
getting a fairly weak bound (no detection), but this is
the first such measurement in clusters that I know of.

This shear needs to be converted to "delta sigma" as
measured for the clusters and I have done this.

Talks

Lots of talks here this week. Two 1.5 hour talks by
Leonard Susskind and a talk by Roger Penrose, plus
a job talk by Andrew Zentner and Alice Shapley is
talking tomorrow.

Susskind and Penrose both talked about things we
don't understand. Susskind discussed, for example,
how everyone loves the eternal inflation, bubble
nucleation, pocket universe theory, but at this point
its just a neat idea with no conceptual or mathematical
foundation. He tried to show that there are indications
that there might exist such a foundation.

Penrose talked about what happened before the big bang.
All the math breaks down there both classically and
the best quantum efforts, so you have to come up with
something new. His talk centered around the concept
that, at a nearly infinite time from now when lambda has
accelerated everything away and if all particles have decayed
away and black holes have evaporated, and somehow you
could get rid of electrons too, then there would be only
photons in the universe. In that case time has no meaning
at all since photons don't experience times. You can't build
clocks. This is an interesting idea. But then he made a huge
leap and said that if suddenly the phase space is rescaled so that
it appears to be a very dense universe, then it looks
like the big bang and everything starts over again.

Erin

Space-time curvature

Today I had an interesting discussion with Andrei
Gruzinov about space-time curvature. I
see clearly now that most of the un-intuitive aspects of
GR (even black holes) have the same source as the
un-intuitive aspects of special relativity. The fact that light
needs no medium for propagation leads to SR, and there
is little conceptually new in GR other than the equivalence
principle. The difference between SR and GR comes from the
fact that accelerations due to sources (such as gravity, EM)
have spatial gradients and this manifests itself as curvature
when you write down the covariant formulation.

There are some aspects of GR that seem really new, for
example the expansion of space. Andrei pointed
out that for an infinite universe you can always reformulate
the problem as particles in space given initial velocities flying
apart; nothing new there. But I don't think a closed universe
with positive curvature can be reformulated in that way
because it is not simply connected, so eventually the particles
will mix and this will produce observational differences.
The thing is, GR is a local theory, it doesn't tell us anything
about the global properties of the universe. We may
need something new altogether to address whether, for
example, the universe can have non-trivial topology.

Erin

More tests

I decided to run some of these tests on our
cluster sample and I definitely see a difference
with the better resolved sources. Now I want
to see how few sources I can cut and still get
the improvement. More tomorrow.

Numpy

Today there was a thread on the Numpy
discussion list about the fee based
documentation. People there made some
good points about this. This guy is actually
writing a book here, and its not like I haven't
bought books before; I own the Perl book
for example and used it all the time when I
was writing a lot of Perl code.

On the other hand, what is really needed for this numpy
thing is really just a list of all the available
commands as much as anything. This I can
do from the source perhaps, and I might actually
do that once they get all the good stuff from numarray
ported over. We'll see.

Erin

Trends in the signal and Python disappointment

Trends

I'm seeing trends of the signal with the resolution of
the source galaxies, the distance (in arcseconds) from
the source to the lens. I see no trends with the seeing,
the redshift of the source, the redshift of the lens, or
deblending (at least the blended flag).

The resolution thing is odd because earlier tests showed
no dependence on it. It may be correlated with the galaxy
density, but in that case there is another variable besides
deblending involved and I don't have a guess yet as to
what it is.

Python

I've been really getting into numerical python and thinking
it is our way to get out from under the IDL yoke. I've
been disappointed, however, with some trends I have seen
among the developers. The first thing I saw is that the
only documentation for Numpy is available only if you
fork over some cash. This seems outrageous considering
one of the whole points of developing this stuff is to get
away from fees like this with Matlab and IDL. That said,
the guy who is charging for this stuff is also one of the
most helpful contributors to the mailing list so it may
all work out in the end. I just think it is a bad example
to set.

The second hit came when reading up on the wonderful
PyTables stuff that is being developed. PyTables is a
python+C library to access the HDF5 api. Most of this is
free, but I found out that the most powerful extensions,
which allow efficient complex searches of the tables, are only going
to be available in a commercial product unless someone else
develops something similar. This crap is happening because
PyTables is released under a BSD license. This license
is not compatible with the GPL (also copyleft) and so is not free
software as I see it. BTW, can someone explain how the Python
license is compatible with GPL yet someone can then license
their Python code under BSD?

I have already put all my personal code under the
GPL which is probably the only thing I will ever license under.
I also will be discussing with my collaborators if the sdssidl
code can also be distributed under GPL.