Erin Sheldon's Research Blog

version 1.5.0 of sdssidl released

2009-07-22T17:08:00.003-05:00

I haven't posted in a while. Lots has happened. I think I'll start slow:

Amy Kimball and I just released v1.5.0 of sdssidl. You can get it here:

http://code.google.com/p/sdssidl/

Here is a look at the changes:

v1.5.0
---------------------------------------
bug fixes:
 gcc 4.3 has turned many warnings into errors.  Hundreds of fixes were
  applied to fixe these mostly harmless problems.
 skiplines works properly now for ascii_read() C++ DLM.
   pro/util/match_dup.pro: Bug fix; code checks for overlapping ranges of
       input arrays.
   pro/astro/spheredist.pro: Bug fix; output fixzed so values are between 0
   and 2!pi.
 pro/sdss/gc2csurvey: misnamed procedure internally.  Incorrect docs.

New Files:
   pro/util/in.pro: Check if a value is contained in an array or matches
       a scalar.
   pro/util/tostring.pro:  Convert idl variables to strings.
   pro/util/fromstring.pro:  Convert from idl strings to variables.
 pro/plotting/plegend.pro:  A fork from legend.pro that supports color
  strings, no default box, box color, etc.

minor changes:
 pro/files/which.pro:
  1: Search for Dynamically Loadable module if no files are found.
     If /dlm is set search onlyl for those.
  2: /show: display the first match found.
  3: Now use file_search, to avoid a bug in idl 7.0.
 pro/idlstruct_files/idlstruct2fits.pro: support ascii
 pro/plotting/pplot.pro: support color strings.
 pro/plotting/textoidl.pro:  Added \sun and made roman font !3
 pro/plotting/plot_box.pro: support color strings.
 pro/plotting/ploth.pro:  support asinh scaling with tvasinh.pro
 pro/htm/htm_match.pro:  Fixed incorrect docs
 pro/fits/mrdfits_multi.pro:  Allow specify extension.
 pro/util/doc_method.pro: Use file_which
 pro/sdss/sdss_objname.pro: Use prefix=,suffix=
 pro/sdss/sdss_flags__define.pro: flagname, plotting
 pro/sdss/sdss_files__define.pro: sensible dealing with reruns.  Also
  allow nyu/boss ordering of rerun
 pro/sdss/make_cmodel_counts.pro: removed preselection with flags
 pro/probgal/bayes_combine_gri.pro:  Use sdss_flag_select

Using atlas and blas on mac OS X

2008-12-05T11:47:00.008-05:00

This post might seem kind of specific, but I've decided its better to blog about what I'm actually doing than not blog at all. I spend most of my time on technical things in order to get to the higher level physics ideas. And this information is pretty useful.

A simple way to get atlas/blas working on mac OS X using the libraries that come with XCode is to link the headers to the "usual" place under /usr. This way you don't have to keep remembering where the header file is, and its buried deep.

sudo ln -s /System/Library/Frameworks/Accelerate.framework/Versions/Current/Frameworks/vecLib.framework/Versions/Current/Headers/cblas.h /usr/include/cblas.h

And remember when you link to get the order right: -lcblas -latlas

Erin

DES weak lensing pipeline

2008-09-29T16:20:00.005-05:00

I'm working on the lensing pipeline for estimating the shear from galaxy shapes in DES. We've already got a piece in place for measuring shapelet expansions of PSF stars. Now we need to finish a few other things. It breaks down like this currently:

Mike has a code working for measuring the pre-PSF shear from a stack of images. The inputs are the positions of objects in the images and shapelets representations of the PSF in each image.
I've got the following data-management related pieces in place
- Can automatically locate all coadd and single epoch (SE) images on the local machine given only a tilename. This involves interaction with the database and all the pieces of metadata must be in place, so this was actually a non-trivial test of the system.
- Can do the forward and reverse coordinate transformations image <-> sky based on the WCS info in the header including the distortion model. This is key considering our plan of action: for the shear measurement we will take the list of coadd objects and go back to the original SE images that were used in the coadd creation. So we need to take the position and bounding box from the coadd, convert to sky coordinates, and then convert back to image coordinates in the individual images. Due to the polynomial distortion model of scamp I implemented a simple root solver for this. Slow but accurate.

We still need to implement the following:

PSF interpolation scheme. An optimal way to interpolate the PSF info. This will be a Jarvis&Jain style scheme that will improve over time.
Figure out how to extract the proper PSF information and package it up in a way that is usable.
Connect my data management tools, WCS transformation tools, and Mike's shear code into something that can automatically process a DES coadd tile, extract shear info, and get that into the database. At first this will do something kind of fake for the PSF, like taking that of the closest star but eventually will use the full PSF interpolation scheme.

Erin

Brookhaven job

2008-04-25T11:11:00.002-05:00

I have accepted a position at Brookhaven National Lab. I posted about it in my other blog

M/L modeling; papers accepted

2008-04-23T09:45:00.002-05:00

M/L Modeling

I'm creating some new measurements that make it simpler for Jeremy Tinker to model our mass-to-light ratio (M/L) results. To create his models he uses Zehavi's correlation functions for r-band selected samples. In my M/L paper I used i-band selected galaxies. It turns out the only way this modeling was going to get done is if I repeat my measurements for r-band limited galaxies. It took a week of running on a cluster of computers here at NYU and I now have the basic light measurements done. This was much faster than the previous measurements because I only measured the light correlation functions to 2Mpc. Now I need to do the analysis, which I hope to get to after my trip to Brookhaven this week.

Papers Accepted

Paper I, the cluster lensing measurement paper, and Paper III, the M/L paper were both accepted to ApJ this week. I've spent the last few days working with ApJ to create downloadable data tables that will accompany the papers, and accompanying "preview" tables to go in the papers. I also plan to make the data available online with links placed with the arXiv abstract.

Cluster Centers and Mass Scatter

2008-02-27T09:34:00.007-05:00

Centers

One of the uncertainties in understanding the lensing results for the MaxBCG cluster sample is the center of mass. We know that some of the time our chosen center, the location of the brightest cluster galaxy, is not at the mass peak or the center of mass. On the other hand, the theoretical predictions for dark matter mass profiles are predicted around the mass peak.

We have put together two alternative centers, both based on finding a peak in smoothed density maps. One is in the number density and the other is in the luminosity weighted number density. It looks like we can flag bad centers when neither of these new centers agrees with the BCG location. Evidence comes from the X--ray concentration and also from lensing. I compared the mass profiles for "good" and "bad" centers at fixed cluster luminosity and found significant differences in the profiles. We still haven't figured out what is the best center, but we can now flag bad ones with some level of confidence.

Mass Scatter

The variance in the mass measurements we get from lensing are the convolution of many different sources of scatter. There is the scatter in mass at fixed richness, which is what we are interested in. But on top of this is the scatter from measurement noise sigma_err and the intrinsic variance among galaxy shapes for the background sources sigma_SN. These two sources of scatter are well known. But even on top of that is the variance from unknown sources of systematic error. We can guess at what these are, such as variance in the signal calibration as a function of observing conditions.

I looked at the inferred scatter sigma_tot^2 - sigma_err^2 - sigma_SN^2. I found that the residual scatter is much larger than the expected sigma_m^2 based on other measurements, such as X--ray properties and velocity measurements.

Next I want to figure out how to estimate other sources of variance. The goal is to design a measurement that does not also include the variance in mass of the lenses.

Lensing papers submitted to ApJ

2007-09-20T11:14:00.000-05:00

All three lensing papers have been submitted to ApJ. We now have eight papers out based on the MaxBCG cluster catalogs: the catalog papers (I,II), a cosmology analysis based on the number counts and selection function (III), a velocity dispersion paper (IV), a stacked X-ray luminosity paper (V) and three lensing papers (VI, VII, VIII). Soon, a ninth paper by Sarah Hansen on galaxy populations in the MaxBCG clusters will appear.

Lensing papers on astro-ph

2007-09-17T17:42:00.000-05:00

The lensing papers were posted on astro-ph last week. Here are the links:


  Paper I:   Measurements        astro-ph 0709.1153
  Paper II:  Modeling/Inversions astro-ph 0709.1159
  Paper III: M/L                 astro-ph 0709.1162

There are still questions about calibration at the 10% level. We did the best we could with the data we had, I know that. But over the year since the initial measurements were made there have undoubtedly been improvements in our understanding of the photozs. But this is part of science; we have to publish sometime and we will incrementally improve our understanding as we go. Because this uncertainty comes in as a calibration, an overall amplitude, we can always re-calibrate the results in the future.

Erin

Collaborations

2007-08-29T17:56:00.002-05:00

I've learned something extremely important about collaborations. It is something that people have often tried to tell me but I didn't have the experience to understand fully. Please note this is not a knock on any of my collaborators but simply a general observation:


You must set hard deadlines for the completion of papers.

I think at first this statement can seem either obvious or irrelevant to many people, but I'll explain why it is certainly not irrelevant.

First of all, it doesn't matter how good your work is if you don't communicate that work to the outside world efficiently. You won't be able to continue your good work because you won't be supported. And if you are in a collaboration your work depends on others, and you must consider others in order to produce efficiently.

This has been really hard for me to accept, because internally my motivation is to follow my interests, and that is it. This is true for most scientists I think, especially after they finish graduate school.

But when you join a collaboration most people are interested in lots projects but they themselves are the lead on at most one or two of those projects. But they have excellent ideas concerning all of the projects. This is where the fundamental problem arises: how do you get people to contribute work to your project on your timescale?

What usually happens is you think of a project and start working on it. You tell your collaborators about it. You think of all the most important issues right away and deal with them. Then you start thinking of not so obvious issues and work on them. Then the work starts to feel like it's ready to share with the larger community. You must do this in order to be supported. You write it up and send it off to collaborators. For the first time, your collaborators start to really think about the project, and they have many good ideas. Most of them you already thought of but there are some ideas you have yet to address. Addressing these new issues often requires work from you, but it often requires a fair amount of work from the individual collaborator because only they fully understand it. So you end up delaying publication for a while, sometimes a long while.

If only everyone involved knew from the beginning that the paper would be released on an exact day, with no possible delay, they would have been forced to think about the project from it's inception in order to contribute. They would have realized that some input was needed from them up front. They would have done this work while you were working. Most of these issues would have been dealt with by the time the first draft was written.

Collaborating is worth while because I learn so much by seeing other people's perspectives. And everyone has such varied skills that the work is always better for it. The theory is that a deadline is an impersonal rule that once established should help to keep the ideas and work flowing more naturally and efficiently.

Erin

M/L paper really close

2007-08-26T18:53:00.000-05:00

I know I said it was close before, but unless something comes up it should be out in a couple of weeks. If things move fast it could come out with papers I and II, which should come out this week.

For those who want a preview: the M/L goes as M^0.33 within r200, the M/L reaches an asymptotic value on large scales (20Mpc) and when you multiply that number by the luminosity density you get 0.20 +/- 0.02. There is a factor of bias on that too, but we don't know what it is exactly other than it should be of order unity.

Erin

Brian May: Observing at La Palma?

2007-07-24T11:29:00.000-05:00

I just found out that Brian May of Queen is finishing up his PhD in astrophysics this year. His thesis is on zodiacal dust and in fact he is observing at La Palma tonight (and last night)! I know this isn't my research, but its a good story.

Here are some entries from his blog.

Erin

M/L paper close

2007-06-19T01:28:00.000-05:00

I have finished all the analysis and plots for the M/L paper. I just need to finish the writing. I think this paper might be ready to put an the archive at the same time as the other two MaxBCG lensing papers.

Erin

Anti-correlations

2007-06-17T11:48:00.000-05:00

Made the discovery this week in aspen that certain types of galaxies are actually anti-correlated with MaxBCG clusters on certain scales. These are very luminous galaxies, and the cross-correlations are negative near the virial radius. Considering that more luminous galaxies tend to me more strongly clustered this was a bit of a surprise.

My guess is that this is due to the cluster finder. The cluster finder looks for groups of luminous red galaxies, ranks them by likelihood, and "percolates", meaning that smaller clusters are not allowed to live within r200 of larger clusters. Then if you choose a certain ngals200 this might cause a feature near r200 in the luminous galaxies. I'm checking simulations. I see that the feature is there for MaxBCG run on the sim, although the effect is weaker. Now I'm going to run on the halos and see if the feature persists.

Erin

Agreement

2007-05-31T14:17:00.000-05:00

All the mass estimates for the clusters are now in basic agreement. The photoz fix brought up the lensing masses and they now agree with the dynamical and other masses. What I like is that we didn't try to make them agree, but bugs were found along the way that changed the answer; we would have fixed the bugs even if they lowered the masses.

Analysis on the ICL stuff is beginning in earnest now; Tim is looking at the outputs and I expect he will generate some science soon. If I get a chance I'll also look into it. I'm interested in getting the total optical light in the system for comparison with the total mass from lensing, both in the relative amount and relative distribution within the clusters.

Draft finished

2007-05-10T22:31:00.000-05:00

Finished the near-final draft of the lensing paper. Now it's time to turn attention to running simulated data through the galaxy cross-correlation code for comparison with the clusters. I don't know yet what we will learn from this, but it is certain to be useful. I'm hoping to use the new computing cluster of our west coast friend which will kick some serious butt.

Cluster lensing finished

2007-05-08T16:43:00.000-05:00

The final lensing measurements for the maxbcg clusters are finished. If there are any other issues that arise I will not address them in this paper. This is non-negotiable. For example the photozs are now fixed; other photoz methods may give different results. This just comes from choosing a method. I would not be surprised if we re-calibrated these results in the future as our knowledge of the source redshift distribution improves, but it won't happen before publication of these results.

The results after fixing the photoz bug have higher amplitude by about 15%.

I've finished making the new plots, now I'm re-working the text. This will take a little work because I need to write up how I have used our estimates of the photoz error distribution in the measurement itself rather than as a correction after the fact. This is the same formalism we used in the 2004 paper but now we actually have a good idea what the error distributions are.

Dave Johnston and I are still planning to release this paper and his analysis paper at the same time. I want this to happen before June.

Simulations for Intracluster Light

2007-04-12T10:25:00.000-05:00

I worked on simulating the ICL for our pipeline. This is just creating fake sersic profiles ala the measurements of Gonzales et al., so it should be straightforward. I found, however, that as with many things in astronomy I spent all my time just trying to figure out the units and conventions. There is the usual -2.5log crap for the surface brightness limits, but also the notation for Sersic profiles makes no sense. I wasted a lot of time thinking r_e really was the effective radius when people actually mean the half light radius.

In figuring all this out, I wrote pure IDL code for converting r0->r50 (roots of the incomplete gamma function) and making images of Sersic profiles. The images don't properly integrate over the pixels on small scales for large n, small r_e (in pixels) but they are fine for these purposes. I faked it by putting in a core.

P.S.
28 mags/arcsec^2 = 6.3e-3 nmgy/arcsec^2

P.P.S.
mags/arcsec^2 = worst unit ever

Photozs

2007-04-01T12:47:00.000-05:00

Rachel and a student at Princeton are working on a lensing project using the MaxBCG clusters. We did some comparisons and saw considerable disagreement between the recovered signals. It turns out that during the last iteration of the photozs a bug was introduced that made the colors look really wrong and the photozs too high. This was not in their estimator, but a difference in magnitude systems between training set and my catalog. It was just a confusion of conventions, but of course those can have disturbing consequences

By the way, does anyone read this blog besides my mom?

Erin

Finally 30Mpc + python cgi

2007-03-06T23:39:00.000-05:00

The cluster-light cross correlations to 30Mpc finally finished. Now I have mass-to-light ratios over the whole range. I started looking at them and I'm seeing that in some cluster richness bins we may not be seeing the M/L asymptote. This is in the lower richness bins. At first this didn't make sense but maybe it is because the mis-centering is worse in those cases. It will be interesting to compare what we get when we try to model in these offsets.

Mulin and I also set up python CGI on the web server. This will be useful because writing these scripts in python is much easier and quicker than using php or perl. The server is old and decrepit, so we'll have to repeat all this when it is replaced in the next couple of weeks.

read_objmask

2007-02-27T21:43:00.000-05:00

I wrote a code to read objmask info for SDSS objects from the fpatlas files. This came down to modifying the atlas reading code to get multiple objects and copy their objmask info into an output structure. I then linked this to IDL using the DLM mechanism.

I was motivated to do this because we are going to start working on intracluster light in the MaxBCG and Percolation clusters. This gives a first look at the area associated with each object in a frame, which we will need to mask out in order to measure just the ICL. We will probably use a different area once we decide on the best masking algorithm.

Erin

30 Mpc M/L

2007-02-21T23:39:00.000-05:00

Today I started a bunch of jobs on the computubg cluster here "mafalda" here at NYU. These should take a few days to finish, and I'll run another set after that. These are the last of the jobs for calculating the light and number densities of galaxies around the MaxBCG clusters. I had these to 10Mpc but now I'll have them to 30Mpc along with the lensing masses, so I'll have M/L over that entire range. If things look good I'll start writing that paper in a week or so.

Erin

Change of plans

2007-02-20T15:21:00.000-05:00

Sarah Hansen and I are going to combine our papers on MaxBCG cluster-galaxy cross correlations with Sarah as first author. I have written up the method, described the selection etc., and made a bunch of plots. Risa didn't like that there was so much overlap between the two papers. I think she is probably right. Sarah is going to merge the sections from my paper into hers.

Erin

Switching to svn

2007-02-08T20:09:00.000-05:00

My CVS archive at UChicago is no longer available, so I talked to the sysadmin here at NYU and we set up an SVN repository for all the code. This is actually kind of important. SVN makes managing code across many different machines and many different developers simple, more simply than CVS. And I feel like I have a good shot at keeping it available for a long time here unlike at Chicago. It will be easier to move now if needed too.

The Aspen meeting is next week and the SDSS cluster folk are going to be there in nearly full force. The SZ cluster people will be there too so we may here some news from SZA.

Wanted Distractions

2007-02-02T00:18:00.000-05:00

Decided to put aside the cluster correlations and work on accessing a database through the web using Python. Over the break I put my Mom's recipes into a database web applicatioin using python. I looked into doing something like that with a larger data set, it is as straighforward as it sounds. The only trick will be convincing the sysadmin that it is secure enough.

There has been an exchange in the cluster group about how to model the lensing signal. It has become clear that mis-centering of cluster sample relative to the mass could be a real interpretational issue. Any mis-centering is basically degenerate with the halo concentration. Eduardo Rozo and Dave Johnston have ideas on how to constrain the mis-centering outside of the lensing signal to break the degeneracy.

I think this centering issue could severely limit the usefullness of the M/L measurements on halo scales. We can only account for the mis-centering with modeling, but we don't want to assume a form for the light distribution, certainly not NFW. One of the points is to see how different the profiles are in mass and light. On the other hand, if we can constain the mis-centering outside of these measurements the very high S/N light measurements could be recovered non-parametrically.

Erin

Lot's of counting

2007-01-29T17:43:00.000-05:00

I've been counting the galaxies around the MaxBCG clusters. I decided that while I was at it I might as well count the luminosity. And while I was at it, why just count the galaxies as a function of radius? Why not count them as a function of their other two basic observables, color and luminosity? So I now have this cube of counts as a function of radius, color (g-r) and luminosity, background subtracted and edge corrected. And the same for luminosity.

The original motivation for this was to measure mass-to-light ratios by comparing the excess luminosity to the excess mass from the lensing measurements. But I think this cube is going to yield a lot of science beyond that. I'm writing up the basic measurements and doing statistics like luminosity functions, blue fractions and such. Sarah Hansen is going to move along the lines of here 2005 paper with better S/N. Blanton and Hogg both have some good ideas which will either make it into my paper or separate papers.

Erin
P.S. I'm going to start posting regularly again. Seriously.

It's away!

2006-08-27T12:10:00.000-05:00

Posted the cluster lensing paper to ApJ on Friday.
Not much else to say, I've been working on this
project, on and off and amongst other things, for
a few years now. Some science is going to come
of this work. Within the group there is already
one paper in prep. and at least two or three others
that should be completed within the year.

Milestone

2006-07-31T19:37:00.000-05:00

I just posted the cluster lensing paper to the collaboration
for three weeks of scrutiny; then off to AJ.

I have this tendency to wait on things like this until they
feel ready. For some reason that takes longer for me than
other people, unless they just move on with unreadiness.
Even though nothing major may have changed in months,
I just keep working on them until they feel right. This
time the feeling coincided well with the timescale the
cluster group set for releasing papers. Deadlines can be
very useful, especially toward the end of a project.

I'm going to tweak some more on this draft but it is
basically done. I'm ready to move on. I'll probably
spend some time on DES stuff and begin working on
some interpretation of these results, probably on
M/L profiles.

Erin

Long time coming

2006-06-15T23:32:00.000-05:00

Is that fact that I've been doing a lot of research
a good excuse for neglecting my research blog?

I've been very busy. Ben Koester and I have been
going back and forth trying to make the maxBCG
cluster catalog all that it can be. As usual, 95%
of the work goes into the most difficult 5%. I think
I can say this: we understand things well enough
that we know that we can basically stop working
on it (for now). We also know what data is usable.

With the new and improved catalog I've been running
the lensing code and Morad has been measuring the
number density and luminosity profiles. Morad and
Sarah both plan to write Science papers from Morad's
work. In the short term, I am most interested in extracting
the r200-Ngals relationship from the luminosity profiles.
With this we have a good size estimate within which
we can can re-measure the cluster properties. This is an
update of Sarah's original measurement of r200-ngals for
the new algorithm. The new algorithm is significantly different,
at least in concept; we will see if the r200-ngals scaling
is different.

It will be interesting to compare this r200 to the r200 we
will get from the mass profiles. We have both the mass
and luminosity profiles, so we will get mass-to-light
ratios as a function of radius. Sarah and Morad are interested
in galaxy-formation and such topics and this should provide
an excellent starting point for that as well.

I have measured the lensing around very low
luminosity galaxies using a catalog from Michael Blanton.
This should be very interesting since there are few reliable
statistical mass measurements at these luminosities. Currently
Michael basically just has the number density argument. I have
also run Chris Miller's cluster catalog through. It is an
extremely well understood catalog from spectroscopic data.
The S/N isn't great because they are few in number and nearby
in space, making their lensing strength weak. A factor of
two improvement in number is on the way with the processing
of dr4.

Finally, I completed the set of shapelet decompositions for the
DES simulations. Now I hope to push to higher redshift by
looking at the GOODS fields. The statistics are poor, but this
will possibly give us a handle on morphology evolution.

Erin

Shapelet Catalogs & Clusters

2006-05-10T23:20:00.000-05:00

SDSS Shapelets

I've been working on shapelet decompositions for
all spectroscopic galaxies from SDSS DR4, in all
5 SDSS banspasses. The primary motivator is
realistic simulations for the Dark Energy Survey,
but there can be a lot of science in this catalog as well.
Basically you transform the image into a two
dimensional basis, sort of like a fourier transform.
From that you can define a reconstruction. We
are using the reconstructions to make realistic
simulated galaxies for the sim. There is even
a technique to perturb the reconstruction in a
way that doesn't just produce random junk. This
way the hundreds of millions of galaxies simulated won't be
just be copies of the hundred thousand I decomposed.

I will write this up after I'm done, even if it is just to
post on astro-ph. They are all running now and will
be finished in a few days.

Clusters

I've been looking a lot at the maxbcg cluster catalog.
It is an excellent catalog and is very well tested at the
high mass end. I've been identifying issues that
don't really affect the high mass end but do affect the
low mass end. This is important to me because there
is a lot of lensing signal in these low mass objects and
if we can construct a believable catalog there it will
be great for lensing and will in general allow a more
complete examination of structures larger than
individual galaxies. With SDSS we can really do this.
Berlind has done this with the spectroscopic sample at
low redshift and this is our chance to do a uniformly
selected sample to higher redshift. Because of the
extra volume this will also just have a lot more statistical
power in certain ways.

Workshop & IDL extensions

2006-04-28T22:52:00.000-05:00

Cluster Workshop

This week we had a maxBCG cluster workshop in AA.
It was very productive from my point of view. We clarified
the science we want to do, defined tasks that need to
be completed, and actually did some work. I got to some
of the sub-sampling that had been on the back burner while
I worked on systematics. These were pretty enlightening,
including a feature seen that an L200 luminosity split in a tNgals
2Mpc bin resulted in very weird profiles as you might expect,
while L200 in an N200 bin was much cleaner. At this point
I'm really liking N200 measures and I think the work that
Eduardo and Morad are both doing to improve this will be
fruitful.

I also did some X-ray clusters and, although the sample is
small and S/N not terrible exciting, it points to some interesting
future work, such as going back to the RASS and looking for
lower flux sources like we did back in the 42 clusters paper.

IDL extensions

I've been getting tired of the crappy file reading
support IDL has, and the slowness of fits and its
inability to properly deal with 64-bit integers. A while
back I wrote a suite of routines to read and write structures
to a file. These handle all the basic data types of IDL
and can write/read in either the native byte order
or another. This makes reading at least 3 times faster
than fits on an little endian machine.

But what is really needed is the ability to extract
individual rows and columns for both memory
and speed reasons. You can't do this efficiently
with the build in routines. I wrote binary_read and ascii_read
C routines, linked to IDL with the DLM mechanism, which
allow efficient reading of data from these types of files into
structures. You can extract just the data you need fast.

Also polished off ascii_write which writes a structure to an
ascii file. Many times faster than looping and doing printf statements.
The idlstruct routines can all work without these, but if these are
compiled it makes them much more powerful.

I realized as I was finishing off binary_read that this is
approaching the ability of databases. I thought about how
I could modify those simple Goddard database routines, which
just work with the most basic read/write abilities of IDL,
to make an extremely efficient native IDL database system.
Don't know if it's worth it since I have an interface to postgres,
but it would have the advantage of not requiring another
program. It would never have all the features. And, besides,
I'm supposed to be migrating to Python!

Anyway, the sdss idl (aka sdssidl, umsdss_idl) code, C/C++
and documentation are here:

http://cheops1.uchicago.edu/idlhelp/sdssidl/umich_idl.html

Erin

Intrinsic alignments & Lots 'o talks

2006-04-13T22:22:00.000-05:00

Intrinsic Alignments

Finished a measurement of intrinsic alignments
in clusters. Used Berlind's spectroscopic clusters
sample cross correlated with the main sample. I'm
getting a fairly weak bound (no detection), but this is
the first such measurement in clusters that I know of.

This shear needs to be converted to "delta sigma" as
measured for the clusters and I have done this.

Talks

Lots of talks here this week. Two 1.5 hour talks by
Leonard Susskind and a talk by Roger Penrose, plus
a job talk by Andrew Zentner and Alice Shapley is
talking tomorrow.

Susskind and Penrose both talked about things we
don't understand. Susskind discussed, for example,
how everyone loves the eternal inflation, bubble
nucleation, pocket universe theory, but at this point
its just a neat idea with no conceptual or mathematical
foundation. He tried to show that there are indications
that there might exist such a foundation.

Penrose talked about what happened before the big bang.
All the math breaks down there both classically and
the best quantum efforts, so you have to come up with
something new. His talk centered around the concept
that, at a nearly infinite time from now when lambda has
accelerated everything away and if all particles have decayed
away and black holes have evaporated, and somehow you
could get rid of electrons too, then there would be only
photons in the universe. In that case time has no meaning
at all since photons don't experience times. You can't build
clocks. This is an interesting idea. But then he made a huge
leap and said that if suddenly the phase space is rescaled so that
it appears to be a very dense universe, then it looks
like the big bang and everything starts over again.

Erin

Space-time curvature

2006-04-11T23:22:00.000-05:00

Today I had an interesting discussion with Andrei
Gruzinov about space-time curvature. I
see clearly now that most of the un-intuitive aspects of
GR (even black holes) have the same source as the
un-intuitive aspects of special relativity. The fact that light
needs no medium for propagation leads to SR, and there
is little conceptually new in GR other than the equivalence
principle. The difference between SR and GR comes from the
fact that accelerations due to sources (such as gravity, EM)
have spatial gradients and this manifests itself as curvature
when you write down the covariant formulation.

There are some aspects of GR that seem really new, for
example the expansion of space. Andrei pointed
out that for an infinite universe you can always reformulate
the problem as particles in space given initial velocities flying
apart; nothing new there. But I don't think a closed universe
with positive curvature can be reformulated in that way
because it is not simply connected, so eventually the particles
will mix and this will produce observational differences.
The thing is, GR is a local theory, it doesn't tell us anything
about the global properties of the universe. We may
need something new altogether to address whether, for
example, the universe can have non-trivial topology.

Erin

More tests

2006-04-04T23:53:00.000-05:00

I decided to run some of these tests on our
cluster sample and I definitely see a difference
with the better resolved sources. Now I want
to see how few sources I can cut and still get
the improvement. More tomorrow.

Numpy

Today there was a thread on the Numpy
discussion list about the fee based
documentation. People there made some
good points about this. This guy is actually
writing a book here, and its not like I haven't
bought books before; I own the Perl book
for example and used it all the time when I
was writing a lot of Perl code.

On the other hand, what is really needed for this numpy
thing is really just a list of all the available
commands as much as anything. This I can
do from the source perhaps, and I might actually
do that once they get all the good stuff from numarray
ported over. We'll see.

Erin

Trends in the signal and Python disappointment

2006-04-03T22:46:00.000-05:00

Trends

I'm seeing trends of the signal with the resolution of
the source galaxies, the distance (in arcseconds) from
the source to the lens. I see no trends with the seeing,
the redshift of the source, the redshift of the lens, or
deblending (at least the blended flag).

The resolution thing is odd because earlier tests showed
no dependence on it. It may be correlated with the galaxy
density, but in that case there is another variable besides
deblending involved and I don't have a guess yet as to
what it is.

Python

I've been really getting into numerical python and thinking
it is our way to get out from under the IDL yoke. I've
been disappointed, however, with some trends I have seen
among the developers. The first thing I saw is that the
only documentation for Numpy is available only if you
fork over some cash. This seems outrageous considering
one of the whole points of developing this stuff is to get
away from fees like this with Matlab and IDL. That said,
the guy who is charging for this stuff is also one of the
most helpful contributors to the mailing list so it may
all work out in the end. I just think it is a bad example
to set.

The second hit came when reading up on the wonderful
PyTables stuff that is being developed. PyTables is a
python+C library to access the HDF5 api. Most of this is
free, but I found out that the most powerful extensions,
which allow efficient complex searches of the tables, are only going
to be available in a commercial product unless someone else
develops something similar. This crap is happening because
PyTables is released under a BSD license. This license
is not compatible with the GPL (also copyleft) and so is not free
software as I see it. BTW, can someone explain how the Python
license is compatible with GPL yet someone can then license
their Python code under BSD?

I have already put all my personal code under the
GPL which is probably the only thing I will ever license under.
I also will be discussing with my collaborators if the sdssidl
code can also be distributed under GPL.

Latest systematics tests and the inversion paper

2006-03-31T00:38:00.000-05:00

Systematics

I ran a bunch of test, none of which
were conclusive, but I did see something
connected to the resolution parameter. This
isn't surprising. I saw a bigger effect
however when I just limited the smallest radius
in arcsec that sources would be used. This again
suggests deblending problems.

So I looked at the flags and in fact deblended
objects are included in the Princeton catalog,
when I thought they had been cut in what Rachel
gave me.

So now I'm remaking the postgres table to include
the flags so I can test this.

Dave's Inversion Paper

I read Dave's new draft. The content is excellent;
it holds much of what we have learned over the
last few years about weak lensing and stacking
in particular. I hope people read it because it
is sort of the Bible for stacking from the theory
point of view.

Erin

Wasted time

2006-03-29T23:27:00.000-05:00

I wasted a few hours trying to figure out why
I couldn't link C programs to IDL on the evolve
machine. At first I thought it was because it is
running IDL 6.2, but then I got it to work on
another machine with 6.2. Then I thought it must be because
it is a 64-bit machine, but then I got it to work
on a different 64-bit machine. Then I figure it
out: for some reason IDL is running in 32-bit mode
on that machine, so it can't work with 64-bit libraries.
This is not the default behavior, so it must be
some kind of configuration error. I've written
to the sysadmin to see what's up.

I did get something done today: I finished
creating the new source catalog tables in the
postgres database and I'm ready to do some
systematics tests tomorrow.

Erin

Searching the sphere + looking for systematics

2006-03-28T23:31:00.000-05:00

Searching

I hacked together a ra/dec searching code which uses the
htm (hierarchical triangular mesh) to match things on
the sphere. It is extremely fast. There is significant overhead
in finding the index for each input ra/dec position which is
a killer for small lists. For my work I always carry this
index around with all my objects, so I only generate it once.
But for a stand alone piece of code it is actually rather
annoying if not prohibitive. I don't know a workaround
for this yet.

Systematics

I've been trying to track down systematics and haven't had any
luck with the easy stuff as I have described below. So what
I have done is remake the catalogs with more info
and stuffed it into my postgres database so I can very quickly
select source samples with different cuts and run them through the
lensing code. I think this is a better approach than adding
lots more info to the input catalog read by the lensing code
and then doing cuts within because the memory gets prohibitive
for the cheopsen at Chicago (limited to 2Gb). I'm finishing this
up now so tomorrow I'll start running the tests.

The first one I want to try is cuts on how well the galaxies are
resolved. I have done this test before and it passed, but maybe
the small number statistics of the LRGs and the high Ngal bin
is more sensitive to certain regions of sky where there is bad
seeing for example. Note the masking will be a bit messed
up if a resolution cut removes all objects from a given region
of sky.

Erin

Parameter quoting == information reduction

2006-03-25T10:10:00.000-05:00

The discussion yesterday about the quoted
best fitting parameters brings up a larger issue. We
always try to reduce our results to a few numbers,
but that discussion illustrates that in fact doing so can
be a significant reduction in information. In fact, the
very act of creating a likelihood function does so
in a way that does not necessarily match intuition.

The function exp(chi^2) only follows
intuition if the data are Gaussian in the first place. What
does it mean if the resulting likelihood function is highly
non-Gaussian? Certainly the minimum chi^2 does follow
intuition; it is the model that is closest to the data given that
metric. But understanding the "error"
on that quantity using these techniques is more a matter of
definition than anything else; if you define the exp(chi^2) as
the probability of a given parameter, then you can draw random
values from that distribution and define your confidence regions
based on the range of parameters about the best fit that contain
some percentage of the random points. Fine, but the fact is
exp(chi^2) isn't even what we would normally define as a
probability except under certain conditions.

So why bother with all the error estimation using this function
if you end up with a skewed distribution like WMAP had with
the optical depth? I think it's fine if everyone looks at the
likelihoods and understands them. You are not really looking
at a likelihood; the breadth of that measure does indicate
something about how well constrained your model is, but
it is not clear how that translates into an intuitive feel of
confidence.

I think the only way you could really get a meaningful "confidence"
is to have N independent data sets and repeat the best fit and ask
about percentages. This tells you about the error on the independent
sets. People rarely do this because the error on these sets is
roughly sqrt(N) larger than the overall dataset. People prefer to use
bootstrap or jackknife techniques because it then artificially gives
you sqrt(N) better error estimates (I'm guilty too). Of course, if
everything is Gaussian, then in fact the error on the overall set
is sqrt(N) smaller.

Erin

3 hours of talks; still odd features in signal

2006-03-25T00:24:00.000-05:00

Two 1.5 hour talks by Spergel today. Learned that just
about all the "significant" differences we see in the new
papers come from choices in analysis. For example, they
quoted the mean in the first year paper but they quote
maximum likelihoods in this paper. Many of the
distributions were heavily scewed so it makes a big
difference. In fact, the ML value for the optical depth
in the first year results was 0.1 and it is in the new results
as well. But the mean changed a lot, from 0.17 to 0.1.
It's not clear why the made the change.

I tried to isolate the odd features in the signal, which
shows up in the LRGs and the high Ngal clusters. I split the
LRGs by redshift but the signal remains. I also tried a buffer
that the sources had to be zlens + zbuffer (0.1 and 0.2) but that
made absolutely no difference at all. I also ran with the other
catalog instead of the princeton but this made no difference
either (it never has). The catalog has no objects where were
deblended, but this could still be the problem and we just don't
have a simple indicator.

The feature is not physical, so I must figure out a way to
isolate it. I'm out of ideas at the moment.

Erin

Tidying up

2006-03-24T01:02:00.000-05:00

I finished generalizing all the code so that adding
new samples and sub samples takes only a few minutes.
Ran the jackknifing for sample 11, 20kpc - 10Mpc. Am
running sample12 (30 Mpc) through the lensing code, will
probably finish late tomorrow or early saturday.

I want to isolate the odd features I see in the profiles.
One thing I want to try is removing some of the interlopers
by putting a buffer zone in redshift. This is a bit harder
to model in the photoz bias corrections, but if it works I'll
put in the time to do it right.

Tomorrow I'll start some of the new randoms running nice 19 on
jet or evolve.

Erin

Distances in cosmology

2006-03-23T00:15:00.000-05:00

Distances

I implemented a bunch of routines to calculate cosmological
distances. I was motivated to do this 1) because I wanted to
generate a volume limited set of random redshifts, so I needed
a formula for the volume element and 2) I wanted some routines
that worked for more general cosmologies; I only had flat universe
code. Still don't have evolving dark energy code, but I'll write
that when I need it.

So I started from the bottom up and wrote an integration routine
to calculate the comoving distance, then all the steps needed to get
to the angular diameter distance (see Hogg astro-ph/9905116, which
mostly comes from Peebles 1993 book).

One thing I noted about that paper: I believe there is no
need to go to equation 19, which is not general. Because
you can calculate Dc, the comoving distance, between two different
redshifts, call it Dc_12, then you can implement the difference of two
DM's (comoving transverse distance) simply by using the Dc_12 in
formula 16. This will also work in any curvature and so is
general. I tested that it works for the cases where equation 19 holds
(curvature >= 0).

Now that I have this stuff going I'm generating 48 sets of 500,000
randoms from z=0.04 to 0.35. I'll set them running through the lensing
code in the background. I don't need these at the
moment since the randoms I have cover the redshift range for maxBCG,
but it will be good to have them for the future.

Corrected Lensing Profiles

Yesterday I finished to code to match the histograms of redshift, so
today I was able to do the random point corrections to the maxBCG
lensing profiles. These look OK, but there are some worrying features
in the profiles. The features are similar to what I saw in the LRG profiles.
Tomorrow I will do the jackknifing so we can do some more meaningful
analysis. I will also do sub-sampling in the "paper" versions of the
catalog. These are the clusters which will be published in Ben's catalog
paper. Essentially it is an Ngals cut and a redshift cut.

Erin

Better idea for matching randoms

2006-03-21T17:54:00.000-05:00

A better idea for the random points (see last entry):

Take the bin in redshift for which the histogram of the lenses
is highest relative to the randoms. For that bin we will keep
100% of the randoms. Normalize the ratio of lens to random histograms
to 1 at this bin. Then the value of the ratio in the other bins is the
fraction of randoms you will keep from that bin. This maximizes the
number of randoms you will use and reproduces the histogram.

I'm matching to randoms as I write this.

Erin

First post: random points

2006-03-20T23:31:00.000-05:00

This is my first post on my research blog, so let's get to it:

Hogg convinced me that I don't need to create new random points for
each sample I do. My thought was that each should
be statistically independent, which they won't be if I re-use random
points. But because the randoms are so well-determined now (20
million points) the statistical fluctuations in their determination
are no longer important anyway. This also allows me to just
set randoms running in the background with redshifts drawn
randomly from a "volume limited" distribution, and then use
them when I need them.

The trick is getting the redshifts correct for the randoms when I use
them with a specific lens sample. What I plan to do is just draw from
the randoms until the histogram of random redshifts is N times
the histogram of lens redshifts. This will preserve the rough shape
of the redshift distribution but not use exactly the same redshifts.

I'm also running a new cluster lens sample right now which should be
done by mid-day tomorrow; more on that when the time comes.

Erin