I cut down the number of maps made by the pipeline from 80 to 30 by merging regions with severe overlap. I don't think this will result in any net change in efficiency of the mapping process because a few of the maps will go over to swap, but it will reduce the amount of overlap / noisy edges and make debugging simpler. It should also reduce hard drive usage once I decide that I'm satisfied with the results and remove the old (v0.6) redundant data. As part of this process, the reference fields for coalignment have been re-set to the v0.6.2 coadds + some (crappy) mosaics from v0.6.2 so that we no longer need to rely on 3x1's for the alignment. See /data/bgps/releases/v0.7/readme.txt for details (reproduced below) Created 01/14/09 Version 0.7 does not exist yet. This is the 'release notes' indicating what will change -Overlapping redundant fields will be eliminated Mapping of field name to area covered, followed by number of observations included: gemob1: l189-l192 11 - 2 = 9 w5: l135-l138 46 = 46 l133: l133-l134 34 - 1 = 33 l111: l110-l111 88 = 88 l086: l085-l086 17 = 17 l082: l080-l084 33 = 33 l077: l075-l080 43 - 1 = 42 l072: l070-l074 10 - 2 = 8 l065: l059-l069 19 = 19 l055: l053-l058 10 - 1 = 9 l050: l048-l052 19 = 19 l045: l043-l047 20 - 4 = 16 l040: l038-l042 33 - 4 = 27 l035: l034-l037 53 = 53 l032: l031-l033 61 = 61 l030: l030 61 - 3 = 58 l029: l028-l029 53 - 4 = 49 l024: l021-l027 29 = 29 l018: l015-l021 29 - 3 = 26 l012: l011-l014 7 = 7 l009: l008-l010 11 = 11 l006: l005-l007 15 - 1 = 14 l003: l002-l004 25 - 1 = 24 l001: l001 20 = 20 l000: l000 23 - 1 = 23 l359: l359 14 = 14 l357: l356-l358 9 - 1 = 8 l354: l353-l355 3 = 3 l351: l350-l352 9 = 9
Gem OB1
With Miranda's help, I discovered some serious errors in Gem OB1 and fixed them.
- L189 was using L134 data for no apparent reason
- many September 2007 observations have the WRONG rotation angle in place in array_params! Unfortunately it's not clear yet which ones suffer from this problem: I need a list of when rotangle was/was not used, and it looks like it'll be a pain to fix the problem.
- GemOB1link wasn't mapped, it will be now.
Pointing model failure
I attempted to map L111, L033, and W3/4/5 with the pointing model corrections applied. That was a failure. Two images are attached to illustrate the problem - relative pointing is clearly not correct (middle of L33, bottom left of L111). So.... possibilities:
- Pointing models are incorrect
- Pointing model signs have been misinterpreted (don't think this one is possible based on other graphs)
- Uhh... ideas?
Next step, I'll use the individual maps with no ptg mdl and find out what the offsets should be....
Deconvolution vs. Not
I've done some by-eye comparisons of deconvolutions vs no deconvolution. The no-deconvolution clearly does better on the bright sources: probably the deconvolved beam size is a little bit different from (larger? smaller?) the actual source size. Deconvolution does better in putting noise from noisy regions into the noisemap and keeping it out of the astromap. Both might be useful - the deconvolved maps may end up being prettier, but the no-deconvolve maps will probably have more reliable fluxes. This all probably relies on testing / simulation.
Version 0.6 begun
I started on (what I hope will be) v0.6 tonight. New things: -Sigma-rejection flagging in the spatial domain -Inverse-variance weighting
Median drizzling
http://groups.google.com/group/comp.lang.idl-pvwave/browse_thread/thread/762770933591238d Someone going by the name 'wox' introduced me to a spectacularly simple drizzling algorithm for medianing. It will be a little less efficient because it can't take advantage of some 'partial sum' tricks that the average can, but it will median over ALL data points, which is an advantage.
New to-do list, updated things....
do_maptests.pro is running a bunch of different mapping parameters (pca components, deconvolution, etc.) on l000, l002, l003, l033, l083. We'll then run bolocat on it and look for the following: -number of sources found -flux in sources found as a function of n_pca -size of sources as function of n_pca -BADNESS, e.g. blurring / unremoved atmosphere / oversubtracted sources bolocat2reg makes a region file out of a bolocat catalog. I'd like to make a separate ds9 region file that includes the actual pointing error + the centroiding error rather than just the elliptical fit to a given source; that will be more useful for finder charts. My current goal is to get a nice set of images I can combine to release as a poster to Jason Glenn's student who is making pretty posters for publicity purposes; probably to promote CCAT. Umm.... I blame the late hour for the alliteration. That goal also means I've been using an IRAF task to do some mosaicing (easier than writing the IDL code AGAIN): mscstack l001_5pca_map09.fits,l002_5pca_map09.fits,l000_5pca_map09.fits,l359_5pca_map09.fits,l003_5pca_map09.fits GCCOMBINE_5pca.fits lthresh=-1l001_13pca_map09_scuba_aligned.fits,l002_13pca_map09_scuba_aligned.fits,l000_13pca_map09_scuba_aligned.fits,l359_13pca_map09_scuba_aligned.fits,l003_13pca_map09_scuba_aligned.fits GCCOMBINE.fits lthresh=-1mscstack l029_13pca_map09_scuba_aligned.fits,l030_13pca_map09_scuba_aligned.fits,l031_13pca_map09_scuba_aligned.fits,l032_13pca_map09_scuba_aligned.fits,l033_13pca_map09_scuba_aligned.fits,l034_13pca_map09_scuba_aligned.fits L33COMBINE.fits lthresh=-1 hsig=5 lsig=5
outer galaxy fields done
they're small files logs/log_101008_coalign.log:/scratch/adam_work/l059/l059_13pca took 286.08181 sec.logs/log_101008_coalign.log:/scratch/adam_work/l062/l062_13pca took 390.88015 sec.logs/log_101008_coalign.log:/scratch/adam_work/l065/l065_13pca took 296.88441 sec.logs/log_101008_coalign.log:/scratch/adam_work/l068/l068_13pca took 1136.2623 sec.logs/log_101008_coalign.log:/scratch/adam_work/l071/l071_13pca took 899.41245 sec.logs/log_101208_coalign.log:/scratch/adam_work/l189p5/l189p5_13pca took 375.06840 sec.logs/log_101208_coalign.log:/scratch/adam_work/l192/l192_13pca took 403.81112 sec.logs/log_101208_coalign.log:/scratch/adam_work/l060/l060_13pca took 90.815500 sec.logs/log_101208_coalign.log:/scratch/adam_work/l069/l069_13pca took 160.17454 sec.
Maps with problems
l002: fuzzy stuff - bad sky sub? l033: misalignment l004: possible misalignment? Pattern recognition fails me here l354: misalignment
Mapping update
I made a lot of mistakes in my script, so progress is about 12 computer hours behind what it should be. These maps are done:
/scratch/adam_work/l003/l003_13pca_map09.fits/scratch/adam_work/l006/l006_13pca_map09.fits/scratch/adam_work/l009/l009_13pca_map09.fits/scratch/adam_work/l018/l018_13pca_map09.fits/scratch/adam_work/l021/l021_13pca_map09.fits/scratch/adam_work/l024/l024_13pca_map09.fits/scratch/adam_work/l027/l027_13pca_map09.fits/scratch/adam_work/l030/l030_13pca_map09.fits/scratch/adam_work/l033/l033_13pca_map09.fits/scratch/adam_work/l036/l036_13pca_map09.fits/scratch/adam_work/l039/l039_13pca_map09.fits/scratch/adam_work/l042/l042_13pca_map09.fits/scratch/adam_work/l044/l044_13pca_map09.fits/scratch/adam_work/l048/l048_13pca_map09.fits/scratch/adam_work/l050/l050_13pca_map09.fits/scratch/adam_work/l054/l054_13pca_map09.fits/scratch/adam_work/l057/l057_13pca_map09.fits
Things I've learned so far: There is an observation (a master?) in L003 that should be the galactic center master. L018 has some really bad noise that needs fixing. The wackiness I saw in L024 seems to be gone now. L044 has the same noise stripes as L357.
/scratch/adam_work/l018/l018_13pca took 1118.3684 sec./scratch/adam_work/l018/l018_13pca_nooffs took 1106.6423 sec./scratch/adam_work/l021/l021_13pca took 1703.7266 sec./scratch/adam_work/l021/l021_13pca_nooffs took 1716.7393 sec./scratch/adam_work/l024/l024_13pca took 1622.3164 sec./scratch/adam_work/l024/l024_13pca_nooffs took 1672.3263 sec./scratch/adam_work/l027/l027_13pca took 1376.1939 sec./scratch/adam_work/l027/l027_13pca_nooffs took 1383.4335 sec./scratch/adam_work/l030/l030_13pca took 2487.1311 sec./scratch/adam_work/l030/l030_13pca_nooffs took 2543.1740 sec./scratch/adam_work/l036/l036_13pca took 549.76983 sec./scratch/adam_work/l036/l036_13pca_nooffs took 548.84132 sec./scratch/adam_work/l039/l039_13pca took 299.96889 sec./scratch/adam_work/l039/l039_13pca_nooffs took 299.16787 sec./scratch/adam_work/l042/l042_13pca took 366.34139 sec./scratch/adam_work/l042/l042_13pca_nooffs took 364.31475 sec./scratch/adam_work/l044/l044_13pca took 427.26923 sec./scratch/adam_work/l044/l044_13pca_nooffs took 424.42403 sec./scratch/adam_work/l048/l048_13pca took 313.86412 sec./scratch/adam_work/l048/l048_13pca_nooffs took 303.52545 sec./scratch/adam_work/l050/l050_13pca took 421.22899 sec./scratch/adam_work/l050/l050_13pca_nooffs took 477.16070 sec./scratch/adam_work/l054/l054_13pca took 315.14965 sec./scratch/adam_work/l054/l054_13pca_nooffs took 342.69137 sec.