So, despite the fact that I clearly have a distortion mapper working, applying the distortion maps that come out of it do NOT improve the peak or FWHM of the data.
filling factors
to do: histogram per square degree, plot cumulative > some bin number (100 mJy, 300 mJy, 1 Jy) vs l, then do it again for b=+/-.1, +/-.3.
distortion mapping
Between sign errors, failures to fit, and.... I really don't know WHAT was going on, I finally figured out how to get the cursed distortion mapping to work. I still haven't even started testing, unfortunately. I don't know why I can fit the way the OP did, fitting R and Theta, and I apparently can't fit x,y. It's frustrating.
Flagger images
James requested sample images from the flagger for the methods paper. Below are images + description: .. image:: http://3.bp.blogspot.com/_lsgW26mWZnU/SXHsZcqegtI/AAAAAAAAEs4/3PjUtU2IXtQ/s400/sample_waterfall_070911_o15_highFnoisepng.png This is a "noise-dominated" scan in the sense that the high/low pixels are set by noise, not signal. Despite the clear high frequency noise, this image actually maps out pretty well - I think the high-noise bolometers get downweighted and the high/low pixels probably get clipped by my hot pixel rejection procedure. .. image:: http://3.bp.blogspot.com/_lsgW26mWZnU/SXHsY7Rgq3I/AAAAAAAAEsY/0NT8IB26JUo/s400/flagger_marked_source_050708_o15.png An image of the galactic center with a source southeast of center identified. .. image:: http://1.bp.blogspot.com/_lsgW26mWZnU/SXHsYyHCEGI/AAAAAAAAEsg/3iwoklY0--0/s400/sample_waterfall_050708_o15_glitchandsources.png A scan from the GC image above. I forgot to mark the source, I should go back and do that. The glitch is obvious. .. image:: http://1.bp.blogspot.com/_lsgW26mWZnU/SXHsZHVZ4OI/AAAAAAAAEso/pOXwGFn7rGg/s400/sample_waterfall_050708_o15_glitchflagged.png I drew a box to flag out the region affected by the glitch. .. image:: http://4.bp.blogspot.com/_lsgW26mWZnU/SXHsZG_MmWI/AAAAAAAAEsw/RA6tqAWMnLM/s400/sample_waterfall_050708_o15_glitchgone.png This is what happens when I redraw after flagging out the glitch - the colorbar is rescaled and no more glitch. .. image:: http://4.bp.blogspot.com/_lsgW26mWZnU/SXIDILh3hfI/AAAAAAAAEtQ/cMuOhlFO_fA/s400/sample_waterfall_050708_o15_glitchandsources_marked.png Timestream with glitches and sources marked (one pixel in the map is hit by 3 different points in this scan). .. image:: http://4.bp.blogspot.com/_lsgW26mWZnU/SXIDH3uIjHI/AAAAAAAAEtI/vijTeEWf_nA/s400/sample_waterfall_050708_o15_glitchandsources_gray.png Grayscale version of above (ok, I lied about grayscale being impossible) with a different pixel marked. .. image:: http://1.bp.blogspot.com/_lsgW26mWZnU/SXIDHbYLbiI/AAAAAAAAEtA/nSh2rM6GhMc/s400/flagger_marked_source_footprint_050708_o15.png A zoom-in around the 'kidney bean' source with the Bolocam footprint overlaid and a pixel marked. Note that this pixel corresponds to the 3 points in the color waterfall above.
0709 rotators fixed, latest run
0709 11-13 needed to be 83.88 degrees, they're all fixed now. I flagged the l060 070911 maps; I'm not convinced that was weather, there were some fluctuations up to 2400 Jy! Maybe a cloud would do that though.
Problems in the latest run:
l359 - missing files? Reading files from /scratch/sliced/INFILES/l359\_infile.txt FIELD v0.7\_l359 BEGUN at Fri Jan 16 20:11:34 2009 MRDFITS: File access error % HEULER: ERROR - First parameter must be a FITS header or astrometry structure l012 - missing files? % READFITS: ERROR - Unable to locate file /scratch/adam\_work/l012/060614\_o10\_raw\_ds5.nc\_indiv13pca\_map01.fi ts
More problem maps
the l060-l062 range has some severe noise problems and rotator issues. They need to be fixed before release.
Uh?
I'm not sure the error reported yesterday is reproducible. Tonight, I started a complete re-mapping of the entire survey. The new mapping will include beam-smoothed maps and more complete header parameters. I need to re-check the L111 mapping. It looks like the region with NGC 7538 in it did alright, but the southeast region didn't do so well. The combined maps are probably better than in any previous iteration, though. Tomorrow's tests will include L024 presumably. It's time to get to work on this stuff. There is definitely a reasonably efficient way to go through all of this stuff; the most difficult part is post-facto error checking. Here are some awk commands I found useful (the \'s are because of VI): `` awk '{printf( $0); if ($2 != 0){ printf(" %4i%4i",$4-443,$5-207) }; if($4!=0){printf(" %4i,%4i",$2-272,$3-742)}; printf("n") }' align/l111_fitslist_shiftfind.txt`` awk '{print $6,$7}' align/l111_fitslist_shiftfind.txt > align/shifts_all_lb.txt One important point: I had to add a parameter to the header file to say whether the current pointing model was used or not. It matters when applying the offsets. Note that since I'm using the pointing model to make all the individual maps, we actually have to stick with this current set of pointing models for all coaligned maps. The current models:
# Definition of coefficientts:# start/end mjd are the start/end modified Julian date of a given pointing model# the 'a' coefficients are for the AZIMUTH OFFSET, the b coefficients are for the ZENITH ANGLE OFFSET# a0/b0 are constants (e.g. the mean)# [ab][12] are the 1st and 2nd coefficients of Azimuth. They have been fiated to zero for most of# the past year or three.# [ab][34] are the 1st and 2nd coefficients of Zenith Angle.# A 'pointing model' is therefore something like this:# azoff = a0 + a3*alt + a4*alt^2# altoff = b0 + b3*alt + b4*alt^2## It is important that the start_mjd/end_mjd be in ascending order## WARNING: LATER THAN JULY 2007 DEFAULTS TO JULY 2007 WHICH WILL PROBABLY RESULT IN ERRORS!# I don't have a September 2007 model yet.## start_mjd end_mjd a0 a1 a2 a3 a4 b0 b1 b2 b3 b4 realdate 53522.5 53582.5 -9.2413685 0.0 0.0 -0.0066354359 -0.0015110883 7.0392221 0.0 0.0 -0.053635657 -0.00047042481 20050601 53614.5 53643.5 84.969583 0.0 0.0 -2.4339154 0.016300937 126.00164 0.0 0.0 -2.4424431 0.015455417 20050901 53887.5 53947.5 9.5305281 0.0 0.0 -0.053191181 -0.0029300592 0.13425019 0.0 0.0 0.48160040 -0.0092814256 20060601 53979.5 54008.5 -98.980381 0.0 0.0 0.65354164 -0.012414466 52.841380 0.0 0.0 1.6705743 -0.020893018 20060901 54101.5 54252.5 -99.078392 0.0 0.0 0.105270 -0.005943491 86.896333 0.0 0.0 0.54257415 -0.011919129 20070101 54252.5 54288 -103.03831 0.0 0.0 0.20972540 -0.0060336987 100.74491 0.0 0.0 0.0099012827 -0.0033331895 20070601# 54288 54313 -99.078392 0.0 0.0 0.10527000 -0.0059434911 86.896333 0.0 0.0 0.54257415 -0.011919129 20070707 54288 54313 -98.803883 0.0 0.0 0.11810246 -0.0051207995 91.720516 0.0 0.0 0.18953269 -0.0078189793 20070707 54313 54500 -99.078392 0.0 0.0 0.10527000 -0.0059434911 86.896333 0.0 0.0 0.54257415 -0.011919129 20070707mjd2date,53522.5 ,y,m,d & print,y,m,dmjd2date,53614.5 ,y,m,d & print,y,m,dmjd2date,53887.5 ,y,m,d & print,y,m,dmjd2date,53979.5 ,y,m,d & print,y,m,dmjd2date,54101.5 ,y,m,d & print,y,m,dmjd2date,54252.5 ,y,m,d & print,y,m,dmjd2date,54288 ,y,m,d & print,y,m,d
New mapping procedure
I cut down the number of maps made by the pipeline from 80 to 30 by merging regions with severe overlap. I don't think this will result in any net change in efficiency of the mapping process because a few of the maps will go over to swap, but it will reduce the amount of overlap / noisy edges and make debugging simpler. It should also reduce hard drive usage once I decide that I'm satisfied with the results and remove the old (v0.6) redundant data. As part of this process, the reference fields for coalignment have been re-set to the v0.6.2 coadds + some (crappy) mosaics from v0.6.2 so that we no longer need to rely on 3x1's for the alignment. See /data/bgps/releases/v0.7/readme.txt for details (reproduced below) Created 01/14/09 Version 0.7 does not exist yet. This is the 'release notes' indicating what will change -Overlapping redundant fields will be eliminated Mapping of field name to area covered, followed by number of observations included: gemob1: l189-l192 11 - 2 = 9 w5: l135-l138 46 = 46 l133: l133-l134 34 - 1 = 33 l111: l110-l111 88 = 88 l086: l085-l086 17 = 17 l082: l080-l084 33 = 33 l077: l075-l080 43 - 1 = 42 l072: l070-l074 10 - 2 = 8 l065: l059-l069 19 = 19 l055: l053-l058 10 - 1 = 9 l050: l048-l052 19 = 19 l045: l043-l047 20 - 4 = 16 l040: l038-l042 33 - 4 = 27 l035: l034-l037 53 = 53 l032: l031-l033 61 = 61 l030: l030 61 - 3 = 58 l029: l028-l029 53 - 4 = 49 l024: l021-l027 29 = 29 l018: l015-l021 29 - 3 = 26 l012: l011-l014 7 = 7 l009: l008-l010 11 = 11 l006: l005-l007 15 - 1 = 14 l003: l002-l004 25 - 1 = 24 l001: l001 20 = 20 l000: l000 23 - 1 = 23 l359: l359 14 = 14 l357: l356-l358 9 - 1 = 8 l354: l353-l355 3 = 3 l351: l350-l352 9 = 9
Gem OB1
With Miranda's help, I discovered some serious errors in Gem OB1 and fixed them.
- L189 was using L134 data for no apparent reason
- many September 2007 observations have the WRONG rotation angle in place in array_params! Unfortunately it's not clear yet which ones suffer from this problem: I need a list of when rotangle was/was not used, and it looks like it'll be a pain to fix the problem.
- GemOB1link wasn't mapped, it will be now.
Back to work
I started working again yesterday, mostly looking at the 31PCA reductions I did. I have no strong opinions yet on their quality. It's time to work towards a final data release, though, and to that end, I'm going to start removing duplicate fields. e.g. l023 and l024 have identical data, so only one of them should ever be used.
