Not entirely, but I realized my code had become... clunky, at best. I'm rewriting the wrappers using structs. I HATE IDL structs, but they're necessary.... if only they would make them dynamically modifiable. Weighting worked to some degree, the mapping is pretty much done.
Implementing weighting
Not as easy as it ought to be. I think I need to do a few things: 1. check and make sure there are no more of those !@#$!@#$#@% different sized array subtractions/multiplications. 'weight' and 'best_astro_model' need to have the same size & shape in mem_iter_pc 2. I guess just check and make sure stuff works. The weighted mean I'm using appears to be right: sum(weight * value) / sum(weight) I hate making lists that end up being two items....
Latest re-run of the pipeline
Wow. A lot of errors this time around. bash: /scratch/adam_work/l020/l020_fitslist.txt: Permission deniedMRDFITS: Image array (1634,560) Type=Real*4% HEULER: FITS header is already in Galactic: nothing changed% READFITS: ERROR - Unable to locate file 060628_o32_raw_ds5.nc_indiv13pca_map01.fits% Variable is undefined: HDIMAGE.% Execution halted at: IMAGE_SHIFTS 44 /home/milkyway/student/ginsbura/bgps_pipeline/alignment/image_shifts.pro% COALIGN_FIELD 31 /home/milkyway/student/ginsbura/bgps_pipeline/wrappers/coalign_field.pro% $MAIN$ Updated radec_offsets in /scratch/sliced_polychrome/l034/050708_o21_raw_ds5.ncWriting offsets to -0.00192975 -9.22609e-09 (ncfile was -0.000337092)% ALIGNMENT_OFFSETS_TO_NCDF: Could not find file % Execution halted at: ALIGNMENT_OFFSETS_TO_NCDF 4 /home/milkyway/student/ginsbura/bgps_pipeline/preproc/alignment_offsets_to_ ncdf.pro% WRITE_IMSHIFTS 11 /home/milkyway/student/ginsbura/bgps_pipeline/alignment/write_imshifts.pro% COALIGN_FIELD 34 /home/milkyway/student/ginsbura/bgps_pipeline/wrappers/coalign_field.pro% IMAGE_SHIFTS 44 /home/milkyway/student/ginsbura/bgps_pipeline/alignment/image_shifts.pro% COALIGN_FIELD 31 /home/milkyway/student/ginsbura/bgps_pipeline/wrappers/coalign_field.pro% $MAIN$ /scratch/adam_work/l354/070709_o16_raw_ds5.nc_indiv13pca took 110.42429 sec.MRDFITS: File access error% HEULER: ERROR - First parameter must be a FITS header or astrometry structure% Execution halted at: HEULER 78FIELD l351 COMPLETED at Sat Nov 8 05:34:01 2008. Took 1122.3564 seconds% Program caused arithmetic error: Floating underflow% Program caused arithmetic error: Floating illegal operandFIELD l052 BEGUN at Sat Nov 8 05:34:01 2008% FILE_LINES: Error opening file. File: /scratch/sliced/l052/l052_infile.txt No such file or directory% Error occurred at: READCOL 141FIELD l359 COMPLETED at Sat Nov 8 05:41:34 2008. Took 452.60051 seconds% Program caused arithmetic error: Floating underflow% Program caused arithmetic error: Floating illegal operandFIELD l066 BEGUN at Sat Nov 8 05:41:34 2008% FILE_LINES: Error opening file. File: /scratch/sliced/l066/l066_infile.txt No such file or directory% Error occurred at: READCOL 141FIELD l072 BEGUN at Sat Nov 8 05:41:34 2008% FILE_LINES: Error opening file. File: /scratch/sliced/l072/l072_infile.txt No such file or directory% Error occurred at: READCOL 141MRDFITS: Image array (628,561) Type=Real*4% HEULER: FITS header is already in Galactic: nothing changed% HEULER: FITS header is already in Galactic: nothing changed/scratch/adam_work/l086/060627_o23_raw_ds5.nc_indiv13pca_map01.fits shifts in pixels: -9.77712 11.8034 shifts in arcsec l/b: 70.395277 84.984804 delta-ra (as): 19.007974 delta-dec (as): -108.70337 x/y error in pixels: 3.99631 0.0701244 ra/dec error: -0.0079926204 0.00014024887 gal l/b error: -0.0079926204 0.00014024887% HEULER: FITS header is already in Galactic: nothing changedWARNING! x,y offs were too large: 203.757 -0.224758 so a different method is being attemptedWARNING! x,y offs were too large on the SECOND TRY too: -74.7339 -159.448 so a third method is being attemptedThis third method is meant to deal with small overlap regions that may be confused by high noise, it restrictsto a maximum 15 pixel offset (really less than that in practice) and therefore should be checked carefully./scratch/adam_work/l086/060627_o26_raw_ds5.nc_indiv13pca_map01.fits shifts in pixels: -1.38579 1.39157 shifts in arcsec l/b: 9.9777001 10.019338 delta-ra (as): 1.1521393 delta-dec (as): -14.093081 x/y error in pixels: 0.0814682 0.0882818 ra/dec error: -0.00016293635 0.00017656364 gal l/b error: -0.00016293635 0.00017656364% HEULER: FITS header is already in Galactic: nothing changedWARNING! x,y offs were too large: 9.13210 11.3905 so a different method is being attemptedWARNING! x,y offs were too large on the SECOND TRY too: -79.0260 20.0907 so a third method is being attemptedThis third method is meant to deal with small overlap regions that may be confused by high noise, it restrictsto a maximum 15 pixel offset (really less than that in practice) and therefore should be checked carefully./scratch/adam_work/l086/070707_o49_raw_ds5.nc_indiv13pca_map01.fits shifts in pixels: -9.13210 -11.3905 shifts in arcsec l/b: 65.751123 -82.011258 delta-ra (as): -105.06683 delta-dec (as): 3.1919479 x/y error in pixels: 0.486307 0.950778 ra/dec error: -0.00097261386 0.0019015568 gal l/b error: -0.00097261386 0.0019015568% HEULER: FITS header is already in Galactic: nothing changedWARNING! x,y offs were too large: 0.384630 13.1890 so a different method is being attemptedWARNING! x,y offs were too large on the SECOND TRY too: 205.434 -23.9600 so a third method is being attemptedThis third method is meant to deal with small overlap regions that may be confused by high noise, it restrictsto a maximum 15 pixel offset (really less than that in practice) and therefore should be checked carefully./scratch/adam_work/l086/070708_o37_raw_ds5.nc_indiv13pca_map01.fits shifts in pixels: -0.384630 -13.1890 shifts in arcsec l/b: 2.7693373 -94.960636 delta-ra (as): -74.063617 delta-dec (as): 59.509407 x/y error in pixels: 2.29135 0.933383 ra/dec error: -0.0045826990 0.0018667649 gal l/b error: -0.0045826990 0.0018667649% HEULER: FITS header is already in Galactic: nothing changedWARNING! x,y offs were too large: -143.139 10.8130 so a different method is being attempted% Subscript range values of the form low:high must be >= 0, < size, with low <= high: CORR.% Execution halted at: PIXSHIFT 107ALL PREPROC ... output is /scratch/adam_work/l133/070911_o25_raw_ds5.nc_indiv13pca READING IN 1 FILES FROM /scratch/sliced/l133/070911_o25_raw_ds5.ncReading file /scratch/sliced/l133/070911_o25_raw_ds5.ncApplying pointing model with alt/az off -0.024687220 -0.029154389NO OFFSETS APPLIED3.6277909 sec.No values flagged as bad!AC_BOLOS has 0.0 NAN points, which is 0.0 of totalFlatfielding (polynomial subtraction) ...0.92621708 sec.No values flagged as bad!AC_BOLOS has 0.0 NAN points, which is 0.0 of totalPrepare Map ... 1.4727209 sec.% CHECK_FITS: NAXIS keywords in FITS header have been updatedDone preproc ... 11.113711 sec.ITERATION NUMBER 0 ...SKY SUBTRACTION - MEDIAN0.15759706 sec.BEGINNING PCA SUBTRACTION OF COMPONENTS 0 TO 13 ... 0.68130398 sec.Flatfielding (polynomial subtraction) ...0.96450186 sec.% CHECK_FITS: NAXIS keywords in FITS header have been updated% CHECK_FITS: NAXIS keywords in FITS header have been updated% Program caused arithmetic error: Floating underflowDeconvolution iteration 0: 1.2357290 sec.Deconvolution iteration 1: 0.93198013 sec.Deconvolution iteration 2: 0.93152189 sec.Deconvolution iteration 3: 0.93164992 sec.Chi2 for iteration 0 is 1.22593e+07 with 2.07110e+06 degrees of freedomFINISHED ITERATION 0 13.803532 sec.ITERATION NUMBER 1 ...SKY SUBTRACTION - MEDIAN0.15691710 sec.BEGINNING PCA SUBTRACTION OF COMPONENTS 0 TO 13 ... % TRIQL: Too many iterations in tqli.% Error occurred at: EIGENQL 122 /usr/local/itt/idl70/lib/eigenql.pro% PCA_SUBTRACT 19/scratch/adam_work/l135p1/060917_o12_raw_ds5.nc_indiv13pca took 30.478553 sec.MRDFITS: File access error% HEULER: ERROR - First parameter must be a FITS header or astrometry structure% Execution halted at: HEULER 78MRDFITS: Image array (1630,555) Type=Real*4% HEULER: FITS header is already in Galactic: nothing changed% HEULER: FITS header is already in Galactic: nothing changed/scratch/adam_work/l068/070910_ob3_raw_ds5.nc_indiv13pca_map01.fits shifts in pixels: -0.0271645 -0.0270424 shifts in arcsec l/b: 0.19558409 -0.19470519 delta-ra (as): -0.26866187 delta-dec (as): -0.062403024 x/y error in pixels: 0.000193238 0.00126581 ra/dec error: -3.8647498e-07 2.5316247e-06 gal l/b error: -3.8647498e-07 2.5316247e-06% HEULER: FITS header is already in Galactic: nothing changedWARNING! x,y offs were too large: 34.8510 -5.72119 so a different method is being attemptedWARNING! x,y offs were too large on the SECOND TRY too: 142.704 76.4267 so a third method is being attemptedThis third method is meant to deal with small overlap regions that may be confused by high noise, it restrictsto a maximum 15 pixel offset (really less than that in practice) and therefore should be checked carefully./scratch/adam_work/l068/070911_ob8_raw_ds5.nc_indiv13pca_map01.fits shifts in pixels: 15.0101 3.42143 shifts in arcsec l/b: -108.07276 24.634271 delta-ra (as): 78.221561 delta-dec (as): 78.545365 x/y error in pixels: 0.0114306 0.184410 ra/dec error: -2.2861189e-05 0.00036881918 gal l/b error: -2.2861189e-05 0.00036881918% HEULER: FITS header is already in Galactic: nothing changedWARNING! x,y offs were too large: 56.2296 14.6994 so a different method is being attemptedWARNING! x,y offs were too large on the SECOND TRY too: 728.753 -184.087 so a third method is being attemptedThis third method is meant to deal with small overlap regions that may be confused by high noise, it restrictsto a maximum 15 pixel offset (really less than that in practice) and therefore should be checked carefully./scratch/adam_work/l068/070911_ob9_raw_ds5.nc_indiv13pca_map01.fits shifts in pixels: -56.2296 -14.6994 shifts in arcsec l/b: 404.85292 -105.83592 delta-ra (as): -304.51436 delta-dec (as): -286.88182 x/y error in pixels: 7.32951 0.00354093 ra/dec error: -0.014659024 7.0818565e-06 gal l/b error: -0.014659024 7.0818565e-06% HEULER: FITS header is already in Galactic: nothing changedWARNING! x,y offs were too large: 3.31224 -13.8321 so a different method is being attempted% Subscript range values of the form low:high must be >= 0, < size, with low <= high: CORR.% Execution halted at: PIXSHIFT 107/scratch/adam_work/l111/060605_o48_raw_ds5.nc_indiv13pca took 10.098406 sec.MRDFITS: File access error% HEULER: ERROR - First parameter must be a FITS header or astrometry structure% Execution halted at: HEULER 78Writing offsets to /scratch/sliced/l027/060616_o10_raw_ds5.nc 0.00120866 -0.00287235 (ncfile was 060616_o10_raw_ds5.nc)Updated radec_offsets in /scratch/sliced/l027/060616_o10_raw_ds5.ncWriting offsets to 5.77312e-05 -2.39760e-06 (ncfile was 0.000409983)% ALIGNMENT_OFFSETS_TO_NCDF: Could not find file % Execution halted at: ALIGNMENT_OFFSETS_TO_NCDF 4
BGPS Photo
New to-do list, updated things....
do_maptests.pro is running a bunch of different mapping parameters (pca components, deconvolution, etc.) on l000, l002, l003, l033, l083. We'll then run bolocat on it and look for the following: -number of sources found -flux in sources found as a function of n_pca -size of sources as function of n_pca -BADNESS, e.g. blurring / unremoved atmosphere / oversubtracted sources bolocat2reg makes a region file out of a bolocat catalog. I'd like to make a separate ds9 region file that includes the actual pointing error + the centroiding error rather than just the elliptical fit to a given source; that will be more useful for finder charts. My current goal is to get a nice set of images I can combine to release as a poster to Jason Glenn's student who is making pretty posters for publicity purposes; probably to promote CCAT. Umm.... I blame the late hour for the alliteration. That goal also means I've been using an IRAF task to do some mosaicing (easier than writing the IDL code AGAIN): mscstack l001_5pca_map09.fits,l002_5pca_map09.fits,l000_5pca_map09.fits,l359_5pca_map09.fits,l003_5pca_map09.fits GCCOMBINE_5pca.fits lthresh=-1l001_13pca_map09_scuba_aligned.fits,l002_13pca_map09_scuba_aligned.fits,l000_13pca_map09_scuba_aligned.fits,l359_13pca_map09_scuba_aligned.fits,l003_13pca_map09_scuba_aligned.fits GCCOMBINE.fits lthresh=-1mscstack l029_13pca_map09_scuba_aligned.fits,l030_13pca_map09_scuba_aligned.fits,l031_13pca_map09_scuba_aligned.fits,l032_13pca_map09_scuba_aligned.fits,l033_13pca_map09_scuba_aligned.fits,l034_13pca_map09_scuba_aligned.fits L33COMBINE.fits lthresh=-1 hsig=5 lsig=5
outer galaxy fields done
they're small files logs/log_101008_coalign.log:/scratch/adam_work/l059/l059_13pca took 286.08181 sec.logs/log_101008_coalign.log:/scratch/adam_work/l062/l062_13pca took 390.88015 sec.logs/log_101008_coalign.log:/scratch/adam_work/l065/l065_13pca took 296.88441 sec.logs/log_101008_coalign.log:/scratch/adam_work/l068/l068_13pca took 1136.2623 sec.logs/log_101008_coalign.log:/scratch/adam_work/l071/l071_13pca took 899.41245 sec.logs/log_101208_coalign.log:/scratch/adam_work/l189p5/l189p5_13pca took 375.06840 sec.logs/log_101208_coalign.log:/scratch/adam_work/l192/l192_13pca took 403.81112 sec.logs/log_101208_coalign.log:/scratch/adam_work/l060/l060_13pca took 90.815500 sec.logs/log_101208_coalign.log:/scratch/adam_work/l069/l069_13pca took 160.17454 sec.
flagging
most of what's left now is flagging. I updated the flagger, flag_manual.pro. Check it out for documentation
Maps with problems
l002: fuzzy stuff - bad sky sub? l033: misalignment l004: possible misalignment? Pattern recognition fails me here l354: misalignment
Mapping update
I made a lot of mistakes in my script, so progress is about 12 computer hours behind what it should be. These maps are done:
/scratch/adam_work/l003/l003_13pca_map09.fits/scratch/adam_work/l006/l006_13pca_map09.fits/scratch/adam_work/l009/l009_13pca_map09.fits/scratch/adam_work/l018/l018_13pca_map09.fits/scratch/adam_work/l021/l021_13pca_map09.fits/scratch/adam_work/l024/l024_13pca_map09.fits/scratch/adam_work/l027/l027_13pca_map09.fits/scratch/adam_work/l030/l030_13pca_map09.fits/scratch/adam_work/l033/l033_13pca_map09.fits/scratch/adam_work/l036/l036_13pca_map09.fits/scratch/adam_work/l039/l039_13pca_map09.fits/scratch/adam_work/l042/l042_13pca_map09.fits/scratch/adam_work/l044/l044_13pca_map09.fits/scratch/adam_work/l048/l048_13pca_map09.fits/scratch/adam_work/l050/l050_13pca_map09.fits/scratch/adam_work/l054/l054_13pca_map09.fits/scratch/adam_work/l057/l057_13pca_map09.fits
Things I've learned so far: There is an observation (a master?) in L003 that should be the galactic center master. L018 has some really bad noise that needs fixing. The wackiness I saw in L024 seems to be gone now. L044 has the same noise stripes as L357.
/scratch/adam_work/l018/l018_13pca took 1118.3684 sec./scratch/adam_work/l018/l018_13pca_nooffs took 1106.6423 sec./scratch/adam_work/l021/l021_13pca took 1703.7266 sec./scratch/adam_work/l021/l021_13pca_nooffs took 1716.7393 sec./scratch/adam_work/l024/l024_13pca took 1622.3164 sec./scratch/adam_work/l024/l024_13pca_nooffs took 1672.3263 sec./scratch/adam_work/l027/l027_13pca took 1376.1939 sec./scratch/adam_work/l027/l027_13pca_nooffs took 1383.4335 sec./scratch/adam_work/l030/l030_13pca took 2487.1311 sec./scratch/adam_work/l030/l030_13pca_nooffs took 2543.1740 sec./scratch/adam_work/l036/l036_13pca took 549.76983 sec./scratch/adam_work/l036/l036_13pca_nooffs took 548.84132 sec./scratch/adam_work/l039/l039_13pca took 299.96889 sec./scratch/adam_work/l039/l039_13pca_nooffs took 299.16787 sec./scratch/adam_work/l042/l042_13pca took 366.34139 sec./scratch/adam_work/l042/l042_13pca_nooffs took 364.31475 sec./scratch/adam_work/l044/l044_13pca took 427.26923 sec./scratch/adam_work/l044/l044_13pca_nooffs took 424.42403 sec./scratch/adam_work/l048/l048_13pca took 313.86412 sec./scratch/adam_work/l048/l048_13pca_nooffs took 303.52545 sec./scratch/adam_work/l050/l050_13pca took 421.22899 sec./scratch/adam_work/l050/l050_13pca_nooffs took 477.16070 sec./scratch/adam_work/l054/l054_13pca took 315.14965 sec./scratch/adam_work/l054/l054_13pca_nooffs took 342.69137 sec.
Scripting the Whole Survey
First, discovered more fields with some sort of failure:
ls -d l[0-3][0-9][0-9] | sed 's:\(.*\):ls \1/*_map01.fits > \1/\1_infile.txt:' | bashls: l004/*_map01.fits: No such file or directoryls: l017/*_map01.fits: No such file or directoryls: l025/*_map01.fits: No such file or directoryls: l108/*_map01.fits: No such file or directoryls: l135/*_map01.fits: No such file or directoryls: l136/*_map01.fits: No such file or directoryls: l137/*_map01.fits: No such file or directoryls: l138/*_map01.fits: No such file or directoryls: l192/*_map01.fits: No such file or directory
Also, that command was a total screwup.
ls -d l[0-3][0-9][0-9] | sed 's:\(.*\):ls /scratch/adam_work/\1/*_map01.fits > \1/\1_fitslist.txt:' | bashls: /scratch/adam_work/l004/*_map01.fits: No such file or directoryls: /scratch/adam_work/l017/*_map01.fits: No such file or directorybash: line 12: l020/l020_fitslist.txt: Permission deniedls: /scratch/adam_work/l025/*_map01.fits: No such file or directoryls: /scratch/adam_work/l108/*_map01.fits: No such file or directoryls: /scratch/adam_work/l135/*_map01.fits: No such file or directoryls: /scratch/adam_work/l136/*_map01.fits: No such file or directoryls: /scratch/adam_work/l137/*_map01.fits: No such file or directoryls: /scratch/adam_work/l138/*_map01.fits: No such file or directoryls: /scratch/adam_work/l192/*_map01.fits: No such file or directory
Now that those files exist, it should be possible to run a set of super-scripts like this: coalign_field,'l057','070719_o29',sliced_dir='sliced_polychrome',premap=0 coalign_field,'l351','070725_ob3' coalign_field,'l354','070724_o10' coalign_field,'l357','070724_ob3' coalign_field,'l000','070719_o14' coalign_field,'l003','070718_o16' coalign_field,'l006','070715_ob5' coalign_field,'l009','070717_ob5' coalign_field,'l012','070715_o10' coalign_field,'l015','070714_o36' coalign_field,'l018','070717_o10' coalign_field,'l021','070715_o15' coalign_field,'l024','070717_o15' coalign_field,'l027','070715_o20' coalign_field,'l030','070717_o20' coalign_field,'l033','070718_ob5',sliced_dir='sliced_polychrome' coalign_field,'l036','070715_o25',sliced_dir='sliced_polychrome' coalign_field,'l039','070717_o25',sliced_dir='sliced_polychrome' coalign_field,'l042','070715_o30',sliced_dir='sliced_polychrome' coalign_field,'l044','070718_o24',sliced_dir='sliced_polychrome' coalign_field,'l048','070717_o30',sliced_dir='sliced_polychrome' coalign_field,'l050','070718_o29',sliced_dir='sliced_polychrome' coalign_field,'l054','070724_o28',sliced_dir='sliced_polychrome' coalign_field,'l057','070719_o29',sliced_dir='sliced_polychrome' where premap=0 means I'm not re-mapping the whole field, the sliced_dir='slice_polychrome' keyword is for those fields that do not have a regular sliced directory. This kind of thing ought to be really, really helpful when mapping the fields whose masters are not in the field: I'll have to modify the 'coalign_field' code to search in a different directory, though. Yearghhh.... last command was bad too.
ls -d l[0-3][0-9][0-9] | sed 's:\(.*\):ls /scratch/adam_work/\1/0*_map01.fits > \1/\1_fitslist.txt:' | bashls: /scratch/adam_work/l004/0*_map01.fits: No such file or directoryls: /scratch/adam_work/l017/0*_map01.fits: No such file or directorybash: line 12: l020/l020_fitslist.txt: Permission deniedls: /scratch/adam_work/l025/0*_map01.fits: No such file or directoryls: /scratch/adam_work/l108/0*_map01.fits: No such file or directoryls: /scratch/adam_work/l135/0*_map01.fits: No such file or directoryls: /scratch/adam_work/l136/0*_map01.fits: No such file or directoryls: /scratch/adam_work/l137/0*_map01.fits: No such file or directoryls: /scratch/adam_work/l138/0*_map01.fits: No such file or directoryls: /scratch/adam_work/l192/0*_map01.fits: No such file or directory