Not as easy as it ought to be. I think I need to do a few things: 1. check and make sure there are no more of those !@#$!@#$#@% different sized array subtractions/multiplications. 'weight' and 'best_astro_model' need to have the same size & shape in mem_iter_pc 2. I guess just check and make sure stuff works. The weighted mean I'm using appears to be right: sum(weight * value) / sum(weight) I hate making lists that end up being two items....
BGPS Photo
New to-do list, updated things....
do_maptests.pro is running a bunch of different mapping parameters (pca components, deconvolution, etc.) on l000, l002, l003, l033, l083. We'll then run bolocat on it and look for the following: -number of sources found -flux in sources found as a function of n_pca -size of sources as function of n_pca -BADNESS, e.g. blurring / unremoved atmosphere / oversubtracted sources bolocat2reg makes a region file out of a bolocat catalog. I'd like to make a separate ds9 region file that includes the actual pointing error + the centroiding error rather than just the elliptical fit to a given source; that will be more useful for finder charts. My current goal is to get a nice set of images I can combine to release as a poster to Jason Glenn's student who is making pretty posters for publicity purposes; probably to promote CCAT. Umm.... I blame the late hour for the alliteration. That goal also means I've been using an IRAF task to do some mosaicing (easier than writing the IDL code AGAIN): mscstack l001_5pca_map09.fits,l002_5pca_map09.fits,l000_5pca_map09.fits,l359_5pca_map09.fits,l003_5pca_map09.fits GCCOMBINE_5pca.fits lthresh=-1l001_13pca_map09_scuba_aligned.fits,l002_13pca_map09_scuba_aligned.fits,l000_13pca_map09_scuba_aligned.fits,l359_13pca_map09_scuba_aligned.fits,l003_13pca_map09_scuba_aligned.fits GCCOMBINE.fits lthresh=-1mscstack l029_13pca_map09_scuba_aligned.fits,l030_13pca_map09_scuba_aligned.fits,l031_13pca_map09_scuba_aligned.fits,l032_13pca_map09_scuba_aligned.fits,l033_13pca_map09_scuba_aligned.fits,l034_13pca_map09_scuba_aligned.fits L33COMBINE.fits lthresh=-1 hsig=5 lsig=5
outer galaxy fields done
they're small files logs/log_101008_coalign.log:/scratch/adam_work/l059/l059_13pca took 286.08181 sec.logs/log_101008_coalign.log:/scratch/adam_work/l062/l062_13pca took 390.88015 sec.logs/log_101008_coalign.log:/scratch/adam_work/l065/l065_13pca took 296.88441 sec.logs/log_101008_coalign.log:/scratch/adam_work/l068/l068_13pca took 1136.2623 sec.logs/log_101008_coalign.log:/scratch/adam_work/l071/l071_13pca took 899.41245 sec.logs/log_101208_coalign.log:/scratch/adam_work/l189p5/l189p5_13pca took 375.06840 sec.logs/log_101208_coalign.log:/scratch/adam_work/l192/l192_13pca took 403.81112 sec.logs/log_101208_coalign.log:/scratch/adam_work/l060/l060_13pca took 90.815500 sec.logs/log_101208_coalign.log:/scratch/adam_work/l069/l069_13pca took 160.17454 sec.
flagging
most of what's left now is flagging. I updated the flagger, flag_manual.pro. Check it out for documentation
Maps with problems
l002: fuzzy stuff - bad sky sub? l033: misalignment l004: possible misalignment? Pattern recognition fails me here l354: misalignment
Mapping update
I made a lot of mistakes in my script, so progress is about 12 computer hours behind what it should be. These maps are done:
/scratch/adam_work/l003/l003_13pca_map09.fits/scratch/adam_work/l006/l006_13pca_map09.fits/scratch/adam_work/l009/l009_13pca_map09.fits/scratch/adam_work/l018/l018_13pca_map09.fits/scratch/adam_work/l021/l021_13pca_map09.fits/scratch/adam_work/l024/l024_13pca_map09.fits/scratch/adam_work/l027/l027_13pca_map09.fits/scratch/adam_work/l030/l030_13pca_map09.fits/scratch/adam_work/l033/l033_13pca_map09.fits/scratch/adam_work/l036/l036_13pca_map09.fits/scratch/adam_work/l039/l039_13pca_map09.fits/scratch/adam_work/l042/l042_13pca_map09.fits/scratch/adam_work/l044/l044_13pca_map09.fits/scratch/adam_work/l048/l048_13pca_map09.fits/scratch/adam_work/l050/l050_13pca_map09.fits/scratch/adam_work/l054/l054_13pca_map09.fits/scratch/adam_work/l057/l057_13pca_map09.fits
Things I've learned so far: There is an observation (a master?) in L003 that should be the galactic center master. L018 has some really bad noise that needs fixing. The wackiness I saw in L024 seems to be gone now. L044 has the same noise stripes as L357.
/scratch/adam_work/l018/l018_13pca took 1118.3684 sec./scratch/adam_work/l018/l018_13pca_nooffs took 1106.6423 sec./scratch/adam_work/l021/l021_13pca took 1703.7266 sec./scratch/adam_work/l021/l021_13pca_nooffs took 1716.7393 sec./scratch/adam_work/l024/l024_13pca took 1622.3164 sec./scratch/adam_work/l024/l024_13pca_nooffs took 1672.3263 sec./scratch/adam_work/l027/l027_13pca took 1376.1939 sec./scratch/adam_work/l027/l027_13pca_nooffs took 1383.4335 sec./scratch/adam_work/l030/l030_13pca took 2487.1311 sec./scratch/adam_work/l030/l030_13pca_nooffs took 2543.1740 sec./scratch/adam_work/l036/l036_13pca took 549.76983 sec./scratch/adam_work/l036/l036_13pca_nooffs took 548.84132 sec./scratch/adam_work/l039/l039_13pca took 299.96889 sec./scratch/adam_work/l039/l039_13pca_nooffs took 299.16787 sec./scratch/adam_work/l042/l042_13pca took 366.34139 sec./scratch/adam_work/l042/l042_13pca_nooffs took 364.31475 sec./scratch/adam_work/l044/l044_13pca took 427.26923 sec./scratch/adam_work/l044/l044_13pca_nooffs took 424.42403 sec./scratch/adam_work/l048/l048_13pca took 313.86412 sec./scratch/adam_work/l048/l048_13pca_nooffs took 303.52545 sec./scratch/adam_work/l050/l050_13pca took 421.22899 sec./scratch/adam_work/l050/l050_13pca_nooffs took 477.16070 sec./scratch/adam_work/l054/l054_13pca took 315.14965 sec./scratch/adam_work/l054/l054_13pca_nooffs took 342.69137 sec.
Scripting the Whole Survey
First, discovered more fields with some sort of failure:
ls -d l[0-3][0-9][0-9] | sed 's:\(.*\):ls \1/*_map01.fits > \1/\1_infile.txt:' | bashls: l004/*_map01.fits: No such file or directoryls: l017/*_map01.fits: No such file or directoryls: l025/*_map01.fits: No such file or directoryls: l108/*_map01.fits: No such file or directoryls: l135/*_map01.fits: No such file or directoryls: l136/*_map01.fits: No such file or directoryls: l137/*_map01.fits: No such file or directoryls: l138/*_map01.fits: No such file or directoryls: l192/*_map01.fits: No such file or directory
Also, that command was a total screwup.
ls -d l[0-3][0-9][0-9] | sed 's:\(.*\):ls /scratch/adam_work/\1/*_map01.fits > \1/\1_fitslist.txt:' | bashls: /scratch/adam_work/l004/*_map01.fits: No such file or directoryls: /scratch/adam_work/l017/*_map01.fits: No such file or directorybash: line 12: l020/l020_fitslist.txt: Permission deniedls: /scratch/adam_work/l025/*_map01.fits: No such file or directoryls: /scratch/adam_work/l108/*_map01.fits: No such file or directoryls: /scratch/adam_work/l135/*_map01.fits: No such file or directoryls: /scratch/adam_work/l136/*_map01.fits: No such file or directoryls: /scratch/adam_work/l137/*_map01.fits: No such file or directoryls: /scratch/adam_work/l138/*_map01.fits: No such file or directoryls: /scratch/adam_work/l192/*_map01.fits: No such file or directory
Now that those files exist, it should be possible to run a set of super-scripts like this: coalign_field,'l057','070719_o29',sliced_dir='sliced_polychrome',premap=0 coalign_field,'l351','070725_ob3' coalign_field,'l354','070724_o10' coalign_field,'l357','070724_ob3' coalign_field,'l000','070719_o14' coalign_field,'l003','070718_o16' coalign_field,'l006','070715_ob5' coalign_field,'l009','070717_ob5' coalign_field,'l012','070715_o10' coalign_field,'l015','070714_o36' coalign_field,'l018','070717_o10' coalign_field,'l021','070715_o15' coalign_field,'l024','070717_o15' coalign_field,'l027','070715_o20' coalign_field,'l030','070717_o20' coalign_field,'l033','070718_ob5',sliced_dir='sliced_polychrome' coalign_field,'l036','070715_o25',sliced_dir='sliced_polychrome' coalign_field,'l039','070717_o25',sliced_dir='sliced_polychrome' coalign_field,'l042','070715_o30',sliced_dir='sliced_polychrome' coalign_field,'l044','070718_o24',sliced_dir='sliced_polychrome' coalign_field,'l048','070717_o30',sliced_dir='sliced_polychrome' coalign_field,'l050','070718_o29',sliced_dir='sliced_polychrome' coalign_field,'l054','070724_o28',sliced_dir='sliced_polychrome' coalign_field,'l057','070719_o29',sliced_dir='sliced_polychrome' where premap=0 means I'm not re-mapping the whole field, the sliced_dir='slice_polychrome' keyword is for those fields that do not have a regular sliced directory. This kind of thing ought to be really, really helpful when mapping the fields whose masters are not in the field: I'll have to modify the 'coalign_field' code to search in a different directory, though. Yearghhh.... last command was bad too.
ls -d l[0-3][0-9][0-9] | sed 's:\(.*\):ls /scratch/adam_work/\1/0*_map01.fits > \1/\1_fitslist.txt:' | bashls: /scratch/adam_work/l004/0*_map01.fits: No such file or directoryls: /scratch/adam_work/l017/0*_map01.fits: No such file or directorybash: line 12: l020/l020_fitslist.txt: Permission deniedls: /scratch/adam_work/l025/0*_map01.fits: No such file or directoryls: /scratch/adam_work/l108/0*_map01.fits: No such file or directoryls: /scratch/adam_work/l135/0*_map01.fits: No such file or directoryls: /scratch/adam_work/l136/0*_map01.fits: No such file or directoryls: /scratch/adam_work/l137/0*_map01.fits: No such file or directoryls: /scratch/adam_work/l138/0*_map01.fits: No such file or directoryls: /scratch/adam_work/l192/0*_map01.fits: No such file or directory
Alignment success stories
Two success stories, one failure story.
Successes: L111, L024. On the left is the aligned map, on the right the unaligned map.
Failure: L024. What's going on here? Is it an issue of that noisy observation, or the fact that we're using a 5x1 (I think?) as a reference?
Progress on alignment, ideas for the next few steps
Alignment: I did a more thorough test of the aligner, and found an error that was getting me the wrong results. I'm still getting distinctly different results than I got with imalign, but I'm not convinced they're wrong. One problem I have to deal with is the ambiguity between pointing model applied / no pointing model applied. In retrospect, I should have done ALL of the maps with no pointing model: it would be simpler to maintain a self-consistent set of alignment corrections. However, since that isn't what I did - and there isn't enough memory to host both data sets at the moment - I've dealt with the problem by adding all of the offsets to the header. Hopefully now it will be obvious if and what offsets have been applied, both pointing model and 'manual', from header keywords. Sadly, it was a pain to get that to work - IDL's _extra does not work the same way as normal keyword passing, which is disappointing and frustrating because it made me rename a lot of variables that should not have been renamed. IF this alignment fails to produce perfect results (which I don't really expect to happen - it will just take time to debug), it is still a good first step, and all we'd really need to do is use bolocat to pull out sources from the reference image and then match those in the others, doing the same thing imalign does. It wouldn't be difficult but it would be time consuming to program. Had a bunch of ideas during my insomnia last night, but I'm afraid I forgot most of them... One idea was related to the use of MAD as opposed to STDDEV: it may allow for a more robust sigma-rejection for automated flagging. But that's difficult. Another thought was using the maps to auto-identify glitches. To do this, I'd make a combined map and an individual map, hastrom them, and subtract the combined from the individual. We could then probably pretty easily spot any glitches; they should be the only outliers left, in principle. I'll test that idea, but it actually looks less promising than I'd hoped. Things are not coming out right in preliminary tests - huge residuals.