most of what's left now is flagging. I updated the flagger, flag_manual.pro. Check it out for documentation
Articles in the bgps category
Maps with problems
l002: fuzzy stuff - bad sky sub? l033: misalignment l004: possible misalignment? Pattern recognition fails me here l354: misalignment
Mapping update
I made a lot of mistakes in my script, so progress is about 12 computer hours behind what it should be. These maps are done:
/scratch/adam_work/l003/l003_13pca_map09.fits/scratch/adam_work/l006/l006_13pca_map09.fits/scratch/adam_work/l009/l009_13pca_map09.fits/scratch/adam_work/l018/l018_13pca_map09.fits/scratch/adam_work/l021/l021_13pca_map09.fits/scratch/adam_work/l024/l024_13pca_map09.fits/scratch/adam_work/l027/l027_13pca_map09.fits/scratch/adam_work/l030/l030_13pca_map09.fits/scratch/adam_work/l033/l033_13pca_map09.fits/scratch/adam_work/l036/l036_13pca_map09.fits/scratch/adam_work/l039/l039_13pca_map09.fits/scratch/adam_work/l042/l042_13pca_map09.fits/scratch/adam_work/l044/l044_13pca_map09.fits/scratch/adam_work/l048/l048_13pca_map09.fits/scratch/adam_work/l050/l050_13pca_map09.fits/scratch/adam_work/l054/l054_13pca_map09.fits/scratch/adam_work/l057/l057_13pca_map09.fits
Things I've learned so far: There is an observation (a master?) in L003 that should be the galactic center master. L018 has some really bad noise that needs fixing. The wackiness I saw in L024 seems to be gone now. L044 has the same noise stripes as L357.
/scratch/adam_work/l018/l018_13pca took 1118.3684 sec./scratch/adam_work/l018/l018_13pca_nooffs took 1106.6423 sec./scratch/adam_work/l021/l021_13pca took 1703.7266 sec./scratch/adam_work/l021/l021_13pca_nooffs took 1716.7393 sec./scratch/adam_work/l024/l024_13pca took 1622.3164 sec./scratch/adam_work/l024/l024_13pca_nooffs took 1672.3263 sec./scratch/adam_work/l027/l027_13pca took 1376.1939 sec./scratch/adam_work/l027/l027_13pca_nooffs took 1383.4335 sec./scratch/adam_work/l030/l030_13pca took 2487.1311 sec./scratch/adam_work/l030/l030_13pca_nooffs took 2543.1740 sec./scratch/adam_work/l036/l036_13pca took 549.76983 sec./scratch/adam_work/l036/l036_13pca_nooffs took 548.84132 sec./scratch/adam_work/l039/l039_13pca took 299.96889 sec./scratch/adam_work/l039/l039_13pca_nooffs took 299.16787 sec./scratch/adam_work/l042/l042_13pca took 366.34139 sec./scratch/adam_work/l042/l042_13pca_nooffs took 364.31475 sec./scratch/adam_work/l044/l044_13pca took 427.26923 sec./scratch/adam_work/l044/l044_13pca_nooffs took 424.42403 sec./scratch/adam_work/l048/l048_13pca took 313.86412 sec./scratch/adam_work/l048/l048_13pca_nooffs took 303.52545 sec./scratch/adam_work/l050/l050_13pca took 421.22899 sec./scratch/adam_work/l050/l050_13pca_nooffs took 477.16070 sec./scratch/adam_work/l054/l054_13pca took 315.14965 sec./scratch/adam_work/l054/l054_13pca_nooffs took 342.69137 sec.
Scripting the Whole Survey
First, discovered more fields with some sort of failure:
ls -d l[0-3][0-9][0-9] | sed 's:\(.*\):ls \1/*_map01.fits > \1/\1_infile.txt:' | bashls: l004/*_map01.fits: No such file or directoryls: l017/*_map01.fits: No such file or directoryls: l025/*_map01.fits: No such file or directoryls: l108/*_map01.fits: No such file or directoryls: l135/*_map01.fits: No such file or directoryls: l136/*_map01.fits: No such file or directoryls: l137/*_map01.fits: No such file or directoryls: l138/*_map01.fits: No such file or directoryls: l192/*_map01.fits: No such file or directory
Also, that command was a total screwup.
ls -d l[0-3][0-9][0-9] | sed 's:\(.*\):ls /scratch/adam_work/\1/*_map01.fits > \1/\1_fitslist.txt:' | bashls: /scratch/adam_work/l004/*_map01.fits: No such file or directoryls: /scratch/adam_work/l017/*_map01.fits: No such file or directorybash: line 12: l020/l020_fitslist.txt: Permission deniedls: /scratch/adam_work/l025/*_map01.fits: No such file or directoryls: /scratch/adam_work/l108/*_map01.fits: No such file or directoryls: /scratch/adam_work/l135/*_map01.fits: No such file or directoryls: /scratch/adam_work/l136/*_map01.fits: No such file or directoryls: /scratch/adam_work/l137/*_map01.fits: No such file or directoryls: /scratch/adam_work/l138/*_map01.fits: No such file or directoryls: /scratch/adam_work/l192/*_map01.fits: No such file or directory
Now that those files exist, it should be possible to run a set of super-scripts like this: coalign_field,'l057','070719_o29',sliced_dir='sliced_polychrome',premap=0 coalign_field,'l351','070725_ob3' coalign_field,'l354','070724_o10' coalign_field,'l357','070724_ob3' coalign_field,'l000','070719_o14' coalign_field,'l003','070718_o16' coalign_field,'l006','070715_ob5' coalign_field,'l009','070717_ob5' coalign_field,'l012','070715_o10' coalign_field,'l015','070714_o36' coalign_field,'l018','070717_o10' coalign_field,'l021','070715_o15' coalign_field,'l024','070717_o15' coalign_field,'l027','070715_o20' coalign_field,'l030','070717_o20' coalign_field,'l033','070718_ob5',sliced_dir='sliced_polychrome' coalign_field,'l036','070715_o25',sliced_dir='sliced_polychrome' coalign_field,'l039','070717_o25',sliced_dir='sliced_polychrome' coalign_field,'l042','070715_o30',sliced_dir='sliced_polychrome' coalign_field,'l044','070718_o24',sliced_dir='sliced_polychrome' coalign_field,'l048','070717_o30',sliced_dir='sliced_polychrome' coalign_field,'l050','070718_o29',sliced_dir='sliced_polychrome' coalign_field,'l054','070724_o28',sliced_dir='sliced_polychrome' coalign_field,'l057','070719_o29',sliced_dir='sliced_polychrome' where premap=0 means I'm not re-mapping the whole field, the sliced_dir='slice_polychrome' keyword is for those fields that do not have a regular sliced directory. This kind of thing ought to be really, really helpful when mapping the fields whose masters are not in the field: I'll have to modify the 'coalign_field' code to search in a different directory, though. Yearghhh.... last command was bad too.
ls -d l[0-3][0-9][0-9] | sed 's:\(.*\):ls /scratch/adam_work/\1/0*_map01.fits > \1/\1_fitslist.txt:' | bashls: /scratch/adam_work/l004/0*_map01.fits: No such file or directoryls: /scratch/adam_work/l017/0*_map01.fits: No such file or directorybash: line 12: l020/l020_fitslist.txt: Permission deniedls: /scratch/adam_work/l025/0*_map01.fits: No such file or directoryls: /scratch/adam_work/l108/0*_map01.fits: No such file or directoryls: /scratch/adam_work/l135/0*_map01.fits: No such file or directoryls: /scratch/adam_work/l136/0*_map01.fits: No such file or directoryls: /scratch/adam_work/l137/0*_map01.fits: No such file or directoryls: /scratch/adam_work/l138/0*_map01.fits: No such file or directoryls: /scratch/adam_work/l192/0*_map01.fits: No such file or directory
Alignment success stories
Two success stories, one failure story.
Successes: L111, L024. On the left is the aligned map, on the right the unaligned map.
Failure: L024. What's going on here? Is it an issue of that noisy observation, or the fact that we're using a 5x1 (I think?) as a reference?
Progress on alignment, ideas for the next few steps
Alignment: I did a more thorough test of the aligner, and found an error that was getting me the wrong results. I'm still getting distinctly different results than I got with imalign, but I'm not convinced they're wrong. One problem I have to deal with is the ambiguity between pointing model applied / no pointing model applied. In retrospect, I should have done ALL of the maps with no pointing model: it would be simpler to maintain a self-consistent set of alignment corrections. However, since that isn't what I did - and there isn't enough memory to host both data sets at the moment - I've dealt with the problem by adding all of the offsets to the header. Hopefully now it will be obvious if and what offsets have been applied, both pointing model and 'manual', from header keywords. Sadly, it was a pain to get that to work - IDL's _extra does not work the same way as normal keyword passing, which is disappointing and frustrating because it made me rename a lot of variables that should not have been renamed. IF this alignment fails to produce perfect results (which I don't really expect to happen - it will just take time to debug), it is still a good first step, and all we'd really need to do is use bolocat to pull out sources from the reference image and then match those in the others, doing the same thing imalign does. It wouldn't be difficult but it would be time consuming to program. Had a bunch of ideas during my insomnia last night, but I'm afraid I forgot most of them... One idea was related to the use of MAD as opposed to STDDEV: it may allow for a more robust sigma-rejection for automated flagging. But that's difficult. Another thought was using the maps to auto-identify glitches. To do this, I'd make a combined map and an individual map, hastrom them, and subtract the combined from the individual. We could then probably pretty easily spot any glitches; they should be the only outliers left, in principle. I'll test that idea, but it actually looks less promising than I'd hoped. Things are not coming out right in preliminary tests - huge residuals.
Failed fields
- l030 - comments were in the form ;; instead of # in the infile
- l055 - no such folder
- l026 - no infile existed; made one
- l028 - /scratch/sliced_polychrome/l028/060605_o[**]_raw_ds5.nc are bad
- l031 - Downsampling seems to have failed for most of this directory. Need to downsample all files and create a new infile
- l035 - /scratch/sliced_polychrome/l035/060605_o[**]_raw_ds5.nc are bad - that's all 060605 observations of l035.
- l036 - /scratch/sliced_polychrome/l036/070702_o25_raw_ds5.nc, /scratch/sliced_polychrome/l036/070705_o21_raw_ds5.nc are bad
- l038 - /scratch/sliced_polychrome/l038/060627_ob3_raw_ds5.nc, /scratch/sliced_polychrome/l038/060627_ob[3567]_raw.nc are bad - had to re-preprocess
- l054 - /scratch/sliced_polychrome/l054/070724_o25_raw_ds5.nc is bad
- l057 - /scratch/sliced_polychrome/l057/070907_o16_raw_ds5.nc is bad
Except for l057, there is no guarantee that the whole observation is bad, BUT the code failed on ncdf_open, so it possible the observation turned bad during downsampling/preprocessing. fixed: 30, 26, 55, 35, 28, 31, 38, 54, 57 so far all observations on 060605 are bad.
Disagreement over definition of l,b coordinates
DS9 does something intuitive: cd0_0 and cd1_1 multiplied by the x,y pixel difference gets you the new l,b coordinate. I think this is very likely to be correct on the galactic plane. IDL does something different. I don't know why different, or really how - it's buried somewhere in wcsxy2sph - but it's definitely different. Which is right?
Broken rotation
I have to echo James in saying how remarkable it is than anything I ever did worked at all! I'm growing more and more certain that I've been double-correcting rotation for the past 2 months, and my maps don't look all that bad! .. image:: http://4.bp.blogspot.com/_lsgW26mWZnU/SMW4oOPcLaI/AAAAAAAADXw/1ARG8iyVXDQ/s400/070717_o15_rotations.png Image caption: Top left - both PA and rotang Top right - neither Bottom left - rotang only Bottom right - PA only I'm running the same sort of comparison on all of the GC and L=24 data. Check out /scratch/adam_work/ROT_TEST for the results. I'll be updating this when I've completed that analysis. Update 6:35 PM: !@#$, etc. New problem child: .. image:: http://1.bp.blogspot.com/_lsgW26mWZnU/SMXFUjLjRGI/AAAAAAAADX4/aynvuycRzGM/s400/060603_o16_rotations.png Image caption (same as above, luckily): Top left - both PA and rotang Top right - neither Bottom left - rotang only Bottom right - PA only So, can anyone tell me which one of THESE is right? Because I can't. Top left is 'correct' except that it's not. Update 7:10 PM: I compared adding/subtracting PA and RA, and I can say this: subtracting PA never works. Adding/subtracting RA works variably. I think I'm done testing parameter space. I haven't had very good luck doing parameter space tests in the past; it usually is more beneficial to take a step back and try to think through the problem. Update 7:30 PM: Fixed. Much as this looks like a short-lived, easily solved problem, it was one of the most subtle and insidious errors I've made. I got extremely lucky that plotting PA and ROTANGLE happened to spark off warning sirens in my mind about the positioning of the variable names. I wouldn't normally look for that.
fresh blood
hate to put it this way, but probably true - bolocam will eat us all. Cara and Marc have both agreed to help out with alignment work. This is good news! Next post will be bad news, just to compensate.
