Alignment: I did a more thorough test of the aligner, and found an error that was getting me the wrong results. I'm still getting distinctly different results than I got with imalign, but I'm not convinced they're wrong. One problem I have to deal with is the ambiguity between pointing model applied / no pointing model applied. In retrospect, I should have done ALL of the maps with no pointing model: it would be simpler to maintain a self-consistent set of alignment corrections. However, since that isn't what I did - and there isn't enough memory to host both data sets at the moment - I've dealt with the problem by adding all of the offsets to the header. Hopefully now it will be obvious if and what offsets have been applied, both pointing model and 'manual', from header keywords. Sadly, it was a pain to get that to work - IDL's _extra does not work the same way as normal keyword passing, which is disappointing and frustrating because it made me rename a lot of variables that should not have been renamed. IF this alignment fails to produce perfect results (which I don't really expect to happen - it will just take time to debug), it is still a good first step, and all we'd really need to do is use bolocat to pull out sources from the reference image and then match those in the others, doing the same thing imalign does. It wouldn't be difficult but it would be time consuming to program. Had a bunch of ideas during my insomnia last night, but I'm afraid I forgot most of them... One idea was related to the use of MAD as opposed to STDDEV: it may allow for a more robust sigma-rejection for automated flagging. But that's difficult. Another thought was using the maps to auto-identify glitches. To do this, I'd make a combined map and an individual map, hastrom them, and subtract the combined from the individual. We could then probably pretty easily spot any glitches; they should be the only outliers left, in principle. I'll test that idea, but it actually looks less promising than I'd hoped. Things are not coming out right in preliminary tests - huge residuals.
New code
Done for the day after this (observing tonight) two new procedures:
alignment_offsets_to_ncdf.pro calcoeffs_to_ncdf.pro
they're really easy, very nearly one-liners. Once they've been used once, you can just use ncdf_varput_scale Modified keyword parameters in do_the_pointing: offsets will always be applied unless the keyword 'no_offsets' is passed into the code (or the top-level wrapper). Since offsets default to zero, this should be OK.
Pointing model success
/scratch/adam_work/plots/models_myptgmdl_0707.ps Look at the left plots in #1 and #4, and ignore the red lines. The RMS is 4.87 in alt, 3.51 in az, using all pointing observations in 0707 (i.e. not splitting it into part1/part2 or refining heavily). Total is 6.003", not bad. I'm calling it quits on pointing refinement for the moment; I'm satisfied using Meredith's pointing models to get science maps co-aligned, then we'll use my model on the PPSes to get the pointing to maximum precision. Also, in the galactic center, the mapping works happily now. No more silly problems with that. It's time for a real Next Step. I think that means mapping nearly everything. Well, here goes!
Updates today
I updated the FITS header writing pieces of the software today, we should now have: N_PCA ITERNUM PPBEAM in all new maps. I'm mapping all of Cygnus degree-by-degree in RA/Dec and Galactic to try to measure offsets... possibly offsets as a function of l/b or ra/dec. I should really be doing this with PPSes, though. My priority list:
- find out what's causing bulk offsets
- beat down noise in my pointing models by rejecting bad sources/points [has to be done from campus]
- test my personal special pointing models
- make maps!
- figure out the rest of pointing
Some good news, some.... not news.
I've been mapping all weekend. It turns out that if you do 25 mapping runs that take ~1hr apiece, that takes ~25 hours. Who knew? Whole observation took 3965.3506 sec. Whole observation took 3071.1403 sec. Whole observation took 4205.3337 sec. Whole observation took 7362.5490 sec. Whole observation took 9313.6064 sec. Whole observation took 3755.2383 sec. Whole observation took 3755.8278 sec. Whole observation took 3760.7032 sec. Whole observation took 3758.5719 sec. Whole observation took 3763.1436 sec. Whole observation took 3766.0066 sec. Whole observation took 3789.5806 sec. Whole observation took 3760.9774 sec. Whole observation took 3762.7124 sec. Whole observation took 3763.1120 sec. Whole observation took 3767.1255 sec. Whole observation took 3766.9085 sec. Whole observation took 3758.5867 sec. Whole observation took 3760.2131 sec. Whole observation took 3755.8963 sec. Whole observation took 3759.9199 sec. Whole observation took 3764.4494 sec. Whole observation took 3766.9686 sec. Whole observation took 3756.5876 sec. The good news is that I haven't encountered any crashes. Smooth sailing so far.
On the list for today....
Since my gigantic mapping run is still going, I'm going to work primarily on developing code for testing the maps, and determining important information about the mapper. The data paper will require an iteration-to-convergence figure, or at least an estimate of degree of convergence.
- Need to develop code to show iteration-to-convergence
- need it for both NxNpca and ascending/descending
- have to use region files? make some sort of region reader?
- pick demonstration sources
- Compare Npca results for bright, faint sources
- Compare iterating with/without deconvolution
- Compare median/average/baseline sky subtraction
- Analyze efficiency...
- Try to figure out what's causing huge offsets. Is it Galactic Coordinate mapping?
-bash-3.00$ less /scratch/adam_work/logs/log_082208_domemiter.log | grep WholeWhole observation took 5122.6795 sec.Whole observation took 4774.3424 sec.Whole observation took 2687.7761 sec.Whole observation took 2102.5577 sec.Whole observation took 3247.5344 sec.Whole observation took 5515.2907 sec.Whole observation took 2877.6389 sec.Whole observation took 2757.0235 sec.Whole observation took 1453.0918 sec.Whole observation took 1459.7892 sec.Whole observation took 1381.3762 sec.Whole observation took 1443.1440 sec.Whole observation took 614.39645 sec.Whole observation took 2001.6709 sec.Whole observation took 1817.1512 sec.Whole observation took 1879.1385 sec.
Note that the longest set there - 5515s = 1.53 hours - was 40 iterations!
- Need to develop code to show iteration-to-convergence
Task List
For the moment, I'm going to leave the pointing alone. This leaves us (me) with a very long list of things to do before the data release:
- Determine ideal iterative mapping strategy for low-flux fields
- Determine ideal iterative mapping strategy for fields with bright sources
- Optimize at least the following:
- scan flatting (polynomial subtraction)
- PCA subtraction
- Pre-PCA sky subtraction (necessary?)
- Deconvolution/No deconvolution
- GENERATE MAPS [top priority, but it can't really happen til after the above]
- Assure consistency with catalogs - e.g. Motte, SCUBA, etc.
- Figure out what needs to go into fits header
- Survive semester before comps [lowest priority]
- sleep [oh, right, real lowest priority]
Sarcasm helps me survive. Almost as much as beer.
Notice anything different?
About milkyway? This post is a red herring, but: I decided to go ahead and run the mapper on Cygnus, L33, L111, and the Galactic Center again. The pointing has gotten to the stage where I'm certain I can't do anything more without a stroke of pure brilliance or a conversation with someone who hasn't touched a thing - e.g. Jason - that suddenly enlightens me. In order to fully reproduce Meredith's results, I'd probably have to go back through and follow her 'pipeline' process step by step as well, and I suspect that, if I had used her method on her RA/Dec maps, I would have come up with exactly the same problem I currently see. Therefore, I won't do anything about it. I think the most appropriate response at this point MAY be to just fit a damned polynomial/sine curve in az and include that as part of the pointing model, but I can't justify where that comes from. All I know is that it's present in Meredith's data as well as my own. What's that next step with the PPSes? We need to do that. More important, though, is getting some image optimization ready FAST. I need to be running this stuff before September! Tonight's run will be a test of numbers of PCA components. At the very least, the v0.5 should have a consistent set of images even if we cut out the high-flux ones. We'll deal with that later. For the high flux objects, e.g. g34.3, since we have so much overlapping data, a simple average/magical baseline subtraction might be just as if not more effective than PCA subtraction, so that's one way around it.
Modifications
Things that have been changed in the past 24 hours that are very significant:
- In apply_pointing_model, the signs of FAZO and FZAO have been swapped. I BELIEVE this is correct but somehow things aren't working out still.
- In do_the_pointing I have changed from "eq2hor" and "hor2eq" to
"my_eq2hor" and "my_hor2eq". These implement two major changes:
- The LST is passed as a parameter rather than calculated within the ASTROLIB code
- The conversion is calculated with BOTH hadec2altaz and getaltaz (and similar for the opposite transformation) and compared for error checking purposes. If they differ by more than one arcsecond, the code will spit out an error message and use getaltaz.
More cleaning modifications
I had removed sigmadeglitch from deline, now it's permanently gone. I think it might be worth exploring re-inserting sigmadeglitch somewhere. My higher-order polynomial fit is creating deeper bowls around sources, so that's a big problem, but if I use a lower order I get the bad streaks. What's the best way to deal with this? I'm thinking perhaps only doing the polysub on the second iteration (i.e. after a source model has been subtracted). Delining runs into some issues now, though, because not all frequencies are sampled (?).