I've been mapping all weekend. It turns out that if you do 25 mapping runs that take ~1hr apiece, that takes ~25 hours. Who knew? Whole observation took 3965.3506 sec. Whole observation took 3071.1403 sec. Whole observation took 4205.3337 sec. Whole observation took 7362.5490 sec. Whole observation took 9313.6064 sec. Whole observation took 3755.2383 sec. Whole observation took 3755.8278 sec. Whole observation took 3760.7032 sec. Whole observation took 3758.5719 sec. Whole observation took 3763.1436 sec. Whole observation took 3766.0066 sec. Whole observation took 3789.5806 sec. Whole observation took 3760.9774 sec. Whole observation took 3762.7124 sec. Whole observation took 3763.1120 sec. Whole observation took 3767.1255 sec. Whole observation took 3766.9085 sec. Whole observation took 3758.5867 sec. Whole observation took 3760.2131 sec. Whole observation took 3755.8963 sec. Whole observation took 3759.9199 sec. Whole observation took 3764.4494 sec. Whole observation took 3766.9686 sec. Whole observation took 3756.5876 sec. The good news is that I haven't encountered any crashes. Smooth sailing so far.
Re: Convergence Tests
Made a little code to check out convergence, but frankly it's pretty easy to just build a mapcube and plot lines in the iteration axis. The code is in bgps_pipeline/postproc/compareiters.pro, and it is not a single program. Comparing deconvolution to no deconvolution, deconvolution is a lot better. Without it, there are much more substantial negative regions. In the GC, my test region, the noise-dominated areas were about the same, though the no-deconvolve map had a little bit more large scale structure. The signal-dominated regions were very nearly uniformly brighter. The deconvolved map was ~3 Jy brighter in SGR B2, or 4%.
On the list for today....
Since my gigantic mapping run is still going, I'm going to work primarily on developing code for testing the maps, and determining important information about the mapper. The data paper will require an iteration-to-convergence figure, or at least an estimate of degree of convergence.
- Need to develop code to show iteration-to-convergence
- need it for both NxNpca and ascending/descending
- have to use region files? make some sort of region reader?
- pick demonstration sources
- Compare Npca results for bright, faint sources
- Compare iterating with/without deconvolution
- Compare median/average/baseline sky subtraction
- Analyze efficiency...
- Try to figure out what's causing huge offsets. Is it Galactic Coordinate mapping?
-bash-3.00$ less /scratch/adam_work/logs/log_082208_domemiter.log | grep WholeWhole observation took 5122.6795 sec.Whole observation took 4774.3424 sec.Whole observation took 2687.7761 sec.Whole observation took 2102.5577 sec.Whole observation took 3247.5344 sec.Whole observation took 5515.2907 sec.Whole observation took 2877.6389 sec.Whole observation took 2757.0235 sec.Whole observation took 1453.0918 sec.Whole observation took 1459.7892 sec.Whole observation took 1381.3762 sec.Whole observation took 1443.1440 sec.Whole observation took 614.39645 sec.Whole observation took 2001.6709 sec.Whole observation took 1817.1512 sec.Whole observation took 1879.1385 sec.
Note that the longest set there - 5515s = 1.53 hours - was 40 iterations!
- Need to develop code to show iteration-to-convergence
TEN!?!
Look at that. Seriously? a 10' offset? That's ridiculous! There is no WAY our pointing models could be off by that much! Even if I got the signs entirely wrong, that's just not possible. What's going on? First theory: Galactic coordinates fail. Any other suggestions?
Task List
For the moment, I'm going to leave the pointing alone. This leaves us (me) with a very long list of things to do before the data release:
- Determine ideal iterative mapping strategy for low-flux fields
- Determine ideal iterative mapping strategy for fields with bright sources
- Optimize at least the following:
- scan flatting (polynomial subtraction)
- PCA subtraction
- Pre-PCA sky subtraction (necessary?)
- Deconvolution/No deconvolution
- GENERATE MAPS [top priority, but it can't really happen til after the above]
- Assure consistency with catalogs - e.g. Motte, SCUBA, etc.
- Figure out what needs to go into fits header
- Survive semester before comps [lowest priority]
- sleep [oh, right, real lowest priority]
Sarcasm helps me survive. Almost as much as beer.
New pointing models
I decided to try to fit the pointing model as a full alt/az or ra/dec dependent thing using mpfitfun and a 12 (or 16) parameter model. It didn't work very well. I could fit out the alt dependence, then independently fit out the az dependence, and see how that goes. I think it will work brilliantly.
Got a sign wrong again
Last night's run failed because I had a wrong sign in the pointing model. Do over time!
Notice anything different?
About milkyway? This post is a red herring, but: I decided to go ahead and run the mapper on Cygnus, L33, L111, and the Galactic Center again. The pointing has gotten to the stage where I'm certain I can't do anything more without a stroke of pure brilliance or a conversation with someone who hasn't touched a thing - e.g. Jason - that suddenly enlightens me. In order to fully reproduce Meredith's results, I'd probably have to go back through and follow her 'pipeline' process step by step as well, and I suspect that, if I had used her method on her RA/Dec maps, I would have come up with exactly the same problem I currently see. Therefore, I won't do anything about it. I think the most appropriate response at this point MAY be to just fit a damned polynomial/sine curve in az and include that as part of the pointing model, but I can't justify where that comes from. All I know is that it's present in Meredith's data as well as my own. What's that next step with the PPSes? We need to do that. More important, though, is getting some image optimization ready FAST. I need to be running this stuff before September! Tonight's run will be a test of numbers of PCA components. At the very least, the v0.5 should have a consistent set of images even if we cut out the high-flux ones. We'll deal with that later. For the high flux objects, e.g. g34.3, since we have so much overlapping data, a simple average/magical baseline subtraction might be just as if not more effective than PCA subtraction, so that's one way around it.
Modifications
Things that have been changed in the past 24 hours that are very significant:
- In apply_pointing_model, the signs of FAZO and FZAO have been swapped. I BELIEVE this is correct but somehow things aren't working out still.
- In do_the_pointing I have changed from "eq2hor" and "hor2eq" to
"my_eq2hor" and "my_hor2eq". These implement two major changes:
- The LST is passed as a parameter rather than calculated within the ASTROLIB code
- The conversion is calculated with BOTH hadec2altaz and getaltaz (and similar for the opposite transformation) and compared for error checking purposes. If they differ by more than one arcsecond, the code will spit out an error message and use getaltaz.
Inconsistent
James: "I think it's fair to say, though, that there is some problem with the simultaneous assumption that CSO coords are geo and that we are applying the ab/nut correction correctly." Yep. The relevant files are in /scratch/adam_work/plots/: models_noabnut_radec_0707.ps models_noabnut_rawcsoptg_radec_0707.ps models_oppositeabnut_radec_0707.ps models_oppositeabnut_rawcso_radec_0707.ps I'm afraid they're pretty confusing. 'noabnut' means no aberration/nutation correction was applied during the mapping process. 'oppositeabnut' means that an aberration/nutation correction that, according to the eq2hor and hor2eq texts, should actually convert heliocentric to geocentric, is being used on data that we believe is starting in a geocentric frame. There are two possibilities: 1. We are wrong 2. eq2hor/hor2eq are wrong. The 'rawcso' files have FAZO/FZAO "subtracted out" (removed) in pages 1 and 3, and FAZO/FZAO added back in on pages 2 and 4. 'rawcso' means that we're looking at the ra/dec the CSO gave without the users' FAZO/FZAO corrections. The non-rawcso have the NCDF ra/dec vectors, with precession correction applied (and some form of ab/nut correction), but the FAZO/FZAO are still present. THESE should be equivalent to Meredith's plots, e.g. where FAZO_SET/FZAO_SET are on the y axis. However, that's only true for pages 1 and 3. Pages 2 and 4 have FAZO/FZAO essentially double-subtracted, so they should be ignored. Pages 1 and 3 of the non-rawcso files should be equivalent to pages 2 and 4 of the rawcso files with the exception that the sigma-rejection used to choose the yellow points is different. So what's in each page? On pages 1 and 2, the red lines are my best fit to the yellow data, which is the black data with an iterative sigma rejection applied (i.e. reject at 1 sigma, recalculate sigma from good data, reject at 2 sigma). The blue lines are Meredith's models. The right side includes only the yellow data points with the red line subtracted. On pages 3 and 4, the same is pretty much true except that ONLY the altitude-dependent line has been subtracted: there is no fit to azimuth. Also, the 'sourcecompare' files are similar. I'd check those out too. I still don't have the extremely low RMS that Meredith saw. I'm going to go through and try to reject bad data points by hand to see if I can get to that level. We'll see.