Done for the day after this (observing tonight) two new procedures:
alignment_offsets_to_ncdf.pro calcoeffs_to_ncdf.pro
they're really easy, very nearly one-liners. Once they've been used once, you can just use ncdf_varput_scale Modified keyword parameters in do_the_pointing: offsets will always be applied unless the keyword 'no_offsets' is passed into the code (or the top-level wrapper). Since offsets default to zero, this should be OK.
/scratch/adam_work/plots/models_myptgmdl_0707.ps Look at the left plots in #1 and #4, and ignore the red lines. The RMS is 4.87 in alt, 3.51 in az, using all pointing observations in 0707 (i.e. not splitting it into part1/part2 or refining heavily). Total is 6.003", not bad. I'm calling it quits on pointing refinement for the moment; I'm satisfied using Meredith's pointing models to get science maps co-aligned, then we'll use my model on the PPSes to get the pointing to maximum precision. Also, in the galactic center, the mapping works happily now. No more silly problems with that. It's time for a real Next Step. I think that means mapping nearly everything. Well, here goes!
I updated the FITS header writing pieces of the software today, we should now have: N_PCA ITERNUM PPBEAM in all new maps. I'm mapping all of Cygnus degree-by-degree in RA/Dec and Galactic to try to measure offsets... possibly offsets as a function of l/b or ra/dec. I should really be doing this with PPSes, though. My priority list:
I've been mapping all weekend. It turns out that if you do 25 mapping runs that take ~1hr apiece, that takes ~25 hours. Who knew? Whole observation took 3965.3506 sec. Whole observation took 3071.1403 sec. Whole observation took 4205.3337 sec. Whole observation took 7362.5490 sec. Whole observation took 9313.6064 sec. Whole observation took 3755.2383 sec. Whole observation took 3755.8278 sec. Whole observation took 3760.7032 sec. Whole observation took 3758.5719 sec. Whole observation took 3763.1436 sec. Whole observation took 3766.0066 sec. Whole observation took 3789.5806 sec. Whole observation took 3760.9774 sec. Whole observation took 3762.7124 sec. Whole observation took 3763.1120 sec. Whole observation took 3767.1255 sec. Whole observation took 3766.9085 sec. Whole observation took 3758.5867 sec. Whole observation took 3760.2131 sec. Whole observation took 3755.8963 sec. Whole observation took 3759.9199 sec. Whole observation took 3764.4494 sec. Whole observation took 3766.9686 sec. Whole observation took 3756.5876 sec. The good news is that I haven't encountered any crashes. Smooth sailing so far.
Since my gigantic mapping run is still going, I'm going to work primarily on developing code for testing the maps, and determining important information about the mapper. The data paper will require an iteration-to-convergence figure, or at least an estimate of degree of convergence.
-bash-3.00$ less /scratch/adam_work/logs/log_082208_domemiter.log | grep WholeWhole observation took 5122.6795 sec.Whole observation took 4774.3424 sec.Whole observation took 2687.7761 sec.Whole observation took 2102.5577 sec.Whole observation took 3247.5344 sec.Whole observation took 5515.2907 sec.Whole observation took 2877.6389 sec.Whole observation took 2757.0235 sec.Whole observation took 1453.0918 sec.Whole observation took 1459.7892 sec.Whole observation took 1381.3762 sec.Whole observation took 1443.1440 sec.Whole observation took 614.39645 sec.Whole observation took 2001.6709 sec.Whole observation took 1817.1512 sec.Whole observation took 1879.1385 sec.
Note that the longest set there - 5515s = 1.53 hours - was 40 iterations!
For the moment, I'm going to leave the pointing alone. This leaves us (me) with a very long list of things to do before the data release:
Sarcasm helps me survive. Almost as much as beer.
About milkyway? This post is a red herring, but: I decided to go ahead and run the mapper on Cygnus, L33, L111, and the Galactic Center again. The pointing has gotten to the stage where I'm certain I can't do anything more without a stroke of pure brilliance or a conversation with someone who hasn't touched a thing - e.g. Jason - that suddenly enlightens me. In order to fully reproduce Meredith's results, I'd probably have to go back through and follow her 'pipeline' process step by step as well, and I suspect that, if I had used her method on her RA/Dec maps, I would have come up with exactly the same problem I currently see. Therefore, I won't do anything about it. I think the most appropriate response at this point MAY be to just fit a damned polynomial/sine curve in az and include that as part of the pointing model, but I can't justify where that comes from. All I know is that it's present in Meredith's data as well as my own. What's that next step with the PPSes? We need to do that. More important, though, is getting some image optimization ready FAST. I need to be running this stuff before September! Tonight's run will be a test of numbers of PCA components. At the very least, the v0.5 should have a consistent set of images even if we cut out the high-flux ones. We'll deal with that later. For the high flux objects, e.g. g34.3, since we have so much overlapping data, a simple average/magical baseline subtraction might be just as if not more effective than PCA subtraction, so that's one way around it.
Things that have been changed in the past 24 hours that are very significant:
I had removed sigmadeglitch from deline, now it's permanently gone. I think it might be worth exploring re-inserting sigmadeglitch somewhere. My higher-order polynomial fit is creating deeper bowls around sources, so that's a big problem, but if I use a lower order I get the bad streaks. What's the best way to deal with this? I'm thinking perhaps only doing the polysub on the second iteration (i.e. after a source model has been subtracted). Delining runs into some issues now, though, because not all frequencies are sampled (?).
On my huge Cygnus run, the fft keeps failing in the deline code. It's actually pretty impressive, but there are 844604 points in the timestream. The prime factorization of 844604 is 2^2 * 211151. This is just damned bad luck, because I think an FFT is extremely inefficient when it can't factorize. What's the best workaround? .... 9 AM update: I've rewritten the deliner to work on a scan-by-scan basis. It's possible that the delining failed in the past because it was essentially removing a constant amplitude at the line frequencies across the observation (or combined observations!) which is not likely to be true.