To make .* in VIM be non-greedy (i.e. match just <a href=x> out of "<a href=x> </a>"), use:
.\{-}
Pointing model success
/scratch/adam_work/plots/models_myptgmdl_0707.ps Look at the left plots in #1 and #4, and ignore the red lines. The RMS is 4.87 in alt, 3.51 in az, using all pointing observations in 0707 (i.e. not splitting it into part1/part2 or refining heavily). Total is 6.003", not bad. I'm calling it quits on pointing refinement for the moment; I'm satisfied using Meredith's pointing models to get science maps co-aligned, then we'll use my model on the PPSes to get the pointing to maximum precision. Also, in the galactic center, the mapping works happily now. No more silly problems with that. It's time for a real Next Step. I think that means mapping nearly everything. Well, here goes!
prepare_map issues
The problem: There is a bulk offset that can range from small to ~22' (largest observed so far) in the combined maps that is not present in the individual maps. Observations: I THINK the offset between ra/dec and l/b is most pronounced when the map boundaries are most different. We have been putting CRPIX in the middle of the map (i.e. number of x/y elements divided by 2), which is not the same position as the median ra/dec or l/b. I figured this might be a problem, but there's not really a way around it - if you just set CRPIX to be 0,0, it doesn't change anything. xy2ad and ad2xy produce self-consistent results, but that doesn't mean anything either. I think the problem is that AD2XY is largely ignorant of CRPIX: it calculates x/y offsets from the CRVAL assuming that the CRVAL is at CRPIX.... but I don't know why that should be wrong. Ideas? More to come if I figure anything out. Update: from doing the individual maps in l/b vs ra/dec, I don't think the offset is between l/b and ra/dec.... something less fundamental. Update 2: still no clues after reading ad2xy and trying to plow through parts of wcssph2xy. gotta be rect_pix_tstream, but I think it's right.... Update 3: I think I have figured out the problem, currently testing. My idea has to do with the fact that all x,y positions in rect_pix_tstream are positive, but ad2xy is not constrained to return positive values. In principle, the two-iteration ad2xy should take care of this, but in the case where the middle of the map in pixels is not the middle of the map in the WCS coordinate system, it is still possible to get negative pixel mappings out of the 2nd iteration. I'm still confused about whether this is really possible - makes my head fuzzy - but I tried implementing a solution where I simply shift the crpix rather than recalculating with AD2XY. Update 4: Based on L111, my test in update 3 fixed the problem. Scuba contours are correct, and reasonably consistent across the field - still need to do the more complete test, e.g. Cygnus Update 5: Cygnus tests still await completion, but a very close inspection of single L111 maps reveals a ~2 pixel difference that could go a long way to explaining my high-RMS pointing calculations. Still doesn't explain the sine curve, but anything helps... Update 6: My correction definitely fixed the problem, EXCEPT there's still an ambiguity at the .5 pixel level. Specifically, is AD2XY treating the pixel center as .5,.5 or 0,0? Ditto IDL. I believe (from my simulated data) that this ambiguity is still causing a problem. Update 7: New correction fixed problem to better than 1" (checked by making a simulated map with 1" pixels) - probably 'perfect' now. Read prepare_map.pro for details. The issue was a difference in pixel centers.
Confirmed error in L/B mapping
I've used my simulated data and confirmed that there is in fact an error in the l/b mapping, but I haven't tracked it down yet. Stay tuned. Update: Something is not self-consistent in prepare_map. I've tried a variety of different things without success yet.
Updates today
I updated the FITS header writing pieces of the software today, we should now have: N_PCA ITERNUM PPBEAM in all new maps. I'm mapping all of Cygnus degree-by-degree in RA/Dec and Galactic to try to measure offsets... possibly offsets as a function of l/b or ra/dec. I should really be doing this with PPSes, though. My priority list:
- find out what's causing bulk offsets
- beat down noise in my pointing models by rejecting bad sources/points [has to be done from campus]
- test my personal special pointing models
- make maps!
- figure out the rest of pointing
Iterating to Convergence
Some sample plots of flux-vs-iteration number in the Galactic Center. The above is for a 13-pca-component map, and the plot is of the northeast peak of SGR B2. The other plots (available in postscript below) are of two other points in the GC, and the last one is of the whole of SGR B2 (it's a very large aperture). ps version here: http://sites.google.com/site/bolocam/pipeline/mapping/iteratetoconvergence.ps?attredirects=0 or /scratch/adam_work/plots/iteratetoconvergence.ps
G34.3 PCA subtraction comparison
From top-left to the blue highlighted box, the 20th iteration of N pca components subtracted is shown. N goes from 1 to 19. The other plots are 'ascending' and 0-pca components with median and baseline subtraction - ask if you want more details on those. 19 iterations recovers a LOT of flux. I haven't generated the plots for flux as a function of iteration, but I have software to do that available. Things to note: at 10 iterations and higher, you start to generate negative bowls. One observation is probably independently responsible for the chunks of glitchy data present even in the 19 PCA component subtraction. While the bowls do worsen with higher number of PCA components, we also find fainter structure (e.g. those tongues to the southeast and northwest).
Some good news, some.... not news.
I've been mapping all weekend. It turns out that if you do 25 mapping runs that take ~1hr apiece, that takes ~25 hours. Who knew? Whole observation took 3965.3506 sec. Whole observation took 3071.1403 sec. Whole observation took 4205.3337 sec. Whole observation took 7362.5490 sec. Whole observation took 9313.6064 sec. Whole observation took 3755.2383 sec. Whole observation took 3755.8278 sec. Whole observation took 3760.7032 sec. Whole observation took 3758.5719 sec. Whole observation took 3763.1436 sec. Whole observation took 3766.0066 sec. Whole observation took 3789.5806 sec. Whole observation took 3760.9774 sec. Whole observation took 3762.7124 sec. Whole observation took 3763.1120 sec. Whole observation took 3767.1255 sec. Whole observation took 3766.9085 sec. Whole observation took 3758.5867 sec. Whole observation took 3760.2131 sec. Whole observation took 3755.8963 sec. Whole observation took 3759.9199 sec. Whole observation took 3764.4494 sec. Whole observation took 3766.9686 sec. Whole observation took 3756.5876 sec. The good news is that I haven't encountered any crashes. Smooth sailing so far.
Re: Convergence Tests
Made a little code to check out convergence, but frankly it's pretty easy to just build a mapcube and plot lines in the iteration axis. The code is in bgps_pipeline/postproc/compareiters.pro, and it is not a single program. Comparing deconvolution to no deconvolution, deconvolution is a lot better. Without it, there are much more substantial negative regions. In the GC, my test region, the noise-dominated areas were about the same, though the no-deconvolve map had a little bit more large scale structure. The signal-dominated regions were very nearly uniformly brighter. The deconvolved map was ~3 Jy brighter in SGR B2, or 4%.
On the list for today....
Since my gigantic mapping run is still going, I'm going to work primarily on developing code for testing the maps, and determining important information about the mapper. The data paper will require an iteration-to-convergence figure, or at least an estimate of degree of convergence.
- Need to develop code to show iteration-to-convergence
- need it for both NxNpca and ascending/descending
- have to use region files? make some sort of region reader?
- pick demonstration sources
- Compare Npca results for bright, faint sources
- Compare iterating with/without deconvolution
- Compare median/average/baseline sky subtraction
- Analyze efficiency...
- Try to figure out what's causing huge offsets. Is it Galactic Coordinate mapping?
-bash-3.00$ less /scratch/adam_work/logs/log_082208_domemiter.log | grep WholeWhole observation took 5122.6795 sec.Whole observation took 4774.3424 sec.Whole observation took 2687.7761 sec.Whole observation took 2102.5577 sec.Whole observation took 3247.5344 sec.Whole observation took 5515.2907 sec.Whole observation took 2877.6389 sec.Whole observation took 2757.0235 sec.Whole observation took 1453.0918 sec.Whole observation took 1459.7892 sec.Whole observation took 1381.3762 sec.Whole observation took 1443.1440 sec.Whole observation took 614.39645 sec.Whole observation took 2001.6709 sec.Whole observation took 1817.1512 sec.Whole observation took 1879.1385 sec.
Note that the longest set there - 5515s = 1.53 hours - was 40 iterations!
- Need to develop code to show iteration-to-convergence