Here's the list from 0507: Streaky offsets: /scratch/adam_work/g34.3/rawcsoptg050706_o47_raw_ds1.nc_indiv3pca_map00.fits .. image:: http://3.bp.blogspot.com/_lsgW26mWZnU/SJtP9wKu4_I/AAAAAAAADNo/gmMEFkzCX-A/s320/050706_o47.png Totally messed up (two sets of images many degrees apart): /scratch/adam_work/g34.3/rawcsoptg050713_o40_raw_ds1.nc_indiv3pca_map00.fits 070709_ob7 : .. image:: http://4.bp.blogspot.com/_lsgW26mWZnU/SJuIDL161GI/AAAAAAAADN4/D4mxFyx1Fr0/s320/070709_ob7_1730m130_peanut.png 070703_o48 : .. image:: http://4.bp.blogspot.com/_lsgW26mWZnU/SJuICzavE_I/AAAAAAAADNw/70Hl5930eU8/s320/070703_o48_3c454.3.png
Looking back at the old pipeline
I've obviously missed something. So to try to figure out what it is, I'm going back to the old pipeline... again... in map_ncdf_reading, lines 441-448, there is something curious that goes back to a definition-of-variables problem: ddec and dra are ADDED to ra and dec to get the new "ra_all" and "dec_all" variables. ddec and dra are calculated from eaz and eel: ERROR OFFSETS in Az and El. Why? What?! I added a new piece of code, correct_eaz_eel.pro. It is extremely short, but extremely necessary.
pro correct_eaz_eel,ra,dec,el,az,eel,eaz,pa dra=-eaz*cos(!dtor*pa)*cos(!dtor*el)+eel*sin(!dtor*pa) ddec=eaz*sin(!dtor*pa)*cos(!dtor*el)+eel*cos(!dtor*pa) dec += ddec/3600. ra += dra/3600. / cos(!dtor*dec) / 15.end
This comes back to the fact that I don't know what ANY of the variables in the NCDF header are supposed to be. Why are "error" variables actually OFFSET variables, and why didn't anyone know about them?
something is wrong in 0507....
Look at that zoomed in. Note that the first 20ish are from 0507. I don't know what's causing the double source. Maybe a 90 degree rotation would do it, I don't know. I do know that I'm applying an array angle of 76.2 degrees for those July observations and 113.6 degrees for the June 05 observations. Maybe that got swapped or something.... It looks like Meredith ran into the same problems when mapping 0506, but not 0507.
comparing mine to hers (I even used the same downsampled/cleand file), it is obvious that something went wrong with the mapping. I tested a number of different fiducial array angles and none of them generated real maps, so the mapping is correct except for offsets in the scan direction, as usual. What the heck.
Distortion (not distortion maps)
I still see distortion in a lot of maps, but not all. I think there is a correlation with altitude ~70+/-2 degrees. So far it's most obvious in the 0606 pointing sources.
Occasional bad centroid
I've gotten some bad centroids for 3c273 that don't make sense.
0507 mapping
Most of the 0507 pointing source maps seem to have failed. Some of them look like rotator and position angle problems, others have multiple copies of sources mapped to different locations. I don't know what's up, but my first bet would be to change the fiducial array angle. After that, I'd check on rotang and then, if desperate, see what the PA is doing.
Mapped, but no dice
Finished mapping what I think are all of the pointing sources. However, the RMS of the centroids has not gone down. Now I need to go through each individual source and image and try to get rid of the baddies. There are presumably some sources with bulk offsets that can be removed to improve the fits. I just have to find them.
Nothing
No progress today. Spent the whole day re-mapping pointing sources and figuring out which observations ARE pointing sources. Mapping STILL not done - it takes a long time when there are hundreds of separate observations. Also, finding some corrupt files and noting them in /scratch/adam_work/texts/toolarge.txt . A lot more of that work to be done.
Observation distribution
-bash-3.00$ wc texts/all_*_observations.txt 0 0 0 texts/all_0505_observations.txt 0 0 0 texts/all_0506_observations.txt 151 151 6039 texts/all_0507_observations.txt 33 33 1377 texts/all_0509_observations.txt 16 16 633 texts/all_0605_observations.txt 135 135 5309 texts/all_0606_observations.txt 0 0 0 texts/all_0607_observations.txt 94 94 3780 texts/all_0609_observations.txt 34 34 1386 texts/all_0705_observations.txt 26 26 1028 texts/all_0706_observations.txt 201 201 7989 texts/all_0707_observations.txt 73 73 2907 texts/all_0709_observations.txt 0 0 0 texts/all_polychrome_0505_observations.txt 222 222 11898 texts/all_polychrome_0506_observations.txt 471 471 24861 texts/all_polychrome_0507_observations.txt 196 196 10346 texts/all_polychrome_0509_observations.txt 43 43 2200 texts/all_polychrome_0605_observations.txt 730 730 38040 texts/all_polychrome_0606_observations.txt 22 22 1195 texts/all_polychrome_0607_observations.txt 471 471 24751 texts/all_polychrome_0609_observations.txt 66 66 3399 texts/all_polychrome_0705_observations.txt 145 145 7533 texts/all_polychrome_0706_observations.txt 1051 1051 54801 texts/all_polychrome_0707_observations.txt 511 511 27049 texts/all_polychrome_0709_observations.txt 4691 4691 236521 total
News, modifications
RA / Dec mapping did change centroid locations, but not really for the better. A deeper analysis is probably necessary, but nevertheless I'm not convinced mapping type is the problem. The list of possible answers to the question of why my pointing models don't match Meredith's:
- The new mapping is shifting the centroid
- The centroiding method I'm using is different / incorrect
- There is still something in the pointing we haven't caught
The lack of systematics suggests that the third option is not correct. The first seems the most likely, but also the most difficult to track down. I don't know where to go with #2. I've updated the code so that the pointing model correction offsets in RA/Dec are written to the FITS header. This will be the standard in all future runs for individual observations. The calculation of this offset post-mapping is straightforward but it's nice to have a redundant error check. The report for Monday will be kind of empty, sadly. However, I think I can say that I'll now move on to testing the previous problems we faced, in particular that we could not match the pointing across whole fields (L111). If THAT is fixed, then at least we know that whatever my pipeline is doing differently (e.g. PA at all times...) is useful if not 100% correct. I'll need to be in the office to test the field mappings, though, because ds9 display doesn't work well over wireless. I'll do a remote run of the individual L111 and W5 maps to compare to previous ones.
