I still see distortion in a lot of maps, but not all. I think there is a correlation with altitude ~70+/-2 degrees. So far it's most obvious in the 0606 pointing sources.
Occasional bad centroid
I've gotten some bad centroids for 3c273 that don't make sense.
0507 mapping
Most of the 0507 pointing source maps seem to have failed. Some of them look like rotator and position angle problems, others have multiple copies of sources mapped to different locations. I don't know what's up, but my first bet would be to change the fiducial array angle. After that, I'd check on rotang and then, if desperate, see what the PA is doing.
Mapped, but no dice
Finished mapping what I think are all of the pointing sources. However, the RMS of the centroids has not gone down. Now I need to go through each individual source and image and try to get rid of the baddies. There are presumably some sources with bulk offsets that can be removed to improve the fits. I just have to find them.
Nothing
No progress today. Spent the whole day re-mapping pointing sources and figuring out which observations ARE pointing sources. Mapping STILL not done - it takes a long time when there are hundreds of separate observations. Also, finding some corrupt files and noting them in /scratch/adam_work/texts/toolarge.txt . A lot more of that work to be done.
Observation distribution
-bash-3.00$ wc texts/all_*_observations.txt 0 0 0 texts/all_0505_observations.txt 0 0 0 texts/all_0506_observations.txt 151 151 6039 texts/all_0507_observations.txt 33 33 1377 texts/all_0509_observations.txt 16 16 633 texts/all_0605_observations.txt 135 135 5309 texts/all_0606_observations.txt 0 0 0 texts/all_0607_observations.txt 94 94 3780 texts/all_0609_observations.txt 34 34 1386 texts/all_0705_observations.txt 26 26 1028 texts/all_0706_observations.txt 201 201 7989 texts/all_0707_observations.txt 73 73 2907 texts/all_0709_observations.txt 0 0 0 texts/all_polychrome_0505_observations.txt 222 222 11898 texts/all_polychrome_0506_observations.txt 471 471 24861 texts/all_polychrome_0507_observations.txt 196 196 10346 texts/all_polychrome_0509_observations.txt 43 43 2200 texts/all_polychrome_0605_observations.txt 730 730 38040 texts/all_polychrome_0606_observations.txt 22 22 1195 texts/all_polychrome_0607_observations.txt 471 471 24751 texts/all_polychrome_0609_observations.txt 66 66 3399 texts/all_polychrome_0705_observations.txt 145 145 7533 texts/all_polychrome_0706_observations.txt 1051 1051 54801 texts/all_polychrome_0707_observations.txt 511 511 27049 texts/all_polychrome_0709_observations.txt 4691 4691 236521 total
News, modifications
RA / Dec mapping did change centroid locations, but not really for the better. A deeper analysis is probably necessary, but nevertheless I'm not convinced mapping type is the problem. The list of possible answers to the question of why my pointing models don't match Meredith's:
- The new mapping is shifting the centroid
- The centroiding method I'm using is different / incorrect
- There is still something in the pointing we haven't caught
The lack of systematics suggests that the third option is not correct. The first seems the most likely, but also the most difficult to track down. I don't know where to go with #2. I've updated the code so that the pointing model correction offsets in RA/Dec are written to the FITS header. This will be the standard in all future runs for individual observations. The calculation of this offset post-mapping is straightforward but it's nice to have a redundant error check. The report for Monday will be kind of empty, sadly. However, I think I can say that I'll now move on to testing the previous problems we faced, in particular that we could not match the pointing across whole fields (L111). If THAT is fixed, then at least we know that whatever my pipeline is doing differently (e.g. PA at all times...) is useful if not 100% correct. I'll need to be in the office to test the field mappings, though, because ds9 display doesn't work well over wireless. I'll do a remote run of the individual L111 and W5 maps to compare to previous ones.
Pointing update
I believe I've completely dealt with the projection issues. There are a lot of new plots available now, pretty much everything for Jun/Jul 05 and July 07. Low on my to-do list is the rest of the pointing model tests. My conclusion now is that I can't match the 3-4" RMS in dAlt and dAz. Even with aggressive sigma-rejection of outliers, I only get down to ~10" RMS (which adds to ~15" RMS). My calculated pointing models differ significantly from Meredith's, but not extremely so for 0707. The code I've used to generate the plots is centroid_plots.pro under plotting/. Using that code, I am able to essentially reproduce each change made in do_the_pointing besides the distortion mapping. Because of the weak dependence of the pointing model on altitude, even errors on the scale of precession won't really hurt the pointing model calculations. The only unanswered question I think may be responsible for some errors - but not enough, I think - is whether eq2hor/hor2eq could be double-correcting for aberration and nutation, but this wouldn't affect the actual data reduction. It WOULD affect my plotting! I think my next test is to try to coadd different epochal data for particular fields, e.g. l001, l033. Not l000 because of the galactic coordinate issue.
Update - Pointing Model
Update - Pointing Model 7/31/08 The pointing model application "works" now, but I have a problem with our model. We don't get down to the <10" RMS offset we're looking for. The mean offsets are all zero, but the spread is more like 20". Example plots on milkyway: [see attached PDFs too]
- ::
- /scratch/adam_work/plots/models_ptgmdl_0506.ps /scratch/adam_work/plots/models_ptgmdl_0707.ps /scratch/adam_work/plots/models_rawcsoptg_0506.ps /scratch/adam_work/plots/models_rawcsoptg_0707.ps
On the left side are plotted "all" of the data points, on the right I've used a two-iteration 3-sigma rejection to eliminate outliers to a small degree.
Top of these plots - as labeled - is altoff vs. alt, bottom is azoff vs. alt.
The red lines are a 2nd order polynomial fit to the data.
The cyan lines are the pointing model corrections calculated by Meredith.
The 'ptgmdl' files have had the pointing model corrections applied. Note that they are centered around offsets of zero.
The 'rawcsoptg' files DO NOT have pointing model corrections applied, and FZAO/FAZO have been REMOVED from the original pointing. Hence, these are RAW CSO TELESCOPE POINTING plots.
Things to note:
- In the 0707 'ptgmdl' set, the az is not quite centered at zero
- the 0506 'ptgmdl' set still has bulk offsets
- the ALTOFF and AZOFF are in delta-coordinate: this means that azoff
should be scaled by dividing by cos(alt) to put the y-axis in consistent units. In most cases, this means that the already large spread at higher altitudes is going to INCREASE. That means the problem is going to get worse... Questions:
- What is the main difference between my plots and Meredith's? i.e. why is my RMS ~an order of magnitude larger?
- Does more outlier rejection make sense? Is there any reason not to trust certain observations if they visible turn out right? PPSes are supposed to be "absolute references" for the 1x1, 3x1 maps OLD PROBLEM: PPS and large maps mapped with same 'pointing model' but ended up with different coordinates
Milkyway /scratch disk usage
-bash-3.00$ du -h --max-depth=1 ../16K ../lost+found80G ../sliced1.3G ../elatov70G ../cleaned58G ../mapped179G ../sliced_polychrome4.0K ../ironsides49G ../coadd_mapped2.1G ../pca_coadd2.8G ../3pca_3iterations11M ../centroid17M ../lost34M ../ptg_maps3.8G ../montage31G ../maps_from_polychrome93M ../ptg_mmd32G ../backup_from_kilauea31G ../sharc1.8M ../distortion6.9G ../fake138G ../adam_work136G ../bgps_dir_from_polychrome815G ../