I started working again yesterday, mostly looking at the 31PCA reductions I did. I have no strong opinions yet on their quality. It's time to work towards a final data release, though, and to that end, I'm going to start removing duplicate fields. e.g. l023 and l024 have identical data, so only one of them should ever be used.
L111 observation notes
In L111, the failures noted in the previous post are specific to September 2005. By comparing the pointing model images to the "raw cso pointing" and "no pointing model" (the latter still have FAZO/FZAO applied, all that has been done to them is aberration/nutation and precession correction) it appears that the Sep 05 model is simply wrong. I haven't had time to check on W3/4/5 and L033, but presumably they have a similar issue. W3/4/5 includes late July 07 and September 06. L033 includes July 05, June 06, and July 07. I will need to generate pointing plots for September 05 and 06, and probably regenerate June 06. I have the rest and I think Meredith's models are fine for those dates - they certainly don't create recognizable errors in the maps. It is possible that my pointing_model.pro has some pointing models (the problematic ones) entered incorrectly. It would be nice to get those double-checked.
Pointing model failure
I attempted to map L111, L033, and W3/4/5 with the pointing model corrections applied. That was a failure. Two images are attached to illustrate the problem - relative pointing is clearly not correct (middle of L33, bottom left of L111). So.... possibilities:
- Pointing models are incorrect
- Pointing model signs have been misinterpreted (don't think this one is possible based on other graphs)
- Uhh... ideas?
Next step, I'll use the individual maps with no ptg mdl and find out what the offsets should be....
Deconvolution vs. Not
I've done some by-eye comparisons of deconvolutions vs no deconvolution. The no-deconvolution clearly does better on the bright sources: probably the deconvolved beam size is a little bit different from (larger? smaller?) the actual source size. Deconvolution does better in putting noise from noisy regions into the noisemap and keeping it out of the astromap. Both might be useful - the deconvolved maps may end up being prettier, but the no-deconvolve maps will probably have more reliable fluxes. This all probably relies on testing / simulation.
4 errors
l047, l083, l359 (l000 is in /usb, l359 is in /scratch), l136p15 Only 83 is a code error, the others are file location errors. l083 is an error in 'pixshift' with subscripting and will require real debugging.
def_user_common modified
The NP doesn't work without USER_COMMON defined because at a single point it makes use of the 'which' function. def_user_common is a disaster, though, because it calls get_screen_size(), which forces an X connection to be launched, which means that if you start a remote session and close X IDL crashes. This has been a constant nagging problem and has cost me ~a few days of work. So I commented out the offending (and offensive) line. The worst of it is, I'm not even convinced there's anything that uses any of that information. The common block is only needed to get paths, so which could have been written better. The pipeline ground to a halt for an unknown reason at an unknown point so I added a lot more timing outputs to try to figure out what's going on. Back to the grind...
flagger
Discovered that the flagger causes some pretty serious problems if you try to run it on a non-coadd for obvious reasons: there's practically no coverage! Individual maps should NOT go through the flagger, so I have added a piece of code that turns off the flagger if only one observation is being used. That code is in map_iter where the flagger is called.
v0.6.1 done
Version 0.6 begun
I started on (what I hope will be) v0.6 tonight. New things: -Sigma-rejection flagging in the spatial domain -Inverse-variance weighting
MAD flagger
; The Mad Flagger; Flag based on the median average deviation within a spatial pixelfunction mad_flagger,data,inds,flags,nsig=nsig t = systime(/sec) f0 = total(where(flags)) if n_e(nsig) eq 0 then nsig = 3 newflags = flags mx=max(inds) vec3=fltarr(mx+1) h=histogram(inds,reverse_indices=ri,OMIN=om) for j=0L,n_elements(h)-1 do begin if ri[j+1] gt ri[j] then begin v_inds = [ri[ri[j]:ri[j+1]-1]] if n_e(v_inds) gt 2 then begin vec = data[v_inds]; vecmad = mad(vec) ; the MAD is WAY too small! I ended up rejecting 8% of points! vecmad = stddev(vec) vecmed = median(vec,/even) madreject = where( (vec gt vecmed + nsig*vecmad) or (vec lt vecmed - nsig*vecmad) ) if (n_e(madreject) gt 0 and total(madreject)) gt 0 then begin reject_inds = v_inds[madreject] newflags[reject_inds] = 1 endif endif endif endfor print,"MAD flagger took ",strc(systime(/sec)-t)," seconds and flagged ",$ strc(round(total(where(newflags)) - f0)),' points' return,newflagsend