Downsampling by a factor of 3 appears not to be an option because this error: % NCDF_VARGET: Requested read is larger than data in dimension 0. Reducing COUNT to 615. % REBIN: Result dimensions must be integer factor of original dimensions % Execution halted at: NCDF_VARGET_SCALE 133 /Users/adam/work/bgps_pipeline/support/ncdf_varget_scale.pro occurs.
More evidence that downsampling causes problems
The captions are pretty much the same as for the previous post, but this is a larger field and the effects are more serious.
Downsampling has serious negative effects on the data
Background: Downsampling is performed using Old Pipeline code called process_ncdf. All BGPS data was downsampled by a factor of 5 before mapping because of data size concerns. I did this 'blindly' (i.e., just accepted that I should) because James said I could. However, I had previously noted that the pointing files could not be done with downsampled data because the beams 'looked funny' or something along those lines; it may also have been a simple map sampling issue in which not all pixels were filled with a downsampled image. Anyway, I decided to go back and quantify the effects. The plots below are from the single "pointing-style" observation of OMC1 from 2009. The units are volts. 'ds1' indicates sampling at 0.02 seconds, 'ds5' indicates sampling every 0.1 seconds. The scan rate was 120"/s.
The beam sizes were measured from the autocorrelation maps. However, because there is structure on many scales in this map, I had to use a rather ad-hoc method to remove the correlated structure. I fitted a gaussian to the elliptical northwest-southeast structure, removed it, then fitted a gaussian to the remaining circular thing in the center, which is approximately the beam. If I fit the "beam" gaussian with an ellipse, I get: Beamsize 1_1: 36.15,26.23 Beamsize 1_5: 48.39,30.21 With a circle: Beamsize 1_1: 29.51 Beamsize 1_5: 35.31
The ds1 and ds5 images compared.
The PSDs of the two images (on identical grids). Note that ds5 loses power at small spatial scales, 50% at 40"!
The pixel-pixel plot with a fit that shows a 10% overall flux loss (best-fit).
BGPS data paper published
*[[http://irsa.ipac.caltech.edu/data/BOLOCAM_GPS/bgps_methods.pdf The Bolocam Galactic Plane Survey I. Survey Description and Data Reduction] `[`[http://arxiv.org/abs/1011.0691 arXiv]`]` *[[http://adsabs.harvard.edu/abs/2010ApJS..188..123R The Bolocam Galactic Plane Survey II. Catalog of the Image Data] `[`[http://arxiv.org/abs/0909.2871 arXiv]`]` *[[http://adsabs.harvard.edu/abs/2010ApJ...717.1157D The Bolocam Galactic Plane Survey III. Characterizing Physical Properties of Massive Star-Forming Regions in the Gemini OB1 Molecular Cloud ] `[`[http://arxiv.org/abs/1005.4969 arXiv]`]` *[[http://adsabs.harvard.edu/abs/2010ApJ...721..137B The Bolocam Galactic Plane Survey IV: λ = 1.1 and 0.35 mm Dust Continuum Emission in the Galactic Center Region]
v2.0 is to v1.0 as ___ is to v1.0
Apparently v2.0 recovers flux better at all scales than v1.0. However, it is noisier. The full comparison is available here. The trustworthy range of flux scales is 1.2-1.6ish. The flux is increased at ALL scales in most images. However, as was noted in comparisons with MAMBO, there may be curvature in the best-fit lines. Also, l=35 does not appear to scale up linearly... G34 is fainter in v2.0? In the low S/N fields, the PSD is more reliable than the pixel-pixel comparison.
PPS comparison completed
The PPS scale factor seems to hold for many different (and ridiculous) aperture sizes. The full results are here.
PPS analysis: suggests a possible solution to the discrepancy
The comparisons I mentioned in the previous post are sort of done. They are pretty suggestive of a solution to the "flux recovery problem" we think must be true. However, even if it is a solution, it doesn't really solve the problem completely. It looks like v1.0 should be scaled up by a factor of 1.3-1.4 (not 1.5). v2.0 is consistent with the PPS sources to within 5%, and might even be slightly too high. The comparison was done by taking the flux in a 60" radius aperture (equivalent to bolocat 120" diameter apertures) and subtracting off the background measured in a 120" radius annulus around the source. Without the background subtraction, these numbers would look very different: in the science fields, most of the sources sit on an extended background. Even though the "background flux" isn't recovered in the PPS fields, it should contribute to the source background because it is involved in the atmosphere subtraction (it's sort of "already subtracted" so you have to subtract from the science fields).
Next step: direct comparison between v1.0 and v2.0. Pixel by pixel, aperture, and powerspectrum
PPS analysis
I've started looking at PPS fields to see if I can glean any additional information about the "flux discrepancy" from them. However, the results are, as usual, unenlightening. There is no consistent increase in flux when 3 PCA components are used instead of 13 PCA components - very plausibly an indication that 13 PCA is not too much to subtract because it's only atmosphere. Similarly, there is no obvious benefit to using a quadratic sky estimator instead of a PCA estimator. I'm using aperture photometry (without background subtraction) on identical fields to perform these comparisons. I've selected (arbitrarily) the l357pps source as my comparison source. The next step (ongoing) is to compare to the co-added maps and crosshatched large-scale maps of the same field. (next step) PPS < single cross-hatched large-scale observation pair < 13PCA full combined map < 3PCA full combined map. Unfortunately, this result implies that the small maps under-recover flux, which suggests that the large maps are too bright, which is the opposite of what we expect. Additionally, lower noise -> more flux recovered? When background subtraction is included, the 3PCA and 13PCA fluxes match nearly perfectly. Despite the failure of this test (PPS < full field), I will do a systematic comparison of PPS sources with 0PCA + masking to the large fields in the hopes that doing so can provide a legitimate estimate of the "scale factor" from treating small and large fields differently.
BGPS Pipeline Published
The BGPS pipeline has officially been released on Google Code.
Aperture Photometry on Herschel-based simulation
Aperture Photometry on isolated and not-so-isolated sources in the Herschel-based BGPS simulation using the L=111 field for the "noise". Depending on the aperture, our flux recovery can be really really low. The images should give an idea of the S/N. Background subtraction means subtracting the median of the image.... it works frighteningly well in most cases.