A demonstration that the spatial transfer function can be improved to ~80% at ~300 arcseconds by reducing the number of PCA components subtracted. However, image fidelity and noise backgrounds suffer, so these sorts of maps are probably less ideal for point source extraction. Below:
0 PCA 3 PCA 10 PCA 15 PCA
Last report was a bit of a fiasco. There were problems all over the place I didn't understand. I still don't but I've fixed them. My best guess at this point is that a pass-by-reference led to an unacceptable modification of an image. That doesn't even make sense - there was no place it could have happened - but, there you have it. So, going through the process step by step. This is the effect of smoothing an image:
Note from the 3rd figure that 100% recovery isn't reached until ~700 arcseconds. Next question: What is happening at large spatial scales in the flat-spectrum simulations?
No obvious problems there.
Hmm, no apparent problem here either, though one might ask why the two curves approach each other in sky05 (alpha=-0.5). So it appears that the reason for the bump up at low frequencies (long wavelengths) must be because of edge effects. After much hassle, I've addressed that by cropping images. Finally, the averaged results:
So we've got an Official Spatial Transfer Function. However, of course, we must note that there is a dependence on the atmosphere amplitude to source amplitude ratio: it appears that large-scale structure is *easier* to recover when the atmosphere is at higher amplitude. This makes sense: it is easier to distinguish faint astrophysical signal from bright atmosphere in this case. The reason I didn't run simulations to test this more is that the S/N ratio on small scales becomes poor for the low astrophysical amplitudes.
Below are plots of the spatial transfer function for a variety of simulation parameters. Each line represents the median (i.e., mean of the center 2 of 4) spatial transfer function of 4 realizations of the listed parameters.
A few unfortunate details are apparent:
Note for comparison that the recovery fraction for point sources is much more nearly 1, at least in the calibrator type maps. We need to wait for Jared's simulations with point sources in a full size map to confirm this, but it seems pretty likely that the STF is different for point sources and extended structures.
Just in case we were wondering, the V1 bolocat is completely inconsistent with a lognormal distribution, but is perfectly consistent (or... at least reasonably consistent...) with a power law distribution.
These plots show power-law fits (red) and lognormal fits (blue) to the data (black). It's pretty obvious that the lognormal is a bad fit, but in case you're unconvinced, the ks test for the "source flux" has a probability 1.6e-7, and it is the highest likelihood by 17 orders of magnitude out of the 4 flux types. By contrast, the simulations are on average (though not uniformly) more consistent with lognormal than powerlaw distributions:
Even in those examples where the KS test is slightly more favorable for the powerlaw distribution, the lognormal is a pretty good fit, and the failure points for the two distributions are in about the same place. The smoothness of the lognormal distribution is required to reproduce the observed distribution.
Note that the first 4 plots are for the whole BGPS survey. What about an individual field? For obvious reasons, I choose l30 again.
This gets to be a little more interesting - apparently the "source flux" has a tendency to pick up the power-law distributed background structure, since it is consistent with a lognormal (but note that it is also consistent with a powerlaw! The ks test doesn't really say definitively which is better). Although the fits look bad at high flux, note that this is a log-log plot and therefore the difference in probability is rather small. What does this all indicate? It's not entirely clear whether individual fields are genuinely more lognormally-distributed or whether the number statistics are just worse. However, even the source flux is consistent with a power-law, while many realizations of the simulations are not. Therefore, we should perform the next logical test - add point sources drawn from a power-law distribution (and a log-normal distribution?) and see what bolocat retrieves. We can at least say now that the point source contribution cannot be ignored, since there is no power-law distribution that can reproduce the observed bolocat flux distribution.
This figure shows the flux recovery of bolocat. The Y axis shows the Input flux (green) and the Filtered flux (blue/cyan/purple) with a few different large-scale filters. The red line is the 1-1 line. Obviously, bolocat doesn't recover everything that was there - we removed a lot of flux, so recovery fractions are in the few % range. However, bolocat does a much better job than the simple filter at recovering positive fluxes. Therefore, the filter function still isn't good enough.
that's all.
I still haven't dealt with http://bolocam.blogspot.com/2011/07/additional-problems.html, or essentiall any of http://bolocam.blogspot.com/2011/07/minor-ongoing-problems.html. However, they're probably related to this problem: % CLEAN_ITER_STRUCT: There were 1 bolometers with high weights: 13.5047 indices: 142 or flagged indices 142 which indicates that in a simulation in which there can be no outliers (in terms of weight/scale), there is one being rejected as an outlier. That indicates that weights are being computed incorrectly, despite the fact that scales look right (so far): Relsens calibration: Scaled to bolometer # 0% RELSENS_CAL_PCA: There were 0 NAN scales and 0 very low scales% RELSENS_CAL_PCA: Scale Median/Mad: 1.0005510+/- 0.010990833 led to 0 scales set to zero for a total of 0 bad bolos% RELSENS_CAL_PCA: Scales avg+/-std = 1.0007304355062783+/- 0.0103252880998329Relsens calibration: Scaled to bolometer # 0% RELSENS_CAL_PCA: There were 0 NAN scales and 0 very low scales% RELSENS_CAL_PCA: Scale Median/Mad: 0.99955294+/- 0.0093122063 led to 0 scales set to zero for a total of 0 bad bolos% RELSENS_CAL_PCA: Scales avg+/-std = 0.9990063123041220+/- 0.0096663438876613
Problem with yesterday's work:
A*B = IFT(FT(A)*FT(B))A*B - A*C != IFT(FT(A)*(FT(B)-FT(C)))
If instead of yesterday's "bandpass filter" you use the convolution - convolution 'filter', the agreement in real space and most of fourier space is much better:
though map20 clearly reproduces some structures better (higher) but is overall less powerful than the filtered map.
I've attempted to model the spatial filter function as a gaussian (or PSF) plus an inverse gaussian. i.e., the high-spatial-frequency components are smoothed with the PSF, and the low spatial frequency components are convolved with a (1-gaussian) high-pass filter. First, the mildly good news: With a 300" FWHM large-scale cutoff, the filter PSD reasonably resembles the iterative map PSD:
Luckily, the double-filter goes a very long way in explaining the scale-free flux loss. In the following diagram, I show the effect of the filter compared to the input map.
The filter only recovers about 75% of the flux at ANY wavenumber. The map does slightly worse at high frequencies, which I can't explain yet. These show the recovery fraction of the iterative maps, a gaussian smoothing function with FWHM=33", and the mid-pass-filter. Map20 (no smooth) has a lot of additional "noise power" at high spatial frequencies; if it wasn't for the telescope filter function, we would apparently have pretty good high-frequency recovery. Hmph. Note that map20 is higher than the filter at some intermediate frequencies, but quite a bit lower at higher frequencies. Also note the moderately poor agreement between the 'smoothed' and 'smoothed (theory)' lines.
Finally, look at the comparison between map20 and fiiltered. The agreement is not bad for positive points; filtered is apparently slightly higher but that can be adjusted. The problem: the filter forces some structures that are negative or zero to be positive. For example, look at the feature at 210,300 that is negative in Map20 but positive in Filtered. In the real (input) map, this feature is lower than its surroundings - it is legitimately negative.
I raised the possibility that the ratty edges of a scan could affect the power spectrum measurements, leading to a scale-free power loss. This doesn't appear to happen... instead, the masking adds some small-scale power and apparently a small amount of large-scale power.
