Adam Ginsburg's blog

• Making maps faster

  The fundamental problem at this point is making the pipeline run faster. At current speeds, with undownsampled data, it may take ~days to process a single map. Ideas for faster processing:

  1. Find out how long it takes to converge to 1%, 5%.... If it only requires 10 iterations, that's a factor of 2 savings over current strategies.
  2. Use downsampled data of some sort if possible. Does DS2 match DS1? How do we measure it? Flux-flux comparison and PSF point-source size measurements are the most important. Need to automate PSF comparison....
  3. Can we use median-combined individual images as a 0th order model? I bet the answer is 'yes' and will probably increase the speed of convergence by a large amount. Tests to run? This is probably needed if we are to split up the 'super-fields' into smaller sub-fields, otherwise overlapping data will be used less effectively.
  4. Find some way to keep bgps.raw, bgps.ra, bgps.dec, and other items that are only used once on the HD during the iterative process. Is there any way to separate out data in a struct in this manner?

---

• Delining - Maps

  First comment - delining has no effect on downsampled data. At least for the 0709 epoch, there were NO lines AT ALL in the data. From 0-5 Hz, it was just empty. So we don't have to worry about that... the problem only affects fully-sampled data. Then, onto map comparisons. Curiously, the noise levels don't drop after delining. They actually go up a bit. This may be because of the effects on PCA cleaning. However, flux levels in the sources go up by 0-10%. As usual, the change in flux changes from field to field without any obvious reason. Example 1: A pointing field. The source is ~2% brighter in the delined version, but otherwise the match between the two is nearly perfect.

  

  Example 2: A bigger map, where the flux recovery is much greater when delining, but the background levels are also higher.

  

---

• Delining and the Cleaning process

  One item I forgot to mention last night was the effects of lines/delining on PCA subtraction. These should be the primary effects on the final map for all epochs except 2010, in which case the primary effect SHOULD be to reduce substantial noise. In the examples below, there are PSDs of whole timestreams (left) and example timestreams from single scans (right). The first thing to note is that the delined timestreams still have correlated components in the line region, but they are suppressed - their amplitudes, and therefore their sort order in the PCA removal scheme, are changed. Since PCA cleaning is by its nature adaptive (the number of components remains fixed, but the order changes), these effects can be significant and dangerous. If the line noise is more correlated, a PCA component will be dedicated to removing it instead of atmospheric signal. Below are examples from l089 (epoch 0709) first. These have less correlated line noise and are more typical of BGPS observations. The first PCA component, the average, does not change much with PCA cleaning. However it is clear that the second component changes substantially, from large-amplitude high-frequency noise to small-amplitude variations that are very likely to describe atmosphere.

  ---

  Example 1 - Zeroing out the lines:

  

  Example 2 - Fitting and removing the lines:

  

  Example 3 - Suppressing the lines with a non-fitted Gaussian:

  

  ---

  The next examples are from December 2010 observations of Uranus. In this case, the correlated noise component is clearly dominant.

  Zeroing lines:

  

  Fitted lines:

  

  Non-fitted gaussian suppression:

  

  ---

  Finally, these two are demonstrations of what you might expect to see for a purely noiseless images of a planet (it was constructed from a PSF). PCA is first:

  

  A single bolometer's timestream and PSD:

  

---

• Delining

  Delining refers to the process of removing electronic noise that is aliased to particular frequencies by the discrete sampling of the data. Typical electronic noise is at 60 Hz with some width. It gets aliased to these frequencies: linefreqs = [10.05+findgen(10)*1.2,10.05-((findgen(7)+1))*1.2] The 10.05 Hz and 20.10 Hz are the worst in most cases, and they are wider. For the above lines, we "remove" a bandwidth of 0.09 Hz by averaging over the neighboring 0.5 Hz on either side. For the wider lines, we remove 0.5 Hz by averaging over the neighboring 1.5 Hz on either side. There are a few new things about the delining process that did not exist in James' old version:

  1. The wide-band lines are removed
  2. A check is performed before removing the lines - they are only removed if the line region mean is 2-sigma above the average region (as computed via median and mad)
  3. The replaced noise is computed via median/mad, and the new noise level is set 5x lower than in the neighboring region

  ---

  Now, some examples:

  

  The timestream (left) and PSD (right) of a "pretty good" bolometer. There is a lot of noise in lines, but note that the peak power is 2-3 orders of magnitude below the PSD peak. In this case the "Power" is in Jy. There is little astrophysical information below ~14" (9 Hz), and there should be none below 7.2" (17 Hz), but there is plausibly information at these scales. It therefore makes sense to save as much information as possible. As can be seen in the delined PSD, the peaks drop by a substantial fraction, but not all the way - that's because these lines are wider than typically observed. Unfortunately, I don't have any really good ideas about how to fix this issue - I think fitting gaussians to each line, while attractive, is going to be prohibitively expensive in both programmer and processor time. However... it would be a very interesting project to undertake. In the timestream, it can be seen that the RMS drops substantially when the lines are filtered out (note that the timestream is strongly dominated by large-scale structure, so 'substantial' is really based on the RMS of the lines removed).

  

  The next two plots look identical to the previous ones. In principle, they include the wide-band delining. However, in this case, the satellite lines to either side of the 10 Hz line are too strong and prevent identification of the 10 Hz line. This is unfortunate, again, but no obvious solution presents itself.

  

  Now we come to a truly problematic bolometer. The lines completely dominate the power spectrum. Narrow line removal fails.

  

  Wide line removal does a much, much better job, dropping the RMS by an order of magnitude.... but the bolometer probably still needs to be removed, since the astrophysical signal is 2-3 orders of magnitude below that.

  

  The 2010 data had much worse line noise and had to be delined. JS accomplished this by throwing out all data above a certain frequency, but I prefer the delining approach. It is clearly effective, but again leaves much to be desired. Should the flagged bandwidth be increased? What about the extra lines around 18 Hz?

  

  Again, the wide line flagging fails because of the satellite lines.

---

• Some pretty neat things about delining....

  Well, I did the work, so might as well come up with some plots... I created a gaussian-fitting based delining code that saves all of the fits in a text file! That's a lot of data to play with, and allows me to draw some conclusions:

  1. For the narrow lines, the linewidth (gaussian sigma) is 0.03 Hz. For the wide lines, it is 0.05.
  2. The amplitudes from December 2010 are ENORMOUS compared to others!
  3. The width distributions of the two are similar, though the lines appear to be significantly wider in July 2009

  I've also gotten to the point that I'm satisfied with how delining with wing suppression works. Wing suppression is 0.4-2x quicker than fitting directly, depending on... not clear what exactly; it depends on epoch, but that could either be because there are more lines found or because of relative data size. Does delining have to be done on whole timestreams, or can it be done on a scan-by-scan basis? This is not entirely clear... the peaks are less suppressed when done on a scan-by-scan basis (presumably because the S/N is low and peaks are therefore not detected), but incorrect suppression (removing a line that's not there) is reduced. For short scans, the scan-by-scan approach is 10x faster; for longer scans it's ~20% faster. I think the scan-by-scan approach is going to be ideal.

  ---

  Examples from l089 (0709): Top row: non-fitted line suppression

  

  Bottom row: fitted line suppression

  

  ---

  Examples from Uranus (1012, really really liney):

  Top row: non-fitted line suppression

  

  Bottom row: fitted line suppression

  

---

• More evidence that downsampling causes problems

  The captions are pretty much the same as for the previous post, but this is a larger field and the effects are more serious.

  

---

• Downsampling has serious negative effects on the data

  Background: Downsampling is performed using Old Pipeline code called process_ncdf. All BGPS data was downsampled by a factor of 5 before mapping because of data size concerns. I did this 'blindly' (i.e., just accepted that I should) because James said I could. However, I had previously noted that the pointing files could not be done with downsampled data because the beams 'looked funny' or something along those lines; it may also have been a simple map sampling issue in which not all pixels were filled with a downsampled image. Anyway, I decided to go back and quantify the effects. The plots below are from the single "pointing-style" observation of OMC1 from 2009. The units are volts. 'ds1' indicates sampling at 0.02 seconds, 'ds5' indicates sampling every 0.1 seconds. The scan rate was 120"/s.

  ---

  

  The beam sizes were measured from the autocorrelation maps. However, because there is structure on many scales in this map, I had to use a rather ad-hoc method to remove the correlated structure. I fitted a gaussian to the elliptical northwest-southeast structure, removed it, then fitted a gaussian to the remaining circular thing in the center, which is approximately the beam. If I fit the "beam" gaussian with an ellipse, I get: Beamsize 1_1: 36.15,26.23 Beamsize 1_5: 48.39,30.21 With a circle: Beamsize 1_1: 29.51 Beamsize 1_5: 35.31

  

  The ds1 and ds5 images compared.

  

  The PSDs of the two images (on identical grids). Note that ds5 loses power at small spatial scales, 50% at 40"!

  

  The pixel-pixel plot with a fit that shows a 10% overall flux loss (best-fit).

---

• BGPS data paper published

   *[[http://irsa.ipac.caltech.edu/data/BOLOCAM_GPS/bgps_methods.pdf The Bolocam Galactic Plane Survey I. Survey Description and Data Reduction] `[`[http://arxiv.org/abs/1011.0691 arXiv]`]`  *[[http://adsabs.harvard.edu/abs/2010ApJS..188..123R The Bolocam Galactic Plane Survey II. Catalog of the Image Data] `[`[http://arxiv.org/abs/0909.2871 arXiv]`]`  *[[http://adsabs.harvard.edu/abs/2010ApJ...717.1157D The Bolocam Galactic Plane Survey III. Characterizing Physical Properties of Massive Star-Forming Regions in the Gemini OB1 Molecular Cloud ] `[`[http://arxiv.org/abs/1005.4969 arXiv]`]`  *[[http://adsabs.harvard.edu/abs/2010ApJ...721..137B The Bolocam Galactic Plane Survey IV: λ = 1.1 and 0.35 mm Dust Continuum Emission in the Galactic Center Region]

---

• v2.0 is to v1.0 as ___ is to v1.0

  

  Apparently v2.0 recovers flux better at all scales than v1.0.  However, it is noisier.  The full comparison is available here.  The trustworthy range of flux scales is 1.2-1.6ish.  The flux is increased at ALL scales in most images.  However, as was noted in comparisons with MAMBO, there may be curvature in the best-fit lines.  Also, l=35 does not appear to scale up linearly... G34 is fainter in v2.0? In the low S/N fields, the PSD is more reliable than the pixel-pixel comparison.

---

• PPS comparison completed

  The PPS scale factor seems to hold for many different (and ridiculous) aperture sizes.  The full results are here.

---

« Page 24 / 36 »