Adam Ginsburg's blog

• Editing metadata in measurement sets

  I needed to update the position of VLA phase calibrator J1744-3116 in my measurement sets. The VLA coordinate is:

  17:33:02.705790 -13.04.49.54823 J2000

  while the ALMA coordinate is

  17:44:23.578227 -31.16.36.29204 ICRS

  and the SIMBAD coordinate is

  17 44 23.57824 -31 16 36.2943 ICRS

  These are separated by a significant amount:

  from astropy import units as u, coordinates
  import numpy as np
  
  vla_coord = coordinates.SkyCoord("17:33:02.705790 -13:04:49.54823", unit=(u.h, u.deg), frame='fk5')
  alma_coord = coordinates.SkyCoord("17:44:23.578227 -31:16:36.29204", unit=(u.h, u.deg), frame='icrs')
  simbad_coord = coordinates.SkyCoord('17 44 23.57824 -31 16 36.2943', unit=(u.h, u.deg), frame='icrs')
  
  alma_coord.separation(vla_coord).to(u.arcsec)
  # <Angle 0.28848595 arcsec>
  
  simbad_coord.separation(vla_coord).to(u.arcsec)
  # <Angle 0.29071513 arcsec>
  
  simbad_coord.separation(alma_coord).to(u.arcsec)
  # <Angle 0.00226614 arcsec>
  

  This is caused by a typo in the VLA calibrator catalog (Lorant Sjouwerman, private communication).

  To fix it:

  tb.open('22A-020.sb41257746.eb41788351.59700.31502699074/22A-020.sb41257746.eb41788351.59700.31502699074.ms/FIELD')
  
  # get the existing PHASE_DIR.  Shape is [coordinates, ?, sourceID]
  phasedir = tb.getcol('PHASE_DIR')
  # figure out which row contains the to-be-modified source
  rownr = np.argmax(tb.getcol('NAME') == 'J1744-3116')
  # modify the phasedir.  CASA expects coordinates to be wrapped at 180 deg
  phasedir[:, 0, rownr] = simbad_coord.fk5.ra.wrap_at(180*u.deg).rad, simbad_coord.fk5.dec.rad
  
  import shutil
  shutil.copytree('22A-020.sb41257746.eb41788351.59700.31502699074/22A-020.sb41257746.eb41788351.59700.31502699074.ms/FIELD', '22A-020.sb41257746.eb41788351.59700.31502699074/22A-020.sb41257746.eb41788351.59700.31502699074.ms/FIELD.backup')
  tb.open('22A-020.sb41257746.eb41788351.59700.31502699074/22A-020.sb41257746.eb41788351.59700.31502699074.ms/FIELD', nomodify=False)
  tb.putcol(columnname='PHASE_DIR', value=phasedir)
  tb.flush()
  tb.close()
  

  Put all together:

  from astropy import units as u, coordinates
  import numpy as np
  
  simbad_coord = coordinates.SkyCoord('17 44 23.57824 -31 16 36.2943', unit=(u.h, u.deg), frame='icrs')
  
  import glob, shutil
  for vis in glob.glob("*/*.ms"):
      shutil.copytree(vis+"/FIELD", vis+"/FIELD.backup")
      tb.open(vis+"/FIELD")
      phasedir = tb.getcol('PHASE_DIR')
      rownr = np.argmax(tb.getcol('NAME') == 'J1744-3116')
      assert rownr != 0
      phasedir[:, 0, rownr] = simbad_coord.fk5.ra.wrap_at(180*u.deg).rad, simbad_coord.fk5.dec.rad
      tb.open(vis+"/FIELD", nomodify=False)
      tb.putcol(columnname='PHASE_DIR', value=phasedir)
      tb.flush()
      tb.close()
  

---

• CASA MPI debugging cont'd

  I continue to try to get MPI to run to completion.

  Using casa-6.5.0-9-py3.8, the latest error on the W51 B3 mosaic (biggest in ALMA-IMF, approximately): 2022-05-03 06:36:06 SEVERE MPICommandServer::command_request_handler_service::CubeMajorCycleAlgorithm::task::MPIServer-2 (file src/code/synthesis/ImagerObjects/CubeMajorCycleAlgorithm.cc, line 137) Exception (std): ArrayBase::validateConformance shape [3] differs from [4]

  This leads to a lot of frozen nodes: 2022-05-03 19:29:17 SEVERE MPIMonitorClient::monitor_status_service::MPIMonitorClient::monitor_status_service::casa Ping status response from server 1 not received in the last 420s. Setting its status to 'timeout'

  This happens during the MakePSF stage.

  On G333.60 B6, which is smaller, the first exception I see is: 2022-05-05 17:31:47 SEVERE MPIMonitorClient::monitor_status_service::MPIMonitorClient::monitor_status_service::casa Ping status response from server 4 not received in the last 85s. Setting its status to 'timeout' but... it looks like the clean actually finished!

  This is an odd one that happened during a major cycle, I think:

  2022-05-13 17:47:25 WARN MPICommandServer::command_request_handler_service::SynthesisImagerVi2::CubeMajorCycle::MPIServer-19 (file src/code/synthesis/ImagerObjects/CubeMajorCycleAlgorithm.cc, line 336) Exception for chan range [144, 151] --- Setting masked pixels to zero for input startmodel : Er ror (Resource deadlock avoided) when acquiring lock on /blue/adamginsburg/adamginsburg/almaimf/workdir/G328.25_B6_spw4_12M_spw4.contcube.model/table.lock

  This one resulted in the residual and image being identical past some point in the spectrum, i.e., it gave up partway through writing to disk. There was NO CASA error! Just a miserable segfault

  [c0711a-s25:60411] * Process received signal * [c0711a-s25:60411] Signal: Segmentation fault (11) [c0711a-s25:60411] Signal code: Address not mapped (1) [c0711a-s25:60411] Failing at address: 0x2d867e04f000

---

• CO2 Monitoring at Conferences: Update in 2022

  I started monitoring CO2 levels at conferences a couple years back, and now that we're having in-person conferences again, I'm doing more monitoring.

  The last I attended was EPOS 2022, which is probably my favorite recurring conference because of its venue, size, and schedule. There is time to talk to everyone you want to about all topics.

  I gave a talk on the small disks in Orion.

  The monitoring results are here:

  

  I missed day 1 because I forgot the USB cable in my room, then I had some electrical connection problems day 2 (the USB-C port on my computer is... finnicky). But besides that, you can see clear trends on day 1 and part of day 2 where the CO2 was rapidly rising to uncomfortable levels.

  The blue zones are the scheduled breaks. During the breaks, the room was almost totally empty and we usually opened the doors and windows.

  I shared the day 2 data with the conference organizers and they then prioritized opening the windows. There was one period I think we opted not to because it started snowing for a few minutes.

  On the last day, I'm not sure why the CO2 didn't go down during the break - it may be that windows were open only one one side of the room, so there was only diffusion, not flow.

---

• ALMA Cycle 9 corrupted zip fix

  If you encounter a bug like this:

  

  "XML (syntax or validity) error" because you hit issue C1_032, the fix is really easy:

  import zipfile
  import os
  
  # not strictly necessary, but important if you want to avoid filling your directory with files
  os.mkdir('recovery')
  os.chdir('recovery')
  
  # replace with your AOT file name
  with zipfile.ZipFile('../corrupted.aot', 'r') as aot:
      # this will extract all the files into the current directory
      aot.extractall()
      files = aot.filelist
  
  with zipfile.ZipFile('../recovered.aot', 'w') as aot:
      for fh in files:
          aot.write(fh.filename)
  

  At least, this worked on the one case where I got bitten.

  This fix is based on Erik Rosolowsky's ALMA proposal tools

---

• Hacking plotms to let pipeline run

  For the ACES project, I'm re-running the ALMA pipeline using MPI.

  However, this results in all runs crashing with the following error:

  2022-04-06 04:22:43 INFO: Executing plotms(vis='uid___A002_Xf512ae_X2626.ms', xaxis='azimuth', yaxis='elevation', spw='25:0~0,27:0~0,29:0~0,31:0~0,33:0~0,35:0~0', antenna='0&&*', avgchannel='9000', avgtime='10', coloraxis='field', customflaggedsymbol=True, flaggedsymbolshape='autoscaling', title='Elevation vs Azimuth for uid___A002_Xf512ae_X2626.ms', plotfile='azel.png', showgui=False, clearplots=True)
  fuse: failed to exec fusermount: No such file or directory
  
  Cannot mount AppImage, please check your FUSE setup.
  You might still be able to extract the contents of this AppImage
  if you run it with the --appimage-extract option.
  See https://github.com/AppImage/AppImageKit/wiki/FUSE
  for more information
  open dir error: No such file or directory
  

  This fails even if I set the global APPIMAGE_EXTRACT_AND_RUN as directed on the appimage.org docs.

  This indicates that the appimage is older than the environmental variable.

  Running the full command "works": /orange/adamginsburg/casa/casa-6.2.1-7-pipeline-2021.2.0.128/lib/py/lib/python3.6/site-packages/casaplotms/__bin__/casaplotms-x86_64.AppImage --appimage-extract but --appimage-extract-and-run gives --appimage-extract-and-run is not yet implemented in version continuous-1-g6f3138f

  So to run plotms, we need to use the AppRun executable in the extracted squashfs-root directory.

  My original hack was to just tell plotms not to run at all, which I accomplished by editing the plotmstool.py file (lib/py/lib/python3.6/site-packages/casaplotms/private/plotmstool.py) to just skip the __launch command:

  def __launch( uri=None ):
      return
  

  However, in writing this post, I've changed to instead modifying the app_path to this:

  app_path = "/tmp/casaplotms/squashfs-root/AppRun"
  if not __os.path.exists(app_path):
      print(f"Did not find extracted path {app_path}")
      app_path = __os.path.join( __os.path.abspath( __os.path.join(__os.path.dirname(__file__),"..") ), '__bin__/casaplotms-x86_64.AppImage')
  

  I haven't tried it but I'm slightly hopeful.

---

• Astronomical Software Development & Career

  I recently visited the University of Maryland to give a colloquium (slides here). This was my first in-person colloquium as faculty because of COVID.

  I had a great meeting with the grad students over lunch, and they asked several good questions. I've thought over some of them a little more and wanted to fill out my answers. I'll paraphrase two questions:

  "How do you balance coding work with other research work?"

  My answer was essentially, "All of my coding work is research work - I don't write code that doesn't contribute to my research." That is broadly true, but not exclusively so, and there are some exceptions.

  All of the coding work I have done has been either to enable my research, support someone else's research, enable my research career, or (in theory) make my life easier.

  The research stuff is mostly obvious - my work on and with data reduction software (see all the CASA-tagged blog posts here, all development on radio-astro-tools, etc), pyspeckit, and most of astroquery. On the astroquery side, a lot of my research work wasn't directly going to papers, but was instead to dig through the archives to see if data were available, or if I needed to obtain new data. It was then useful for supporting observing proposal writing.

  The "enable my research career" component included a lot of side-work on things that aren't directly research, but are research-adjacent. These projects included building tools to deploy my papers to github (which was a bit obsoleted by overleaf), automatically updating reference lists (it turns out I often cite 5-10 arXiv papers in an article, but by publication time, I need to update them to official journal article citations), assemble lists of coauthors, add citation counts to the CV, writing this blog, etc.

  The exceptions are some of the 'pure service' coding. This necessarily had to be the lowest priority most of the time, but this is still career-motivated. Some of the contributions I've made to astropy & other open-source projects are just to improve their code bases, either with bugfixes, added features, or things to improve robustness. Most of my contributions were directly motivated by need, of course - either there was something basic missing, or I was the expert in that particular subtopic. A good example is the J-to-K equivalency; it wasn't directly research-motivated, but was something I found myself needing in day-to-day work.

  The 'pure service' coding also entails maintaining projects. There is still a selfish motivation here: if the code is maintained (if other people are using iti and finding bugs), it is more likely to be functional when I need it next. But, most of this work isn't triggered by my own needs, but by others. The astropy Moore grant now funds some of this work, which helps ensure that I'm motivated to continue the maintenance - but I was doing a lot of this work as a postdoc long before I could be directly paid for it.

  "How do you avoid being typecast as a coder?"

  This question came from students who got to be known in their research collaborations as "good at coding". My answer was basically, "learn to say no", but there's more nuance to add.

  First, there are some solid career paths to follow by being the science coder in a group. Many observatory jobs, for example, would prioritize this coding experience. There are a lot of positions in observatory jobs at places like STSCI, NRAO, NOAO, etc. that value this sort of skill over many others. So if you want to pursue one of these paths, or an industry path, because you <i> enjoy </i> the coding, then great! Do it!

  If you really enjoy the coding, you'll get to do a lot more of it in a job focused on software development than in an R1 research job.

  That said, if you are interested in pursuing an R1 faculty job, you need to strike a more careful balance. It's fine if you're the coder in the group - that can land you a coauthorship on a lot of papers, which is a good thing! However, first and foremost, you need to publish your own (first-author) papers, which means prioritizing your work over collaborative work. Ideally, you'd just do both - that's what's expected of faculty members (faculty members can't choose to prioritize teaching or research, really - they have to do well at both, which often means just putting in more hours). But if you're faced with the choice, a few first-author papers are more important than many co-author papers.

  The way I struck the balance was to focus entirely on research papers during my grad school career, but still do a lot of software support work on the side - I put in more time, but it was all stuff that was useful both for me and for collaborators. Later, in my second postdoc, I started publishing code-only papers; I would advise grad students to do this sooner, though. Since AAS journals now accept code papers, if you want coding to be a big part of your research portfolio, it's a good idea to have a refereed paper on a piece of software. Note that some of the most highly-cited papers in astronomy are code papers, like the DAOPHOT, SEXTRACTOR, and astropy papers.

  Last bit of advice closing this out: Avoid GUI development. That way lays madness.

---

• My blog is now deployed with gh-actions

  This is mostly a test post to prove it.

  I'm now using this gh-deployment script: https://github.com/keflavich/gh-pages-pelican-action (forked from https://github.com/nelsonjchen/gh-pages-pelican-action) in conjunction with gh-actions.

---

• CASA MPI Errors continued in early 2022

  Running 7m+12m combination, the following errors show up:

  Exception: Error in making PSF : Invalid Table operation: Rows cannot be removed from table /blue/adamginsburg/adamginsburg/almaimf/workdir/G333.60_spw5_12M_B6/IMAGING_WEIGHT_1390794_230801956263_230804885819_bwtaper_0_interp_1; its storage managers do not support it
  

  This turns up several times repeatedly:

  G333.60_B6_fullcube_7M12M_5_15827045.log:2022-01-21 18:05:38  WARN    MPICommandServer::command_request_handler_service::SynthesisImagerVi2::CubeMajorCycle::MPIServer-25 (file src/code/synthesis/ImagerObjects/CubeMajorCycleAlgorithm.cc, line 336)      Exception for chan range [496, 499] ---   Error in making PSF : Interpolate1D::operator() data has repeated x values
  G333.60_B6_fullcube_7M12M_5_15868985.log:2022-01-22 00:01:20  WARN    MPICommandServer::command_request_handler_service::SynthesisImagerVi2::CubeMajorCycle::MPIServer-19 (file src/code/synthesis/ImagerObjects/CubeMajorCycleAlgorithm.cc, line 336)      Exception for chan range [496, 499] ---   Error in making PSF : Invalid Table operation: Rows cannot be removed from table /blue/adamginsburg/adamginsburg/almaimf/workdir/G333.60_spw5_12M_B6/IMAGING_WEIGHT_1390794_230801956263_230804885819_bwtaper_0_interp_1; its storage managers do not support it
  G333.60_B6_fullcube_7M12M_5_15891690.log:2022-01-22 14:37:39  WARN    MPICommandServer::command_request_handler_service::SynthesisImagerVi2::CubeMajorCycle::MPIServer-23 (file src/code/synthesis/ImagerObjects/CubeMajorCycleAlgorithm.cc, line 336)      Exception for chan range [496, 499] ---   Error in making PSF : Invalid Table operation: Rows cannot be removed from table /blue/adamginsburg/adamginsburg/almaimf/workdir/G333.60_spw5_12M_B6/IMAGING_WEIGHT_1390794_230801956263_230804885819_bwtaper_0_interp_1; its storage managers do not support it
  G333.60_B6_fullcube_7M12M_5_15914880.log:2022-01-23 02:21:24  WARN    MPICommandServer::command_request_handler_service::SynthesisImagerVi2::CubeMajorCycle::MPIServer-25 (file src/code/synthesis/ImagerObjects/CubeMajorCycleAlgorithm.cc, line 336)      Exception for chan range [496, 499] ---   Error in making PSF : Invalid Table operation: Rows cannot be removed from table /blue/adamginsburg/adamginsburg/almaimf/workdir/G333.60_spw5_12M_B6/IMAGING_WEIGHT_1390794_230801956263_230804885819_bwtaper_0_interp_1; its storage managers do not support it
  

  Looks like it's always the same few channels failing.

  Errors like this one:

  WARN    MPICommandServer::command_request_handler_service::SynthesisImagerVi2::CubeMajorCycle::MPIServer-17 (file src/
  code/synthesis/ImagerObjects/CubeMajorCycleAlgorithm.cc, line 336)        Exception for chan range [1152, 1153] ---   Programmer error: sumwt disk image is non existant
  

  are probably cauased by deleting the .sumwt file in the middle of a tclean run - so that is genuine "user error" (I was deleting the sumwt, psf, and weight files occasionally b/c they are the 'leftovers' when a tclean run fails for bad reasons)

  There are some others that don't have obvious explanations:

  casa_log_line_G010.62_B3_fullcube_7M12M_1_2022-01-21_08_17_16.log:2022-01-21 15:29:42   WARN    MPICommandServer::command_request_handler_service::SynthesisImagerVi2::CubeMajorCycle::MPIServer-15 (file src/ code/synthesis/ImagerObjects/CubeMajorCycleAlgorithm.cc, line 336)        Exception for chan range [534, 535] ---   FilebufIO::readBlock - incorrect number of bytes read for file /blue/adamginsburg/adamgins burg/almaimf/workdir/G010.62_B3_spw1_7M12M_spw1.sumwt/table.f0
  casa_log_line_G010.62_B3_fullcube_7M12M_1_2022-01-21_08_17_16.log:2022-01-21 17:07:41   WARN    MPICommandServer::command_request_handler_service::SynthesisImagerVi2::CubeMajorCycle::MPIServer-31 (file src/ code/synthesis/ImagerObjects/CubeMajorCycleAlgorithm.cc, line 336)        Exception for chan range [1596, 1597] ---   FilebufIO::readBlock - incorrect number of bytes read for file /blue/adamginsburg/adamgi nsburg/almaimf/workdir/G010.62_B3_spw1_7M12M_spw1.sumwt/table.f0
  

  and this:

  casa_log_line_G333.60_B6_fullcube_7M12M_5_2022-01-21_08_17_16.log:2022-01-21 18:05:38   WARN    MPICommandServer::command_request_handler_service::SynthesisImagerVi2::CubeMajorCycle::MPIServer-25 (file src/code/synthesis/ImagerObjects/CubeMajorCycleAlgorithm.cc, line 336)        Exception for chan range [496, 499] ---   Error in making PSF : Interpolate1D::operator() data has repeated x values
  

  This is an old one that remains unsolved:

  casa_log_line_G338.93_B3_fullcube_7M12M_1_2022-01-22_20_25_52.log:2022-01-23 04:59:57   WARN    MPICommandServer::command_request_handler_service::SynthesisImagerVi2::CubeMajorCycle::MPIServer-8 (file src/code/synthesis/ImagerObjects/CubeMajorCycleAlgorithm.cc, line 336) Exception for chan range [2049, 2049] ---   Error in making PSF : A nasty Visbuffer2 error occured...wait
  

  Got this real nice one after the .image was created:

  *** Error in `/blue/adamginsburg/adamginsburg/casa/casa-6.4.3-4/lib/py/bin/python3': malloc(): smallbin double linked list corrupted: 0x00002af7a40d8000 ***
  ======= Backtrace: =========
  /lib64/libc.so.6(+0x7f804)[0x2af73aa03804]
  /lib64/libc.so.6(+0x82f40)[0x2af73aa06f40]
  /lib64/libc.so.6(__libc_malloc+0x4c)[0x2af73aa09b1c]
  /apps/compilers/gcc/9.3.0/lib64/libstdc++.so.6(_Znwm+0x15)[0x2af751076395]
  

  That happened in the middle of major cycle 1, so the data were probably fine, but I'm just going to restart it

---

• CASA Errors encountered while imaging LB data

  I'm trying to reduce some new data from ALMA's longest baselines (C10, B6).

  There are several exciting errors:

  1. The lines were all flagged out. Thankfully, there is a calibrated_final.ms.backup file that does not have the lines flagged out. flagmanager was unable to restore the flags. The flagging comes from the manual flagging done by the QA2 reducer, which was used to make the continuum image. That's fine, but the unflagging step didn't work. The uvcontsub data, as a result, contain no line data. ?????
  2. I can't image the full cube because of errors like this:

  2021-12-08 08:07:34     INFO    SynthesisImagerVi2::defineImage         Shape : [9600, 9600, 1, 1920]Spectral : [2.31563e+11] at [0] with increment [976510]
  2021-12-08 08:07:34     INFO    SynthesisImagerVi2::defineImage         Set Gridding options for [S255IR-SMA1_sci.spw1.cube.I.manual] with ftmachine : gridft
  2021-12-08 08:07:34     INFO    SynthesisImagerVi2::nSubCubeFitInMemory Required memory: 5.468e+05 GB. Available mem.: 166.3 GB (rc, mem. fraction: 70%, memory: -) => Subcubes: 1920. Processes on node: 64.
  2021-12-08 08:07:34     INFO    SynthesisImagerVi2::weight()    Set imaging weights : Briggs weighting: sidelobes will be suppressed over full image
  2021-12-08 08:07:34     INFO    SynthesisImagerVi2::weight()    Doing spectral cube Briggs weighting formula --  norm
  2021-12-08 08:07:34     INFO    task_tclean::SynthesisDeconvolver::setupDeconvolution   Set Deconvolution Options for [S255IR-SMA1_sci.spw1.cube.I.manual] : hogbom
  2021-12-08 08:07:34     WARN    tclean::::casa  Memory available 187904819 kB is very close to amount of required memory 3982754512 kB
  2021-12-08 08:07:34     INFO    task_tclean::SynthesisImager::makePSF   ----------------------------------------------------------- Make PSF ---------------------------------------------
  2021-12-08 08:07:34     INFO    task_tclean::SynthesisImagerVi2::nSubCubeFitInMemory    Required memory: 5.468e+05 GB. Available mem.: 166.3 GB (rc, mem. fraction: 70%, memory: -) => Subcubes: 1920. Processes on node: 64.
  

  Well, there's no error there, but the PSF maker can't handle making even a single frame. It had no trouble making a 9600x9600 MTMFS image for the continuum, though, so it's pretty unclear to me why the cube is getting OOM-killed. My best guess is that the 64 processes are too much for CASA to handle, and I need to run with fewer to "trick" it into not using up that much memory at once. Maybe I just need to make the PSFs in serial mode. I don't want to wait around to make the cubes, let alone clean them, in serial mode.

  3. Parallel failed with this:

  2021-12-08 14:26:19     INFO    tclean::::casa  ##########################################
  2021-12-08 14:26:19     INFO    tclean::::casa  ##### Begin Task: tclean             #####
  2021-12-08 14:26:19     INFO    tclean::::casa  tclean( vis='calibrated_final.ms', selectdata=True, field='S255IR-SMA1', spw='1', timerange='', uvrange='', antenna='', scan='', observation='', intent='', datacolumn='corrected', imagename='S255IR-SMA1_sci.spw1.cube.I.zoom.manual', imsize=[500, 500], cell='0.0042arcsec', phasecenter='', stokes='I', projection='SIN', startmodel='', specmode='cube', reffreq='', nchan=-1, start='', width='', outframe='lsrk', veltype='radio', restfreq=[], interpolation='linear', perchanweightdensity=True, gridder='standard', facets=1, psfphasecenter='', wprojplanes=1, vptable='', mosweight=True, aterm=True, psterm=False, wbawp=True, conjbeams=False, cfcache='', usepointing=False, computepastep=360.0, rotatepastep=360.0, pointingoffsetsigdev=[], pblimit=0.2, normtype='flatnoise', deconvolver='hogbom', scales=[], nterms=2, smallscalebias=0.0, restoration=True, restoringbeam=[], pbcor=True, outlierfile='', weighting='briggs', robust=0.0, noise='1.0Jy', npixels=0, uvtaper=[], niter=10000, gain=0.1, threshold='10mJy', nsigma=0.0, cycleniter=-1, cyclefactor=1.0, minpsffraction=0.05, maxpsffraction=0.8, interactive=False, usemask='user', mask='', pbmask=0.0, sidelobethreshold=3.0, noisethreshold=5.0, lownoisethreshold=1.5, negativethreshold=0.0, smoothfactor=1.0, minbeamfrac=0.3, cutthreshold=0.01, growiterations=75, dogrowprune=True, minpercentchange=-1.0, verbose=False, fastnoise=True, restart=True, savemodel='none', calcres=True, calcpsf=True, psfcutoff=0.35, parallel=True )
  2021-12-08 14:26:19     INFO    tclean::::casa  Verifying Input Parameters
  2021-12-08 14:26:19     INFO    SynthesisImagerVi2::selectData  MS : calibrated_final.ms | Selecting on fields : S255IR-SMA1 | Selecting on spw :1 | [Opened in readonly mode]
  2021-12-08 14:26:19     INFO    SynthesisImagerVi2::selectData    NRows selected : 0
  2021-12-08 14:26:19     SEVERE  tclean::::casa  Task tclean raised an exception of class RuntimeError with the following message: Parallel transport layer not initialized
  2021-12-08 14:26:19     INFO    tclean::::casa  Task tclean complete. Start time: 2021-12-08 09:26:19.103483 End time: 2021-12-08 09:26:19.314353
  2021-12-08 14:26:19     INFO    tclean::::casa  ##### End Task: tclean               #####
  2021-12-08 14:26:19     INFO    tclean::::casa  ##########################################
  

  I'm guessing this is a bad error message and the real error is that the data selection failed. Probably when I copied over the continuum_final.ms.backup file, it uses the original numbering (spw 25, 27, etc.) instead of the new numbering. I guess the renumbering is cased by uvcontsub, which I'm now skipping because it may have broken my data and because I'd rather image the continuum. That proved correct; the error message above just means that I selected a nonexistent SPW.

  Flagged Out Lines

  After re-copying over the calibrated MS and confirming that no lines were flagged out, I re-imaged... and the lines are again missing.

  # show the flags graphically
  flagdata(vis='calibrated_final.ms', mode='summary', spwchan=True, display='both')
  

  

  The flags are coming from tclean, which makes no sense.

  There are messages like these:

  2021-12-08 15:27:19     WARN    MPICommandServer::command_request_handler_service::SIImageStore::getPSFGaussian::MPIServer-51 (file src/code/synthesis/ImagerObjects/SIImageStore.cc, line 2037)        PSF is blank for[C9:P0] [C10:P0] [C11:P0] [C12:P0] [C13:P0] [C14:P0] [C15:P0] [C16:P0] [C17:P0] [C23:P0] [C24:P0] [C25:P0] [C26:P0]
  

  scattered throughout the PSF making.

  The PSF spectrum ends up like this:

  

  Channels are totally flagged out. But in the data, they are not:

  

  Running the tclean without parallel, i.e. with parallel=False, resulted in a smooth, unbroken PSF.

  So, if we dig back into what happened in the parallel tclean....

  2021-12-08 15:26:12     INFO    SynthesisImagerVi2::selectData    NRows selected : 848700
  2021-12-08 15:26:15     INFO    SynthesisImagerVi2::defineImage         Define image coordinates for [S255IR-SMA1_sci.spw0.cube.I.zoom.manual] :
  2021-12-08 15:26:16     INFO    MSTransformRegridder::calcChanFreqs      phaseCenter='Direction: [-0.0535075, 0.949606, 0.308847]'  Channels equidistant in freq
  2021-12-08 15:26:16     INFO    MSTransformRegridder::calcChanFreqs+     Central frequency (in output frame) = 2.345e+11 Hz
  2021-12-08 15:26:16     INFO    MSTransformRegridder::calcChanFreqs+     Width of central channel (in output frame) = 976510 Hz
  2021-12-08 15:26:16     INFO    MSTransformRegridder::calcChanFreqs+     Number of channels = 1920
  2021-12-08 15:26:16     INFO    MSTransformRegridder::calcChanFreqs+     Total width of SPW (in output frame) = 1.8749e+09 Hz
  2021-12-08 15:26:16     INFO    MSTransformRegridder::calcChanFreqs+     Lower edge = 2.33563e+11 Hz, upper edge = 2.35437e+11 Hz
  2021-12-08 15:26:16     INFO    SynthesisImagerVi2::defineImage         Impars : start
  2021-12-08 15:26:16     INFO    SynthesisImagerVi2::defineImage         Shape : [500, 500, 1, 1920]Spectral : [2.33563e+11] at [0] with increment [976510]
  2021-12-08 15:26:16     INFO    SynthesisImagerVi2::defineImage         Set Gridding options for [S255IR-SMA1_sci.spw0.cube.I.zoom.manual] with ftmachine : gridft
  2021-12-08 15:26:16     INFO    SynthesisImagerVi2::nSubCubeFitInMemory Required memory: 1483 GB. Available mem.: 173.6 GB (rc, mem. fraction: 70%, memory: -) => Subcubes: 9. Processes on node: 64.
  2021-12-08 15:26:16     INFO    SynthesisImagerVi2::weight()    Set imaging weights : Briggs weighting: sidelobes will be suppressed over full image
  2021-12-08 15:26:16     INFO    SynthesisImagerVi2::weight()    Doing spectral cube Briggs weighting formula --  norm
  2021-12-08 15:26:16     INFO    task_tclean::SynthesisDeconvolver::setupDeconvolution   Set Deconvolution Options for [S255IR-SMA1_sci.spw0.cube.I.zoom.manual] : hogbom
  2021-12-08 15:26:17     INFO    task_tclean::SynthesisImager::makePSF   ----------------------------------------------------------- Make PSF ---------------------------------------------
  
  ... a bunch of stuff ...
  
  2021-12-08 15:27:11     INFO    MPICommandServer::command_request_handler_service::SynthesisImagerVi2::makePrimaryBeam::MPIServer-54    vi2 : Evaluating Primary Beam model onto image grid(s)
  2021-12-08 15:27:12     INFO    MPICommandServer::command_request_handler_service::SynthesisImagerVi2::makePrimaryBeam::MPIServer-38    vi2 : Evaluating Primary Beam model onto image grid(s)
  2021-12-08 15:27:13     INFO    MPICommandServer::command_request_handler_service::SIImageStore::calcSensitivity::MPIServer-54  [S255IR-SMA1_sci.spw0.cube.I.zoom.manual] Theoretical sensitivity (Jy/bm):c0:none c1:none c2:none c3:none c4:none c5:none c6:none c7:none c8:none c9:none c10:none c11:none c12:none c13:0.00121258 c14:0.00121257 c15:0.00121255 c16:none c17:none c18:none c19:none c20:none c21:none c22:none c23:none c24:none c25:none c26:0.00121253 c27:0.00121253 c28:0.00121252 c29:0.00121251
  2021-12-08 15:27:13     WARN    MPICommandServer::command_request_handler_service::SIImageStore::getPSFGaussian::MPIServer-54 (file src/code/synthesis/ImagerObjects/SIImageStore.cc, line 2037)        PSF is blank for[C0:P0] [C1:P0] [C2:P0] [C3:P0] [C4:P0] [C5:P0] [C6:P0] [C7:P0] [C8:P0] [C9:P0] [C10:P0] [C11:P0] [C12:P0] [C16:P0] [C17:P0] [C18:P0] [C19:P0] [C20:P0] [C21:P0] [C22:P0] [C23:P0] [C24:P0] [C25:P0]
  2021-12-08 15:27:13     INFO    MPICommandServer::command_request_handler_service::SIImageStore::getPSFGaussian::MPIServer-54   Time to fit Gaussian to PSF 0.12
  2021-12-08 15:27:13     INFO    MPICommandServer::command_request_handler_service::SIImageStore::printBeamSet::MPIServer-54     Restoring Beams
  2021-12-08 15:27:13     INFO    MPICommandServer::command_request_handler_service::SIImageStore::printBeamSet::MPIServer-54 +   Pol   Type Chan         Freq     Vel
  2021-12-08 15:27:13     INFO    MPICommandServer::command_request_handler_service::SIImageStore::printBeamSet::MPIServer-54 +     I    Max    0 2.351450e+11 -824.59    0.0254 arcsec x    0.0197 arcsec pa= -6.8506 deg
  2021-12-08 15:27:13     INFO    MPICommandServer::command_request_handler_service::SIImageStore::printBeamSet::MPIServer-54 +     I    Min    1 2.351460e+11 -825.84    0.0000 arcsec x    0.0000 arcsec pa=  0.0000 deg
  2021-12-08 15:27:13     INFO    MPICommandServer::command_request_handler_service::SIImageStore::printBeamSet::MPIServer-54 +     I Median    9 2.351538e+11 -835.82    0.0254 arcsec x    0.0197 arcsec pa= -6.8506 deg
  

  These blank psfs are wrong, there are no flagged data in this data set (no flagged channels). Something about the parallel version of clean is reading the data wrong.

  Solution

  I finally solved this by manually unflagging the channels. Thanks to Brian Svoboda for helping with the antenna selection syntax.

  flagdata(linevis, spw=str(orig_spw), antenna='*&*', mode='unflag')
  

  I don't understand why flagdata was showing the channels as being unflagged, as they were clearly flagged in all data sets (original & backup). I don't understand _when_ they were flagged, either, as it seems they were flagged even before continuum imaging was done.

---

• CASA reading incorrect number of bytes w/MPI

  Running big MPI runs again, I encounter errors like these one:

  2021-12-02 07:02:26      WARN    MPICommandServer::command_request_handler_service::SynthesisImagerVi2::CubeMajorCycle::MPIServer-25 (file src/code/synthesis/ImagerObjects/CubeMajorCycleAlgorithm.cc, line 336)        Exception for chan range [1434, 1445] ---   FilebufIO::readBlock - incorr
  ect number of bytes read for file /blue/adamginsburg/adamginsburg/almaimf/workdir/G327.29_B6_spw7_12M_spw7.sumwt/table.f0
  ##################################
  #############################
  Exception: FilebufIO::readBlock - incorrect number of bytes read for file /blue/adamginsburg/adamginsburg/almaimf/workdir/G327.29_B6_spw7_12M_spw7.sumwt/table.f0
  
  
  2021-12-02 06:03:53        WARN    MPICommandServer::command_request_handler_service::SynthesisImagerVi2::CubeMajorCycle::MPIServer-13 (file src/code/synthesis/ImagerObjects/CubeMajorCycleAlgorithm.cc, line 336)        Exception for chan range [550, 560] ---   FilebufIO::readBlock - incorrect number
   of bytes read for file /blue/adamginsburg/adamginsburg/almaimf/workdir/G327.29_B6_spw4_12M_spw4.sumwt/table.f0
  ##################################
  #############################
  Exception: FilebufIO::readBlock - incorrect number of bytes read for file /blue/adamginsburg/adamginsburg/almaimf/workdir/G327.29_B6_spw4_12M_spw4.sumwt/table.f0
  
  2021-12-02 04:57:21 WARN    MPICommandServer::command_request_handler_service::SynthesisImagerVi2::CubeMajorCycle::MPIServer-9 (file src/code/synthesis/ImagerObjects/CubeMajorCycleAlgorithm.cc, line 336) Exception for chan range [1563, 1574] ---   FilebufIO::readBlock
  - incorrect number of bytes read for file /blue/adamginsburg/adamginsburg/almaimf/workdir/G327.29_B6_spw0_12M_spw0.sumwt/table.f0
  ##################################
  #############################
  Exception: FilebufIO::readBlock - incorrect number of bytes read for file /blue/adamginsburg/adamginsburg/almaimf/workdir/G327.29_B6_spw0_12M_spw0.sumwt/table.f0
  

  This is only quasi-repeatable. For spw4, it happened after the 8th Major Cycle the first time, then the 6th Major Cycle the second time. By repeating the cleans over and over, I was able to eventually get the results to converge. It looks like these crashes are sporadic but don't affect the data.

---

Page 1 / 51 »