ism3d.utils package¶
Submodules¶
ism3d.utils.cli_new module¶
Console script for ism3d.
ism3d.utils.io module¶
-
ism3d.utils.io.dct2fits(dct, outname='dct2fits')[source]¶ save a non-nested dictionary into a FITS binary table note: not every Python object can be dumped into a FITS column,
e.g. a dictionary type can be aded into a column of a astropy/Table, but the Table can’be saved into FITS.
- example:
gmake.dct2fits(dat_dct,save_npy=True)
for npy np.save(outname.replace(‘.fits’,’.npy’),dct)
- note: we decide to save dct to HDF5 by default for a couple of reasons from now.
FITS is always stored in big-endian byte order, will cause some troubles with casacore/galario I/O performance is worse The “table” approach doesn’t support nesting / DatType may changed during the dct2fits->fits2dct process
-
ism3d.utils.io.dct2hdf(dct, outname='dct2hdf')[source]¶ write a Python dictionary object into a HDF5 file. slashes are not allowed in HDF5 object names We repalce any “/” in keys with “|” to avoid a confusion with HDF5 internal structures
-
ism3d.utils.io.export_model(models, outdir='./', includedata=False, outname_exclude=None, outname_replace=None)[source]¶ export a model set into FITS images or Measurement Sets shortname: a string list to get rid of from the original data image name
-
ism3d.utils.io.fits2dct(fits)[source]¶ read a fits table into a none-nested dictionary the fits table must be the “1-row” version from gmake.dct2fits
- References
about FITS byteorder: https://github.com/astropy/astropy/issues/4069
- note: we decide to save dct to HDF5 by default for a couple of reasons from now.
FITS is always stored in big-endian byte order, will cause some troubles with casacore/galario I/O performance is worse The “table” approach doesn’t support nesting / DatType may changed during the dct2fits->fits2dct process
-
ism3d.utils.io.from_hdf5(h5name)[source]¶ read a Python dictionary object from a HDF5 file slashes are not allowed in HDF5 object names We repalce any “|” in keys with “/” to recover the potential file directory paths in keysst
-
ism3d.utils.io.hdf2dct(hdf)[source]¶ read a Python dictionary object from a HDF5 file slashes are not allowed in HDF5 object names We repalce any “|” in keys with “/” to recover the potential file directory paths in keysst
-
ism3d.utils.io.read_data(inp_dct, save_data=False, fill_mask=False, fill_error=False, polaverage=True, dataflag=False, saveflag=True)[source]¶ read FITS/image or MS/visibilities into the dictionary
we set data=
- note:
DATA column shape in nrecord x nchan x ncorr WEIGHT column shape in nrecord x ncorr( supposely this is the “average” value of WEIGHT_SPECTRUM along the channle-axis WEIGHT_SPECTRUM SHAPE in nrecord x nchan x ncorr FLAGS shape in nrecord x nchan x ncorr (so WEIGHT_SPECTRUM is likely ~WEIGHT/NCHAN?) depending on the data model / calibration script
data@ms in complex64 (not complex128)
- For space-saving, we
set DATA-values=np.nan when flag=True (so there is no a “flag” variable for flagging in dat_dct
- when polaverage=True, we only derive stokes-I if both XX/YY (or RR/YY) are Good. If one of them are flagged, Data-Values set to np.nan
this follows the principle in the tclean()/stokes parameter https://casa.nrao.edu/casadocs-devel/stable/global-task-list/task_tclean/parameters http://casacore.github.io/casacore/StokesConverter_8h_source.html
weight doesn’t include the channel-axis (assuming the channel-wise weight variable is neligible
XX=I+Q YY=I-Q ; RR=I+V LL=I-V => I=(XX+YY)/2 or (RR+LL)/2
-
ism3d.utils.io.to_hdf5(value, outname='test.h5', checkname=False, **kwargs)[source]¶ write a Python dictionary object into a HDF5 file. slashes are not allowed in HDF5 object names We repalce any “/” in keys with “|” to avoid a confusion with HDF5 internal structures
- note:
- https://support.hdfgroup.org/HDF5/Tutor/cmdtoolview.html
h5ls -r test.h5
h5dump -n 1 test.h5
- optional kwargs: compression=’gzip’ None, gzip, lzf (+ szip, if installed)
gzip is preferred lzip with high-speed but lower comperssion rate. examples: uvdata.h5 (none2.5GB/lzip2.2GB/gzip2.0GB)
ism3d.utils.meta module¶
Container for internal metadata/configuration variables under the program namespace
inp_dct: input parameter set (using quantities) mod_dct: model parameter set (using quantities) obj_dct: component parameter set (dimensionless, values in internal units)
Input metadata inp_dct (can be hacked) (fixed) Component metadata mod_dct (lightweight) (dynamical( Component Reference model mod_dct (lightweight (dynamical) Data dat_dct (heavy) (transite) Models models (heavy) not nesscarilly filled with data (transite only exporting data)
ism3d.utils.misc module¶
-
ism3d.utils.misc.convert_size(size_bytes)[source]¶ obsolete now we use human_unit()/human_to_string()
-
ism3d.utils.misc.human_to_string(q, format_string='{0.value:0.2f} {0.unit:shortname}', nospace=True)[source]¶ A slightly more fancy version of quality.to_string() add the option of 0.units:shortname in formating string syntax # https://docs.astropy.org/en/stable/units/format.html#astropy-units-format
- format: forwarded to .to_string(format):
options: generic, unscaled, cds, console, fits, latex, latex_inline, ogip, unicode, vounit
- format_string: {0:0.2f} {1}
help format you output when output=’string’
- output:
‘string’: a string represent the input quantity in best-guess unit
For time: try built-in astropy.utils.console.human_time() For file size: one may also use astropy.utils.concolse.human_file_size()
print(human_to_string(q,format_string=’{0.value:0.2f} {0.unit:shortname}’,nospace=False)) print(human_to_string(q,format_string=’{0.value:0.2f}{0.unit:shortname}’,nospace=True)) print(human_to_string(q,format_string=’{0.value:0.2f}{0.unit:cds}’,nospace=True)) print(human_to_string(q,format_string=’{0.value:0.2f} in {0.unit:shortname}’,nospace=True)) print(human_to_string(q,format_string=’{0.value:0.2f} in {0.unit:cds}’,nospace=True))
-
ism3d.utils.misc.human_unit(quantity, return_unit=False, base_index=0, scale_range=None)[source]¶ Sugguest a better unit for the quantity and make it more human readable e.g. 1200 m/s -> 1.2 km/s
- return_unit:
False: return the input quantity in a suggested unit True: just return a suggested unit
- base_index:
the index of the unitbase which we examine its prefix possibility.
For time: try built-in astropy.utils.console.human_time() For file size: one may also use astropy.utils.concolse.human_file_size()
reference: https://docs.astropy.org/en/stable/_modules/astropy/units/core.html
have tried the functions below,they work similar for PrefixUnit, but they dont work well with composited units (e.g. u.km/u.s)
get_current_unit_registry().get_units_with_physical_type(unit) unit.find_equivalent_units(include_prefix_units=True) unit.compose(include_prefix_units=True) might work but the results can be unexpected
- note:
get_units_with_same_physical_type() is a private method, since end users should be encouraged to use the more powerful compose and find_equivalent_units methods (which use this under the hood).
help find the best human readable unit then you can do q.to_string(unit=’*’)
-
ism3d.utils.misc.makekernel(xpixels, ypixels, beam, pa=0.0, cent=None, mode='center', factor=10, verbose=True)[source]¶ mode: ‘center’,’linear_interp’,’oversample’,’integrate’
beam=[bmaj,bmin] FWHM not (bmin,bmaj) pa=east from north (ccw)deli
in pixel-size units
- by default: the resulted kernel is always centered around a single pixel, and the application of
the kernel will lead to zero offset,
- make a “centered” Gaussian PSF kernel:
- e.g. npixel=7, centered at px=3
npixel=8, centered at px=4 so the single peak is always at a pixel center (not physical center) and the function is symmetric around that pixel the purpose of doing this is to avoid offset when the specified kernel size is even number and you build a function peaked at a pixel edge.
is x ks (peak pixel index) 10 x 7(3) okay 10 x 8(4) okay 10 x 8(3.5) offset for convolve (non-fft), odd ks is required (the center pixel is undoubtely index=3)
even ks is not allowed
- for convolve_fft, you need to use cent=ks/2:
technically it’s not the pixel index of image center but the center pixel is “considers”as index=4
- the rule of thumb-up:
cent=floor(ks/2.) or int(ks/2) # int() try to truncate towards zero.
note:
- Python “rounding half to even” rule vs. traditional IDL:
https://docs.scipy.org/doc/numpy/reference/generated/numpy.around.html https://realpython.com/python-rounding/ Python>round(1.5) # 2 Python>round(0.5) # 0 IDL>round(1.5) # 2 IDL>round(0.5) # 1
- “forget about how the np.array is stored, just use the array as it is IDL;
when it comes down to shape/index, reverse the sequence”
About Undersampling Images: http://docs.astropy.org/en/stable/api/astropy.convolution.discretize_model.html
-
ism3d.utils.misc.makepb(header, phasecenter=None, antsize=<Quantity 12. m>)[source]¶ - make a 2D Gaussian image approximated to the ALMA (or VLA?) primary beam
https://help.almascience.org/index.php?/Knowledgebase/Article/View/90/0/90
- note: this is just a apprximate solution, assuming the the pointing is towards the reference pixel in the header
and use the first channel as the reference frequency.
-
ism3d.utils.misc.makepsf(header, beam=None, size=None, mode='oversample', factor=None, norm='peak')[source]¶ make a 2D Gaussian image as PSF, warapping around makekernel() beam: tuple (bmaj,bmin,bpa) quatity in fits convention
otherwise, use header bmaj/bmin/bpa
size: (nx,ny) <– in the FITS convention (not other way around, or so called ij)
norm=’peak’ would be godo for Jy/pix->Jy/beam output
Note: we choose not use Gaussian2DKernel as we want to handle customzed PSF case in upstream (as dirty beam)
-
ism3d.utils.misc.render_component(out, im, scale=1, mode='iadd')[source]¶ in-plane-add model components into spectral-cube or mutil-freq MS see other mode options: e.g. iadd/isub
out=None return a new obect out!=None return a reference point to the updated out
- Note:
this may return a view of numpy if out is a slice
from gmake.discretize import render_component cube=np.zeros((3,3,3))
cube[0,:,:]=render_component(cube[0,:,:],np.ones((3,3)),scale=5) x=render_component(cube[0,:,:],np.ones((3,3)),scale=5) np.may_share_memory(x,cube)
- warning:
cube=render_component(cube[0,:,:],np.ones((3,3)),scale=5) will override your cube values.. one should use instead: cube[0,:,:]=render_component(cube[0,:,:],np.ones((3,3)),scale=5)
if im=[], it will do nothing…
- if you’re not sure out is None, maybe use:
out=render_component(out)
- if you know out is not None, use
render_component(out)
-
ism3d.utils.misc.unit_shortname(unit, nospace=True, options=False)[source]¶ convert unit to shortest name e.g.: (unit_shortname(u.Jy*u.km/u.s,nospace=False)
In [5]: unit_shortname(u.Jy*u.km/u.s,nospace=False) Out[5]: ‘Jy km / s’
In [6]: unit_shortname(u.Jy*u.km/u.s,nospace=True) Out[6]: ‘Jykm/s’
ref: https://docs.astropy.org/en/stable/units/format.html#astropy-units-format
ism3d.utils.utils module¶
General-purpose utility Functions used by other modules
-
ism3d.utils.utils.gal_flat(im, ang, inc, cen=None, interp=True, align_major=False, fill_value=None)[source]¶ translated from IDL/gal_flat.pro im in (nz,ny,nx) cen is (xc,yc)
-
ism3d.utils.utils.gmake_pformat(fit_dct)[source]¶ fill.. p_format : format for values p_format_keys : format for p_name
-
ism3d.utils.utils.imcontsub(imagename, linefile='', contfile='', fitorder=0, linechan=None, contchan=None)[source]¶ linechan / contchan: tuple with [fmin,fmax] in each element
-
ism3d.utils.utils.inp_validate(inp_dct, verbose=False)[source]¶ - validate the input parameters, will do the follows step by step
check if the variable type against its type definition
for quanity, check the physical_type use .is_eqivalent(default_unit)
-
ism3d.utils.utils.logger= <Logger ism3d.utils.utils (DEBUG)>¶ import ast, operator
- binOps = {
ast.Add: operator.add, ast.Sub: operator.sub, ast.Mult: operator.mul, ast.Div: operator.truediv, ast.Mod: operator.mod
}
- def arithmeticEval (s):
node = ast.parse(s, mode=’eval’)
- def _eval(node):
- if isinstance(node, ast.Expression):
return _eval(node.body)
- elif isinstance(node, ast.Str):
return node.s
- elif isinstance(node, ast.Num):
return node.n
- elif isinstance(node, ast.BinOp):
return binOps[type(node.op)](_eval(node.left), _eval(node.right))
- else:
raise Exception(‘Unsupported type {}’.format(node))
return _eval(node.body)
-
ism3d.utils.utils.paste_array(wall, block, loc, method='replace')[source]¶ past a small array into a larger array with shifting works for high dimension or off-edge cases wall/block requires in the same dimension number loc: the index of left-bottom pixel of block in wall.
-
ism3d.utils.utils.pprint(*args, **kwargs)[source]¶ A modified pprint.pprint without sorting dictionary keys
- this may not be neccsary with Py38+ as optional keyword sort_dicts become available
-
ism3d.utils.utils.read_par(inp_dct, par_name, to_value=False)[source]¶ read parameter quantity / values (in default units) from a parameter dictionary this function can handle literal slicing and attribute syntax
- e.g.: par_str[ind_str]@section_name
-
ism3d.utils.utils.read_range(center=0, delta=0, mode='a')[source]¶ modifiy the parameter exploring bounrdary according to the info from opt section a: absolute
-
ism3d.utils.utils.repr_parameter(v)[source]¶ Similiar to the built-in repr(), but can handle Quantity / SkyCoord used for writing input files
-
ism3d.utils.utils.rotate_xy(x, y, angle, xo=0, yo=0)[source]¶ rotate points in a 2D coordinate system angle: rotate points by angle in CCW ref: rotate_xy.pro rot_3d.pro performance consideration:
-
ism3d.utils.utils.sample_grid(spacing, xrange=[- 100, 100], yrange=[- 100, 100], center=None, ratio=1, angle=0)[source]¶ this is loosely based on my sample_grid.pro in IDL with improvement on defining x-/y-range angle in rad ratio=major/minor
-
ism3d.utils.utils.set_threads(num=None)[source]¶ some underline alrgoithm library can use OMP turn on OMP is beneficial for iteration optimizwer but not for parallele optimizer this function can turn on / turn off OMP in modelling alrothjm
- ref:
looks like os.environ will only work before import packages after the packages are imported, changing os.environ can not dynamically switching nthreads
https://software.intel.com/en-us/mkl-linux-developer-guide-mkl-domain-num-threads http://www.diracprogram.org/doc/release-17/installation/mkl.html https://software.intel.com/en-us/blogs/2018/10/18/mkl-service-package-controlling-mkl-behavior-through-python-interfaces
Module contents¶
subpackage utils