calwf3

The WFC3 calibration pipeline used for processing data from the UVIS or IR detectors is called calwf3. While calwf3 is a single executable which can fully process WFC3 data, the pipeline has been designed such that a number of its components can also be invoked as standalone executables. Aside from the calwf3 executable itself, the standalone executable components are: wf3cte, wf3ccd, wf32d, wf3ir, wf3rej, and wf3sum. The first three components apply to UVIS data, wf3ir applies to IR data, and the wf3rej program is used for data from both detectors to combine multiple exposures contained in a CR-SPLIT or REPEAT-OBS set. The wf3sum program sums together the IMSET chips of REPEAT-OBS exposures, and there is no Python wrapper for the wf3sum executable.

The calwf3 processes exposures according to the setting (PERFORM, OMIT, COMPLETE, SKIPPED) of the calibration switch keywords (e.g., FLATCORR, PCTECORR) in the primary header of the input file. The calibration switch keywords directly correspond to calibration processing steps. A flow diagram representing the calibration steps is available in WFC3 Data Handbook.

The calibrated output products generated by calwf3 for UVIS are the FLT and CRJ (when applicable) files, as well as the Charge Transfer Efficiency corrected (CTE-corrected) versions of these files, FLC and CRC. Correspondingly, the output products for the IR detector are the FLT and CRJ (when applicable) files.

HST File Naming Convention available through the archive has a thorough description of how files are named and what those names mean. See also the Types of Output Files From calwf3 section in this document.

Note

For completeness of this discussion, AstroDrizzle functionality has been part of the automated calibration pipeline since 2012. AstroDrizzle removes geometric distortion, corrects for sky background variations, flags cosmic-rays, and combines images with optional subsampling. AstroDrizzle generates the calibrated drizzled data products DRZ and DRC, where the latter is for CTE-corrected images. See the WFC3 Data Handbook or DrizzlePac for more information.

As previously noted, the HSTCAL package is needed to support the calibration interface wrappers contained in the WFC3TOOLS package.

Discussion of the C Version of `calwf3`

Where to Find `calwf3`

The C code for calwf3 is part of HSTCAL package, and the source code can be downloaded from its Git repository Spacetelescope/hstcal. Downloading the source code requires compiling and linking the package with third-party software which can sometimes be tricky. Alternatively, the HSTCAL binaries can be downloaded from conda-forge. High-level release notes for all of the HSTCAL package updates can be found in its Git repository at Releases.

The current WFC3 Data Handbook can be found here. If you have questions not addressed in this documentation or need help with installing or using the software in a timely fashion, please contact the STScI Help Desk. You may also submit a GitHub issue for either the HSTCAL or WFC3TOOLS repositories.

Running `calwf3`

The calwf3 C executable can be run on a single input RAW file or an ASN table listing the members of an association. When processing an association, it retrieves calibration switch and reference file keyword settings from the first image listed in the ASN table. calwf3 does not accept a user-defined list of input images or wildcards on the command line (e.g. *_raw.fits cannot be used to process all raw files in the current directory). A more extensive discussion regarding required input and options for calwf3 follows in the next section.

The wf3ccd, wf32d, wf3cte and wf3ir tasks on the other hand, will accept user-defined input file lists, but they will not accept an association table (_asn.fits) as input. The standalone components wf3rej and wf3sum do not accept lists.

Command Line Options for the `calwf3` C Executable

calwf3 can be called directly from the operating system command line by supplying the executable calwf3.e with an input file and a list of options. This is the same executable that the WFC3TOOLS package wraps with Python code.

   calwf3.e [options] input
   calwf3.e -vts iaa012wdq_raw.fits

   input : str
       Name of input file
       - single filename (_raw.fits or _crj.fits)
       - filename of an ASN table (_asn.fits)

   options
       -d : print optional debugging statements
       -q : print messages only to the trailer file
       -r : print version number and date of software (e.g., Current version: 3.6.2 (May-27-2021))
       -s : save temporary files
       -t : print a detailed time stamp
       -v : print verbose time stamps and information
       -1 : suppress the OpemMP parallel processing for the UVIS CTE correction
   --help : print the syntax for executing this command
--version : print version number of software (e.g., 3.6.2)
--gitinfo : print git information (if it can be obtained)

Note

calwf3 can be run on a filename which represents a single image which is not RAW (_raw.fits) file input. Ideally, you should use the appropriate standalone component (e.g., wf3rej) to process an intermediate product, but calwf3 can perform this processing. This capability of calwf3 should be used with caution.

Running Many Files at the Same Time

The C command line executable only accepts one file at a time, but you can use operating system tools like awk to process many _raw.fits files in a directory:

ls *raw.fits | awk '{print "calwf3.e",$1}' | csh

Discussion of the Python Wrapper `calwf3`

Where to Find `calwf3`

The Python wrapper for calwf3 is part of WFC3TOOLS package, and the source code can be downloaded from the Git repository in the Spacetelescope/wfc3tools area. Alternatively, the package can be downloaded from PyPi.

Running `calwf3` from a Python Session Using WFC3TOOLS

The WFC3TOOLS Python wrappers act as a convenience front-end functions for the C code by allowing you to integrate the invocation of calwf3 with additional Python analysis utilities. In order to use the Python calwf3 wrapper from within the Python environment:

from wfc3tools import calwf3
filename = 'path/to/input/filename.fits'
calwf3(filename, save_tmp=True, verbose=True)

Parameter Options

input : str, default=None
- Single filename (iaa012wdq_raw.fits)
- Filename of an ASN table (ibfma4030_asn.fits)
printtimebool, default=False
If True, print a detailed time stamp.
save_tmpbool, default=False
If True, save temporary files.
verbosebool, optional, default=False
If True, print verbose time stamps and information.
debugbool, default=False
If True, print optional debugging statements.
parallelbool, default=True
If True, run the code with OpemMP parallel processing turned on for the UVIS CTE correction.
log_funcfunc(), default=print()
If not specified, the print function is used for logging to facilitate use in the Jupyter notebook.

Note

The Python calwf3 module does not support all of the command line options available to the C calwf3 executable.

Running Many Files at the Same Time

The recommended method for running calwf3 on many files is to use the calwf3 Python wrapper in the WFC3TOOLS package.

For example:

from wfc3tools import calwf3
from glob import glob

for filename in glob('i*_raw.fits'):
    calwf3(filename)

Displaying Output from `calwf3` in a Jupyter Notebook

When calling calwf3 from a Jupyter notebook or from the Python wrappers, informational text output from the underlying calwf3.e C executable will be passed through print as the program runs and will show up in the user’s cell. This behavior can be customized by passing your own function as the log_func keyword argument to calwf3. As output is read from the underlying program, the calwf3 Python wrapper will call log_func with the contents of each line. The print is an obvious choice for a log function, but this also provides a way to connect calwf3 to the Python logging system by passing the logging.debug function or similar.

If log_func=None is passed, informational text output from the underlying program will be ignored, but the program’s exit code will still be checked for successful completion.

Note

When running in the notebook or from the Python wrappers, the calwf3 module may raise a RuntimeError if the underlying calwf3.e program fails with a non-zero exit code. Review the text output during the calibration call for hints as to what went wrong. Full runtime and error messages are printed to the terminal window and saved in the trailer file (.tra) for every run to help you diagnose the issue.

Types of Files Used as Input to `calwf3`

_asn file: name of an association table
_raw file: name of an individual, uncalibrated exposure
_crj file: name of any sub-product from an association table
_ima file: name of an intermediate step file (IR multiaccum), proceed with caution when using this option

While both CR-SPLIT and REPEAT-OBS exposures from an association get combined using calwf3, dithered observations from an association will be combined using AstroDrizzle. Images taken at a given dither position can be additionally CR-SPLIT (UVIS only) into multiple exposures.

When calwf3 is given an input file, it first discovers which of the above types of files it has been provided, and then ensures the specified input exists. calwf3 then checks to see which DETECTOR was in use for the data acqusition and calls the appropriate processing pipeline, either UVIS or IR.

Association Tables

An association file has a single extension that is a binary FITS table. The table has three columns where the member names (MEMNAME), member types (MEMTYPE), the role which that member plays in the association, and a boolean value for whether the member is present (MEMPRSNT) are displayed. The present value for the product shows “yes” because the data has already been processed and the table updated. calwf3 will check for the existence on disk of each member and then record the type and any products which will be produced from the association.

MEMTYPE	DESCRIPTION
EXP-CRJ	An input CR-SPLIT exposure
EXP-CRn	An input CR-SPLIT exposure for CR-combined image n (multiple sets)
PROD-CRJ	CR-combined output product from a single set
PROD-CRn	CR-combined output product n from multiple sets
EXP-RPT	An input REPEAT-OBS exposure for a single set
EXP-RPn	An input REPEAT-OBS exposure for repeated image n
PROD-RPT	An output product for a REPEAT-OBS combined single set
PROD-RPn	REPEAT-OBS combined output product n from multiple sets
EXP-DTH	An input dithered exposure
PROD-DTH	A dither-combined output product

In order to create a geometrically correct, drizzle-combined product, PROD-DTH exposures are combined only with AstroDrizzle, which executes after calwf3 has finished processing all members.

PROD-RPT and PROD-CRJ products are combined using wf3rej and all output files have the _cr.fits extension.

Here’s an example of what an association table might contain:

#  Table iacr51010_asn.fits[1]  Tue 15:23:02 25-Apr-2017

# row MEMNAME        MEMTYPE        MEMPRSNT
#

IACR51OHQ      EXP-RP1        yes
IACR51OJQ      EXP-RP1        yes
IACR51OKQ      EXP-RP2        yes
IACR51OMQ      EXP-RP2        yes
IACR51010      PROD-DTH       yes
IACR51011      PROD-RP1       yes
IACR51012      PROD-RP2       yes

The association file has four REPEAT-OBS exposures and instructs calwf3 to make three products, one for RP1 members, one for RP2 members, and one dither combination for all of the members.

The IACR51010 will be produced by AstroDrizzle while the IACR51011 and IACR51012 products will be produced by calwf3 using wf3rej.

Intermediate Files

When calwf3 determines that an intermediate product has been given as input, the preferred method is for users to call the stand-alone tasks by hand. However, it will default to looking for the ASN_TAB keyword in the file header and will partially process the table that is specified.

Single Files

As previously noted, the calwf3 processes exposures according to the calibration switch settings in the primary header of the input file. For single exposure processing, the calibration switch keywords, CRCORR and RPTCORR, are set to OMIT by default, as these processes require multiple observations.

UVIS Data Single File FITS Format

The full-frame _raw.fits file for the UVIS detector data contains a primary header data unit (PHDU) with global header information and no data component, as well as multiple extensions in groups of three. Each exposure has a corresponding set of three extensions that are comprised of the science image itself, the error associated with each pixel, and data quality flags for the pixel.

IR Data Single File FITS Format

The input _raw.fits file for the IR detector data also contains a primary header data unit (PHDU) with global header information and no data component, as well as multiple extensions in groups of five. In this case each exposure has a corresponding set of five extensions that are comprised of the science image itself, the error associated with each pixel, data quality flags for the pixel, the number of samples used to calculate the pixel signal, and the accumulated integration time for each pixel. The final output file contains only five extensions, each containing the final values for the entire set of exposures as related to the slope image in the science extension.

Explanation of the FITS Extensions

The science image contains the data from the focal plane array detectors.
The error array contains an estimate of the statistical uncertainty associated with each corresponding science image pixel
The data quality array contains independent flags indicating various status and problem conditions associated with each corresponding pixel in the science image
The sample array (IR ONLY) contains the number of samples used to derive the corresponding pixel values in the science image.
The time array (IR ONLY) contains the effective integration time associated with each corresponding science image pixel value.

Types of Output Files from `calwf3`

The suffixes used for WFC3 raw and calibrated data products closely align to those used by ACS and NICMOS:

SUFFIX	DESCRIPTION	UNITS
_raw	raw data	DN
_rac	UVIS CTE corrected raw data, no other calibration	DN
_asn	association table for observation set
_spt	telescope and WFC3 telemetry and engineering data
_blv_tmp	overscan-trimmed UVIS exposure	DN
_blc_tmp	overscan-trimmed UVIS, CTE corrected exposure	DN
_crj_tmp	uncalibrated, cosmic-ray rejected combined	DN
_crc_tmp	uncalibrated, cosmic-rat rejected, CTE cleaned	DN
_ima	calibrated intermediate IR multiaccum image	\(e^{-}/s\)
_flt	UVIS calibrated exposure	\(e^{-}\)
_flc	UVIS calibrated exposure including CTE correction	\(e^{-}\)
_flt	IR calibrated exposure	\(e^{-}/s\)
_crj	UVIS calibrated, cosmic ray rejected image	\(e^{-}\)
_crj	IR calibrated, cosmic ray rejected image	\(e^{-}/s\)
_crc	UVIS calibrated, CR rejected, CTE cleaned image	\(e^{-}\)
.tra	trailer file, contains processing messages

The DRZ and DRC products are produced with AstroDrizzle which executes once calwf3 completes.

Keyword Usage

calwf3 processing is controlled by the values of keywords in the input image headers. Certain keywords, referred to as calibration switches, are used to control which calibration steps are performed. Reference file keywords indicate which reference files to use in the corresponding calibration steps. Users who wish to perform custom reprocessing of their data may change the values of these keywords in the _raw.fits file primary headers and then rerun the modified file through calwf3. See the WFC3 Data Handbook for a more complete description of these keywords and their values.

Using CRDS to Update Your Reference Files

CRDS is the reference file management software used by STScI for organizing and assigning reference files to datasets. Users can query CRDS to get the best reference files for their data available at the time of the request. The following link explains how you can use this facility via a web interface, Using CRDS to find the best reference files for your data, or from the command line.

UVIS Pipeline

See https://hst-docs.stsci.edu/wfc3dhb/chapter-3-wfc3-data-calibration/3-2-uvis-data-calibration-steps

IR Pipeline

See https://hst-docs.stsci.edu/wfc3dhb/chapter-3-wfc3-data-calibration/3-3-ir-data-calibration-steps

calwf3

Discussion of the C Version of calwf3

Where to Find calwf3

Running calwf3

Command Line Options for the calwf3 C Executable

Running Many Files at the Same Time

Discussion of the Python Wrapper calwf3

Where to Find calwf3

Running calwf3 from a Python Session Using WFC3TOOLS

Parameter Options

Running Many Files at the Same Time

Displaying Output from calwf3 in a Jupyter Notebook

Types of Files Used as Input to calwf3

Association Tables

Intermediate Files

Single Files

UVIS Data Single File FITS Format

IR Data Single File FITS Format

Explanation of the FITS Extensions

Types of Output Files from calwf3

Keyword Usage

Using CRDS to Update Your Reference Files

UVIS Pipeline

IR Pipeline

Discussion of the C Version of `calwf3`

Where to Find `calwf3`

Running `calwf3`

Command Line Options for the `calwf3` C Executable

Discussion of the Python Wrapper `calwf3`

Where to Find `calwf3`

Running `calwf3` from a Python Session Using WFC3TOOLS

Displaying Output from `calwf3` in a Jupyter Notebook

Types of Files Used as Input to `calwf3`

Types of Output Files from `calwf3`