******** Overview ******** A subset of the tools in the WFC3TOOLS package allows the user to employ the same software that STScI uses in the science calibration pipelines to calibrate WFC3 data from both the UVIS and IR detectors. This means the user can reprocess data with the latest software releases and/or reference files at will. This also means the user can customize the actual calibrations applied to the data by modifying the settings (e.g., OMIT vs PERFORM) of calibration steps in the primary header of the RAW FITS file. The set of calibration tools resident in the WFC3TOOLS package are actually thin Python wrappers around C executables. While the C code is performing the heavy lifting, the Python wrapper acts as a convenience front-end function for the C code. In order to use the Python tools which are actually wrappers, the user must also obtain the HSTCAL package which contains the C version of the pipeline programs. .. note:: While a calibration tool will be referenced generically in this discussion as, for example, `calwf3`, the file actually being executed in C is `calwf3.e` and in Python is `calwf3.py/calwf3.pyc`. `calwf3` ******** The WFC3 calibration pipeline used for processing data from the UVIS or IR detectors is called `calwf3`. While `calwf3` is a single executable which can fully process WFC3 data, the pipeline has been designed such that a number of its components can also be invoked as standalone executables. Aside from the `calwf3` executable itself, the standalone executable components are: :ref:`wf3cte`, :ref:`wf3ccd`, :ref:`wf32d`, :ref:`wf3ir`, :ref:`wf3rej`, and `wf3sum`. The first three components apply to UVIS data, :ref:`wf3ir` applies to IR data, and the :ref:`wf3rej` program is used for data from both detectors to combine multiple exposures contained in a CR-SPLIT or REPEAT-OBS set. The `wf3sum` program sums together the IMSET chips of REPEAT-OBS exposures, and there is no Python wrapper for the `wf3sum` executable. The `calwf3` processes exposures according to the setting (PERFORM, OMIT, COMPLETE, SKIPPED) of the calibration switch keywords (e.g., FLATCORR, PCTECORR) in the primary header of the input file. The calibration switch keywords directly correspond to calibration processing steps. A flow diagram representing the calibration steps for the UVIS data is :numref:`calflow`, while the complementary diagram for the IR data is :numref:`irflow`. The calibrated output products generated by `calwf3` for UVIS are the FLT and CRJ (when applicable) files, as well as the Charge Transfer Efficiency corrected (CTE-corrected) versions of these files, FLC and CRC. Correspondingly, the output products for the IR detector are the FLT and CRJ (when applicable) files. `HST File Naming Convention `_ available through the archive has a thorough description of how files are named and what those names mean. See also the :ref:`Types of Output Files From calwf3 ` section in this document. .. note:: For completeness of this discussion, AstroDrizzle functionality has been part of the automated calibration pipeline since 2012. AstroDrizzle removes geometric distortion, corrects for sky background variations, flags cosmic-rays, and combines images with optional subsampling. AstroDrizzle generates the calibrated drizzled data products DRZ and DRC, where the latter is for CTE-corrected images. See the `WFC3 Data Handbook `__ or `DrizzlePac `__ for more information. As previously noted, the HSTCAL package is needed to support the calibration interface wrappers contained in the WFC3TOOLS package. ======================================= Discussion of the C Version of `calwf3` ======================================= Where to Find `calwf3` ====================== The C code for `calwf3` is part of HSTCAL package, and the source code can be downloaded from its Git repository `Spacetelescope/hstcal `_. Downloading the source code requires compiling and linking the package with third-party software which can sometimes be tricky. Alternatively, the HSTCAL binaries can be downloaded from `conda-forge `_. High-level release notes for all of the HSTCAL package updates can be found in its Git repository at `Releases `__. The current WFC3 Data Handbook can be found `here `__. If you have questions not addressed in this documentation or need help with installing or using the software in a timely fashion, please contact the `STScI Help Desk `_. You may also submit a GitHub issue for either the HSTCAL or WFC3TOOLS repositories. Running `calwf3` ================ The `calwf3` C executable can be run on a single input RAW file or an ASN table listing the members of an association. When processing an association, it retrieves calibration switch and reference file keyword settings from the first image listed in the ASN table. `calwf3` does not accept a user-defined *list* of input images or wildcards on the command line (e.g. ``*_raw.fits`` cannot be used to process all raw files in the current directory). A more extensive discussion regarding required input and options for `calwf3` follows in the next section. The :ref:`wf3ccd`, :ref:`wf32d`, :ref:`wf3cte` and :ref:`wf3ir` tasks on the other hand, will accept user-defined input file lists, but they will not accept an association table (``_asn.fits``) as input. The standalone components :ref:`wf3rej` and `wf3sum` do not accept lists. Command Line Options for the `calwf3` C Executable -------------------------------------------------- `calwf3` can be called directly from the operating system command line by supplying the executable `calwf3.e` with an input file and a list of options. This is the same executable that the WFC3TOOLS package wraps with Python code. .. code-block:: shell calwf3.e [options] input calwf3.e -vts iaa012wdq_raw.fits input : str Name of input file - single filename (_raw.fits or _crj.fits) - filename of an ASN table (_asn.fits) options -d : print optional debugging statements -q : print messages only to the trailer file -r : print version number and date of software (e.g., Current version: 3.6.2 (May-27-2021)) -s : save temporary files -t : print a detailed time stamp -v : print verbose time stamps and information -1 : suppress the OpemMP parallel processing for the UVIS CTE correction --help : print the syntax for executing this command --version : print version number of software (e.g., 3.6.2) --gitinfo : print git information (if it can be obtained) .. note:: `calwf3` can be run on a filename which represents a single image which is not RAW (``_raw.fits``) file input. Ideally, you should use the appropriate standalone component (e.g., `wf3rej`) to process an intermediate product, but `calwf3` can perform this processing. This capability of `calwf3` should be used with caution. Running Many Files at the Same Time =================================== The C command line executable only accepts one file at a time, but you can use operating system tools like `awk` to process many ``_raw.fits`` files in a directory: .. code-block:: shell ls *raw.fits | awk '{print "calwf3.e",$1}' | csh ========================================= Discussion of the Python Wrapper `calwf3` ========================================= Where to Find `calwf3` ====================== The Python wrapper for `calwf3` is part of WFC3TOOLS package, and the source code can be downloaded from the Git repository in the `Spacetelescope/wfc3tools `__ area. Alternatively, the package can be downloaded from `PyPi `_. Running `calwf3` from a Python Session Using WFC3TOOLS ====================================================== The WFC3TOOLS Python wrappers act as a convenience front-end functions for the C code by allowing you to integrate the invocation of `calwf3` with additional Python analysis utilities. In order to use the Python `calwf3` wrapper from within the Python environment: .. code-block:: python from wfc3tools import calwf3 filename = 'path/to/input/filename.fits' calwf3(filename, save_tmp=True, verbose=True) Parameter Options ----------------- * **input : str, default=None** * Single filename (iaa012wdq_raw.fits) * Filename of an ASN table (ibfma4030_asn.fits) * printtime : bool, default=False If True, print a detailed time stamp. * save_tmp : bool, default=False If True, save temporary files. * verbose : bool, optional, default=False If True, print verbose time stamps and information. * debug : bool, default=False If True, print optional debugging statements. * parallel : bool, default=True If True, run the code with OpemMP parallel processing turned on for the UVIS CTE correction. * log_func : func(), default=print() If not specified, the print function is used for logging to facilitate use in the Jupyter notebook. .. note:: The Python `calwf3` module does not support all of the command line options available to the C `calwf3` executable. Running Many Files at the Same Time =================================== The recommended method for running `calwf3` on many files is to use the `calwf3` Python wrapper in the WFC3TOOLS package. For example: .. code-block:: python from wfc3tools import calwf3 from glob import glob for filename in glob('i*_raw.fits'): calwf3(filename) Displaying Output from `calwf3` in a Jupyter Notebook ===================================================== When calling `calwf3` from a Jupyter notebook or from the Python wrappers, informational text output from the underlying `calwf3.e` C executable will be passed through `print` as the program runs and will show up in the user's cell. This behavior can be customized by passing your own function as the `log_func` keyword argument to `calwf3`. As output is read from the underlying program, the `calwf3` Python wrapper will call `log_func` with the contents of each line. The `print` is an obvious choice for a log function, but this also provides a way to connect `calwf3` to the Python logging system by passing the `logging.debug` function or similar. If `log_func=None` is passed, informational text output from the underlying program will be ignored, but the program's exit code will still be checked for successful completion. .. note:: When running in the notebook or from the Python wrappers, the `calwf3` module may raise a RuntimeError if the underlying `calwf3.e` program fails with a non-zero exit code. Review the text output during the calibration call for hints as to what went wrong. Full runtime and error messages are printed to the terminal window and saved in the trailer file (.tra) for every run to help you diagnose the issue. ========================================= Types of Files Used as Input to `calwf3` ======================================== * _asn file: name of an association table * _raw file: name of an individual, uncalibrated exposure * _crj file: name of any sub-product from an association table * _ima file: name of an intermediate step file (IR multiaccum), proceed with caution when using this option While both CR-SPLIT and REPEAT-OBS exposures from an association get combined using `calwf3`, dithered observations from an association will be combined using AstroDrizzle. Images taken at a given dither position can be additionally CR-SPLIT (UVIS only) into multiple exposures. When `calwf3` is given an input file, it first discovers which of the above types of files it has been provided, and then ensures the specified input exists. `calwf3` then checks to see which DETECTOR was in use for the data acqusition and calls the appropriate processing pipeline, either UVIS or IR. Association Tables ------------------ An association file has a single extension that is a binary FITS table. The table has three columns where the member names (MEMNAME), member types (MEMTYPE), the role which that member plays in the association, and a boolean value for whether the member is present (MEMPRSNT) are displayed. The present value for the product shows "yes" because the data has already been processed and the table updated. `calwf3` will check for the existence on disk of each member and then record the type and any products which will be produced from the association. ======== ===================================================================== MEMTYPE DESCRIPTION ======== ===================================================================== EXP-CRJ An input CR-SPLIT exposure EXP-CRn An input CR-SPLIT exposure for CR-combined image n (multiple sets) PROD-CRJ CR-combined output product from a single set PROD-CRn CR-combined output product n from multiple sets EXP-RPT An input REPEAT-OBS exposure for a single set EXP-RPn An input REPEAT-OBS exposure for repeated image n PROD-RPT An output product for a REPEAT-OBS combined single set PROD-RPn REPEAT-OBS combined output product n from multiple sets EXP-DTH An input dithered exposure PROD-DTH A dither-combined output product ======== ===================================================================== In order to create a geometrically correct, drizzle-combined product, PROD-DTH exposures are combined only with AstroDrizzle, which executes after `calwf3` has finished processing all members. PROD-RPT and PROD-CRJ products are combined using :ref:`wf3rej` and all output files have the ``_cr.fits`` extension. Here's an example of what an association table might contain: .. code-block:: shell # Table iacr51010_asn.fits[1] Tue 15:23:02 25-Apr-2017 # row MEMNAME MEMTYPE MEMPRSNT # 1 IACR51OHQ EXP-RP1 yes 2 IACR51OJQ EXP-RP1 yes 3 IACR51OKQ EXP-RP2 yes 4 IACR51OMQ EXP-RP2 yes 5 IACR51010 PROD-DTH yes 6 IACR51011 PROD-RP1 yes 7 IACR51012 PROD-RP2 yes The association file has four REPEAT-OBS exposures and instructs `calwf3` to make three products, one for RP1 members, one for RP2 members, and one dither combination for all of the members. The IACR51010 will be produced by AstroDrizzle while the IACR51011 and IACR51012 products will be produced by `calwf3` using :ref:`wf3rej`. Intermediate Files ------------------ When `calwf3` determines that an intermediate product has been given as input, the preferred method is for users to call the stand-alone tasks by hand. However, it will default to looking for the ASN_TAB keyword in the file header and will partially process the table that is specified. Single Files ------------ As previously noted, the `calwf3` processes exposures according to the calibration switch settings in the primary header of the input file. For single exposure processing, the calibration switch keywords, CRCORR and RPTCORR, are set to OMIT by default, as these processes require multiple observations. UVIS Data Single File FITS Format ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The full-frame ``_raw.fits`` file for the UVIS detector data contains a primary header data unit (PHDU) with global header information and no data component, as well as multiple extensions in groups of three. Each exposure has a corresponding set of three extensions that are comprised of the science image itself, the error associated with each pixel, and data quality flags for the pixel. .. _uvis_data_format: .. figure:: ../_static/uvis_data_format.png :align: center :alt: UVIS data raw file format UVIS data raw file format IR Data Single File FITS Format ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The input ``_raw.fits`` file for the IR detector data also contains a primary header data unit (PHDU) with global header information and no data component, as well as multiple extensions in groups of five. In this case each exposure has a corresponding set of five extensions that are comprised of the science image itself, the error associated with each pixel, data quality flags for the pixel, the number of samples used to calculate the pixel signal, and the accumulated integration time for each pixel. The final output file contains only five extensions, each containing the final values for the entire set of exposures as related to the slope image in the science extension. .. _ir_data_format: .. figure:: ../_static/ir_data_format.png :align: center :alt: IR data raw file format IR data raw file format Explanation of the FITS Extensions ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ * The science image contains the data from the focal plane array detectors. * The error array contains an estimate of the statistical uncertainty associated with each corresponding science image pixel * The data quality array contains independent flags indicating various status and problem conditions associated with each corresponding pixel in the science image * The sample array (IR ONLY) contains the number of samples used to derive the corresponding pixel values in the science image. * The time array (IR ONLY) contains the effective integration time associated with each corresponding science image pixel value. .. _target_to_file_extensions: Types of Output Files from `calwf3` =================================== The suffixes used for WFC3 raw and calibrated data products closely align to those used by ACS and NICMOS: ======== ================================================= ==================== SUFFIX DESCRIPTION UNITS ======== ================================================= ==================== _raw raw data DN _rac UVIS CTE corrected raw data, no other calibration DN _asn association table for observation set _spt telescope and WFC3 telemetry and engineering data _blv_tmp overscan-trimmed UVIS exposure DN _blc_tmp overscan-trimmed UVIS, CTE corrected exposure DN _crj_tmp uncalibrated, cosmic-ray rejected combined DN _crc_tmp uncalibrated, cosmic-rat rejected, CTE cleaned DN _ima calibrated intermediate IR multiaccum image :math:`e^{-}/s` _flt UVIS calibrated exposure :math:`e^{-}` _flc UVIS calibrated exposure including CTE correction :math:`e^{-}` _flt IR calibrated exposure :math:`e^{-}/s` _crj UVIS calibrated, cosmic ray rejected image :math:`e^{-}` _crj IR calibrated, cosmic ray rejected image :math:`e^{-}/s` _crc UVIS calibrated, CR rejected, CTE cleaned image :math:`e^{-}` .tra trailer file, contains processing messages ======== ================================================= ==================== **The DRZ and DRC products are produced with** `AstroDrizzle `__ **which executes once** `calwf3` **completes.** Keyword Usage ============= `calwf3` processing is controlled by the values of keywords in the input image headers. Certain keywords, referred to as calibration switches, are used to control which calibration steps are performed. Reference file keywords indicate which reference files to use in the corresponding calibration steps. Users who wish to perform custom reprocessing of their data may change the values of these keywords in the ``_raw.fits`` file primary headers and then rerun the modified file through `calwf3`. See the `WFC3 Data Handbook `_ for a more complete description of these keywords and their values. Using CRDS to Update Your Reference Files ========================================= CRDS is the reference file management software used by STScI for organizing and assigning reference files to datasets. Users can query CRDS to get the best reference files for their data available at the time of the request. The following link explains how you can use this facility via a web interface, `Using CRDS to find the best reference files for your data `_, or from the `command line `_. .. include:: ./uvis_pipeline.inc .. include:: ./ir_pipeline.inc