Step

class stpipe.Step(name=None, parent=None, config_file=None, _validate_kwds=True, **kws)

Bases: object

Create a Step instance.

Parameters:

namestr: The name of the Step instance. Used in logging messages and in cache filenames. If not provided, one will be generated based on the class name.
parentStep: The parent step of this step. Used to determine a fully-qualified name for this step, and to determine the mode in which to run this step.
config_filestr or pathlib.Path: The path to the config file that this step was initialized with. Use to determine relative path names of other config files.
_validate_kwdsbool: Validate given kws against specs/config.
**kwsdict: Additional parameters to set. These will be set as member variables on the new Step instance.

Attributes Summary

`class_alias`
`correction_pars`
`input_dir`
`log`
`log_records`	Retrieve logs from the most recent run of this step.
`make_output_path`	Return function that creates the output path
`prefetch_references`
`reference_file_types`
`spec`
`use_correction_pars`

Methods Summary

`build_config`(input, **kwargs)	Build the ConfigObj to initialize a Step
`call`(args, *kwargs)	Create and run a new instance of the class.
`default_output_file`([input_file])	Create a default filename based on the input name
`default_suffix`()	Return a default suffix based on the step
`export_config`(filename[, include_metadata])	Export this step's parameters to an ASDF config file.
`finalize_result`(result, reference_files_used)	Hook that allows subclasses to set mission-specific metadata on each step result before that result is saved.
`from_cmdline`(args)	Create a step from a configuration file.
`from_config_file`(config_file[, parent, name])	Create a step from a configuration file.
`from_config_section`(config[, parent, name, ...])	Create a step from a configuration file fragment.
`get_config_from_reference`(dataset[, ...])	Retrieve step parameters from reference database
`get_config_reftype`()	Get the CRDS reftype for this step's config reference.
`get_pars`([full_spec])	Retrieve the configuration parameters of a step
`get_ref_override`(reference_file_type)	Determine and return any override for `reference_file_type`.
`get_reference_file`(input_file, ...)	Get a reference file from CRDS.
`get_stpipe_loggers`()	Get the names of loggers to configure.
`load_spec_file`([preserve_comments])
`make_input_path`(file_path)	Create an input path for a given file path
`merge_config`(config, config_file)
`open_model`(init, **kwargs)	Open a datamodel
`prefetch`(*args)	Prefetch reference files, nominally called when self.prefetch_references is True.
`print_configspec`()
`process`(*args)	This is where real work happens.
`remove_suffix`(name)	Remove a known Step filename suffix from a filename (if present).
`run`(*args)	Run handles the generic setup and teardown that happens with the running of each step.
`save_model`(model[, suffix, idx, ...])	Saves the given model using the step/pipeline's naming scheme
`search_attr`(attribute[, default, parent_first])	Return first non-None attribute in step hierarchy
`set_primary_input`(obj[, exclusive])	Sets the name of the master input file and input directory.
`update_pars`(parameters)	Update step parameters

Attributes Documentation

class_alias = None

correction_pars = None

input_dir

log

log_records

Retrieve logs from the most recent run of this step.

Returns:

list of str

make_output_path: Return function that creates the output path

prefetch_references = True

reference_file_types: ClassVar = []

spec

pre_hooks          = list(default=list())        # List of Step classes to run before step
post_hooks         = list(default=list())        # List of Step classes to run after step
output_file        = output_file(default=None)   # File to save output to.
output_dir         = string(default=None)        # Directory path for output files
output_ext         = string()                    # Default type of output
output_use_model   = boolean(default=False)      # When saving use `DataModel.meta.filename`
output_use_index   = boolean(default=True)       # Append index.
save_results       = boolean(default=False)      # Force save results
skip               = boolean(default=False)      # Skip this step
suffix             = string(default=None)        # Default suffix for output files
search_output_file = boolean(default=True)       # Use outputfile define in parent step
input_dir          = string(default=None)        # Input directory

use_correction_pars = False

Methods Documentation

classmethod build_config(input, **kwargs)

Build the ConfigObj to initialize a Step

A Step config is built in the following order:

CRDS parameter reference file
Local parameter reference file
Step keyword arguments

Parameters:

inputstr or None: Input file
kwargsdict: Keyword arguments that specify Step parameters.

Returns:

config, config_fileConfigObj, str: The configuration and the config filename.

classmethod call(*args, **kwargs)

Create and run a new instance of the class.

Gets a config file from CRDS if one is available.

By default, log handlers are added for the duration of the run. To avoid configuring the log, specify configure_log=False in the keyword arguments.

To set configuration parameters, pass a config_file path or keyword arguments. Keyword arguments override those in the specified config_file.

Any positional *args will be passed along to the step’s process method.

Note: this method creates a new instance of Step with the given config_file if supplied, plus any extra *args and **kwargs. If you create an instance of a Step, set parameters, and then use this call() method, it will ignore previously-set parameters, as it creates a new instance of the class with only the config_file, *args and **kwargs passed to the call() method.

default_output_file(input_file=None): Create a default filename based on the input name

default_suffix(): Return a default suffix based on the step

export_config(filename, include_metadata=False)

Export this step’s parameters to an ASDF config file.

Parameters:

filenamestr or pathlib.Path: Path to config file.
include_metadatabool, optional: Set to True to include metadata that is required for submission to CRDS.

finalize_result(result, reference_files_used)

Hook that allows subclasses to set mission-specific metadata on each step result before that result is saved.

Parameters:

resulta datamodel that is an instance of AbstractDataModel or: collections.abc.Sequence One step result (potentially of many).
reference_files_usedlist of tuple: List of reference files used when running the step, each a tuple in the form (str reference type, str reference URI).

static from_cmdline(args)

Create a step from a configuration file.

Parameters:

argslist of str: Commandline arguments

Returns:

stepStep instance

If the config file has a class parameter, the return value will be as instance of that class.

Any parameters found in the config file will be set as member variables on the returned Step instance.

classmethod from_config_file(config_file, parent=None, name=None)

Create a step from a configuration file.

Parameters:

config_filepath or readable file-like object: The config file to load parameters from
parentStep instance, optional: The parent step of this step. Used to determine a fully-qualified name for this step, and to determine the mode in which to run this step.
namestr, optional: If provided, use that name for the returned instance. If not provided, the following are tried (in order): - The name parameter in the config file - The filename of the config file - The name of returned class

Returns:

stepStep instance

If the config file has a class parameter, the return value will be as instance of that class. The class parameter in the config file must specify a subclass of cls. If the configuration file has no class parameter, then an instance of cls is returned.

Any parameters found in the config file will be set as member variables on the returned Step instance.

classmethod from_config_section(config, parent=None, name=None, config_file=None)

Create a step from a configuration file fragment.

Parameters:

configconfigobj.Section instance: The config file fragment containing parameters for this step only.
parentStep instance, optional: The parent step of this step. Used to determine a fully-qualified name for this step, and to determine the mode in which to run this step.
namestr, optional: If provided, use that name for the returned instance. If not provided, try the following (in order): - The name parameter in the config file fragment - The name of returned class
config_filestr or pathlib.Path, optional: The path to the config file that created this step, if any. This is used to resolve relative file name parameters in the config file.

Returns:

stepinstance of cls: Any parameters found in the config file fragment will be set as member variables on the returned Step instance.

classmethod get_config_from_reference(dataset, disable=None, crds_observatory=None)

Retrieve step parameters from reference database

Parameters:

clsstpipe.Step: Either a class or instance of a class derived from Step.
datasetAbstractDataModel or dict: A model of the input file. Metadata on this input file will be used by the CRDS “bestref” algorithm to obtain a reference file. If a dict, crds_observatory must be a non-None value.
disable: bool or None: Do not retrieve parameters from CRDS. If None, check global settings.
crds_observatorystr: Observatory name (‘jwst’ or ‘roman’).

Returns:

step_parametersconfigobj: The parameters as retrieved from CRDS. If there is an issue, log as such and return an empty config obj.

classmethod get_config_reftype()

Get the CRDS reftype for this step’s config reference.

Returns:

str

get_pars(full_spec=True)

Retrieve the configuration parameters of a step

Parameters:

full_specbool: Return all parameters, including parent-specified parameters. If False, return only parameters specific to the step.

Returns:

dict: Keys are the parameters and values are the values.

get_ref_override(reference_file_type)

Determine and return any override for reference_file_type.

Returns:

override_filepath or None.

get_reference_file(input_file, reference_file_type)

Get a reference file from CRDS.

If the configuration file or commandline parameters override the reference file, it will be automatically used when calling this function.

Parameters:

input_filea datamodel that is an instance of AbstractDataModel: A model of the input file. Metadata on this input file will be used by the CRDS “bestref” algorithm to obtain a reference file.
reference_file_typestring: The type of reference file to retrieve. For example, to retrieve a flat field reference file, this would be ‘flat’.

Returns:

reference_filepath of reference file, a string

static get_stpipe_loggers()

Get the names of loggers to configure.

For the base class, the root logger is currently returned, to allow downstream classes time to transition to specifying their loggers. In future builds, the base class logger will default to “stpipe” only.

Returns:

loggerstuple of str: Tuple of log names to configure.

classmethod load_spec_file(preserve_comments=<stpipe.utilities._NotSet object>)

make_input_path(file_path)

Create an input path for a given file path

If file_path has no directory path, use self.input_dir as the directory path.

Parameters:

file_pathstr or obj: The supplied file path to check and modify. If anything other than str, the object is simply passed back.

Returns:

full_pathstr or obj: File path using input_dir if the input had no directory path.

classmethod merge_config(config, config_file)

open_model(init, **kwargs)

Open a datamodel

Primarily a wrapper around DataModel.open to handle Step peculiarities

Parameters:

initobject: The object to open

Returns:

datamodelinstance of AbstractDataModel: Object opened as a datamodel

prefetch(*args): Prefetch reference files, nominally called when self.prefetch_references is True. Can be called explicitly when self.prefetch_refences is False.

classmethod print_configspec()

process(*args): This is where real work happens. Every Step subclass has to override this method. The default behaviour is to raise a NotImplementedError exception.

static remove_suffix(name)

Remove a known Step filename suffix from a filename (if present).

Parameters:

namestr: Filename.

Returns:

str: Filename with any known suffix removed.
str: Separator that delimited the original suffix.

run(*args): Run handles the generic setup and teardown that happens with the running of each step. The real work that is unique to each step type is done in the process method.

save_model(model, suffix=None, idx=None, output_file=None, force=False, **components)

Saves the given model using the step/pipeline’s naming scheme

Parameters:

modela instance of AbstractDataModel: The model to save.
suffixstr: The suffix to add to the filename.
idxobject: Index identifier.
output_filestr: Use this file name instead of what the Step default would be.
forcebool: Regardless of whether save_results is False and no output_file is specified, try saving.
componentsdict: Other components to add to the file name.

Returns:

output_paths[str[, …]]: List of output file paths the model(s) were saved in.

search_attr(attribute, default=None, parent_first=False)

Return first non-None attribute in step hierarchy

Parameters:

attributestr: The attribute to retrieve
defaultobj: If attribute is not found, the value to use
parent_firstbool: If True, allow parent definition to override step version

Returns:

valueobj: Attribute value or default if not found

set_primary_input(obj, exclusive=True)

Sets the name of the master input file and input directory. Used to generate output file names.

Parameters:

objstr, pathlib.Path, or instance of AbstractDataModel: The object to base the name on. If a datamodel, use Datamodel.meta.filename.
exclusivebool: If True, only set if an input name is not already used by a parent Step. Otherwise, always set.

update_pars(parameters)

Update step parameters

Only existing parameters are updated. Otherwise, new keys found in parameters are ignored.

Parameters:

parametersdict: Parameters to update.

Notes

parameters is presumed to have been produced by the Step.get_pars method. As such, the “steps” key is treated special in that it is a dict whose keys are the steps assigned directly as parameters to the current step. This is standard practice for Pipeline-based steps.