AbstractModelLibrary

class stpipe.library.AbstractModelLibrary(init, asn_exptypes=None, asn_n_members=None, on_disk=False, temp_directory=None, **datamodels_open_kwargs)

Bases: ABC

A “library” of models (loaded from an association file).

Do not anger the librarian!

The library owns all models from the association and it will handle opening and closing files.

Models can be “borrowed” from the library (by iterating through the library or “borrowing” a specific model). However the library must be “open” (used in a with context) to borrow a model and the model must be “shelved” before the library “closes” (the with context exits).

>>> with library:
        model = library.borrow(0)  # borrow the first model
        # do stuff with the model
        library.shelve(model, 0)  # return the model

Failing to “open” the library will result in a ClosedLibraryError.

Failing to “return” a borrowed model will result in a BorrowError.

Create a new ModelLibrary based on the provided “init”.

Parameters:

initstr or Path or list (of DataModels): If a string or Path this should point to a valid association file. If a list of models consider continuing to use the list instead of making a library (the list will be more efficient and easier to use).
asn_exptypeslist of str, optional: List of “exptypes” to load from the “init” value. Any association member with a “exptype” that matches (case insensitive) a value in this list will be available through the library.
asn_n_membersint, optional: Number of association members to include in the library. This filtering will occur after “asn_exptypes” is applied. By default all members will be included.
on_diskbool, optional, default=False: When enabled, keep the models in the library “on disk”, writing out temporary files when the models are “shelved”. This option is only compatible with models loaded from an association (not a list of models).
temp_directorystr or Path, optional: The temporary directory in which to store models when “on_disk” is enabled. By default a tempfile.TemporaryDirectory will be created in the working directory.
**datamodels_open_kwargsdict: Keyword arguments that will provided to _datamodels_open when models are opened.

Attributes Summary

`asn`	Association dictionary used to create the library.
`crds_observatory`	Return the name of the observatory as a string
`group_indices`	A `dict` with "group_id" as keys.
`group_names`	A `set` of association member "group_id" values

Methods Summary

`borrow`(index)	"borrow" a model from the library.
`finalize_result`(step, reference_files_used)	Called from `stpipe.Step.run` after `stpipe.Step.process` has completed.
`get_crds_parameters`()	Get the "crds_parameters" from either:
`map_function`(function[, modify])	Call a function once for each model in the library.
`shelve`(model[, index, modify])	"shelve" a model, returning it to the library.

Attributes Documentation

asn

Association dictionary used to create the library.

Note that changes to the models may cause this information to fall “out-of-sync” with the models. For example, borrowing and changing a models “group_id” metadata will not change the “group_id” entry for the member at the same index in the association dictionary returned for this property.

This is “read-only” to reduce the chances that the information in this dictionary will conflict with the model metadata.

crds_observatory

Return the name of the observatory as a string

Returns:

observatorystr

group_indices

A dict with “group_id” as keys. Each value contains a list of indices of members that shared that “group_id”.

Note that this is based on the member entries in the association used to create the ModelLibrary. Updating the “group_id” of a model in the library will NOT change the values returned by this property (and will not change the association data available at asn).

group_names

A set of association member “group_id” values

This property (similar to asn) can fall “out-of-sync” with the model metadata. The names returned here will only reflect the “group_id” at the time of library creation.

Methods Documentation

borrow(index)

“borrow” a model from the library.

Every model “borrowed” from the library must be returned before the library is closed (see the exceptions described below).

For an “on_disk” library a calling this function will cause the corresponding model to be loaded from disk.

Parameters:

indexint: The index of the model within the library. As the library size does not change this is unique to this model for this library.

Returns:

modelDataModel

Raises:

ClosedLibraryError: If the library is not “open” (used in a with context).
BorrowError: If the model at this index is already “borrowed”.

finalize_result(step, reference_files_used)

Called from stpipe.Step.run after stpipe.Step.process has completed. This method will call stpipe.Step.finalize_result for each model in the library.

Parameters:

stepStep: The step calling this method.
reference_files_usedlist: List of reference files used during the execution of this step.

get_crds_parameters()

Get the “crds_parameters” from either:

the first “science” member (based on exptype)
the first model (if no “science” member is found)

If no “science” members are found in the library a UserWarning will be issued.

Returns:

crds_parametersdict: The result of get_crds_parameters called on the selected model.

map_function(function, modify=True)

Call a function once for each model in the library.

Similar to the built-in map function this applies the function “function” to each model in the library by iterating through the models (yielding on each function call).

Parameters:

functioncallable: A function that accepts 2 arguments, model and index
modifybool, optional, default=True: For an “on_disk” library, temporary files will only be written when modify is True. For a library that is not “on_disk” this option has no effect and any modifications made to the model while it was borrowed will persist.

Returns:

result_itergenerator: A generator that will return function results. This can be converted to a list list(result_iter) to get the results of all the function calls (ordered the same as the models in the library).

shelve(model, index=None, modify=True)

“shelve” a model, returning it to the library.

All borrowed models must be “shelved” before the library is closed (see notes about exceptions below).

For an “on_disk” model shelving a model may cause a temporary file to be written (see “modify” below).

Parameters:

modelDataModel: DataModel to return to the library at the provided index.
indexint, optional: The index within the library where the model will be stored. If the same model (checked by id(model)) was previously borrowed and the index is not provided, the borrowed index will be used. If providing a new DataModel (one not borrowed from the library) the index must be provided, and the new model replaces the prior model at that index.
modifybool, optional, default=True: For an “on_disk” library, temporary files will only be written when modify is True. For a library that is not “on_disk” this option has no effect and any modifications made to the model while it was borrowed will persist.

Raises:

ClosedLibraryError: If the library is not “open” (used in a with context).
BorrowError: If an unknown model is provided (without an index) or if the model at the provided index has not been “borrowed”