Job Options

JobOptions

JobOptions are used to store parameters for jobs. They contain all the info necessary for on-the-fly GUI generation.

class pipeliner.job_options.BooleanJobOption(*, label: str, default_value: bool, help_text: str = '', in_continue: bool = False, only_in_continue: bool = False, deactivate_if: JobOptionCondition | None = None, jobop_group: str = 'Main')

Bases: JobOption

Define a job option as a boolean

Parameters:

  • label (str) – A verbose label for the parameter. This is what appears in a run.job file

  • default_value (bool) – The default value

  • help_text (str) – Text that will be displayed in the GUI if help is clicked

  • in_continue (bool) – If this parameter can be modified in a job that is continued

get_boolean() → bool

Get a boolean value

Returns:

The value

Return type:

bool

joboption_type = 'BOOLEAN'
validate() → List[JobOptionValidationResult]

Basic validation of the input parameters

Returns:

The validation

error of one exists

Return type:

JobOptionValidationResult

class pipeliner.job_options.DirPathJobOption(*, label: str, default_value: str = '', help_text: str = '', must_be_in_project: bool = False, must_exist: bool = True, in_continue: bool = False, only_in_continue: bool = False, is_required: bool = False, required_if: JobOptionCondition | None = None, deactivate_if: JobOptionCondition | None = None, validation_regex: str | None = None, regex_error_message: str | None = None, jobop_group: str = 'Main')

Bases: StringJobOption

A JobOption for directory paths

Parameters:

  • label (str) – A verbose label for the parameter.

  • default_value (None)

  • help_text (str) – Text that will be displayed in the GUI if help is clicked.

  • in_continue (bool) – If this parameter can be modified in a job that is continued

  • must_be_in_project (bool) – Does the directory have to be in the project

  • must_exist (bool) – Does this directory have to exist when the job option is validated? Usually true, unless it will be created by a previous command in the job

check_dir() → List[JobOptionValidationResult]

Check that the specified dir exists.

Returns:

An error if the dir does not exist, or a warning if the dir does not exist and must_exist is False

Return type:

List[JobOptionValidationResult]

joboption_type = 'DIRPATH'
validate() → List[JobOptionValidationResult]

Basic validation of the input parameters

class pipeliner.job_options.ExternalFileJobOption(*, label: str, pattern: FileDescription | None = None, default_value: str = '', help_text: str = '', in_continue: bool = False, only_in_continue: bool = False, is_required: bool = False, required_if: JobOptionCondition | None = None, deactivate_if: JobOptionCondition | None = None, validation_regex: str | None = None, regex_error_message: str | None = None, suggestion: str = '', jobop_group: str = 'Main', command_expected: bool = False)

Bases: JobOption

Define a job option as a file name

GUI will open a file browser for the user to find an input

Parameters:

  • label (str) – A verbose label for the parameter. This is what appears in a run.job file

  • default_value (str) – will usually be set from the string and pattern.

  • pattern (str) – Info about the search string for file types. It should be in the format <description> (<example>) e.g.: “MRC files (*.mrc)”

  • help_text (str) – Text that will be displayed in the GUI if help is clicked

  • in_continue (bool) – If this parameter can be modified in a job that is continued

  • suggestion (str) – A hint that will appear in the data entry field if no default value is set, it will not be used. IE: ‘Enter file path here’ If you want the suggestion used as the default then just put it in default_value

  • command_expected (bool) – Is the value expected to be an executable that can be found with shutil.which()? If so, an executable name or a path to an executable file are both acceptable. If not, the value should be a path to an existing file on disk.

check_file() → List[JobOptionValidationResult]

Check that specified input files exist.

If command_expected is True, the value is checked with shutil.which, which allows the name of an executable on the PATH or the path to an executable file.

If command_expected is False, the value should be the path to a file.

Returns:

An error message if the file does not exist

Return type:

str

joboption_type = 'FILENAME'
validate() → List[JobOptionValidationResult]

Basic validation of the input parameters

Returns:

The

validation error of one exists

Return type:

List[JobOptionValidationResult]

class pipeliner.job_options.FileDescription(name: str = 'Any file', exts: List[str] | None = None, exact: bool = False)

Bases: object

Produce a description of files and their extensions

Parameters:

  • name (str) – The type of file IE: ‘Micrograph movies’

  • exts (list) – The acceptable extensions or file search strings, e.g. ‘.txt’ to find *.txt or ‘_half1.star’ for *_half1.star

  • exact (bool) – Match the file name(s) exactly, do not prepend a *

class pipeliner.job_options.FloatJobOption(*, label: str, default_value: str | float | None, suggested_min: float | None = None, suggested_max: float | None = None, step_value: float = 1.0, help_text: str = '', in_continue: bool = False, only_in_continue: bool = False, is_required: bool = False, required_if: JobOptionCondition | None = None, deactivate_if: JobOptionCondition | None = None, hard_min: float | None = None, hard_max: float | None = None, jobop_group: str = 'Main')

Bases: JobOption

Define a job option as a slider for inputting numbers as floats

The min value and max value are for display only, they do not limit the actual values that can be input

Parameters:

  • label (str) – A verbose label for the parameter. This is what appears in a run.job file

  • default_value (float) – The default value for the parameter

  • suggested_min (float) – The suggested minimum value for the slider

  • suggested_max (float) – The suggested maximum value for the slider

  • step_value (float) – The step value for the slider

  • help_text (str) – Text that will be displayed in the GUI if help is clicked

  • in_continue (bool) – If this parameter can be modified in a job that is continued

  • hard_min (float) – An error will be raised if the value is < this value

  • hard_max (float) – An error will be raised if the value is > this value

get_number() → float

Get the value of a JobOption as a number

Returns:

The value

Return type:

float

Raises:
joboption_type = 'FLOAT'
validate() → List[JobOptionValidationResult]

Basic validation check on the job option

Returns:

The validation

error of one exists

Return type:

JobOptionValidationResult

class pipeliner.job_options.InputNodeJobOption(*, label: str, node_type: str, pattern: FileDescription | None = None, default_value: str = '', help_text: str = '', in_continue: bool = False, only_in_continue: bool = False, is_required: bool = False, required_if: JobOptionCondition | None = None, deactivate_if: JobOptionCondition | None = None, validation_regex: str | None = None, regex_error_message: str | None = None, suggestion: str = '', jobop_group: str = 'Main', node_kwds: List[str] | None = None)

Bases: JobOption

Define a job option as a file name, create an input node for it

GUI will open a file browser for the user to find an input

Parameters:

  • label (str) – A verbose label for the parameter. This is what appears in a run.job file

  • node_type (str) – Top level node type for this node

  • default_value (str) – will be set by the specific class method

  • pattern (str) – Info about the search string for file types. It should be in the format <description> (<example>) IE: “MRC files (*.mrc)”

  • help_text (str) – Text that will be displayed in the GUI if help is clicked

  • in_continue (bool) – If this parameter can be modified in a job that is continued

  • suggestion (str) – A hint that will appear in the data entry field if no no default value is set, it will not be used. IE: ‘Enter file path here’ If you want the suggestion used as the default then just put it in default_value

check_file() → List[JobOptionValidationResult]

Check that specified input files exist

Returns:

An error message if the file does not exist

Return type:

str

get_input_nodes(existing_nodes: Dict[str, Node] | None = None) → List[Node]

Get the input node(s) related to this job option.

Most job option types do not create input nodes and so they should rely on this default implementation that returns an empty list.

Options that do create nodes should return a list of the corresponding Node objects.

Parameters:

existing_nodes – A dict of {node_name: Node object} for the current nodes in the pipeline, if available. This allows jobs to re-use existing nodes rather than creating new ones, which can be necessary if the type of the node is uncertain.

joboption_type = 'INPUTNODE'
validate() → List[JobOptionValidationResult]

Basic validation of the input parameters

Returns:

The validation error of one exists

Return type:

List[JobOptionValidationResult]

class pipeliner.job_options.IntJobOption(*, label: str, default_value: int | None, suggested_min: int | None = None, suggested_max: int | None = None, step_value: int = 1, help_text: str = '', in_continue: bool = False, only_in_continue: bool = False, is_required: bool = False, required_if: JobOptionCondition | None = None, deactivate_if: JobOptionCondition | None = None, hard_min: int | None = None, hard_max: int | None = None, jobop_group: str = 'Main')

Bases: JobOption

Define a job option as a slider for inputting numbers as integers

The slider min value and max value are for display only, they do not limit the actual values that can be input

Parameters:

  • label (str) – A verbose label for the parameter. This is what appears in a run.job file

  • default_value (int) – The default value for the parameter

  • suggested_min (int) – The suggested minimum value for the slider

  • suggested_max (int) – The suggested maximum value for the slider

  • step_value (int) – The step value for the slider

  • help_text (str) – Text that will be displayed in the GUI if help is clicked

  • in_continue (bool) – If this parameter can be modified in a job that is continued

  • hard_min (int) – An error will be raised if the value is < this value

  • hard_max (int) – An error will be raised if the value is > this value

Raises:

ValueError – If the suggested_min, suggested_max, default, hard_min , hard_max, or step values are not integers

get_number() → int

Get the value of a JobOption as a number

Returns:

The value

Return type:

int

Raises:
joboption_type = 'INT'
validate() → List[JobOptionValidationResult]

Basic validation check on the job option

Returns:

The validation

error of one exists

Return type:

JobOptionValidationResult

class pipeliner.job_options.JobOption(*, label: str, default_value: Any = '', help_text: str = '', in_continue: bool = False, is_required: bool = False, required_if: JobOptionCondition | None = None, deactivate_if: JobOptionCondition | None = None, only_in_continue: bool = False, jobop_group: str = 'Main')

Bases: object

A JobOption stores a parameter, its value, and info about the GUI for that param

This is a general class with several more specialised class subclasses.

check_dir() → List[JobOptionValidationResult]

Check that the value is an existing directory

Returns:

A JobOptionValidationResult for any errors

Return type:

list

check_file() → List[JobOptionValidationResult]

Check that specified input files exist

check_required_not_empty() → None

Check if the job option is required but has no value

get_boolean() → bool
get_input_nodes(existing_nodes: Dict[str, Node] | None = None) → List[Node]

Get the input node(s) related to this job option.

Most job option types do not create input nodes and so they should rely on this default implementation that returns an empty list.

Options that do create nodes should return a list of the corresponding Node objects.

Parameters:

existing_nodes – A dict of {node_name: Node object} for the current nodes in the pipeline, if available. This allows jobs to re-use existing nodes rather than creating new ones, which can be necessary if the type of the node is uncertain.

get_list() → List[str]
get_number() → int | float
get_string() → str

Get the value of a JobOption as a string

Returns:

The value

Return type:

str

Raises:

ValueError – If the value is required and missing

joboption_type = 'ERROR: no job option type set'
set_string(set_to: str) → None

Set the value of a JobOption to a string

Parameters:

set_to (str) – The string to set the value to

validate() → List[JobOptionValidationResult]

Basic validation of input parameters

will be set by individual JobOption subtypes

class pipeliner.job_options.JobOptionCondition(conditions: List[Tuple[str, str, str | bool | int | float]], operation: Literal['ANY', 'ALL'] = 'ANY')

Bases: object

Definines conditions for JobOption based on other JobOptions

Used to determine if a JobOption should be required or deactivated

check_condition_is_met(jobops: Dict[str, JobOption]) → Tuple[bool, List[Tuple[str, str, str | bool | int | float]], List[Tuple[str, str, str | bool | int | float]]]

Has the condition been met?

Parameters:

jobops (dict) – The JobOptions dict for the job

Returns:

(bool: Has the condition been met?, List[conditions that passed],

List[conditions that failed]

Return type:

Tuple

class pipeliner.job_options.JobOptionValidationResult(result_type: Literal['warning', 'error', 'info'], raised_by: List[JobOption], message: str)

Bases: object

A class for handling validation of joboptions

type

One of ‘warning’, ‘error’ or ‘info’

Type:

str

raised_by

List of JobOption objects that raised the warning/error. Generally this will only be one JobOption, unless the warning/error was raised from comparing two or more JobOptions

Type:

list

message

Description of the error/warning

Type:

str

class pipeliner.job_options.MultiExternalFileJobOption(*, label: str, pattern: FileDescription | None = None, default_value: str = '', help_text: str = '', in_continue: bool = False, only_in_continue: bool = False, is_required: bool = False, required_if: JobOptionCondition | None = None, deactivate_if: JobOptionCondition | None = None, validation_regex: str | None = None, regex_error_message: str | None = None, suggestion: str = '', jobop_group: str = 'Main', allow_upload: bool = False, command_expected: bool = False)

Bases: JobOption

Define a job option as multiple file names

GUI will open a file browser for the user to find an input for each one. The value for this should be a string of values separated by ::: which will be converted into a list by: [x.strip() for x in the_str.split(“:::”)]

Parameters:

  • label (str) – A verbose label for the parameter. This is what appears in a run.job file

  • default_value (str) – will usually be set from the string and pattern

  • pattern (str) – Info about the search string for file types. It should be in the format <description> (<example>) e.g.: “MRC files (*.mrc)”

  • help_text (str) – Text that will be displayed in the GUI if help is clicked

  • in_continue (bool) – If this parameter can be modified in a job that is continued

  • suggestion (str) – A hint that will appear in the data entry field if no no default value is set, it will not be used. IE: ‘Enter file path here’ If you want the suggestion used as the default then just put it in default_value

  • allow_upload (bool) – Should a file upload be allowed for this input? If yes a copy of the file will end up in the project. This should be turned off for external programs and files that are expected to be very large like ML models.

  • command_expected (bool) – Are the values expected to be executables that can be found with shutil.which()? If so, an executable name or a path to an executable file are both acceptable. If not, each value should be a path to an existing file on disk.

check_file() → List[JobOptionValidationResult]

Check that specified input files exist and are of the right type

For each input file that does not exist, a JobOptionValidationResult of type “error” is returned.

If command_expected is True, each value is checked with shutil.which, which allows the name of an executable on the PATH or the path to an executable file.

If command_expected is False, each value should be the path to a file.

Returns:

class:~pipeliner.job_options.JobOptionValidationResult]: if any of the files do not exist

Return type:

List[

get_list() → List[str]

Return the items in self.value as a list of strings

Returns:

The items in the JobOptions comma separated string

as a list, with any white space at the beginning and/or end removed

Return type:

List[str]

joboption_type = 'MULTIFILENAME'
validate() → List[JobOptionValidationResult]

Basic validation of the input parameters

Returns:

The validation errors of any exist

Return type:

List[JobOptionValidationResult]

class pipeliner.job_options.MultiInputNodeJobOption(*, label: str, node_type: str, pattern: FileDescription | None = None, default_value: str = '', help_text: str = '', in_continue: bool = False, only_in_continue: bool = False, is_required: bool = False, required_if: JobOptionCondition | None = None, deactivate_if: JobOptionCondition | None = None, validation_regex: str | None = None, regex_error_message: str | None = None, suggestion: str = '', jobop_group: str = 'Main', node_kwds: List[str] | None = None)

Bases: JobOption

Define a job option as a file name, create an input node for it

GUI will open a file browser for the user to find an input

Parameters:

  • label (str) – A verbose label for the parameter. This is what appears in a run.job file

  • node_type (str) – Toplevel node type for this node.

  • node_kwds – Optional[List[Optional[List[str]]]]: keywords for each toplevel type in node_types. The index of the toplevel type is used to match the kwds to toplevel type.

  • default_value (str) – will be set by the specific class method

  • pattern (str) – Info about the search string for file types. It should be in the format <description> (<example>), e.g. “MRC files (*.mrc)”

  • help_text (str) – Text that will be displayed in the GUI if help is clicked

  • in_continue (bool) – If this parameter can be modified in a job that is continued

  • suggestion (str) – A hint that will appear in the data entry field if no no default value is set, it will not be used. IE: ‘Enter file path here’ If you want the suggestion used as the default then just put it in default_value

check_file() → List[JobOptionValidationResult]

Check that specified input files exist

For each input file that does not exist, a JobOptionValidationResult of type “error” is returned.

Returns:

class:~pipeliner.job_options.JobOptionValidationResult]: if any of the files do not exist

Return type:

List[

get_input_nodes(existing_nodes: Dict[str, Node] | None = None) → List[Node]

Get the input node(s) related to this job option.

Most job option types do not create input nodes and so they should rely on this default implementation that returns an empty list.

Options that do create nodes should return a list of the corresponding Node objects.

Parameters:

existing_nodes – A dict of {node_name: Node object} for the current nodes in the pipeline, if available. This allows jobs to re-use existing nodes rather than creating new ones, which can be necessary if the type of the node is uncertain.

get_list() → List[str]
joboption_type = 'MULTIINPUTNODE'
validate() → List[JobOptionValidationResult]

Basic validation of the input parameters

Returns:

The validation results (if any)

Return type:

List[JobOptionValidationResult]

class pipeliner.job_options.MultiStringJobOption(*, label: str, default_value: str = '', help_text: str = '', in_continue: bool = False, only_in_continue: bool = False, is_required: bool = False, required_if: JobOptionCondition | None = None, deactivate_if: JobOptionCondition | None = None, validation_regex: str | None = None, regex_error_message: str | None = None, jobop_group: str = 'Main')

Bases: JobOption

Create a job option object for a multiple strings. For things like keyword lists

The value for this should be a string containing values separated by ::: which will be converted into a list by: [x.strip() for x in the_str.split(“:::”)]

The only reason there can’t just be a self.get_list() function in a regular StringJobOption is the GUI must handle them differently.

For regex validation all values in the list of inputs must match the regex

Parameters:

  • label (str) – A verbose label for the parameter.

  • default_value (None)

  • help_text (str) – Text that will be displayed in the GUI if help is clicked.

  • in_continue (bool) – If this parameter can be modified in a job that is continued

get_list() → List[str]

Return the items in self.value as a list of strings

Returns:

The items in the JobOptions separated string

as a list, with any white space at the beginning and/or end removed

Return type:

List[str]

joboption_type = 'MULTISTRING'
validate() → List[JobOptionValidationResult]

Basic validation of the input parameters

class pipeliner.job_options.MultipleChoiceJobOption(*, label: str, choices: List[str], default_value_index: int, help_text: str = '', in_continue: bool = False, only_in_continue: bool = False, is_required: bool = True, required_if: JobOptionCondition | None = None, deactivate_if: JobOptionCondition | None = None, jobop_group: str = 'Main')

Bases: JobOption

Define a job option as a pull down menu

Radio is a misnomer, the GUI will display a pull down menu with the options

Parameters:

  • label (str) – A verbose label for the parameter. This is what appears in a run.job file

  • choices (list) – A list str options for the menu

  • default_value_index (int) – Index of the initial option for the radio; also used as the default value

  • help_text (str) – Text that will be displayed in the GUI if help is clicked

  • in_continue (bool) – If this parameter can be modified in a job that is continued

Raises:

ValueError – If the index of the initial option is not in the radio menu

joboption_type = 'MULTIPLECHOICE'
validate() → List[JobOptionValidationResult]

Basic validation of the input parameters

Returns:

The validation error of one exists

Return type:

List[JobOptionValidationResult]

class pipeliner.job_options.SearchStringJobOption(*, label: str, default_value: str = '', help_text: str = '', must_be_in_project: bool = False, in_continue: bool = False, only_in_continue: bool = False, is_required: bool = False, required_if: JobOptionCondition | None = None, deactivate_if: JobOptionCondition | None = None, validation_regex: str | None = None, regex_error_message: str | None = None, jobop_group: str = 'Main')

Bases: StringJobOption

A JobOption for search strings

Parameters:

  • label (str) – A verbose label for the parameter.

  • default_value (None)

  • help_text (str) – Text that will be displayed in the GUI if help is clicked.

  • in_continue (bool) – If this parameter can be modified in a job that is continued

check_file() → List[JobOptionValidationResult]

Check that at least one input file exists

Returns:

Error if no files found

Return type:

List[JobOptionValidationResult]

get_list() → List[str]
joboption_type = 'SEARCHSTRING'
validate() → List[JobOptionValidationResult]

Basic validation of the input parameters

class pipeliner.job_options.StringJobOption(*, label: str, default_value: str = '', help_text: str = '', in_continue: bool = False, only_in_continue: bool = False, is_required: bool = False, required_if: JobOptionCondition | None = None, deactivate_if: JobOptionCondition | None = None, validation_regex: str | None = None, regex_error_message: str | None = None, jobop_group: str = 'Main')

Bases: JobOption

Create a job option object for a string parameter.

Parameters:

  • label (str) – A verbose label for the parameter.

  • default_value (None)

  • help_text (str) – Text that will be displayed in the GUI if help is clicked.

  • in_continue (bool) – If this parameter can be modified in a job that is continued

joboption_type = 'STRING'
validate() → List[JobOptionValidationResult]

Basic validation of the input parameters