Job Options

JobOptions

JobOptions are used to store parameters for jobs. They contain all the info necessary for on-the-fly GUI generation.

class pipeliner.job_options.BooleanJobOption(*, label: str, default_value: bool, help_text: str = '', in_continue: bool = False, only_in_continue: bool = False, deactivate_if: JobOptionCondition | None = None, jobop_group: str = 'Main')

Bases: JobOption[bool]

Define a job option as a boolean

Parameters:

label (str) – A verbose label for the parameter. This is what appears in a run.job file
default_value (bool) – The default value
help_text (str) – Text that will be displayed in the GUI if help is clicked
in_continue (bool) – If this parameter can be modified in a job that is continued

get_boolean() → bool

Get a boolean value

Returns:: The value
Return type:: bool

joboption_type = 'BOOLEAN'

class pipeliner.job_options.DirPathJobOption(*, label: str, default_value: str = '', help_text: str = '', must_be_in_project: bool = False, must_exist: bool = True, in_continue: bool = False, only_in_continue: bool = False, is_required: bool = False, required_if: JobOptionCondition | None = None, deactivate_if: JobOptionCondition | None = None, validation_regex: str | None = None, regex_error_message: str | None = None, jobop_group: str = 'Main')

Bases: StringJobOption

A JobOption for directory paths

Parameters:

label (str) – A verbose label for the parameter.
default_value (None)
help_text (str) – Text that will be displayed in the GUI if help is clicked.
in_continue (bool) – If this parameter can be modified in a job that is continued
must_be_in_project (bool) – Does the directory have to be in the project
must_exist (bool) – Does this directory have to exist when the job option is validated? Usually true, unless it will be created by a previous command in the job

check_dir() → List[JobOptionValidationResult]

Check that the specified dir exists.

Returns:: An error if the dir does not exist, or a warning if the dir does not exist and must_exist is False
Return type:: List[JobOptionValidationResult]

joboption_type = 'DIRPATH'

validate() → List[JobOptionValidationResult]: Basic validation of the input parameters

class pipeliner.job_options.ExternalFileJobOption(*, label: str, pattern: FileDescription | None = None, default_value: str = '', help_text: str = '', in_continue: bool = False, only_in_continue: bool = False, is_required: bool = False, required_if: JobOptionCondition | None = None, deactivate_if: JobOptionCondition | None = None, validation_regex: str | None = None, regex_error_message: str | None = None, suggestion: str = '', jobop_group: str = 'Main', command_expected: bool = False)

Bases: StringJobOption

Define a job option as a file name

GUI will open a file browser for the user to find an input

Parameters:

label (str) – A verbose label for the parameter. This is what appears in a run.job file
default_value (str) – will usually be set from the string and pattern.
pattern (str) – Info about the search string for file types. It should be in the format <description> (<example>) e.g.: “MRC files (*.mrc)”
help_text (str) – Text that will be displayed in the GUI if help is clicked
in_continue (bool) – If this parameter can be modified in a job that is continued
suggestion (str) – A hint that will appear in the data entry field if no default value is set, it will not be used. IE: ‘Enter file path here’ If you want the suggestion used as the default then just put it in default_value
command_expected (bool) – Is the value expected to be an executable that can be found with shutil.which()? If so, an executable name or a path to an executable file are both acceptable. If not, the value should be a path to an existing file on disk.

check_file() → List[JobOptionValidationResult]

Check that specified input files exist.

If command_expected is True, the value is checked with shutil.which, which allows the name of an executable on the PATH or the path to an executable file.

If command_expected is False, the value should be the path to a file.

Returns:: An error message if the file does not exist
Return type:: str

joboption_type = 'FILENAME'

validate() → List[JobOptionValidationResult]

Basic validation of the input parameters

Returns:

The: validation error of one exists

Return type:

List[JobOptionValidationResult]

class pipeliner.job_options.FileDescription(name: str = 'Any file', exts: List[str] | None = None, exact: bool = False)

Bases: object

Produce a description of files and their extensions

Parameters:

name (str) – The type of file IE: ‘Micrograph movies’
exts (list) – The acceptable extensions or file search strings, e.g. ‘.txt’ to find *.txt or ‘_half1.star’ for *_half1.star
exact (bool) – Match the file name(s) exactly, do not prepend a *

class pipeliner.job_options.FloatJobOption(*, label: str, default_value: float | None = None, suggested_min: float | None = None, suggested_max: float | None = None, step_value: float = 1.0, help_text: str = '', in_continue: bool = False, only_in_continue: bool = False, is_required: bool = False, required_if: JobOptionCondition | None = None, deactivate_if: JobOptionCondition | None = None, hard_min: float | None = None, hard_max: float | None = None, jobop_group: str = 'Main', handle_minus_one_as_none: bool = False)

Bases: JobOption[float | None]

Define a job option as a slider for inputting numbers as floats

The min value and max value are for display only, they do not limit the actual values that can be input

Parameters:

label (str) – A verbose label for the parameter. This is what appears in a run.job file
default_value (float) – The default value for the parameter
suggested_min (float) – The suggested minimum value for the slider
suggested_max (float) – The suggested maximum value for the slider
step_value (float) – The step value for the slider
help_text (str) – Text that will be displayed in the GUI if help is clicked
in_continue (bool) – If this parameter can be modified in a job that is continued
hard_min (float) – An error will be raised if the value is < this value
hard_max (float) – An error will be raised if the value is > this value
handle_minus_one_as_none (bool) –
‘None’ values will be interpreted as -1.0, keeping with Relion’s format for default values. Specifically, what this means is that if the value is set to be any “empty” value (i.e. None, -1.0, “” or “-1.0”) then it will appear empty in Doppio, and it will return these values:
```
* .value -> None
* .value_string -> "-1.0"
* .get_string() -> "-1.0"
* .get_number() -> -1.0
```
This parameter cannot be used in combination with is_required=True.

get_number() → float

Get the value of a JobOption as a number

Returns:

The value

Return type:

float

Raises:

ValueError – If the value cannot be converted to a float
ValueError – If the value is required and missing

joboption_type = 'FLOAT'

validate() → List[JobOptionValidationResult]

Basic validation check on the job option

Returns:

The validation: error of one exists

Return type:

JobOptionValidationResult

class pipeliner.job_options.InputNodeJobOption(*, label: str, node_type: str, pattern: FileDescription | None = None, default_value: str = '', help_text: str = '', in_continue: bool = False, only_in_continue: bool = False, is_required: bool = False, required_if: JobOptionCondition | None = None, deactivate_if: JobOptionCondition | None = None, validation_regex: str | None = None, regex_error_message: str | None = None, suggestion: str = '', jobop_group: str = 'Main', node_kwds: List[str] | None = None)

Bases: StringJobOption

Define a job option as a file name, create an input node for it

GUI will open a file browser for the user to find an input

Parameters:

label (str) – A verbose label for the parameter. This is what appears in a run.job file
node_type (str) – Top level node type for this node
default_value (str) – will be set by the specific class method
pattern (FileDescription) – Info about the search string for file types.
help_text (str) – Text that will be displayed in the GUI if help is clicked
in_continue (bool) – If this parameter can be modified in a job that is continued
suggestion (str) – A hint that will appear in the data entry field if no no default value is set, it will not be used. IE: ‘Enter file path here’ If you want the suggestion used as the default then just put it in default_value

check_file() → List[JobOptionValidationResult]

Check that specified input files exist

Returns:: An error message if the file does not exist
Return type:: str

get_input_nodes(existing_nodes: Dict[str, Node] | None = None) → List[Node]

Get the input node(s) related to this job option.

Most job option types do not create input nodes and so they should rely on this default implementation that returns an empty list.

Options that do create nodes should return a list of the corresponding Node objects.

Parameters:: existing_nodes – A dict of {node_name: Node object} for the current nodes in the pipeline, if available. This allows jobs to re-use existing nodes rather than creating new ones, which can be necessary if the type of the node is uncertain.

joboption_type = 'INPUTNODE'

validate() → List[JobOptionValidationResult]

Basic validation of the input parameters

Returns:: The validation error of one exists
Return type:: List[JobOptionValidationResult]

class pipeliner.job_options.IntJobOption(*, label: str, default_value: int | None = None, suggested_min: int | None = None, suggested_max: int | None = None, step_value: int = 1, help_text: str = '', in_continue: bool = False, only_in_continue: bool = False, is_required: bool = False, required_if: JobOptionCondition | None = None, deactivate_if: JobOptionCondition | None = None, hard_min: int | None = None, hard_max: int | None = None, jobop_group: str = 'Main', handle_minus_one_as_none: bool = False)

Bases: JobOption[int | None]

Define a job option as a slider for inputting numbers as integers

The slider min value and max value are for display only, they do not limit the actual values that can be input

Parameters:

label (str) – A verbose label for the parameter. This is what appears in a run.job file
default_value (int) – The default value for the parameter
suggested_min (int) – The suggested minimum value for the slider
suggested_max (int) – The suggested maximum value for the slider
step_value (int) – The step value for the slider
help_text (str) – Text that will be displayed in the GUI if help is clicked
in_continue (bool) – If this parameter can be modified in a job that is continued
hard_min (int) – An error will be raised if the value is < this value
hard_max (int) – An error will be raised if the value is > this value
handle_minus_one_as_none (bool) –
‘None’ values will be interpreted as -1, keeping with Relion’s format for default values. Specifically, what this means is that if the value is set to be any “empty” value (i.e. None, -1, “” or “-1”) then it will appear empty in Doppio, and it will return these values:
```
* .value -> None
* .value_string -> "-1"
* .get_string() -> "-1"
* .get_number() -> -1
```
This parameter cannot be used in combination with is_required=True.

Raises:

ValueError – If the suggested_min, suggested_max, default, hard_min , hard_max, or step values are not integers

get_number() → int

Get the value of a JobOption as a number

Returns:

The value

Return type:

int

Raises:

ValueError – If the value cannot be converted to a int
ValueError – If the value is required and missing

joboption_type = 'INT'

validate() → List[JobOptionValidationResult]

Basic validation check on the job option

Returns:

The validation: error of one exists

Return type:

JobOptionValidationResult

class pipeliner.job_options.JobOption(*, label: str, default_value: T | str, help_text: str = '', in_continue: bool = False, is_required: bool = False, required_if: JobOptionCondition | None = None, deactivate_if: JobOptionCondition | None = None, only_in_continue: bool = False, jobop_group: str = 'Main')

Bases: Generic[T]

A JobOption stores a parameter, its value, and info about the GUI for that param

This is a general class with several more specialised subclasses.

check_dir() → List[JobOptionValidationResult]

Check that the value is an existing directory

Returns:: A JobOptionValidationResult for any errors
Return type:: list

check_file() → List[JobOptionValidationResult]: Check that specified input files exist

property default_value_string: str: Read-only property to get the job option’s default value as a string.

get_boolean() → bool

get_input_nodes(existing_nodes: Dict[str, Node] | None = None) → List[Node]

Get the input node(s) related to this job option.

Most job option types do not create input nodes and so they should rely on this default implementation that returns an empty list.

Options that do create nodes should return a list of the corresponding Node objects.

Parameters:: existing_nodes – A dict of {node_name: Node object} for the current nodes in the pipeline, if available. This allows jobs to re-use existing nodes rather than creating new ones, which can be necessary if the type of the node is uncertain.

get_list() → List[str]

get_number() → int | float

get_string() → str

Get the value of a JobOption as a string.

Returns:: The value
Return type:: str
Raises:: ValueError – If check_required is True and the value is required and missing

property is_blank: bool: Check if the JobOption has been left blank (value is empty str or None)

joboption_type = 'ERROR: no job option type set'

validate() → List[JobOptionValidationResult]

Basic validation of input parameters

This simple implementation will typically be called and extended by individual JobOption subtypes.

property value: T

Get the value of this job option.

The value is returned in its native internal type T, for example a FloatJobOption will return a float and a MultiStringJobOption will return a list[str]. If you need the value as a string, call get_string(), or use the value_string property if you need the string form of the value without checking if it is required and empty.

property value_string: str

Read-only property to get the job option’s value as a string.

In most situations you should use the get_string() method instead. Only use this if you specifically need a value even if the job option is required and the value is empty.

Bases: object

Definines conditions for JobOption based on other JobOptions

Used to determine if a JobOption should be required or deactivated

OPERATORS = {'!=': <built-in function ne>, '<': <built-in function lt>, '<=': <built-in function le>, '=': <built-in function eq>, '>': <built-in function gt>, '>=': <built-in function ge>}

Has the condition been met?

Parameters:

jobops (dict) – The JobOptions dict for the job

Returns:

(bool: Has the condition been met?, List[conditions that passed],: List[conditions that failed]

Return type:

Tuple

class pipeliner.job_options.JobOptionValidationResult(result_type: Literal['warning', 'error', 'info'], raised_by: List[JobOption], message: str)

Bases: object

A class for handling validation of joboptions

type

One of ‘warning’, ‘error’ or ‘info’

Type:: str

raised_by

List of JobOption objects that raised the warning/error. Generally this will only be one JobOption, unless the warning/error was raised from comparing two or more JobOptions

Type:: list

message

Description of the error/warning

Type:: str

class pipeliner.job_options.MultiExternalFileJobOption(*, label: str, pattern: FileDescription | None = None, default_value: List[str] | None = None, help_text: str = '', in_continue: bool = False, only_in_continue: bool = False, is_required: bool = False, required_if: JobOptionCondition | None = None, deactivate_if: JobOptionCondition | None = None, validation_regex: str | None = None, regex_error_message: str | None = None, suggestion: str = '', jobop_group: str = 'Main', allow_upload: bool = False, command_expected: bool = False)

Bases: MultiStringJobOption

Define a job option as multiple file names

GUI will open a file browser for the user to find an input for each one. The value for this should be a list of strings. The value_string (which is written in job.star/run.job files) contains the values separated by ::: which will be converted into a list by: [x.strip() for x in the_str.split(“:::”)]

Parameters:

label (str) – A verbose label for the parameter. This is what appears in a run.job file
default_value (List[str]) – value will usually be set from the string and pattern
pattern (FileDescription) – Info about the search string for file types.
help_text (str) – Text that will be displayed in the GUI if help is clicked
in_continue (bool) – If this parameter can be modified in a job that is continued
suggestion (str) – A hint that will appear in the data entry field if no no default value is set, it will not be used. IE: ‘Enter file path here’ If you want the suggestion used as the default then just put it in default_value
allow_upload (bool) – Should a file upload be allowed for this input? If yes a copy of the file will end up in the project. This should be turned off for external programs and files that are expected to be very large like ML models.
command_expected (bool) – Are the values expected to be executables that can be found with shutil.which()? If so, an executable name or a path to an executable file are both acceptable. If not, each value should be a path to an existing file on disk.

check_file() → List[JobOptionValidationResult]

Check that specified input files exist and are of the right type

For each input file that does not exist, a JobOptionValidationResult of type “error” is returned.

If command_expected is True, each value is checked with shutil.which, which allows the name of an executable on the PATH or the path to an executable file.

If command_expected is False, each value should be the path to a file.

Returns:: class:~pipeliner.job_options.JobOptionValidationResult]: if any of the files do not exist
Return type:: List[

joboption_type = 'MULTIFILENAME'

validate() → List[JobOptionValidationResult]

Basic validation of the input parameters

Returns:: The validation errors of any exist
Return type:: List[JobOptionValidationResult]

class pipeliner.job_options.MultiInputNodeJobOption(*, label: str, node_type: str, pattern: FileDescription | None = None, default_value: List[str] | None = None, help_text: str = '', in_continue: bool = False, only_in_continue: bool = False, is_required: bool = False, required_if: JobOptionCondition | None = None, deactivate_if: JobOptionCondition | None = None, validation_regex: str | None = None, regex_error_message: str | None = None, suggestion: str = '', jobop_group: str = 'Main', node_kwds: List[str] | None = None)

Bases: MultiStringJobOption

Define a job option as a file name, create an input node for it

GUI will open a file browser for the user to find an input

Parameters:

label (str) – A verbose label for the parameter. This is what appears in a run.job file
node_type (str) – Toplevel node type for this node.
node_kwds – Optional[List[Optional[List[str]]]]: keywords for each toplevel type in node_types. The index of the toplevel type is used to match the kwds to toplevel type.
default_value (List[str]) – will be set by the specific class method
pattern (FileDescription) – Info about the search string for file types.”`
help_text (str) – Text that will be displayed in the GUI if help is clicked
in_continue (bool) – If this parameter can be modified in a job that is continued
suggestion (str) – A hint that will appear in the data entry field if no no default value is set, it will not be used. IE: ‘Enter file path here’ If you want the suggestion used as the default then just put it in default_value

check_file() → List[JobOptionValidationResult]

Check that specified input files exist

For each input file that does not exist, a JobOptionValidationResult of type “error” is returned.

Returns:: class:~pipeliner.job_options.JobOptionValidationResult]: if any of the files do not exist
Return type:: List[

get_input_nodes(existing_nodes: Dict[str, Node] | None = None) → List[Node]

Get the input node(s) related to this job option.

Most job option types do not create input nodes and so they should rely on this default implementation that returns an empty list.

Options that do create nodes should return a list of the corresponding Node objects.

Parameters:: existing_nodes – A dict of {node_name: Node object} for the current nodes in the pipeline, if available. This allows jobs to re-use existing nodes rather than creating new ones, which can be necessary if the type of the node is uncertain.

joboption_type = 'MULTIINPUTNODE'

validate() → List[JobOptionValidationResult]

Basic validation of the input parameters

Returns:: The validation results (if any)
Return type:: List[JobOptionValidationResult]

class pipeliner.job_options.MultiStringJobOption(*, label: str, default_value: List[str] | None = None, help_text: str = '', in_continue: bool = False, only_in_continue: bool = False, is_required: bool = False, required_if: JobOptionCondition | None = None, deactivate_if: JobOptionCondition | None = None, validation_regex: str | None = None, regex_error_message: str | None = None, jobop_group: str = 'Main')

Bases: JobOption[List[str]]

Create a job option object for multiple strings.

The value for this should be a list of strings. The value_string (which is written in job.star/run.job files) contains the values separated by ::: which will be converted into a list by: [x.strip() for x in the_str.split(“:::”)]

Empty strings are removed from the list.

For regex validation all values in the list of inputs must match the regex.

Parameters:

label (str) – A verbose label for the parameter.
default_value (None)
help_text (str) – Text that will be displayed in the GUI if help is clicked.
in_continue (bool) – If this parameter can be modified in a job that is continued

get_list() → List[str]

Return the items in self.value as a list of strings

Returns:

The items in the JobOptions separated string: as a list, with any white space at the beginning and/or end removed

Return type:

List[str]

joboption_type = 'MULTISTRING'

validate() → List[JobOptionValidationResult]: Basic validation of the input parameters

class pipeliner.job_options.MultipleChoiceJobOption(*, label: str, choices: List[str], default_value_index: int | None = None, default_value: str | None = None, help_text: str = '', in_continue: bool = False, only_in_continue: bool = False, is_required: bool = True, required_if: JobOptionCondition | None = None, deactivate_if: JobOptionCondition | None = None, jobop_group: str = 'Main')

Bases: StringJobOption

Define a job option as a pull down menu

Parameters:

label (str) – A verbose label for the parameter. This is what appears in a run.job file
choices (List[str]) – A list of options for the menu
default_value_index (int) – Index of the default value in the choices list. This was previously the only way to select the default value, but now it is preferred to just set default_value directly instead. If both are set, default_value will take priority and default_value_index will be ignored.
default_value (str) – The default value. If given, this must either be an empty string (to start with no option selected) or it must be exactly equal to one of the items in the choices list.
help_text (str) – Text that will be displayed in the GUI if help is clicked
in_continue (bool) – If this parameter can be modified in a job that is continued

Raises:

ValueError – If neither default_value nor default_value_index is set.
ValueError – If default_value_index is set out of range for the number of choices.
ValueError – If default_value is set to a non-empty string and is not in the list of choices.

joboption_type = 'MULTIPLECHOICE'

validate() → List[JobOptionValidationResult]

Basic validation of the input parameters

Returns:: The validation error of one exists
Return type:: List[JobOptionValidationResult]

class pipeliner.job_options.SearchStringJobOption(*, label: str, default_value: str = '', help_text: str = '', must_be_in_project: bool = False, in_continue: bool = False, only_in_continue: bool = False, is_required: bool = False, required_if: JobOptionCondition | None = None, deactivate_if: JobOptionCondition | None = None, validation_regex: str | None = None, regex_error_message: str | None = None, jobop_group: str = 'Main')

Bases: StringJobOption

A JobOption for search strings

Parameters:

label (str) – A verbose label for the parameter.
default_value (str)
help_text (str) – Text that will be displayed in the GUI if help is clicked.
in_continue (bool) – If this parameter can be modified in a job that is continued

check_file() → List[JobOptionValidationResult]

Check that at least one input file exists

Returns:: Error if no files found
Return type:: List[JobOptionValidationResult]

get_list() → List[str]

joboption_type = 'SEARCHSTRING'

validate() → List[JobOptionValidationResult]: Basic validation of the input parameters

class pipeliner.job_options.StringJobOption(*, label: str, default_value: str = '', help_text: str = '', in_continue: bool = False, only_in_continue: bool = False, is_required: bool = False, required_if: JobOptionCondition | None = None, deactivate_if: JobOptionCondition | None = None, validation_regex: str | None = None, regex_error_message: str | None = None, jobop_group: str = 'Main')

Bases: JobOption[str]

Create a job option object for a string parameter.

Parameters:

label (str) – A verbose label for the parameter.
default_value (None)
help_text (str) – Text that will be displayed in the GUI if help is clicked.
in_continue (bool) – If this parameter can be modified in a job that is continued

joboption_type = 'STRING'

validate() → List[JobOptionValidationResult]: Basic validation of the input parameters