Star File Utilities

class pipeliner.starfile_handler.BodyFile(fn_in: str)

Bases: StarFile

A STAR file that lists the bodies in a multibody refinement

bodycount

Number bodies in the file

Type:

int

class pipeliner.starfile_handler.JobStar(fn_in: str)

Bases: StarFile

A class for STAR files that define a pipeliner job parameters

jobtype

the job type, converted from the relion nomenclature if necessary

Type:

str

joboptions

The joboptions {name: value} all values are strings regardless of joboption type

Type:

dict

is_continue

is the job a continuation

Type:

bool

is_tomo

is the job tomography (might be removed when relon compatobility is deprecated)

Type:

bool

all_options_as_dict() → Dict[str, str]

Returns a dict of all the parameters of a job.star file

The dict contains both the job options and the running options. All values in the dict are strings.

make_substitutions(conversions: Dict[str, str])

Substitute one job name or file name for another wherever it appears

Parameters:

conversions (Dict[str, str]) – {“old job name”: “new job name”}

modify(params_to_change: dict, allow_add=False)

Change multiple values in a job.star from a dict

Parameters:

  • params_to_change (dict) – The Parameters to change in the template in the format {param_name: new_value}

  • allow_add (bool) – Can new joboptions be added?

Raises:

ValueError – If an attempt is made to add new fields with allow_add=False

write(outname: str | PathLike[str] = '')

Write a STAR file from the JobStar

Parameters:

outname (str or Path) – name of the output file; will overwrite the original file if none provided

class pipeliner.starfile_handler.StarFile(fn_in: str | PathLike[str])

Bases: object

A superclass for all types of STAR files used by the pipeliner

file_name

The STAR file

Type:

str or Path

column_as_list(blockname: str | None, colname: str) → List[str]

Return a single column from a block as a list

Parameters:

  • blockname (str) – The name of the block to use

  • colname (str) – The name of the column to get

Returns:

The values from that column

Return type:

list

column_headers(block: str = '') → List[str]

Get all the column headers for a specific block.

count_block(blockname: str | None = None) → int

Count the number of items in a block that only contains a single loop

This is the format in most relion data STAR files

Parameters:

blockname (str) – The name of the block to count

Returns:

The count

Return type:

int

get_block(blockname: str | None = None) → Block

Get a block from the STAR file

Parameters:

blockname (str) – The name of the block to get. Use None if the file has a single unnamed block

Returns:

The desired block or None if not found

Return type:

(gemmi.cif.Block)

get_block_names() → List[str]
loop_as_list(block: str = '', columns: List[str] | None = None, headers: bool = False) → List[List[str]]

Returns a set of columns from a STAR file loop as a list

Parameters:

  • block (str) – The name of the block to get the data from

  • columns (list) – Names of the columns to get, if None then all columns are returned

  • headers (bool) – Should the 1st row of the returned list be the header names

Returns:

The column data one row per sublist. If headers=True

the first sublist contains the headers

Return type:

List[List[str]]

Raises:

  • ValueError – If the specified block is not found

  • ValueError – If the specified block does not contain a loop

  • ValueError – If any of the specified columns are not found

pairs_as_dict(block: str = '') → Dict[str, str]

Returns paired values from a STAR file as a dict

Parameters:

block (str) – The name of the block to get the data from

Returns:

{parameter: value}

Return type:

dict

Raises:

  • ValueError – If the specified block is not found

  • ValueError – If the specified block is a loop and not a pair-value

pipeliner.starfile_handler.convert_old_relion_pipeline(fn_in: str)

Convert a pipeline STAR file from RELION 3 or 4 to ccpem-pipeliner format.

The file is also checked and corrected if it contains any reserved words which would cause Gemmi’s CIF parser to fail when the file is read.

Note that this function reads the whole file several times, but pipeline STAR files are never expected to be so large that this would take a significant amount of time.

Returns:

True if the file was converted, or False otherwise.

Raises:

ValueError – if the pipeline file is invalid or its version cannot be determined

pipeliner.starfile_handler.fix_reserved_words(fn_in)

Make sure the STAR file doesn’t contain any illegally used reserved words

Overwrites the original file if it is corrected. The old file is saved as filename.orig.

Returns:

True if the file was corrected, or False if it contained no reserved words and did not need correcting.

pipeliner.starfile_handler.get_particle_box_size_from_starfile(starfile: str) → int

Get the image size in pixels from a starfile from the optics group(s)

assumes the images are square and all the same size

Parameters:

starfile (str) – path to starfile

Returns:

edge length in px

Return type:

int

pipeliner.starfile_handler.get_pixel_size_from_starfile(starfile: str) → float

Get the pixel size from a starfile from the optics group(s)

Parameters:

starfile (str) – path to starfile

Returns:

pixel size

Return type:

float

pipeliner.starfile_handler.get_starfile_loop_headers(in_file: str) → Dict[str, List[str]]

Get the headers for each loop in each block in a STAR file

Parameters:

in_file (str) – The STAR file to operate on

Returns:

{block_name: [header1, header2, … headerN]} Headers

appear in the column order of the STAR file.

Return type:

Dict[str, List[str]]

pipeliner.starfile_handler.interconvert_box_star_coords(fn_in: str, out_name: str | None = None) → str

Interconvert .box and .star coordinate files

Parameters:

  • fn_in (str) – The file to convert

  • out_name (str) – Dir to write the output file into, if None writes it the same place as the input, with the same name just switching the extension

Returns:

Name of the file written

Return type:

str

Raises:

ValueError – If the file is not .box or .star

pipeliner.starfile_handler.quotate_starfile_line(in_line: str) → str | None

Adds the necessary quotation to a line to make it compatible with gemmi

Parameters:

in_line (str) – The line to operate on

Returns:

The line, properly quotated, or None if no changes to the line are needed

Return type:

str

pipeliner.starfile_handler.read_relion_optimiser_starfile(fn_in) → Dict[str, str]

Special function for reading optimiser.star files written by relion

These files have issues with duplicate entries because of a Relion5 bug, which precludes use of gemmi because it raises an error if it encounters duplicate entires If a duplicate entry exists it uses the value from the last instance.

Parameters:

fn_in (str) – Name of the optimiser file

Returns:

The param-value pairs from the file

Return type:

Dict[str, str]

Raises:

ValueError – If the file isn’t an optimiser file

Starfile writing utilities

Writes STAR files in the same style as RELION

pipeliner.star_writer.star_quote(val: str) → str

Quote a string value as needed for writing to a STAR file.

If the value contains only characters that can be safely written without quoting, and does not begin with a reserved word of the STAR format, it is returned unchanged.

Otherwise, the value is passed to gemmi.cif.quote() for quoting, which handles all of the more complicated quoting logic that is sometimes required.

pipeliner.star_writer.write(doc: Document, filename: str | PathLike[str])

Write a Gemmi CIF document to a RELION-style STAR file.

The file is written as an atomic operation. This ensures that any processes reading it will always see a valid version (old or new) and not a half-written new file.

The document is written to a temporary file first, then after the writing is complete and the file has been flushed to disk, the target file is replaced with the temporary one. The temporary file will always be removed even if the replacement is unsuccessful.

Note: it is still possible for data to be lost using this function, if two processes try to write to the file at the same time. In that case, one of the new versions of the file will be kept and the other will not.

Parameters:
pipeliner.star_writer.write_jobstar(in_dict: dict, out_fn: str | PathLike[str])

Write a job.star file from a dictionary of options

Parameters:

  • in_dict (dict) – Dict of job option keys and values

  • out_fn (str or Path) – Name of the file to write to

pipeliner.star_writer.write_to_stream(doc: Document, out_stream: IO[str])

Write a Gemmi CIF document to an output stream using RELION’s output style.

Parameters:

  • doc (gemmi.cif.Document) – The data to write out

  • out_stream (str) – The name of the file to write the data to

Utils for OneDep (PDB and EMDB) mmCIF files

class pipeliner.deposition_tools.cif_handler.CifHandler(input: str | Document)

Bases: object

add_atom_type_parent_entry(block: str = '') → None

Adds the _atom_type.symbol entry as a stub

This is the minimum required for this parent category. A loop with a single atom_type.symbol entry.

Parameters:

block (str) – The block to add the entry to. Assumes sole block if empty

add_chemcomp_parent_entry(block: str = '') → None

Adds the missing _atom_site.label_comp_id parent category as a stub

This is the minimum required for this parent category

Parameters:

block (str) – The block to add the entry to. Assumes sole block if empty

add_entry(entry_id: str, block: str = '')

Add the entry ID for pair to a block

Parameters:

  • entry_id (str) – The entry ID

  • block (str) – block name to operate on if empty assumes the cif has only one block

remove_entry_ids(block_name: str | None = None)

Remove the entry.id field from a block in a pair or loop

This field will be written by the pipeliner and all cross-refs updated

Parameters:

block_name (str) – block name to operate on if empty assumes the cif has only one block

replace_characters_in_field_values(subs: Dict[str, str], fields: list[str], block: str = '') → None

Substitute characters in specific fields

Useful for removing non-alpha numerica characters in blocks fields that can’t accept them

Parameters:

  • subs (Dict[str, Union[int, str, float]]) – Substitution dictionary

  • fields (List[str]) – List of fields to make the substitutions in

  • block (str) – Block to make the substitutions in, if empty assumes a sole block

update_entry_ids(entry_id: str, blocks: List[str] | None = None) → None

Update the .entry_id field for every category in a block

Parameters:

  • entry_id (str) – The entry ID to use for the updates

  • blocks (List[str]) – The blocks to update the entries for, updates all blocks if None

write(file_name: str = '', options: WriteOptions | None = None)

Write a CIF file

Parameters:

  • file_name (Optional[str]) – The name of the file to write. Defaults to overwriting the existing file if none given.

  • options (gemmi.cif.WriteOptions) – Options to pass to write. See the gemmi docs.

pipeliner.deposition_tools.cif_handler.add_block_uids(block: Block) → Tuple[Block, Dict[Tuple[str, str], str]]

Replace all int ids in main category id fields with UIDs

Parameters:

block (gemmi.cif.Block) – The cif block to operate on

Returns:

The new cif block, and a

dict of the substitutions made {(field, id): uuid }

Return type:

Tuple[gemmi.cif.Block, Dict[Tuple[str, str], str]]

pipeliner.deposition_tools.cif_handler.add_id_tuple_entry(id_entry: Tuple[str, str], ids_dict: Dict[Tuple[str, str], str]) → str

Add an entry to the ids dictionary made by block_as_all_loop_format()

If the field is one that needs to be replaced by a UID it is added to the dict and the UID returned. If it doesn’t then return an empty string.

Parameters:

  • id_entry (Tuple[str, str]) – The entry to be added. (category.field, id) EG: (em_image_precessing.id, 1)

  • ids_dict (Dict[Tuple[str, str], str]) – The dictionary of ids and substitutions an entry will be added if appropriate.

pipeliner.deposition_tools.cif_handler.assign_new_int_ids(block: Block) → Block

Update all UID format ids in a block to inetger ids

Ids that are not UUUID4s sre ignored

Parameters:

block (gemmi.cif.Block) – The cif block to operate on

Returns:

The updated cif block

Return type:

gemmi.cif.Block

pipeliner.deposition_tools.cif_handler.combine_blocks(bl1: Block, bl2: Block, rename: str) → Block

Assumes both blocks contain only loops

Parameters:
Returns:

The combined block

Return type:

gemmi.cif.Block

pipeliner.deposition_tools.cif_handler.combine_cifs(cif_1: Document | str, cif_2: Document | str) → Document

Combine two cif files

Parameters:
Returns:

the combined cif file

Return type:

gemmi.cif.Document

pipeliner.deposition_tools.cif_handler.combine_loops(block1: Block, tags1: List[str], block2: Block, tags2: List[str]) → Tuple[List[List[str]], List[str]]

Combine two cif loops into a single loop Normalizes the tags and order, any fields present in on loop but not the other are give ‘?’ as values

Parameters:

  • block1 (gemmi.cif.Block) – The block comtaining the first loop

  • tags1 (List[str]) – The tags present in the first loop

  • block2 (gemmi.cif.Block) – The block comtaining the second loop

  • tags2 (List[str]) – The tags present in the second loop

Returns:

The combined loop data (List[List[str]]) and

its tags (List[str])

Return type:

Tuple[List[List[str]], List[str]]

pipeliner.deposition_tools.cif_handler.find_loop_from_tags(block: Block, tags: List[str]) → Loop | None

Find a loop from a cif block with a possibly incomplete list of tags

Parameters:

  • block (gemmi.cif.Block) – The cif block to look in

  • tags (List[str]) – The tags to look for. The first loop containing any tag in the list will be returned

Returns:

The found loop or None of none found

Return type:

Optional[gemmi.cif.Loop]

pipeliner.deposition_tools.cif_handler.flatten_cif(cif_doc: Document) → Document

Convert a cif document in to a single block and all fields into loops

Note that this returns a new document, the old one is not modified in-place.

Parameters:

cif_doc (gemmi.cif.Document) – The cif document to flatten

Returns:

The flattened cif document

Return type:

gemmi.cif.Document

pipeliner.deposition_tools.cif_handler.update_block_cross_refs(block: Block, id_updates: Dict[Tuple[str, str], str]) → Block

Update all id cross-references in a block

Parameters:

  • block (gemmi.cif.Block) – The cif block to operate on

  • id_updates (Dict[Tuple[str, str], str]) – The dict mapping field ids to values

Returns:

The updated cif block

Return type:

gemmi.cif.Block

pipeliner.deposition_tools.cif_handler.update_cif_ids_and_crossrefs(cifdoc: Document) → Document

Update all ids and cross-references in a cif doc

Parameters:

cifdoc (gemmi.cif.Document) – The cif document to operate on

Returns:

The updated cif document

Return type:

gemmi.cif.Document